Diagnosing automation failures from logs and output

When an automation run fails, the logs and command output it produced are your evidence. This article covers how to read that evidence methodically — a repeatable diagnostic procedure, the handful of error signatures that name a root cause on sight, and the layer-by-layer map that turns a symptom into a fix.

A repeatable method

Under pressure the temptation is to guess. Resist it and work the same five steps every time:

1. Locate the event. Which run, which device, what time? A correlation ID or a timestamp lets you line up records across systems. (If your logging emits a shared run_id per run, this step is trivial — see the logging strategy article.)
2. Find the first error. A single failure cascades into many downstream messages. The earliest meaningful error usually holds the cause; everything after it is fallout.
3. Read it literally. The exact exception type, HTTP status, or device message names the problem. Don't paraphrase it in your head — the specific words are the diagnosis.
4. Correlate. Match the automation log against the device's own syslog for the same moment. Two independent records of one event confirm what actually happened.
5. Classify and act. Almost every failure falls into a small number of recognizable categories, and the category points straight at the fix.

The discipline that matters most is step 2: read to the first error, not the last line. In a multi-step log the final line is typically a downstream symptom, not the cause.

Recognizable failure signatures

Most failures announce themselves. Learn these signatures and the message maps directly to a category — and the category maps directly to a fix.

Signature	Category	Fix
401 / 403, "Authentication failed"	Authentication	Wrong/expired credentials, or too little privilege
"Connection timed out" / "No route to host"	Reachability	Network path, ACL, or the device is down
429 Too Many Requests	Rate limiting	Back off and retry with jitter
RESTCONF 400 / 409, or % Invalid input	Rejected config	Body didn't match the model, or collided with existing state
"certificate verify failed"	TLS trust	Untrusted, expired, or mismatched certificate

The single most valuable distinction here is authentication versus reachability, because they look similar and demand completely different fixes. A 401/403 means the device answered and rejected your credentials (RFC 9110 §15.5.2/§15.5.4) — an identity or permission problem. A timeout or "no route to host" means you never reached it — a path, ACL, or device-down problem. One is fixed with a credential; the other with a route. Confusing them wastes an incident.

Worked example: reading an Ansible failure

An Ansible run fails on one device. Read the output down to the first meaningful line:

TASK [Push interface config] ***************************************
fatal: [r3]: FAILED! => {"changed": false,
  "msg": "Got a 401 from RESTCONF endpoint at https://r3/restconf/data"}

PLAY RECAP *********************************************************
r3 : ok=2  changed=0  unreachable=0  failed=1

The signature is unmistakable. A 401 is authentication — not reachability (the device answered) and not bad config (the request never got that far). Note unreachable=0 in the recap: the device was reached but rejected the request. The fix is the credentials or token for r3, not the playbook logic or the network path. Confirm by correlating with r3's syslog for a failed login at the same timestamp.

Worked example: read the traceback upward

A Python traceback ends with a summary line, but the first exception is the one that names the cause. Consider:

Traceback (most recent call last):
  ...
  File "deploy.py", line 44, in connect
    conn = device.connect(timeout=10)
netmiko.exceptions.NetmikoTimeoutException:
    Connection to 10.0.0.7 timed out
RuntimeError: deployment aborted

The final line, RuntimeError: deployment aborted, is a generic wrapper — a downstream symptom. The real cause is the NetmikoTimeoutException above it: a connection timeout to 10.0.0.7, which is a reachability problem. Fixing the RuntimeError handler would change nothing; the device was unreachable. Correlate with that device's syslog to confirm it was down or unreachable at that moment.

Worked example: a RESTCONF conflict

Not every non-2xx is auth or syntax. A RESTCONF write can return:

HTTP/1.1 409 Conflict
{ "errors": { "error": [ {
    "error-tag": "data-exists",
    "error-message": "object already exists: vlan 10" } ] } }

The 409 with the data-exists error-tag is defined by RFC 8040 §4.4.1: a POST to create a resource that already exists must fail this way. This is neither a credential problem nor malformed data — the operation collided with existing state, usually a POST (create) used where a PUT (replace) was intended. The fix is the HTTP method, or reconciling the object that's already there.

Exit codes and the layer map

When a pipeline job fails, the exit code localizes it fast: zero is success, any non-zero value is failure, and CI marks the stage red on the first non-zero code. A tool that "worked" but exited non-zero — a linter that found issues, a test that failed — still fails the stage, which surprises people who read the human-readable output and miss the code.

Once you have the first error, name the layer it lives in. That single classification usually points straight at the fix:

Symptom	Most likely layer
Connection refused / timeout / no route	Transport / reachability (device, port, firewall)
Authentication failed / 401 / permission denied	Credentials / AAA / RBAC
TLS / "certificate verify failed"	PKI / trust
400 / malformed payload / model error	Data / schema (YANG, JSON encoding)
429 Too Many Requests	Rate limiting — back off and retry (RFC 6585 §4)
Traceback in your own code	The automation logic itself

Key takeaways

• Work the method every time: locate the event, find the first error, read it literally, correlate the automation log with the device syslog, then classify and act.
• Read to the first error, not the last line. Later messages are usually cascading effects; a Python RuntimeError at the bottom often hides the real exception above it.
• Learn the signatures. 401/403 = authentication; timeout / no route = reachability; 429 = rate limit; RESTCONF 400/409 or % Invalid input = rejected config; "certificate verify failed" = TLS trust.
• Never confuse authentication with reachability — one is fixed with a credential, the other with a route.
• Trust the exit code. Non-zero fails the stage even when the output looks fine, and it tells CI where to stop.

---

Sources: RFC 9110 — HTTP Semantics (401/403), RFC 8040 — RESTCONF Protocol (§4.4.1, error responses), RFC 6585 — Additional HTTP Status Codes (429), Netmiko exceptions.