Skip to content

Change validation with pyATS

A playbook that reports success only tells you the commands were sent — not that the network is doing what you intended. This article covers how to close that gap with pyATS, Cisco's Python test and validation framework, and its companion library Genie: how to describe your devices in a testbed, snapshot operational state before and after a change, and prove with a diff that the change did exactly what it should and nothing else.

pyATS and Genie

pyATS (Python Automated Test System) is the test harness — it connects to devices, runs test cases, and produces reports. Genie is the library layer that turns raw device output into structured data through a large set of vendor parsers and feature models. Together they answer the question that matters after any change: did it work, and did it break anything else?

The key idea is that Genie normalizes messy CLI text into predictable Python dictionaries. Once state is structured, you can compare two snapshots mechanically instead of eyeballing show output, and you can assert on a specific field instead of grepping for a string.

The testbed file

Everything starts with a testbed — a YAML file describing the devices, how to reach them, and the credentials to use. Crucially, secrets are never written into the file; they are referenced from the environment with pyATS's %ENV{...} markup, so the testbed can be committed to Git safely and the real password is injected at run time from a vault or a masked CI variable.

# testbed.yaml — device inventory and connection details
testbed:
  name: core-testbed
devices:
  r1:
    os: iosxe
    type: router
    credentials:
      default:
        username: automation
        password: "%ENV{DEVICE_PASSWORD}"
    connections:
      cli:
        protocol: ssh
        ip: 10.0.0.1

Before relying on a testbed, confirm it is well-formed and every device is reachable:

pyats validate testbed --testbed-file testbed.yaml

The learn / diff workflow

The single most useful pyATS feature for change validation is snapshot and compare. You learn the relevant features into a "before" snapshot, apply your change, learn an "after" snapshot, then diff the two. Because Genie stores each feature as structured data, the diff reports precisely what moved in operational state — Cisco's own "compare network states" quickstart is built entirely around this loop.

# 1) snapshot BEFORE the change
pyats learn ospf bgp interface platform \
  --testbed-file testbed.yaml --output before/

# 2) ... apply your automation change (Ansible, RESTCONF, Terraform, ...) ...

# 3) snapshot AFTER the change
pyats learn ospf bgp interface platform \
  --testbed-file testbed.yaml --output after/

# 4) compare the two snapshots
pyats diff before/ after/

Each feature named after learn (ospf, bgp, interface, platform, …) is collected across every device in the testbed and written to the output directory as structured data. The diff then reads like a source-control diff — additions, deletions, and changes — scoped to operational state:

--- before/connection_iosxe_r1_ospf.txt
+++ after/connection_iosxe_r1_ospf.txt
 'areas':
   '0.0.0.0':
     'interfaces':
+      'GigabitEthernet3':
+        'state': 'up'

A diff limited to exactly what you intended — here, GigabitEthernet3 joining OSPF area 0 — is objective proof the change landed. The real value shows up when an unexpected line appears: a route that vanished, a neighbor that dropped. That is collateral damage from your change, surfaced during the change window instead of discovered by users hours later. A before/after diff that shows the intended QoS policy plus a dropped OSPF adjacency on an unrelated interface has just caught a bug an ACL typo would otherwise have hidden.

This is the difference between "reported success" and "provably correct."

Structured parsing with Genie

The same normalization powers ad-hoc checks. pyats parse runs a command and returns Genie's structured representation, so you assert on a field rather than a substring:

pyats parse "show ip interface brief" \
  --testbed-file testbed.yaml --devices r1

{
  "interface": {
    "GigabitEthernet1": { "ip_address": "10.0.0.1", "status": "up", "protocol": "up" },
    "GigabitEthernet3": { "ip_address": "unassigned", "status": "down", "protocol": "down" }
  }
}

Scripted tests for the pipeline

Learn/diff is ideal at a keyboard during a change window. For repeatable validation that a pipeline can gate on, write a scripted test with pyATS's AEtest framework: connect, parse state, and assert. A failed assertion fails the job — exactly the gate a post-deploy stage needs.

# interface_checks.py
from pyats import aetest
from genie.testbed import load


class CommonSetup(aetest.CommonSetup):
    @aetest.subsection
    def connect(self):
        self.parent.parameters["tb"] = tb = load("testbed.yaml")
        tb.devices["r1"].connect()


class VerifyInterface(aetest.Testcase):
    @aetest.test
    def gi1_is_up(self):
        tb = self.parent.parameters["tb"]
        out = tb.devices["r1"].parse("show ip interface brief")
        if out["interface"]["GigabitEthernet1"]["status"] != "up":
            self.failed("GigabitEthernet1 is not up")

A single script runs from a job file, which pyats run job executes through the easypy runner. easypy runs each task, aggregates pass/fail, and produces a consolidated report and log archive — the artifacts a CI stage stores and a human reviews.

# job.py  ->  pyats run job job.py --testbed-file testbed.yaml
from pyats.easypy import run


def main(runtime):
    run(testscript="interface_checks.py", runtime=runtime)

Wired into a pipeline, this is what turns "I ran a test on my laptop" into "every change is validated automatically": snapshot the network, let the deploy run, snapshot again, and fail the stage if the diff — or an AEtest assertion — is not what you intended.

Key takeaways

  • pyATS validates changes; Genie parses. pyATS connects, tests, and reports; Genie's parsers turn CLI output into structured data you can compare and assert on.
  • A testbed YAML defines devices, connections, and credentials — and keeps secrets out of the file with %ENV{...}, so it is safe to commit.
  • The core workflow is pyats learn → change → pyats learn → pyats diff. Snapshot operational state before and after, then compare.
  • The diff proves both halves: that the intended change happened, and that nothing unexpected moved.
  • Script it with AEtest and run it as a job to make validation a repeatable, pass/fail gate in a CI/CD pipeline.

Sources: pyATS & Genie on Cisco DevNet, Compare network states — pyATS getting-started quickstart, pyATS key concepts and glossary.