Skip to content

Building a telemetry collector stack with Telegraf, InfluxDB, and Grafana

Model-driven telemetry gets structured operational data off the device. This article is about where that data lands: the collector. You'll build the classic open-source TIG pipeline — Telegraf ingests, InfluxDB stores, Grafana visualizes — and wire it to both styles of subscription so the stream you configured on the box actually becomes a dashboard you can watch.

If you haven't set up the device side yet, start with model-driven telemetry; this picks up where that leaves off, at the receiver.

The collector's one job

A telemetry subscription pushes YANG-modeled data keyed by its sensor path. The collector's job is to receive that stream, write it to a time-series database, and make it queryable. The important property — and the whole reason model-driven telemetry beats SNMP polling — is that the data arrives already structured. Telegraf doesn't screen-scrape or regex show output; it maps the YANG path straight to a measurement and its leaves to fields. No parsing step means no fragile parser to maintain.

Three components, each doing one thing:

Component Role
Telegraf Ingest — receives or subscribes to the stream, writes line protocol out
InfluxDB Store — a time-series database holding the samples with retention
Grafana Visualize — queries InfluxDB and renders dashboards and alerts

Telegraf: the ingest layer

Telegraf is a plugin-driven agent: input plugins bring data in, output plugins send it on. For telemetry you pick an input that matches the direction of your subscription, and the influxdb_v2 output to store it.

Dial-out (the device connects to you). Use the cisco_telemetry_mdt input. It's a service input — it listens on a port and accepts the gRPC stream the device dials out to. It consumes MDT from IOS XR, IOS XE, and NX-OS over TCP or gRPC:

# telegraf.conf — receive gRPC dial-out MDT
[[inputs.cisco_telemetry_mdt]]
  transport = "grpc"
  service_address = ":57500"   # must match the receiver port in the subscription

That service_address port is exactly the receiver ip address ... 57500 your IOS XE subscription points at. The device initiates the connection; Telegraf accepts it.

Dial-in (you subscribe to the device). Use the vendor-agnostic gnmi input. Telegraf opens a gNMI Subscribe session to the device and requests paths dynamically — nothing is pre-configured on the box:

# telegraf.conf — subscribe to a device over gNMI (dial-in)
[[inputs.gnmi]]
  addresses = ["10.0.0.1:57400"]
  username  = "telemetry"
  password  = "${DEVICE_PASS}"   # from the environment, never inline
  encoding  = "proto"
  redial    = "10s"              # reconnect if the session drops

  [[inputs.gnmi.subscription]]
    name              = "if_counters"
    origin            = "openconfig-interfaces"
    path              = "/interfaces/interface/state/counters"
    subscription_mode = "sample"   # or "on_change"
    sample_interval   = "10s"

The subscription_mode and sample_interval here are the dial-in equivalents of the cadence you'd set on a dial-out subscription: sample is periodic, on_change sends only on change. Enable TLS with tls_enable = true and the usual tls_ca/tls_cert/tls_key options — note the old enable_tls option was renamed to tls_enable in Telegraf 1.27.0.

Dial-out over gNMI

Newer Telegraf releases also ship a gnmi_listener input for devices that push over gNMI (gNMI dial-out and some vendor dial-out protocols). The gnmi input above is the dial-in subscriber; cisco_telemetry_mdt is the Cisco dial-out receiver. Match the plugin to who opens the connection.

The output. Whichever input you use, one output stores it. InfluxDB 2.x takes line protocol over its /api/v2/write endpoint, and Telegraf's influxdb_v2 plugin speaks it:

[[outputs.influxdb_v2]]
  urls         = ["http://influxdb:8086"]
  token        = "${INFLUX_TOKEN}"
  organization = "netops"
  bucket       = "telemetry"

InfluxDB: buckets and retention

In InfluxDB 2.x a bucket combines what 1.x split into a database and a retention policy. Every point you write lands in a bucket, and the bucket's retention period decides how long it lives — set it to 0 for no expiry, or a finite window so high-rate counters don't fill the disk forever.

Retention is the knob that keeps a telemetry store from growing without bound. A sensible pattern is a short-retention bucket for raw high-frequency samples and a longer one for downsampled rollups, but even a single bucket with a 30- or 90-day window is enough to start. The other thing to watch is series cardinality: every unique combination of measurement and tag values is a distinct series, and telemetry tagged per-interface across a large fleet multiplies fast. Keep high-churn values (timestamps, counters) as fields, not tags.

Grafana: from stored series to a dashboard

Add InfluxDB as a Grafana data source and point it at the same org, token, and default bucket. For InfluxDB 2.x the query language is Flux:

# Flux: interface input rate over the last hour
from(bucket: "telemetry")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "interfaces" and r._field == "in-octets")
  |> derivative(unit: 1s, nonNegative: true)

One gotcha if you prefer the older InfluxQL instead of Flux: InfluxDB 2.x requires a DBRP mapping (database/retention-policy → bucket) before InfluxQL queries resolve, or Grafana reports "retention policy not found." Flux needs no such mapping, so it's the path of least resistance for a new v2 stack.

From there, build panels for the signals that matter — interface utilization, CPU, protocol adjacencies, environment sensors — and set Grafana alerts on the ones you'd page on.

Operating the stack

A few things separate a demo from something you'd run:

  • Secrets stay out of config. Telegraf expands ${VAR} from its environment, so pull device passwords and the InfluxDB token from the environment or a secret store, never inline in telegraf.conf. (See secure coding practices.)
  • Encrypt the stream. Enable TLS on both the device subscription and the Telegraf input. Telemetry carries real operational detail about your network.
  • Store ≠ alert. InfluxDB is your historical record; Grafana alerting (or a dedicated alerting path) is how a human finds out. Don't confuse a full dashboard with a working alert — decide deliberately what pages someone.
  • Run it as code. The whole stack is three containers; define it with a Docker Compose file and keep telegraf.conf in Git so the collector is as reproducible as the automation feeding it.

Key takeaways

  • The collector turns a pushed telemetry stream into a queryable dashboard; TIG = Telegraf (ingest) → InfluxDB (store) → Grafana (visualize).
  • Because the data arrives structured by its YANG path, Telegraf writes clean labeled series with no parsing — the core payoff of model-driven telemetry.
  • Match the Telegraf input to the subscription direction: cisco_telemetry_mdt receives Cisco dial-out, gnmi performs dial-in, and the influxdb_v2 output stores both.
  • In InfluxDB 2.x a bucket bundles database + retention; set retention to bound growth and keep high-churn values as fields to control cardinality.
  • Grafana queries v2 with Flux (InfluxQL needs a DBRP mapping first); store everything, but alert deliberately.
  • Keep secrets in the environment, enable TLS end to end, and define the stack as code.

Sources: Telegraf cisco_telemetry_mdt input plugin, Telegraf gnmi input plugin, Telegraf influxdb_v2 output plugin, InfluxDB 2.x buckets and retention, Use Grafana with InfluxDB, gNMI specification (OpenConfig).