IS-IS Watcher bootcamp runbook

1. How this runbook works

For each exercise: make one change, predict the IS-IS result, observe the Watcher event, then find the same fact in Topolograph Monitoring, in the SDK, and - for the correlation exercise - in the agent's answer. Restore the lab before the next independent exercise.

isis01 is a six-router FRR domain in one area (49.0001, AS 65100). router2-router3 runs Level-1-2; router3 to router6 and router3's leg to the router4/router5 LAN are Level-2-only. The Watcher rides a Level-2 adjacency on router1 and resolves each System ID to its hostname through the IS-IS dynamic-hostname TLV, so events name router2, router3, router6 rather than 0100.1001.000x.

1. The Watcher CSV line is the deterministic source event.
2. Monitoring and the SDK prove the event was ingested and is queryable.
3. The agent is used only to correlate several events into one incident, never as a replacement for the source line.

2. Bringing up the six-router lab

Run the public GRE-based isis01 topology from the IS-IS Watcher repository. prepare.sh also creates the isis-br-dr bridge and loads the MPLS kernel modules that IS-IS TE needs.

First check whether a lab is already running with sudo clab inspect --all. If a stale isis01 is listed, tear it down with sudo clab destroy --topo isis01.clab.yml --cleanup from containerlab/isis01/ before deploying again - name the topology file so no other lab is touched.

Command

cd containerlab/isis01
sudo clab inspect --all
sudo clab destroy --topo isis01.clab.yml --cleanup   # only if a stale isis01 is listed
sudo ./prepare.sh
sudo clab deploy --topo isis01.clab.yml
sudo docker logs clab-isis01-isis-watcher
sudo tail -f watcher/logs/watcher1.isis.log

Rollback: When you finish the runbook, remove the lab with sudo clab destroy --topo isis01.clab.yml --cleanup from containerlab/isis01/.

3. Network event record format

The runbook uses three event families: host, metric, and network. Read event_object as the changed object, event_status as the transition, and event_detected_by as the advertising or detecting router. graph_time is the watcher's own label for the run and selects the Topolograph graph.

IS-IS lines carry a level field (1 or 2) that OSPF lines do not - it is the third field, right after watcher_name. Read the metric line above as one sentence: at 2026-09-07T06:55:14Z the watcher lab-isis01 saw router3 re-advertise its Level-1 link toward router2 with the metric changed from 10 to -1 (adjacency lost), on the interface addressed 192.168.23.2 in area 49.0001 / AS 65100. The identity behind router2 is its NET / System ID 49.0001.0100.1001.0002.00; the watcher prints the hostname because every router advertises the dynamic-hostname TLV.

A plain host or network up/down line carries no cost fields, so Fluent Bit forwards only the paired changed line. Topolograph reads an adjacency going down from the metric line whose new_cost is -1, and coming back from the one whose old_cost is -1.

1. Fields in order: watcher_time, watcher_name, level, event_name, event_object, event_status, [cost fields], event_detected_by, graph_time, area_num, asn, [local_ip, remote_ip | subnet_type, int_ext_subtype], sesid, srcid.
2. area_num 49.0001 and asn 65100 identify the routing domain; sesid is the watcher session, srcid the source router System ID (router1, 0100.1001.0001).

Watcher CSV

host:    2026-09-07T06:55:14.754Z,lab-isis01,1,host,router2,down,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.23.2,192.168.23.1,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
metric:  2026-09-07T06:55:14.756Z,lab-isis01,1,metric,router2,changed,old_cost:10,new_cost:-1,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.23.2,192.168.23.1,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
network: 2026-09-07T06:55:14.761Z,lab-isis01,1,network,192.168.23.0/24,changed,old_cost:10,new_cost:-1,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001

4. Where events come from

One IS-IS change produces one Watcher CSV line per level. Fluent Bit forwards it to Topolograph, which stores it and exposes it through the Monitoring page, the event API, and the SDK. Each exercise below checks the same fact at every level available to you.

1. isis01 Watcher log -> Fluent Bit CSV parser -> Topolograph ingestion -> Monitoring page + event API -> Topolograph SDK
2. Monitoring page: OSPF/IS-IS Real-Time Monitoring. Pick the graph by its timestamp in Choose the graph, set the From/To window in UTC, turn on the L1 and L2 toggles, click Find logs; the New/Old Subnets, Up/Down Links and Changed metric toggles filter what is listed.
3. The graph selector lists every topology snapshot the watcher reported - that is the topology the watcher sent.

SDK request

from topolograph import Topolograph

topo = Topolograph(url="http://<your-topolograph>:8080",
                   username="<email>", password="<password>")
graph = topo.graphs.get(latest=True)
print(graph.graph_time, graph.protocol, graph.hosts)

SDK output

07Sep2026_06h49m43s_6_hosts isis {'count': 6}

5. Cost change on a point-to-point link

Change router2 eth1 toward router3. IS-IS metrics are directed, so the reverse router3-to-router2 metric must not be described as the same scalar. router2 eth1 has no explicit isis metric, so it starts at the wide-metric default of 10. router2-router3 is a Level-1-2 circuit, so the change is advertised at both levels.

Command

sudo docker exec clab-isis01-router2 vtysh \
  -c 'conf t' -c 'interface eth1' -c 'isis metric 222'

Verify the change. Open each source available in your environment and check it against the examples below.

# router2: interface eth1 / isis metric 222
2026-09-07T06:52:15.035Z,lab-isis01,1,metric,router3,changed,old_cost:10,new_cost:222,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:52:15.037Z,lab-isis01,1,network,192.168.23.0/24,changed,old_cost:10,new_cost:222,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:52:15.045Z,lab-isis01,2,metric,router3,changed,old_cost:10,new_cost:222,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
# rollback (isis metric 10):
2026-09-07T06:52:37.514Z,lab-isis01,1,metric,router3,changed,old_cost:222,new_cost:10,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:52:37.526Z,lab-isis01,2,metric,router3,changed,old_cost:222,new_cost:10,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001

adj = graph.events.get_adjacency_events(
    start_time="2026-09-07T06:52:14Z", end_time="2026-09-07T06:52:20Z")
net = graph.events.get_network_events(
    start_time="2026-09-07T06:52:14Z", end_time="2026-09-07T06:52:20Z")
for e in adj["adjacency_cost_change_events"]:
    print(e.event_object, e.event_detected_by, e.old_cost, "->", e.new_cost, "L" + str(e.level_number))
for e in net["network_cost_change_events"]:
    print(e.event_object, e.old_cost, "->", e.new_cost, "L" + str(e.level_number))

router3 router2 10 -> 222 L1
router3 router2 10 -> 222 L2
3ffe::192:168:23:2/127 10 -> 222 L1
192.168.23.0/24 10 -> 222 L1
3ffe::192:168:23:2/127 10 -> 222 L2
192.168.23.0/24 10 -> 222 L2

Rollback: Run isis metric 10 (or no isis metric) on router2 eth1 and confirm the inverse 222 -> 10 metric and network events.

6. Detecting internal and external prefix events

Run each prefix experiment independently. Unlike OSPF, IS-IS advertises a loopback with the mask it is configured with and costs it at the interface metric - a /24 stays a /24 at cost 10, it is not collapsed to a /32 host route. router6 is Level-2-only, so its prefixes appear at L2 alone.

1. 6a. On router2, interface lo / ip address 192.168.123.1/24; observe 192.168.123.0/24 up and cost -1 -> 10 at L1 and L2.
2. 6b. On router6, interface lo / ip address 10.10.36.6/24; observe 10.10.36.0/24 up and cost -1 -> 10 at L2.
3. 6c. On router6, no ip route 6.6.6.6/32 192.168.36.3; observe 6.6.6.6/32 down and cost 11 -> -1 at L2. FRR redistributes it into IS-IS without the external bit, so the watcher tags it internal.

Command

sudo docker exec clab-isis01-router2 vtysh \
  -c 'conf t' -c 'interface lo' -c 'ip address 192.168.123.1/24'

Verify the change. Open each source available in your environment and check it against the examples below.

# 6a router2: interface lo / ip address 192.168.123.1/24
2026-09-07T06:52:52.074Z,lab-isis01,1,network,192.168.123.0/24,up,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:52:52.075Z,lab-isis01,1,network,192.168.123.0/24,changed,old_cost:-1,new_cost:10,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
# 6b router6: interface lo / ip address 10.10.36.6/24
2026-09-07T06:53:39.099Z,lab-isis01,2,network,10.10.36.0/24,up,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:53:39.099Z,lab-isis01,2,network,10.10.36.0/24,changed,old_cost:-1,new_cost:10,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
# 6c router6: no ip route 6.6.6.6/32 192.168.36.3
2026-09-07T06:54:14.533Z,lab-isis01,2,network,6.6.6.6/32,down,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:54:14.533Z,lab-isis01,2,network,6.6.6.6/32,changed,old_cost:11,new_cost:-1,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001

for start, end in [("2026-09-07T06:52:50Z", "2026-09-07T06:52:55Z"),
                   ("2026-09-07T06:53:37Z", "2026-09-07T06:53:42Z"),
                   ("2026-09-07T06:54:12Z", "2026-09-07T06:54:17Z")]:
    net = graph.events.get_network_events(start_time=start, end_time=end)
    for e in net["network_up_down_events"]:
        print(e.event_object, e.event_status, e.event_detected_by,
              f"{e.old_cost} -> {e.new_cost}", "L" + str(e.level_number), e.subnet_type)

192.168.123.0/24 changed router2 -1 -> 10 L1 internal
192.168.123.0/24 changed router2 -1 -> 10 L2 internal
10.10.36.0/24 changed router6 -1 -> 10 L2 internal
6.6.6.6/32 changed router6 11 -> -1 L2 internal

Rollback: 6a: no ip address 192.168.123.1/24 on router2 interface lo. 6b: no ip address 10.10.36.6/24 on router6 interface lo. 6c: ip route 6.6.6.6/32 192.168.36.3 on router6. Confirm the inverse event after each.

7. Detecting a connectivity loss and its recovery

Shut router2 eth1, inspect the correlated event set, then restore the interface with no shutdown. Because router2-router3 is Level-1-2, every line appears once for L1 and once for L2.

Command

# down
sudo docker exec clab-isis01-router2 vtysh \
  -c 'conf t' -c 'interface eth1' -c 'shutdown'
# recovery
sudo docker exec clab-isis01-router2 vtysh \
  -c 'conf t' -c 'interface eth1' -c 'no shutdown'

Verify the change. Open each source available in your environment and check it against the examples below.

# router2: interface eth1 / shutdown
2026-09-07T06:55:14.754Z,lab-isis01,1,host,router2,down,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.23.2,192.168.23.1,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:55:14.756Z,lab-isis01,1,metric,router2,changed,old_cost:10,new_cost:-1,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.23.2,192.168.23.1,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:55:14.760Z,lab-isis01,1,metric,router3,changed,old_cost:10,new_cost:-1,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:55:14.761Z,lab-isis01,1,network,192.168.23.0/24,changed,old_cost:10,new_cost:-1,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
# router2: interface eth1 / no shutdown
2026-09-07T06:55:39.545Z,lab-isis01,1,metric,router3,changed,old_cost:-1,new_cost:10,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:55:39.607Z,lab-isis01,1,metric,router2,changed,old_cost:-1,new_cost:10,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.23.2,192.168.23.1,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001

down = graph.events.get_adjacency_events(
    start_time="2026-09-07T06:55:14Z", end_time="2026-09-07T06:55:16Z")
for e in down["all_host_up_down_events"]:
    print(e.event_object, "detected_by", e.event_detected_by,
          f"{e.old_cost} -> {e.new_cost}", "L" + str(e.level_number))

router2 detected_by router3 10 -> -1 L1
router3 detected_by router2 10 -> -1 L1
router2 detected_by router3 10 -> -1 L2
router3 detected_by router2 10 -> -1 L2

Rollback: no shutdown on router2 eth1 restores the interface; wait for the host, network and metric recovery events at both levels.

8. Reporting on a broadcast transit segment

router6 eth1 faces router3 on a broadcast (LAN) circuit. router6 carries isis priority 100 against router3's 64, so router6 is the DIS and originates the pseudonode LSP. The circuit is Level-2-only, so every line is L2. router6 eth1 has no explicit isis metric, so the baseline is the default 10.

Command

# cost
sudo docker exec clab-isis01-router6 vtysh \
  -c 'conf t' -c 'interface eth1' -c 'isis metric 66'
# then, separately: shutdown, then no shutdown
sudo docker exec clab-isis01-router6 vtysh \
  -c 'conf t' -c 'interface eth1' -c 'shutdown'
sudo docker exec clab-isis01-router6 vtysh \
  -c 'conf t' -c 'interface eth1' -c 'no shutdown'

Verify the change. Open each source available in your environment and check it against the examples below.

# router6: interface eth1 / isis metric 66   (rollback: isis metric 10)
2026-09-07T06:56:47.139Z,lab-isis01,2,network,192.168.36.0/24,changed,old_cost:10,new_cost:66,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:56:47.140Z,lab-isis01,2,metric,router3,changed,old_cost:10,new_cost:66,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:57:04.561Z,lab-isis01,2,metric,router3,changed,old_cost:66,new_cost:10,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
# router6: interface eth1 / shutdown
2026-09-07T06:57:22.037Z,lab-isis01,2,network,192.168.36.0/24,changed,old_cost:10,new_cost:-1,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:57:22.055Z,lab-isis01,2,host,router6,down,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.36.3,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:57:22.055Z,lab-isis01,2,metric,router6,changed,old_cost:10,new_cost:-1,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.36.3,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
# router6: interface eth1 / no shutdown
2026-09-07T06:57:55.962Z,lab-isis01,2,host,router6,up,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.36.3,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:57:55.973Z,lab-isis01,2,metric,router3,changed,old_cost:-1,new_cost:66,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001

cost = graph.events.get_adjacency_events(
    start_time="2026-09-07T06:56:46Z", end_time="2026-09-07T06:56:49Z")
for e in cost["adjacency_cost_change_events"]:
    print("cost", e.event_object, e.event_detected_by, f"{e.old_cost} -> {e.new_cost}", "L" + str(e.level_number))
flap = graph.events.get_adjacency_events(
    start_time="2026-09-07T06:57:21Z", end_time="2026-09-07T06:58:00Z")
for e in flap["all_host_up_down_events"]:
    print("updown", e.event_object, e.event_detected_by, f"{e.old_cost} -> {e.new_cost}", "L" + str(e.level_number))

cost router3 router6 10 -> 66 L2
updown router6 router3 10 -> -1 L2
updown router6 router3 -1 -> 10 L2
updown router3 router6 -1 -> 10 L2

Rollback: isis metric 10 (or no isis metric) on router6 eth1 restores the baseline 10; no shutdown restores the adjacency. Confirm the inverse metric events.