Helpdesk story

How do I find what changed when a site drops every Thursday?

To find what changed when a site drops on a schedule, line up the outage times against the event and change history. In this story the Memphis tunnels drop every Thursday around 9:40, the event history shows the first link-down was seven weeks ago, and a QoS rule scheduled for Thursday 9:30 to 10:30 was added that same week. The history is kept for 18 months, so with the event log set to look back 18 months the first occurrence is still on record.

Watch the ticket get answered

Ticket #4490 · Tier 2 · Thu 9:44 AM
Memphis dropped again. It did this last Thursday too.

Every Thursday, around 9:40.

30 days of tunnel status at a glance. The Memphis tunnels drop on every Thursday in the window.

30 days agoThu (today)
  • normal
  • tunnel down: 5 days

It started seven weeks ago. Never before.

The collector keeps 18 months of firewall events. With the event log set to look back 18 months, Memphis WAN link-down events start seven weeks ago, every Thursday since.

First link-down: 7 weeks ago

18 months ago12 months6 monthsToday
  • other events
  • Memphis WAN link-down

A change from that same week.

With config changes set to the last 90 days and Memphis-850 picked, one change lands that same week.

Memphis-850 · QoS policy rule "Backup-Window"Added by m.chen · 7 weeks ago
added schedule: Thu 09:30–10:30 "backup-window"
added source: backup-server destination: any
added class: 1

Found in one sitting: a weekly QoS rule starving the tunnel.

Step by step

Ticket #4490 · Tier 2 · Thu 9:44 AM: “Memphis dropped again. It did this last Thursday too.”

30 days of tunnels: Every Thursday, around 9:40.

The 30-day tunnel strip shows the Memphis tunnels dropping on every Thursday in the window, around 9:40.

  • Timeline: Thu (today).

18 months of events: It started seven weeks ago. Never before.

With the event log set to look back 18 months, the event history shows the first WAN link-down seven weeks ago, with one every Thursday since and none before.

  • First link-down: 7 weeks ago.

What changed: A change from that same week.

With config changes set to the last 90 days and Memphis-850 picked, the change history shows a QoS rule, "Backup-Window", added by m.chen that same week. It is scheduled for Thursday 9:30 to 10:30, which matches the drops.

  • Memphis-850 · QoS policy rule "Backup-Window"
  • Added by m.chen · 7 weeks ago
  • + schedule: Thu 09:30–10:30 "backup-window"
  • + source: backup-server destination: any
  • + class: 1

The answer: Found in one sitting: a weekly QoS rule starving the tunnel.

How Firewall Fleet Health spots this

  • Shows 30 days of tunnel status for every site, so a weekly pattern is easy to see.
  • Keeps 18 months of firewall events and config changes, so you can find the first time it happened.
  • Shows each configuration change with what was added, who made it and when.
  • The Events tab holds the firewall event log and the config change history, filterable by firewall. Set each one to look back 10, 30 or 90 days, 12 months or 18 months, so you can line up when a problem started with what changed that week.

Common questions

How do I find what changed when a site drops every Thursday?

Find the first outage in the event history, then look at the configuration changes from that week. Firewall Fleet Health keeps 18 months of firewall events and config changes, and shows each change with its author and time.

How long are tunnel status and event history kept?

Firewall Fleet Health keeps 30 days of tunnel status and 18 months of firewall events and config changes.

See your own fleet this way

Explore the demo fleet yourself.

Schedule a demo Explore the live demo (opens in a new tab)