Failover that actually fails over: testing the second internet line

Why most failover has never worked

Second internet lines are bought in a moment of concern and then forgotten. The circuit is installed, somebody confirms it can reach the internet, it gets plugged into the firewall, and the project closes. Nobody disconnects the primary to see what happens, because doing that on purpose feels reckless and there is never a good moment.

So the arrangement sits there for years in an untested state. Then the primary fails, and one of a small number of things goes wrong: the second line was never configured as a route at all, or it was configured but the firewall needs a manual change, or it had quietly stopped working months earlier and nobody was monitoring it. The outage becomes two problems at once, which is the worst possible time to debug a configuration.

The uncomfortable truth is that an untested failover is roughly as useful as an untested backup, and for the same reason. The thing you are relying on has never been asked to do its job. We make the same argument about restores on backup that actually restores, and it applies here with less effort required to fix it.

What automatic failover actually means on a firewall

Automatic failover is a configuration on the device where both lines terminate, not a property of having two lines. The firewall needs both circuits presented to it, a way of deciding that the primary has failed, and rules describing what happens next.

The detection part is where most of the subtlety lives. A firewall that only watches whether the physical link is up will not notice the common failure, which is a line that is electrically fine but no longer carrying traffic. Detection therefore has to be based on reaching something beyond the provider, which means a probe to an external address, with sensible timings so a brief wobble does not trigger a switch and a genuine outage does.

The rules then say what moves. Usually everything, but not always: some offices deliberately keep a payment terminal or a voice service on a specific path. Whatever the decision, it should be written down rather than living in one engineer's memory, which is part of what a managed support arrangement is for.

Two circuits terminating in two separate routers, with no single device seeing both, cannot fail over automatically at all. That arrangement is more common than it should be, usually because the second line was added later by whoever installed it, and it is the configuration most likely to be discovered during an outage. If each line has its own router and nothing joins them, the office has two internet connections and no failover.

The test: disconnecting the primary on purpose

The test itself is almost disappointingly simple, which is why there is no excuse for not having done it. Pick a window when an interruption is tolerable, tell people it is happening, and physically disconnect the primary circuit at the firewall. Then watch.

Do it physically rather than by disabling the interface in software where you can. Pulling the cable simulates the real failure more honestly, and it catches configurations that depend on the interface being administratively up. Then leave it disconnected for longer than feels comfortable, at least fifteen or twenty minutes, because some problems only appear once sessions start timing out and applications begin retrying.

Test from more than one vantage point while it is down. A laptop on the office network is the obvious one, but also check that a remote worker can still reach whatever they normally reach, that email is still flowing rather than queuing somewhere, and that anything customer facing still answers. An office that keeps working internally while its public services are unreachable has half a failover, and that distinction only shows up if somebody looks from outside.

• How long until traffic flowed again, measured rather than estimated.
• Whether anybody had to do anything, and what.
• Which applications recovered on their own and which needed restarting.
• What happened to any call in progress.
• Whether the network separation still held on the backup path.
• Whether monitoring actually alerted somebody, or whether the outage was silent.

What breaks even when failover works correctly

A successful failover is not invisible, and expecting it to be sets people up to think it failed. When the path changes, your public address usually changes with it, and anything holding a long-lived session notices.

Voice calls in progress will normally drop. A call that is already connected is a session tied to an address, and the handset has to register again on the new path. Expect the call to end and the phone to come back within a short time. The same applies to video meetings, though most clients reconnect on their own.

VPN tunnels drop and re-establish. Remote desktop sessions drop. Large uploads restart. Anything with a strict address allowlist at the far end may be refused entirely, which is the failure that surprises people most: a bank portal or a client system that only accepts connections from your primary address will reject you from the backup until the second address is registered with them. That is worth finding out during a test rather than during an incident.

Printers and local devices are the opposite surprise: they usually carry on without noticing, because nothing about an internal print job depends on the route to the internet. Knowing which category each system falls into is most of the value of a test, because it tells you what to warn people about in advance next time rather than fielding the same questions during the incident.

Asymmetric capacity, and deciding what to shed

A backup path is usually slower than the primary, whether it is a smaller circuit, a mobile connection or a satellite link. If everything that normally runs simply keeps running, the backup saturates and the office concludes failover does not work when in fact it worked and was then overwhelmed.

So decide in advance what stops. Cloud backups, large file synchronisation and software update downloads are the usual candidates, and they are the right ones because none of them is urgent for the few hours an outage lasts. What should keep working is email, voice, the line of business application and anything customer facing.

On a mobile backup there is a second reason to shed load, which is cost. A metered connection carrying a full day of office traffic including backups produces a bill nobody budgeted for. The rule can be configured once on the firewall rather than remembered each time.

Failback is a second event

When the primary returns, traffic has to move back, and that transition is a second interruption that many offices have never considered. Done badly it is more disruptive than the original failure, because it tends to happen unannounced in the middle of the working day.

The two decisions are whether failback is automatic or manual, and how long the primary must be stable before it is trusted again. A line that is flapping, coming up and going down repeatedly, will move traffic back and forth and produce an afternoon of brief outages that is worse than staying on the backup. A hold timer prevents that, and it is a setting rather than a project.

Our preference for a small office is automatic failback with a conservative hold timer, because manual failback depends on somebody remembering, and what actually happens is that the office runs on the backup line for three weeks until the bill arrives.

Monitoring belongs in this conversation too. A failover that happens silently means the office runs on a backup path, possibly a metered one, with nobody aware that the primary has failed. The alert matters as much as the switch, and it should go to a person rather than only into a dashboard that gets looked at weekly.

Writing it down, and how often to retest

The output of a failover test is a short document, not a feeling that it went fine. It should say what was tested, on what date, how long the switch took, what dropped, what had to be restarted, and what was changed as a result. The value of that page is entirely in the next incident, when somebody who was not there needs to know what normal looks like.

Retest when something changes and otherwise once or twice a year. Changing either provider, replacing the firewall, moving office or adding a voice system are all reasons to test again, because each of them can invalidate what the last test proved. Twice a year is enough for a stable office, and it takes under an hour.

• Both providers, both circuit references, and which physical route each takes into the building.
• How failure is detected, and the timings configured.
• Measured switch time from the last test, and the date it was measured.
• What is deliberately paused on the backup path.
• The failback rule and its hold timer.
• Any external system that allowlists your address and therefore needs the second one registered.

When two lines are not the right answer

Testing often reveals that the second line is not independent. If both circuits share a building riser or a street duct, the test will pass happily and the real event will still take both down, because the thing that fails is upstream of the firewall entirely. Asking each provider which riser and which duct they use is free, and it is the question that decides whether any of this configuration matters.

Where the answer is that both share a path, the options are a genuinely different route if the building has one, a mobile connection on the firewall, or a satellite link, which shares no duct, riser or exchange with a cable. That comparison is set out on the satellite internet and Starlink page, and the wider multi-site version on the multi-location IT support page. What each provider can genuinely deliver to your unit is worth checking directly with du and e& rather than assuming.

The decision itself, and the question of whether your existing second line is independent at all, is set out in when Starlink is a real second line.

Frequently asked questions

Written by Srinivasan Ramachandran, Founder, Listonics · Published Oct 11, 2026 · Last reviewed October 11, 2026 · 2,097 words