What breaks when a site loses its clock

Time and name resolution are usually designed as though they arrive from somewhere else. A DHCP scope hands out a resolver and a time source, both of them point outward, and the arrangement works for exactly as long as the path outward is available.

At a site expected to keep working when its uplink is not, they stop being upstream services and become local dependencies. The failures they produce rarely announce themselves as time or name resolution problems, which is most of what makes them expensive.

The symptom does not say “time”

Kerberos-style authentication rejects a request whose timestamp falls outside a configured skew window. One-time passcodes are derived from a counter based on the clock, and a validator accepts only a small number of steps either side of its own. Certificate validation compares the current time against a validity period. Signed DNS responses carry their own inception and expiry timestamps. Correlation across systems does not fail at all, which is worse: the timeline is simply wrong, and nothing reports an error.

These do not fail together and they do not fail at the same point. The tolerances differ by orders of magnitude: a skew window measured in minutes, a passcode step measured in tens of seconds, a certificate validity period measured in months. A relatively small error can break one mechanism while the others continue to work. A much larger error can make several fail at once, and by then the support queue is full of problems that look unrelated to each other.

A clock with no discipline free-runs. How fast it diverges depends on the oscillator, the hardware and the temperature, and there is no universal figure worth quoting. The architectural question is not the rate. It is the relationship between that rate and the tightest tolerance the site depends on, and how long the site is expected to hold without an external source.

Distributing the time source

DHCP can advertise a time source. RFC 2132 defines Option 42 as a list of NTP server addresses in order of preference. What it defines is what the server advertises. It places no obligation on the client, and that distinction is the whole of the problem.

Both ends of the range are documented. Microsoft's Windows Time service documentation defines its time sources as the domain hierarchy or a configured NTP server list, set through the registry or Group Policy, and does not document DHCP Option 42 as an input. By contrast, systemd-networkd explicitly documents DHCP-provided NTP servers as enabled by default, with those servers taking precedence over statically configured ones.

One protocol option, very different documented client behavior. Distribution has to be verified at the client, not inferred from the DHCP scope.

The related habit is hardcoding public time sources into endpoint images or configuration baselines. It looks robust, because the addresses are stable and well known. It is fragile in precisely the case being designed for: the site loses the path to those addresses at the same moment it loses everything else, and every endpoint is left holding a time source it can no longer reach.

Relaying is not being an authority

Edge devices frequently have a time server setting, and the presence of that setting is easy to read as a guarantee that the device can serve time to the site. It is not the same claim.

Two capabilities are involved. One is passing time along from an upstream authority the device can currently reach. The other is remaining a usable authority for the site after that upstream is gone. A device can do the first without doing the second, and the configuration for both can look identical, because the difference only appears in a state that does not exist while everything is working.

Feature naming varies and so does behavior. The operational way to verify which behavior a device provides is to remove its upstream source and watch what it tells its clients afterward. That is a test, not a datasheet question.

Internal consistency is not accurate time

A time service that has lost its upstream sources has two honest options. It can report itself as unsynchronized, in which case clients seek another acceptable source and, if none is available, eventually free-run on their own oscillators. Or it can be configured to keep serving, holding the site together on its own clock.

The second is often what an isolated-site design is trying to achieve. Implementation-specific features such as orphan or local-reference modes can provide that behavior. What they provide needs stating precisely, because the configuration can sound more authoritative than it really is. It keeps the site internally consistent: clients continue to agree with the local server and therefore with each other, which is what skew windows and cross-system correlation actually require. It does not manufacture correct UTC. The whole site drifts together, away from the outside world, at whatever rate the local oscillator dictates.

Those are two different requirements, and a design should say which one it is buying. A site can remain internally consistent for a long outage and still be wrong enough to fail validation against an outside system once connectivity returns.

Name resolution has the same shape

A site usually contains services that do not need the internet but do need names. If every resolver at the site ultimately depends on a resolver outside it, name resolution has an uplink dependency whether or not the service being resolved needs the internet.

A cache is not a substitute. It answers for names that have already been asked for, for as long as their records remain valid, and it caches failures as well as successes. A resolver that happens to be warm when the uplink drops looks like a working design for a while and then stops being one, which is the least useful failure mode available: it postpones the symptom past the point where anyone connects it to the outage.

What survives isolation is a resolver that is locally authoritative for internal names, holding the data rather than the memory of having fetched it. That is an architectural property, and it does not follow from caching.

Test the site with the uplink down

Take the site's external uplink away deliberately, in a maintenance window, and look at what the clients still have.

Note the thing not to test. Watching whether clocks still look correct proves nothing over a short window: drift is far too slow to be visible in minutes, so a site whose time architecture is entirely broken will show perfectly reasonable clocks for the whole of the test. The question is not whether the time is still right. It is whether the client still has a valid local source of it.

Four things to establish:

• The local time service still reports a source state consistent with the disconnected design that was intended, rather than having quietly declared itself unsynchronized.
• Clients are selecting that local source, checked in the client's own view of which source it is using.
• No tested client has silently fallen back to free-running on its own oscillator, or to an external source it can no longer reach.
• Internal names resolve from data the site genuinely holds, verified against a name that was not looked up before the uplink was removed, so a warm cache cannot answer on the architecture's behalf.

Anything that fails is an uplink dependency the design did not intend. The alternative is finding it during a real outage, which is the one occasion when nobody is free to investigate it.

Explore this expertise: Network & Infrastructure

All technical notes