MJ12

The Infrastructure That Starts the Infrastructure

Categories: [Field Notes], [Infrastructure]
Tags: [infrastructure], [dns], [identity], [virtualization], [networking], [resilience], [architecture]

Modern infrastructure has become very good at consolidation.

Virtualization allows us to move services away from dedicated hardware. Central identity removes local accounts. DNS becomes authoritative and dynamic. Configuration management keeps systems consistent. Monitoring provides a common operational view. Certificates, secrets and authentication can all be centrally managed.

Each of those decisions is reasonable on its own.

Taken far enough, however, they can create an interesting recovery problem: the infrastructure required to recover the environment is hosted inside the environment that needs to be recovered.

This is the bootstrap problem.

The Circular Dependency Is Usually Invisible

During normal operation, the dependency chain is easy to overlook because everything is already running.

A hypervisor resolves the storage target through DNS. An administrator authenticates through central identity. Automation connects to hosts using names instead of addresses. Monitoring confirms the state of the cluster. Management interfaces use certificates issued by an internal authority. Time synchronisation keeps authentication and certificate validation reliable.

None of this is unusual.

Now consider the same environment after a complete site outage.

The hypervisors start first, but the DNS servers are virtual machines. The DNS servers are stored on shared storage. The storage management interface is normally accessed by hostname. Administrative authentication depends on the identity service, which is another VM. The automation system that normally restores configuration is also unavailable.

Individually, every system may be highly available.

Collectively, the recovery sequence has become circular.

A simplified version looks like this:

Virtualization cluster
        |
        v
   Core services
  DNS / Identity
  NTP / Automation
        |
        v
   Storage access
   Administration
   Service discovery
        |
        +------------------+
                           |
                           v
                Virtualization cluster

The diagram is deliberately uncomfortable.

Nothing in it is technically wrong. The problem is that the dependency direction changes during recovery.

Core Services Have a Different Job During an Incident

DNS is a good example because it is normally treated as an application-level service.

During recovery, it is closer to infrastructure.

The same is true for time synchronisation, identity, management access and sometimes certificate services. These systems are not necessarily more important than the applications running above them, but they have a different operational role: other systems require them before normal administration can resume.

That distinction matters.

A DNS service can be perfectly redundant across three virtual machines and still be unavailable after a platform-wide outage if all three instances depend on the same virtualization or storage layer.

Similarly, an identity system may have several replicas but still provide no useful redundancy during recovery if every replica is unreachable for the same reason.

Redundancy only protects against failures that are actually separated by the architecture.

Three copies inside one failure domain are still one failure domain.

This Does Not Mean Returning Everything to Physical Servers

The obvious reaction is to move every infrastructure service back onto dedicated hardware.

That is rarely necessary.

The objective is not physical independence for its own sake. The objective is to remove circular recovery dependencies.

A practical design may use a small number of deliberately placed infrastructure services outside the primary workload dependency chain.

One DNS and time service might live inside the main virtualization environment, while another exists on a separate utility host or at another site. Identity replicas can be distributed so that loss of one compute environment does not remove every authentication path. Management devices can use static addressing and local emergency access even if production DNS is unavailable.

The important detail is not where these services run.

It is whether enough of them remain available to begin recovery.

This is a subtle but useful difference between service availability and bootstrap availability.

A service may normally require a sophisticated redundant architecture. Bootstrap availability often requires something much smaller and deliberately boring.

Recovery Should Work With Fewer Dependencies Than Production

Normal operations benefit from abstraction.

We want hostnames instead of addresses. Central authentication instead of local passwords. Automation instead of manual configuration. Dynamic discovery instead of static mappings.

Recovery is different.

Every additional dependency creates another condition that must already be satisfied before the next step can begin.

This does not mean abandoning abstraction. It means retaining a minimal path underneath it.

For example, an administrator recovering a virtualization environment should ideally still be able to determine whether the core pieces are alive without depending on the entire service stack:

ping <management-address>

dig @<known-dns-address> example.internal

chronyc sources

curl -k https://<management-address>/

ip route

These are intentionally ordinary commands.

During a complicated incident, ordinary is useful.

The ability to reach a known management address directly can be more valuable than a sophisticated service-discovery mechanism that currently depends on the DNS server we are trying to start.

Identity Deserves Particular Attention

Central identity improves security and administration considerably, but it introduces another bootstrap dependency.

If every hypervisor, storage system, network appliance and management server requires central authentication, the loss of identity services can turn an infrastructure failure into an administrative lockout.

The answer is not to abandon central identity.

It is to preserve controlled emergency access.

That may mean local break-glass credentials on critical management systems, console access that does not depend on the identity platform, or carefully protected service accounts used only during recovery.

Those mechanisms should not become convenient alternatives to normal authentication. Their purpose is precisely the opposite: they exist for the rare case where normal authentication cannot exist yet.

Recovery mechanisms are most useful when they are boring, documented and almost never used.

Time Is More Important Than It Looks

Time synchronisation deserves a place in the same discussion because many other services quietly depend on it.

Authentication protocols, certificates, distributed databases, clustered services and log correlation all become more difficult when clocks diverge significantly.

NTP therefore occupies an unusual position in the dependency tree. It rarely attracts attention while working, but unreliable time can create failures that appear unrelated to time itself.

A resilient environment should have more than one usable time source, and at least one should remain reachable during the early stages of infrastructure recovery.

This is particularly important when recovery involves systems that validate certificates or use time-sensitive authentication.

A failed DNS server is obvious.

A clock that is several minutes wrong can be considerably less obvious.

The Real Test Is a Cold Start

The easiest way to discover circular dependencies is not by studying an architecture diagram.

It is by asking a simple question:

If the complete site lost power right now, what would we need in order to start the first useful service?

Then continue from there.

What does storage require?

What do the hypervisors require?

What does DNS require?

What does identity require?

Can administrators authenticate before identity is available?

Can storage be reached without DNS?

Can networking be managed without the virtualization cluster?

Can the environment establish reliable time before certificate validation becomes important?

The answers usually reveal the actual recovery architecture more accurately than a list of redundant components.

A service that survives the loss of one node may still be poorly positioned for a complete cold start.

Those are different resilience problems.

Small Infrastructure Can Be Extremely Valuable

There is also an architectural argument for keeping some infrastructure deliberately small.

Not every foundational service needs to share the same compute platform as the largest workloads.

A pair of modest utility systems running DNS, time or other basic network services can provide disproportionate operational value if they remain available while the primary environment is being recovered.

Likewise, a second site does not need to reproduce the entire production environment in order to help with bootstrap.

A small independent set of core services can provide DNS resolution, time, remote access and enough management capability to make recovery of the larger site considerably easier.

This is one of the few areas where architectural simplicity is itself a resilience feature.

The less a bootstrap service requires, the more useful it becomes when everything else is uncertain.

What We Are Trying to Achieve

The objective is not to create a separate physical platform for every infrastructure service, nor to avoid virtualization, central identity or automated management.

Those technologies make infrastructure substantially easier to operate.

The objective is to make sure that recovery does not depend on completing the recovery first.

DNS, identity, time, storage access and management services deserve special treatment because they help establish the conditions under which the rest of the environment can operate normally.

A resilient design therefore needs more than redundancy. It needs a clear bootstrap path.

The useful question is not simply whether a core service has two or three instances.

It is whether at least one of those instances will still be available when the platform underneath the others is not.

That is the difference between an environment that is highly available once it is running and one that can reliably find its way back from being completely stopped.