MJ12

High Availability Is More Than a Running VM

Categories: [Field Notes], [Infrastructure]
Tags: [virtualization], [xcp-ng], [high-availability], [storage], [architecture], [operations]

A virtual machine can recover successfully while the service it belongs to remains unavailable. The difference matters more than it first appears.

High availability is one of those terms that sounds precise until several different parts of an infrastructure start using it at the same time.

A virtualization platform may provide HA. Storage may be redundant. An application may run on several nodes. A database may replicate. A load balancer may have a standby instance.

All of those statements can be true, while the service as a whole can still fail.

The reason is simple: availability does not belong to a single component. It exists across the complete path between the user and the data the user is trying to reach.

Where Hypervisor HA Ends

At the virtualization layer, high availability solves a very specific problem.

If a physical host disappears, can the workloads running on it continue somewhere else?

For a virtual machine stored on shared storage, the answer may be straightforward. Another host can access the same disks, recreate the VM and start it again.

That is an important capability, but it is still only one part of recovery.

A VM using local storage has a different failure model. If its host disappears, the virtual disk may disappear with it from the perspective of the rest of the cluster. That does not make local storage a poor design choice. For some workloads, particularly distributed systems or applications that already replicate their own data, local NVMe can be exactly the right solution.

The important point is that the responsibility for recovery has moved.

With shared storage, part of that responsibility sits with the virtualization and storage platforms. With a distributed application using local disks, more of it sits with the application itself.

Both approaches can provide resilient services. They simply achieve it differently.

This is why the useful question is not whether a system uses shared or local storage.

The useful question is: which layer is responsible for recovering the service when something disappears?

A Running VM Is Only the Beginning

There is another assumption that appears frequently during recovery planning: if the virtual machines have started, the environment has recovered.

That is rarely enough.

Consider an application that depends on DNS, authentication, a database and a message queue. After a complete restart, the application server may boot much faster than the database. The operating system starts, the application process starts and the hypervisor reports a healthy running VM.

The application still cannot work.

Some software will retry its failed connections until the dependency becomes available. Some will require a process restart. Some will enter a degraded state that is not obvious from the infrastructure layer.

The same issue appears with identity services, DNS, storage mounts, external APIs and reverse proxies.

This turns startup order into something more important than an operational convenience. It becomes part of the architecture.

A recovery procedure should therefore describe services and dependencies, not merely virtual machines.

Storage must be available before applications can use it. Core network services must be available before systems can resolve or authenticate. Databases and queues must be sufficiently healthy before applications begin relying on them. Only after those layers are working does it make sense to verify the user-facing service.

The order will differ between environments, but the principle remains the same.

Recovery should follow dependencies.

Automatic Startup Is Not the Same as HA

Automatic VM startup is useful, but it addresses another problem.

It defines what happens when a host boots normally.

High availability defines what happens when a host fails unexpectedly.

Those events may eventually result in the same VM being powered on, but the conditions around them are very different.

In XCP-ng, for example, it is easy to inspect some of the behaviour assigned to a virtual machine:

UUID=$(xe vm-list \
  name-label="Example VM" \
  is-control-domain=false \
  --minimal)

xe vm-param-get uuid="$UUID" param-name=actions-after-shutdown
xe vm-param-get uuid="$UUID" param-name=actions-after-reboot
xe vm-param-get uuid="$UUID" param-name=actions-after-crash

The commands are not particularly interesting by themselves.

What matters is knowing what behaviour is expected before an incident occurs.

What should happen after a controlled host reboot?

What should happen after an unexpected host failure?

Can the workload run elsewhere?

Does its data exist elsewhere?

Will its dependencies be ready when it starts?

Those questions describe the actual recovery model far better than a single HA checkbox.

Monitoring Has the Same Boundary Problem

The same distinction applies to monitoring.

A virtualization platform can correctly report that a host is healthy, a VM is running and its virtual disks are attached.

That tells us something useful about the infrastructure.

It does not tell us whether the application works.

A monitoring platform closer to the operating system can confirm that the process is running and the expected TCP port is listening.

That is better, but still incomplete.

The next layer may confirm that an HTTP endpoint responds.

A synthetic test can go further and perform the same transaction that an actual user or external system performs.

Each step moves monitoring closer to the real service.

This is especially important after recovery. A green hypervisor dashboard should not be considered confirmation that everything above it has also recovered.

The most useful validation is performed at the highest meaningful layer.

If users authenticate through the service, test authentication.

If the service exposes an API, perform a representative API request.

If an application writes to a database, confirm that the transaction succeeds.

That is much closer to proving availability than confirming that a VM has an uptime counter.

The Architecture Changes Even When the Diagram Does Not

Another reason recovery deserves regular attention is that infrastructure evolves.

A workload moves to another host. Storage is replaced. A new authentication system is introduced. A reverse proxy appears in front of an application. DNS changes. A database is moved onto a different platform.

Each individual change may be reasonable.

Over time, however, the original recovery assumptions can become inaccurate.

An architecture diagram created during implementation may still look correct while the real dependency chain has quietly changed underneath it.

This is where controlled failure testing becomes valuable.

Not because intentionally breaking systems is an objective in itself, but because it reveals whether the environment still behaves according to its design.

A planned hypervisor shutdown can show which workloads genuinely restart elsewhere.

A storage-path failure can show whether redundancy behaves as expected.

Starting an application before its database can show whether the application recovers automatically or requires intervention.

Temporarily removing a dependency can show whether monitoring detects the failure at the correct layer.

These tests turn assumptions into evidence.

What We Are Trying to Achieve

High availability should not mean attempting to make every component impossible to lose.

That quickly becomes expensive and often creates unnecessary complexity.

A better objective is to understand where failure can occur, which layer is expected to handle it and how recovery will be verified.

Hypervisor HA remains important. Shared storage remains important. Application clustering, database replication, automatic startup and monitoring remain important.

They simply solve different parts of the same problem.

A resilient service appears when these layers are designed together rather than treated as independent features.

The final measure of recovery is therefore not that a host returned to the cluster or that a virtual machine reached the running state.

It is that the service returned to a known, verified and usable state.