MJ12

If Monitoring Fails With the Service, It Was Never Enough

Categories: [Field Notes], [Observability]
Tags: [monitoring], [observability], [elastic], [synthetic-monitoring], [infrastructure], [architecture], [resilience]

Monitoring systems are usually designed around a simple question: what information do we need in order to understand the state of our infrastructure and applications? We collect metrics, logs, traces, application telemetry and synthetic measurements, and then use them to build a picture of how the environment behaves.

There is another question that deserves equal attention: where does the monitoring system itself live?

It is common for monitoring infrastructure to share the same virtualization platform, storage, network and power infrastructure as the systems it observes. Under normal conditions this is perfectly reasonable. The monitoring platform receives data, dashboards are available and alerts are generated as expected.

The limitation only becomes obvious when the shared infrastructure fails. If the application and the monitoring platform disappear at the same time, the system responsible for explaining the incident has become part of the incident itself.

That does not make the monitoring design incorrect, but it does expose an important architectural dependency.

Monitoring Has a Failure Domain

Most monitoring discussions concentrate on data collection. CPU utilisation, memory pressure, database response times, application transactions, logs and security events all provide useful information, but every one of those signals depends on a path between the monitored system and the place where the information is stored and analysed.

Consider a conventional environment where the application, monitoring backend and log platform all run on the same virtualization cluster. They may be separate virtual machines, use separate services and even reside on different VLANs, yet they still share the same underlying failure domain.

A simplified view might look like this:

Application
    |
    v
Monitoring Agents
    |
    v
Virtualization Cluster
    |
    v
Shared Storage
    |
    v
Datacenter Network

If the virtualization cluster or storage platform becomes unavailable, both the service and its monitoring may disappear at approximately the same time. The absence of telemetry then becomes difficult to interpret. The application may have failed, the collector may have failed, the network may have failed, or the complete platform may simply no longer be available.

This is why monitoring architecture should be considered in terms of failure domains, just as we already do with compute, storage and networking.

Internal and External Monitoring Answer Different Questions

Internal monitoring remains essential because it provides information that cannot be obtained from outside the environment. It can show process state, JVM behaviour, storage latency, database contention, packet loss, memory pressure or application errors. This level of detail is often what allows an incident to be diagnosed correctly.

Its limitation is that internal monitoring usually depends on the same infrastructure it measures.

An agent running inside a virtual machine can provide excellent telemetry until the hypervisor hosting that virtual machine fails. Once the VM is no longer running, the monitoring agent cannot explain what is happening because it has disappeared along with the workload.

An independent observer solves a different problem. It does not necessarily know why the application is failing, but it can determine whether the application is reachable and usable from outside its immediate failure domain.

For a web service, an external test might validate DNS resolution, establish a TLS connection, request an application endpoint and confirm that the response is correct. A more advanced synthetic transaction could authenticate to the application or execute an API request representative of normal use.

The value is not simply that another monitoring tool exists. The value is that the observation originates somewhere else.

A simple availability test may therefore be more useful when executed remotely than a much more sophisticated test executed from the same host as the application:

curl \
  --silent \
  --show-error \
  --fail \
  --max-time 10 \
  https://example.org/health

Running this command locally confirms that the application can answer locally. Running it from another network confirms that the application can be reached through the path that users actually depend upon.

The command is identical. The observation point changes the meaning of the result.

Missing Telemetry Is a Signal

Monitoring systems tend to focus on values that have been received. High CPU usage, increased latency or failed transactions are easy to represent because data exists to describe them.

Missing data deserves the same attention.

If a host normally reports every thirty seconds and then stops reporting entirely, that absence is itself meaningful. The difficulty is determining what disappeared.

The monitored system may be down. The agent may have stopped. A network path may be unavailable. The collector may be failing. The monitoring backend may be unavailable.

When all of these components share the same infrastructure, distinguishing between them becomes difficult. Independent monitoring provides a second perspective that can help determine whether telemetry loss is local to the monitoring platform or part of a wider service failure.

This is also where synthetic monitoring becomes particularly useful. Infrastructure monitoring may confirm that a VM is running and that an application process exists, while a synthetic transaction can confirm whether the complete service still performs the operation expected by its users.

The two views are complementary rather than interchangeable.

Internal telemetry explains behaviour. External monitoring verifies availability.

Monitoring Should Follow the Architecture

Monitoring is often added after the main infrastructure has already been designed. This can result in a topology where every component is technically monitored but all monitoring components share the same dependencies as production.

For many environments that may still be acceptable. Complete physical separation of monitoring infrastructure is neither necessary nor economical in every case.

What matters is deciding how much independent visibility is required.

A small environment may need nothing more than an external availability check for its most important services. A larger environment may operate its monitoring backend in another cluster or site. An organisation with several datacenters may deliberately place monitoring collectors and synthetic probes in separate locations so that one site can observe another.

The architecture might then resemble this:

Site A
    Application
    Database
    Local telemetry agents

Site B
    Monitoring backend
    Log storage
    Alerting

Independent observer
    Synthetic checks
    External availability tests

The exact implementation is less important than the principle. Not every part of the monitoring chain should automatically inherit the same failure domain as the system being monitored.

The amount of independence should reflect the importance of the service and the operational consequences of losing visibility.

The Most Useful Test Is Closest to the User

There is another practical consequence of this design.

Infrastructure monitoring often confirms that a server is up, its CPU load is normal and its storage is healthy. Application monitoring may confirm that a process is running and listening on the expected port. These are all useful observations, but none of them necessarily prove that the service is functioning correctly.

A reverse proxy can be healthy while its upstream application is failing. An application can be healthy while authentication is unavailable. Authentication can succeed while the database has entered a state where writes fail.

The most meaningful verification therefore happens at the highest layer that can reasonably be tested.

If the service exposes an API, perform a representative API request. If users authenticate through the application, test authentication. If the application depends on a database operation, validate a transaction that reaches the database.

This does not replace lower-level monitoring. It provides context for it.

When an external synthetic test begins failing and internal telemetry shows database latency increasing several minutes earlier, the combination of those signals provides a much stronger incident timeline than either source alone.

Likewise, if the synthetic test fails at exactly the same moment that all internal telemetry disappears, the scope of the failure is immediately clearer.

What We Are Trying to Achieve

The purpose of observability is not simply to collect as much telemetry as possible. It is to retain enough reliable information to understand what happened when a service stops behaving as expected.

Internal monitoring provides depth. Logs, metrics and traces explain what is happening inside the environment. External monitoring provides perspective by verifying the service from outside its immediate infrastructure boundary. Synthetic monitoring connects both views by testing the transaction that ultimately matters.

A resilient monitoring design does not require every component to live in a completely independent datacenter. It does require a conscious decision about which failures the monitoring system itself must survive.

The same question used when designing any resilient service therefore applies equally well to observability:

What happens to our visibility when one of its dependencies disappears?

If the answer is that the monitoring platform disappears together with the service, the environment may still be well monitored during normal operation.

It simply has less independent evidence available at the moment when that evidence is most valuable.