Field Notes
The Problem With a Field Called User - Security data normalization makes correlation possible. It can also change what an event appears to mean.
Categories:
[Field Notes],
[Security]
Security monitoring depends on our ability to combine information from systems that were never designed to speak the same language. Firewalls, operating systems, identity platforms, applications and cloud services all produce events describing different activities, using different formats and often different terminology.
Normalization attempts to solve this problem by giving the data a common structure. Instead of building a separate detection rule for every product, we map events into a consistent set of fields and use those fields for searches, correlation, dashboards and alerting.
It is an entirely reasonable approach and one of the foundations of modern SIEM engineering.
There is, however, an interesting problem hidden inside it. Two events can contain the same field name without describing the same thing. Even more concerning, two perfectly valid data models can assign different meanings to that field.
When that happens, the data can be technically correct while the conclusions drawn from it are wrong.
The Infrastructure That Starts the Infrastructure - DNS, identity, time and management services should not become circular dependencies during recovery.
Categories:
[Field Notes],
[Infrastructure]
Modern infrastructure has become very good at consolidation.
Virtualization allows us to move services away from dedicated hardware. Central identity removes local accounts. DNS becomes authoritative and dynamic. Configuration management keeps systems consistent. Monitoring provides a common opera...
If Monitoring Fails With the Service, It Was Never Enough - Observability is only useful if enough of it survives the failure it is supposed to explain.
Categories:
[Field Notes],
[Observability]
Monitoring systems are usually designed around a simple question: what information do we need in order to understand the state of our infrastructure and applications? We collect metrics, logs, traces, application telemetry and synthetic measurements, and then use them to build a picture of how the e...
High Availability Is More Than a Running VM
Categories:
[Field Notes],
[Infrastructure]
A virtual machine can recover successfully while the service it belongs to remains unavailable. The difference matters more than it first appears.
High availability is one of those terms that sounds precise until several different parts of an infrastructure start using it at the same time.
A vir...
A Green Patch Run Proves Less Than We Think - Why installing updates is only one part of a successful maintenance process.
Categories:
[Field Notes],
[Automation]
A routine maintenance operation finishes successfully. The automation reports that every host was reachable, packages were updated and no tasks failed. The execution summary is green, the logs are available and the maintenance activity appears complete.
From an automation perspective, everything worked.
From an operational perspective, however, we have only confirmed that the requested actions were executed successfully. We have not necessarily established that the updated systems are running the expected software, that their applications recovered correctly or that the services they provide remain fully functional.
This distinction is particularly relevant as infrastructure teams increasingly automate operating system maintenance using tools such as Ansible. Installing packages is relatively straightforward. Establishing that the maintenance operation achieved its intended outcome requires a different level of attention.
The challenge is not necessarily making automation more sophisticated. It is deciding what we actually consider a successful result.
A Direct Link Is Not a Network - Point-to-point 25, 40 or 100 GbE can be excellent for migration, replication and storage traffic, but only if we treat it as a deliberate failure domain rather than a shortcut.
Categories:
[Field Notes],
[Infrastructure]
There is something very attractive about connecting two virtualization hosts directly with a pair of high-speed network interfaces.
No switch. No spanning tree. No intermediate fabric. No uplink contention. Just a cable between two machines and a large amount of bandwidth available for migration,...