MJ12

A Direct Link Is Not a Network

Categories: [Field Notes], [Infrastructure]
Tags: [networking], [virtualization], [xcp-ng], [proxmox], [storage], [migration], [architecture], [resilience]

There is something very attractive about connecting two virtualization hosts directly with a pair of high-speed network interfaces.

No switch. No spanning tree. No intermediate fabric. No uplink contention. Just a cable between two machines and a large amount of bandwidth available for migration, replication, storage traffic or anything else that benefits from a short path.

For small clusters, lab environments and even some production systems, this can be an excellent design.

It can also become a surprisingly confusing one if the direct link starts accumulating responsibilities that nobody originally intended it to have.

The problem is not performance. The problem is topology.

Why Direct Links Are So Appealing

A dedicated point-to-point connection solves several practical problems at once.

Live migration traffic no longer needs to share the primary switching fabric. Replication between two storage nodes can use bandwidth that is completely independent from normal server traffic. Backup or synchronization jobs can move large amounts of data without competing with user-facing services.

The design can also be very efficient economically. If two hosts already contain spare 25, 40 or 100 GbE interfaces, a direct cable may provide more usable bandwidth than adding another switch layer purely for east-west traffic.

For two-node environments, the topology is also easy to understand:

Host A
  |
  |  Dedicated high-speed link
  |
Host B

There is nothing inherently wrong with this.

The difficulty begins when the dedicated link becomes more than a dedicated link.

A migration interface gains a route to a storage network. A storage interface becomes useful for cluster communication. A backup job is pointed at it because the path is faster. Another service later starts depending on that route because it happens to work.

Eventually the cable between two hosts has quietly become part of the infrastructure fabric.

At that point, it needs to be treated like one.

Bandwidth Is Only One Property of a Network

When we evaluate network paths, bandwidth tends to dominate the conversation.

A 40 GbE direct connection looks objectively better than a 10 GbE switched path if the requirement is simply to move data between two machines.

But a network path also has failure behaviour, visibility, routing semantics and operational dependencies.

A switched network usually provides a topology that other systems can participate in. A point-to-point link does not. It connects exactly the systems at either end, and every service using that path inherits that constraint.

That distinction matters during failures.

If Host A disappears, the direct path disappears with it. If Host B is replaced, reinstalled or temporarily moved, the topology changes. If routing has been configured so that another network is reachable through that peer, the failure affects more than the cable itself.

The physical design may therefore be simple while the logical design becomes surprisingly complicated.

This is particularly easy to miss in virtualization platforms because several types of traffic coexist on the same hosts. Management, VM traffic, storage, migration, replication and cluster communication may all be present, but they do not necessarily have the same availability requirements.

The fastest path is not automatically the correct path for all of them.

A Dedicated Link Should Have a Dedicated Purpose

The cleanest point-to-point designs tend to have one defining property: the reason for the link can be explained in a single sentence.

For example:

This link carries live migration traffic between these two hypervisors.

Or:

This link carries storage replication between these two nodes.

That clarity matters more than it sounds.

Once the purpose is known, the expected failure behaviour becomes much easier to reason about. If the link fails, migrations become slower or unavailable. If replication uses another path as fallback, we know exactly what happens next. If no fallback exists, that decision is explicit rather than accidental.

Problems usually begin when the description becomes longer:

This is the migration link, but storage also uses it, and there is a route through it to another network, and backups use it because it is faster, and cluster traffic sometimes goes over it as well.

At that point, the direct connection has stopped being a dedicated link and started behaving like an undocumented transit network.

That is where simplicity disappears.

Routing Is Where Things Become Interesting

A point-to-point network often needs very little configuration.

For IPv4, a /31 is a convenient example:

Host A: 192.0.2.0/31
Host B: 192.0.2.1/31

With the link configured, the most useful validation is often not a throughput benchmark but confirmation that traffic is actually using the path we think it is using.

On Linux, a few simple checks can tell us quite a lot:

ip -br addr

ip route get 192.0.2.1

ping -I <direct-interface> 192.0.2.1

ethtool <direct-interface>

The important command here is often ip route get.

It shows the route the kernel intends to use for a specific destination. That becomes extremely valuable once several interfaces, bridges, storage networks and management networks exist on the same host.

A system may have a perfectly healthy 40 GbE direct link while traffic silently uses another interface because routing prefers a different path.

The opposite problem is more dangerous: traffic may unexpectedly depend on the direct link because a route was added months earlier and later forgotten.

Both situations can survive unnoticed for a long time.

Redundant Cables Do Not Automatically Create a Redundant Path

It is also tempting to solve the problem by connecting two direct links between the hosts.

Physically, that looks redundant.

Logically, it may or may not be.

Two interfaces can be bonded, used as separate routed paths or assigned different roles. Each option has different behaviour when a cable, NIC or host fails.

The important distinction is that two cables between the same two machines still share a major failure domain: the machines themselves.

If the purpose of the second link is protection against a cable or interface failure, that may be entirely sufficient.

If the expectation is protection against a host failure, it is not.

This sounds obvious when written down, but architecture frequently becomes confusing because physical redundancy and service redundancy are discussed as though they were the same thing.

They are not.

The right question is always: which failure are we trying to survive?

The Switch Is Not Always the More Complicated Option

Direct connections are often chosen because a switch seems like unnecessary complexity.

Sometimes that is correct.

But there is a point where the opposite becomes true.

If several hosts need to communicate over the same high-speed network, or if storage, migration and replication traffic must move between more than two systems, a proper switched fabric may actually simplify the architecture.

The topology becomes explicit. Every host participates in the same network. Routing becomes easier to understand. Monitoring can observe the fabric as a whole rather than a collection of private links.

The switched design may contain more hardware, but less operational ambiguity.

This is an important distinction in infrastructure design.

Fewer devices do not always mean a simpler system.

Sometimes a dedicated network switch is the component that removes complexity.

Where Direct Connections Work Very Well

None of this is an argument against direct links.

Quite the opposite.

They are particularly effective when the topology is genuinely point-to-point and the purpose is narrow.

Two storage nodes replicating directly between one another are a good example. Two virtualization hosts with a dedicated migration path are another. A temporary high-speed path used while moving several terabytes of data can also be perfectly reasonable.

In these situations, the lack of a switch is a feature.

The design remains understandable because the topology matches the workload.

The architecture becomes less attractive when the link starts providing connectivity for services that logically belong to a wider network.

That is usually the point where it is worth asking whether the direct connection is still solving a problem or merely postponing the addition of proper network infrastructure.

What We Are Trying to Achieve

The objective is not to avoid direct high-speed links.

They can provide excellent performance, useful isolation and very good value, particularly in small virtualization or storage environments.

The objective is to keep their purpose explicit.

A direct connection should have a clearly defined responsibility, predictable routing and understood failure behaviour. Services that depend on it should do so deliberately rather than because the path happened to be available.

The important distinction is between a link and a network.

A link connects two systems.

A network provides connectivity as part of a wider architecture.

Both can be the correct answer.

The mistake is allowing the first one to quietly become the second without redesigning it accordingly.