On End-to-End Arguments in System Design
Every few years, I revisit one of my favorite papers END-TO-END ARGUMENTS IN SYSTEM DESIGN and this time I decided I will write an exposition of it to maybe inspire others to read it but also serve as a set of notes for my future self.
Imagine we have an application that holds bank accounts for Alice and Bob. Alice wants to transfer $50 from her account to Bob’s.
This is the current state:
Alice: $100
Bob : $50
How would you implement the transfer operation? (Think about this for a bit. No, really, think about it to internalize the problem).
Let’s start with a simple implementation:
Transfer $50 from Alice to Bob.
Loading the experiment…
At first glance, it appears to work. Alice now has $50, Bob has $100, and their combined balance is still $150.
But what exactly does “Transfer complete” mean?
Before examining the implementation, let’s define the result we expect from a correct transfer:
The state shown by the interactive example above is consistent with those conditions. But it demonstrates only the simplest successful case. It hides everything that happened between clicking Transfer $50 and seeing the updated balances.
To decide whether we can trust the words “Transfer complete”, we need to open the box and examine the operation step by step.
The first step is straightforward: Alice’s request must reach the application.
Suppose the application confirms that it received the request. Is that enough to conclude that the transfer happened?
Send a $50 transfer request to the application.
Loading the experiment…
Well, “Request received” tells Alice only that her request reached the application. It does not tell her whether the application performed the transfer. The application could fail after receiving the request but before changing either balance. Acknowledging delivery is not the same as acknowledging completion.
So when is the application justified in saying “Transfer complete”? Let’s look inside the application and follow the transfer step by step.
Follow the transfer inside the application, one step at a time.
Loading the experiment…
Now we can give “Transfer complete” a more precise meaning: the application has produced the state required by the transfer. Alice has $50, Bob has $100, and their combined balance remains $150. The application knows that the transfer completed but Alice does not observe the balances directly; she learns the result through the application’s response.
What happens if the transfer completes, but that response never reaches Alice?
Follow a completed transfer whose response never reaches Alice.
Loading the experiment…
The transfer has succeeded. Alice now has $50, Bob has $100, and the application knows that the operation completed. Alice, however, received no response.
From her perspective, several things might have happened:
Silence does not allow Alice to distinguish between these possibilities. The actual state of the application and Alice’s knowledge of that state are two different things.
What should Alice do when she cannot tell whether the transfer happened?
A natural response is to try again. If the original request was lost, retrying may be exactly what is needed. But what if the original transfer succeeded and only the response was lost?
Follow an explicit retry after Alice loses the first completion response.
Loading the experiment…
Alice intended to transfer $50 once. The application received two requests and applied both of them.
From the application’s point of view, the second request looked like a new request. Nothing told it that Alice was repeating an earlier attempt whose outcome she did not know.
Retrying solved one problem and introduced another:
The network may be able to detect when it has delivered the same packet twice. But these were not necessarily the same packets. Alice created a new request because she did not receive a response.
The two requests were different messages, but they represented the same intended transfer.
How can the application distinguish two transfers from two attempts to perform one transfer?
Alice can assign an identifier to the transfer and reuse it whenever she retries the same operation.
For example:
Transfer ID: transfer-7F3A
From: Alice
To: Bob
Amount: $50
The identifier belongs to the logical transfer, not to an individual attempt to send it.
Retry transfer-7F3A and observe the application return its previous result.
Loading the experiment…
When the first request arrives, the application records
transfer-7F3A and performs the transfer. If Alice retries using the
same identifier, the application can recognize that the transfer has
already been applied. Instead of moving another $50, it returns the
result of the existing operation.
The final balances remain:
Alice: $50
Bob: $100
A lower layer can suppress duplicates that it creates or recognizes. It cannot generally know that two separate requests express one piece of user intent. Only the application understands that both attempts refer to the same logical transfer.
This does not magically guarantee that every operation executes exactly once under every possible failure, but it gives the application enough information to recognize this retry and avoid applying the transfer again.
We have dealt with repeated requests. What about related requests whose order matters?
Suppose Alice submits the transfer and then, before learning its result, asks the application to cancel it.
For this simplified application, we will use the following rule:
If the cancellation is accepted before the transfer completes, the transfer must not take effect.
The network delivers the two messages in this order:
Is that enough to guarantee that their effects occur in the required order?
Compare the order commands are sent, received, and applied.
Loading the experiment…
The network can preserve the order in which messages are presented to the application. It cannot guarantee how the application schedules, coordinates, or completes the resulting work.
These are three different orders:
Ordered delivery establishes the second. The application cares about the third.
Only the application knows what relationship must hold between a transfer and its cancellation. It must enforce that relationship where the meaning of those operations is understood.
Suppose delivery, retries, duplicate handling, and ordering now work as intended. Does that prove that the final result is correct?
Imagine that every observation so far reports success:
Run the reported checks, then inspect the final account state.
Loading the experiment…
And yet the final balances are:
Alice: $50
Bob: $90
Every report may have been accurate within its own narrow scope. The request may have arrived intact and in the expected order.
The final application state is still wrong.
The combined balance is now $140. Ten dollars have disappeared.
To detect this, the application must validate the result using information that belongs to the application:
A transport checksum cannot check these conditions. It can determine whether bytes changed while being transmitted. It does not know what Alice’s and Bob’s balances should be.
This has the same shape as the careful file-transfer example in the original paper.
A network may deliver every packet correctly, but the final file can still be corrupted by memory, software, storage, or another failure along the path. The receiving application must therefore check the final stored file against what the sender intended to transfer.
Correct delivery of the parts does not prove correctness of the final result.
Security follows the same pattern.
Suppose we encrypt the request while it travels across the network.
Compare exposed and encrypted transfer data across the network path.
Loading the experiment…
Encryption prevents an observer on that protected network path from reading the request. That is valuable, but it does not satisfy the application’s complete security requirement.
The endpoint may still:
The network can protect data while it crosses a particular boundary but it cannot control what an endpoint does before encryption or after decryption. Only the endpoints know the identities, permissions, data, and trust relationships that determine whether the operation is secure. Network encryption remains useful. It is simply not the complete end-to-end security guarantee.
We have repeatedly seen that a lower layer cannot establish the application’s complete notion of success.
That does not mean lower-layer mechanisms are useless.
Enable lower-layer mechanisms and compare their local benefits.
Loading the experiment…
Retransmission can recover from a lost packet without requiring Alice to repeat the entire operation.
Checksums can detect corruption early.
Ordered delivery can save the application from reconstructing a useful message sequence.
Duplicate filtering can remove common duplicates before they reach the application.
Encryption can protect traffic across untrusted networks.
These mechanisms reduce the frequency and cost of failures. They make the common case faster and more reliable.
What they don’t do, however, is remove the need for the final application-level check. Even with every lower-layer mechanism enabled, the application must still determine whether:
Lower layers help perform the work but they do not define what successful work means.
We can now state the idea that has been hiding inside the example.
When only the endpoints possess enough information to implement a function completely and correctly, the complete guarantee must be implemented there.
The relevant endpoints are determined by the function we are trying to guarantee. In this example, they are the parts of the application that know Alice’s intent and can validate the resulting balances.
For our transfer:
Only the application can know that one intended $50 transfer moved money from Alice to Bob while preserving the application’s required invariants. This does not mean that every function should exist only at the highest layer. A partial implementation in a lower layer may still be worthwhile when it improves performance, handles common failures cheaply, or provides a useful shared service.
The important distinction is between:
The former may be good engineering. The latter mistakes a partial guarantee for a complete one.
We can now return to the phrase that was present in the introduction:
Transfer complete
The intended logical transfer has been identified. Alice has been debited by $50. Bob has been credited by $50. Their combined balance remains $150. A retry has not created a second transfer. The application’s ordering and security requirements have been satisfied, and its final state has been validated.
No acknowledgement from one intermediate layer can establish all of that. The endpoint that understands the operation must make the final judgment.
That is the end-to-end argument:
When correctness depends on application knowledge, the complete guarantee must be established at the relevant endpoints. Lower layers may duplicate parts of the function when doing so improves performance or reduces common failures.