RELIABILITY · POSTGRESQL · .NET
Duplicate-safe workflow service
I started this because a retry looks harmless until it creates the same work twice. If a client loses a response, it cannot know whether the server failed or succeeded just before the connection dropped. Sending the request again is reasonable. Creating a second workflow is not.
The code, scripts, and run outputs live in a separate repository.
Stack C# / .NET / PostgreSQL / Docker / Python
What I wanted to find out
I wanted to see the difference between a service that simply accepts every request and one that can recognise a retry. After that, I wanted to keep the workflow record and its follow-up event from drifting apart when something goes wrong.
The question I am working with is simple: what changes when I add an idempotency key and an outbox to a workflow service?
What I tried
I began with the wrong version on purpose. The first endpoint did not look at a key at all. I sent the same request five times and got five separate workflow records back. That gave me something concrete to compare against.
Next I added an Idempotency-Key. The service stores the key alongside a fingerprint of the request. If the same request comes back with the same key, it returns the first workflow instead of making another one. If somebody reuses the key with different input, it responds with a conflict rather than guessing.
I then sent twenty copies of one keyed request at the same time. One request created the workflow. Nineteen others found it and reused it. That run is useful because it exercises the database uniqueness race, not just a neat sequential example.
The second half of the project is about the event that follows workflow creation. I added an outbox table and wrote the workflow row and the pending event in one PostgreSQL transaction. In a controlled rollback, neither row stayed in the database. I also added a dispatcher experiment: a controlled failure leaves the event pending with an error and attempt count; the next run can publish it.
What happened in the runs
- Five unkeyed repeats created five workflow records.
- Five keyed repeats created one workflow record.
- Twenty concurrent keyed repeats still resolved to one workflow record.
- A rollback left no workflow and no outbox event behind.
- A failed dispatch stayed pending and was published after a retry.
Experiments
Naive request handler
Five repeated requests without a key created five workflow records.
Idempotency key
Five matching replays with one key returned the original workflow instead of creating new records.
Transactional outbox
A workflow and its pending event committed together; a controlled rollback left neither record behind.
Concurrent idempotency
Twenty simultaneous requests with one key resolved to one workflow: one creation and nineteen replays.
Outbox retry
A controlled delivery failure left the event pending; the next dispatch published it.
Run captures
Four captures from the September 8 local run. The full evidence set and the raw JSON records remain in the repository.
E01 / persisted recordsFive replays, five workflow records.
E02 / key conflictA changed request body is rejected.
E04 / concurrent replayTwenty requests resolved to one workflow.
E05 / retry recoveryThe pending event was published on retry.
What is still open
I have not tested a real message broker, multiple service instances, consumer-side deduplication, or a hard process kill in the middle of a database operation. I also would not describe this as exactly-once processing. Those are separate problems, and I would rather leave them open than write past the evidence.
If you want to inspect it
The repository has the C# service, PostgreSQL Docker setup, Python runners, raw JSON from the runs, diagrams, and five automated tests. These are the files I would start with: