Learning Reliable System Design from a Tiny API Glitch
System-level engineer building reliable backend systems with a focus on performance, correctness, and real-world constraints. I work across APIs, databases, networking, and infrastructure, enjoy understanding how systems behave under load and failure, and write to break down complex backend and distributed-systems concepts through practical, real-world learnings.
Today, I was working on an admin panel feature.
Simple task.
One button.
One API.
👉 “Create Policy”
I clicked the button.
Policy created ✅
Clicked again (by mistake)…
Another policy created ❌
Clicked again (slow network)…
Boom — three duplicate records.
No error.
No crash.
Just silent damage.
That day, I didn’t just fix a bug.
I learned why idempotency exists — and why every backend engineer must understand it.
Let me explain idempotency in the simplest, most practical way possible.
What Is Idempotency? (Plain English Version)
An operation is idempotent if doing it once or doing it 10 times gives the same final result.
That’s it.
No jargon.
No theory.
Real-world analogy
Pressing an elevator button multiple times → elevator still comes once
Turning a light ON multiple times → light stays ON
Setting your name to
"Ankan"again and again → name doesn’t change
That’s idempotent behavior.
Why I Faced the Problem (Real Story)
In my case:
User clicked Create Policy
Network was slow
User clicked again
Frontend sent multiple requests
Backend happily created multiple policies
From the backend’s point of view:
“I received requests, so I executed them.”
From the product’s point of view:
“Why the hell do I have duplicate data?”
This is where idempotency saves you from yourself.
Idempotency in Backend APIs (Simple Explanation)
❌ Non-idempotent API
POST /create-policy
Every request = new record
Click twice = two records
✅ Idempotent API
PUT /policy/{policyId}
Same request, same ID = same result
No duplicates.
HTTP Methods & Idempotency (Important for Interviews)
| Method | Idempotent? | Why |
| GET | ✅ Yes | Just reads data |
| PUT | ✅ Yes | Replaces resource |
| DELETE | ✅ Yes | Deleting again does nothing |
| POST | ❌ No | Creates new resource |
POST is dangerous if you don’t handle idempotency manually.
The Hidden Enemy: Network, Retries & Double Clicks
Even if:
Your UI is perfect
Your users are careful
Still, duplicates happen because of:
Slow internet
Mobile retries
Load balancers
User impatience
Browser refresh
Backend must be defensive. Always.
How I Fixed It (Practical Approach)
1. Introduced an Idempotency Key
Frontend sends a unique key:
X-Idempotency-Key: abc123
Backend logic:
If key already exists → return old response
If not → process and store key
Same request = same result.
2. Used Unique Constraints (Database Level)
For example:
Policy name
Policy + domain combination
If duplicate comes → DB blocks it.
This is real protection, not just code-level hope.
3. Disabled Button on UI (But Never Trust UI Alone)
Yes, frontend disabling helps.
But frontend is not security.
Backend must assume:
“User can do anything.”
Why Idempotency Is a Senior-Level Concept
Idempotency is not about syntax.
It’s about thinking in failure scenarios.
Senior engineers ask:
What if request retries?
What if client crashes?
What if load balancer repeats request?
What if user double clicks?
Junior engineers assume:
“It won’t happen.”
Reality says:
“It WILL happen.”
Where Idempotency Is Absolutely Mandatory
Payment APIs
Order creation
Policy creation
Signup / onboarding
Webhooks
Cron jobs
Distributed systems
Anywhere money or data integrity is involved.
Mental Model to Remember Forever
If repeating an operation can break your system,
your system is fragile.
Idempotency makes systems boring, predictable, and safe —
and boring systems scale the best.
Final Thoughts (Hard Truth)
Idempotency is not optional.
It’s not “extra”.
It’s not “later”.
If you’re building APIs and not thinking about idempotency,
you’re one retry away from a production bug.
I learned this the hard way.
You don’t have to.