Why Adding Kafka Can Make Your System Worse

I've seen this architecture decision go wrong more than once: "This needs to scale. Let's put Kafka in the middle."

It sounds like a good decision. Until the system becomes harder to operate than the problem it was supposed to solve.

Suddenly you have producers, topics, partitions, consumer groups, offset management, retries, dead-letter handling, schema evolution, monitoring, and replay strategies to maintain — for a problem that might have been solved with a database transaction and a background worker.

That's the uncomfortable truth about event-driven architecture. Kafka is powerful. But powerful doesn't mean necessary.

Before introducing an event stream, I ask:

1. Does the producer really need to be decoupled from the consumer? If not, a direct call may be simpler.
2. Do we need independent consumers? If only one service reacts, an event bus may add unnecessary complexity.
3. Do we need replayable history? If not, a simpler queue may be enough.
4. Can the team operate it reliably? Architecture isn't just about runtime performance — it's also about operational ownership.

This is where experienced engineers think differently. Junior thinking asks: "What technology can solve this?" Senior thinking asks: "What's the simplest architecture that satisfies the requirement?"

Event-driven architecture can give you decoupling, independent scaling, asynchronous processing, replayability, and multiple consumers. But you also pay for eventual consistency, operational complexity, debugging difficulty, duplicate processing, ordering problems, and schema evolution.

So the lesson isn't "don't use Kafka." It's: Don't introduce distributed complexity until the problem actually requires it.

The best architecture isn't the one with the most infrastructure. It's the one with the fewest moving parts that can still handle the future you actually need.

What's the most over-engineered component you've seen added to a production system?