Samuel Edwards
|
September 23, 2025
%206.png)
If a modern software system is a city, message brokers are the subway tunnels where most of the real movement happens. Producers drop off payloads, consumers pick them up, and the broker keeps everything moving without a pileup. That tidy picture raises a spicy question. Who is actually in charge here.
Is it the broker that routes and buffers, or the applications that send and receive. For teams exploring automation consulting, the answer matters because control shapes reliability, cost, and how fast you can ship. Let’s open the control panel and see whose hands are on which knobs.
A message broker sits in the middle, receiving messages, storing them for a while, and delivering them to interested consumers. Producers publish events, commands, or jobs. Consumers subscribe, process, and respond. The broker enforces contracts about delivery, ordering, retention, and access control.
If the broker is the train dispatcher, producers and consumers are the commuters. The dispatcher sets the schedule and routes trains to the right tracks. The commuters still decide when to arrive, where to go, and whether to bring coffee. Power is shared, and that balance is your architecture.
Producers control message shape and frequency. They decide whether to batch or send one by one, whether to compress, and whether to embed context or include a reference. Consumers control back pressure and idempotency. They choose how quickly to pull, how to deduplicate, and what to do when something is missing.
If producers are too chatty, queues swell. If consumers misbehave, retries spike. The broker can help, but it cannot fix poor judgment at the edges. Healthy systems embrace that truth and put discipline at both ends.
Brokers arbitrate. They are good at routing, fan-out, buffering, and durability. They enforce security rules and quotas. They publish metrics about lag and throughput. They also present a temptation.
When a broker offers stream processing, transactional writes, or stateful filtering, teams sometimes shift business rules into the infrastructure. That move feels powerful until it becomes a maze. Keep the broker focused on transport and coordination. Let services decide behavior, because that is where meaning lives.
Control is not the same as authority. Authority is who can change a rule. Control is who actually shapes the outcome. In a messaging system, outcome is shaped by message contracts, consumer behavior, and operational guardrails. If contracts are fuzzy, consumers reverse engineer payloads and drift ensues.
If consumers skip idempotency, retries become landmines. If guardrails are missing, bursts drown critical workloads. The broker can announce the schedule. The people waiting on the platform still determine whether trains leave full or empty.
Think of the broker as coordination. It guarantees that messages move from A to B with certain semantics. Control belongs to the services that build the messages and interpret them. Coordination without control is chaos. Control without coordination is gridlock. Architects earn their pay by choosing minimal broker features that unlock maximal service autonomy. The right blend keeps teams moving independently without stepping on each other’s toes.
A stable message contract is the strongest lever you have. Define fields clearly, document optional versus required, and version with purpose. Prefer additive changes that preserve old readers. Adopt a schema registry or at least a disciplined process for changes. Reject breaking edits at the door. Good contracts reduce the broker to a fast, reliable pipe instead of a diplomatic crisis hotline. If you want control, write it down and keep it consistent.
Many teams chase absolute ordering. The dream goes like this. If we keep perfect order, downstream logic will be simple. Reality is less kind. Distributed systems reorder things. Networks hiccup. Consumers scale. Even brokers that promise ordering often mean partition ordering, not global. You can rely on per-key order if you partition carefully. You cannot rely on the universe behaving. Better to design for tolerance and recover gracefully.
If a customer record must process changes in sequence, partition by customer identifier. Accept that two different customers can interleave. If something arrives early or late, include version numbers or event times so the consumer can compare and decide. Store enough state to detect out-of-order updates. Use reconciliation to clean up rare edge cases. This puts control in the service code, where context and intent exist, instead of pretending the wire will be kind forever.
At-least-once delivery is common. It means duplicates happen. At-most-once trades away retries for speed. Exactly-once is a careful dance that usually hides complexity somewhere else. You can simplify your life by assuming at-least-once and writing idempotent consumers. Think create-or-update rather than create-only.
Use natural keys or idempotency keys to collapse duplicates. If you rely on exactly-once, verify every integration point, including databases and cron-like jobs that quietly retry. Control belongs to the design that handles the messy parts instead of hoping they never show up.
A good broker is strict on movement and humble on meaning. It should make routing choices, keep messages flowing, and expose clear metrics. It should not decide business outcomes. Avoid putting conditional business logic into routing keys that grow more cryptic with each requirement.
Keep filtering rules mechanical. Save decisions with domain context for consumers that know what the fields mean. Your brain belongs in the app. The pipes should just be very good pipes.
Some brokers excel at simple queues. Others shine with high-throughput streams. Choose based on the jobs you actually need. If you need delayed delivery or message priorities, make sure those features are first class and operationally mature.
If retention windows matter, understand compaction and tombstones. If multiprotocol support is attractive, test clients in the languages your teams use. Control feels like confidence. Confidence comes from choosing features you will use well, not buying a Swiss Army knife you never open.
Without light governance, messaging turns into a junk drawer. Topics multiply. Naming gets creative. Sensitive data sneaks into payloads that travel everywhere. Governance does not mean red tape. It means a few social contracts that everyone follows. Write down a naming glossary.
Require schemas for new topics. Set clear service-level agreements (SLAs) for lag and delivery, then publish dashboards that match them. Treat topic creation like API creation. If a topic is an interface, it deserves care.
Lag, throughput, and error rates are your heartbeat. Publish them where on-call engineers actually look. Track consumer group health and error queues. Alert on growing backlogs and repeated retries. Add correlation identifiers so a single message can be traced across services. Observability does not just help with outages. It gives you the power to tune behavior under load. Control without feedback is just a guess written down nicely.
Pub or sub is your friend when you want loose coupling. Let interested parties subscribe and evolve independently. Work queues fit when you have jobs that must be done once per item, with explicit acknowledgment and back pressure.
Event sourcing works when you need an audit trail or want to rebuild state from history. Each pattern sets the broker’s job and the service’s job with clear boundaries. When patterns are crisp, handoffs are clean. Clean handoffs feel like control.
If the broker starts storing long-term business state because it is convenient, stop and ask why. If you are squeezing business rules into routing keys to avoid code changes, you are paying debt with interest. If consumers are too fragile to handle duplicates, ordering quirks, or version bumps, you have let comfort undermine resilience.
Tooling that makes it easy to ship can also make it easy to forget which side owns meaning. The antidote is simple. Put decisions in services, put movement in the broker, and keep your contracts sharp.
Ownership decides response time. If nobody owns a topic, nobody fixes it. Keep stewardship clear and visible. Documentation reduces fear. A short, current doc beats a sprawling novel that nobody reads. Practicing failure builds courage. Fire drills for lag, dropped consumers, and bad messages create muscle memory before the pager screams. People hold the true reins. The broker just follows orders.
So who is actually in charge. The honest answer is shared custody with boundaries. The broker controls how messages move. Services control what messages mean and how they are handled. Contracts, idempotency, and observability give teams leverage that lasts beyond the next feature sprint.
Choose broker features for fit, not flash. Keep business logic in code that owns context. Write schemas that can grow and still make sense. When you treat the broker as a superb traffic system instead of the city council, you get reliable flow, sane teams, and fewer nights spent staring at a graph that looks like a heart monitor.
