RabbitMQ guide

What RabbitMQ does in production

RabbitMQ decouples producers from consumers. A web API can publish a job and return immediately; workers process the queue at their own pace. Common uses: order processing, email dispatch, ETL triggers, microservice async calls, and buffering traffic spikes. The protocol is primarily AMQP 0-9-1 (also MQTT, STOMP via plugins).

RabbitMQ vs Kafka

Both move data between services asynchronously, but they solve different problems. RabbitMQ is a message broker: messages are routed to queues, delivered to consumers, and typically removed after ack. It excels at task queues, RPC-style patterns, and flexible routing (exchanges, bindings, per-message delivery). The broker tracks which consumer got which message.

Apache Kafka is an event log (streaming platform): producers append records to topics partitioned for scale; consumers read at an offset and messages are retained for a configurable time (replay, multiple consumer groups, audit trail). Kafka is built for high-throughput event streams, analytics pipelines, and “publish once, read many times” — not for deleting a job once one worker finishes it.

Rule of thumb: use RabbitMQ when you need work distribution, complex routing, and per-message acknowledgements (order jobs, email workers). Use Kafka when you need durable event history, replay, stream processing, or very high ingest volume (click streams, CDC, log aggregation). Some systems use both — Kafka for the event backbone, RMQ for task dispatch to workers.

Core concepts

• Producer — publishes messages to an exchange
• Exchange — routes messages to queues by type and bindings
• Queue — buffers messages until consumers acknowledge them
• Binding — link between exchange and queue (with optional routing key)
• Consumer — receives messages and sends ack (acknowledgement) when done
• Virtual host (vhost) — logical namespace isolating exchanges/queues/users

Exchange types

• direct — route by exact routing key match
• fanout — broadcast to all bound queues
• topic — pattern match on routing keys (orders.*)
• headers — route by message header attributes

How a message flows

1. Connect — client opens TCP to broker (default 5672)
2. Publish — producer sends message to exchange with routing key
3. Route — exchange copies message to matching queue(s)
4. Deliver — broker pushes to consumer (or consumer pulls)
5. Ack — consumer acks; broker removes message from queue

Without acks (auto-ack), messages disappear on deliver even if processing fails — prefer manual ack in production workers.

Management dashboard

Enable the management plugin for the web UI at http://host:15672 (default user guest on localhost only). The dashboard is your first stop for monitoring and troubleshooting:

• Queues tab — message count, publish/deliver rates, consumer count
• Connections / Channels — who is connected, leaks, blocked clients
• Overview — cluster health, message rates, resource alarms

Watch queues for steadily increasing Ready counts — producers outpacing consumers. Also watch Unacked — messages delivered but not yet acknowledged (stuck or slow handlers).

Queue growth and backpressure

A healthy queue fluctuates near zero between bursts. A queue that always grows signals trouble:

• Consumers crashed or not running
• Consumers too slow (scale out, optimize handler, tune prefetch)
• Poison messages — handler throws, nack/requeue loops
• Downstream dependency slow (DB, API) blocking acks

Set max-length policies or TTL to prevent unbounded disk use. Use dead-letter exchanges (DLX) for messages that fail repeatedly.

Users, permissions, and networking

Users are scoped to vhosts with configure/write/read permissions. Default vhost is /. Remote clients need a non-guest user — guest is localhost-only by default. TLS and SASL for production; firewall port 5672 (AMQP) and 15672 (management) as needed.

Clustering (brief)

RMQ clusters share metadata; queues live on individual nodes (classic queues) or replicate (quorum queues in modern versions). A single node failure can make queues on that node unavailable until recovery. Prefer quorum queues for new HA designs; understand where your queues live.

Key files and service

• Config — /etc/rabbitmq/rabbitmq.conf (modern cuttlefish format)
• Enabled plugins — /etc/rabbitmq/enabled_plugins
• Data — /var/lib/rabbitmq/
• Logs — /var/log/rabbitmq/ or journal
• Service — rabbitmq-server

Learning resources

• RabbitMQ documentation — rabbitmq.com/docs
• Management plugin — rabbitmq.com — management
• Production checklist — rabbitmq.com — production
• Quorum queues — rabbitmq.com — quorum queues