RabbitMQ vs Kafka

When it comes to machine to machine messaging there are so many choices. From proprietary to open source the list is so long and filled with competing products. But there are two choices which are widely used by the tech sphere. 

RabbitMQ which was introduced in 2007 is a traditional smart queue for dumb consumers where as Apache Kafka which was introduced in 2011 is a cutting edge dumb queue for smart consumers. Lets have a look at the difference between the two while evaluating the pros and cons.

Beginning

RabbitMQ

• A “traditional” message broker that implements variety of messaging protocols developed using Erlang.
• It was one of the first open source message brokers to achieve a reasonable level of features, client libraries, dev tools, and quality documentation.
• RabbitMQ was originally developed to implement AMQP, an open wire protocol for messaging with powerful routing features. With the advent of AMQP, cross-language flexibility became real for open source message brokers.

Apache Kafka

• Developed in Scala and started out at LinkedIn as a way to connect different internal systems. 
• Kafka is well adopted today within the Apache Software Foundation ecosystem of products and is particularly useful in event-driven architecture. 

Architecture

RabbitMQ

• Designed as a general purpose message broker, with point to point, request/reply and pub-sub communication styles patterns.
• It uses a smart broker / dumb consumer model, focused on consistent delivery of messages to consumers that consume at a roughly similar pace as the broker keeps track of consumer state.
• It is mature, performs well when configured correctly, is well supported (client libraries Java, .NET, node.js, Ruby, PHP and many more languages) and has dozens of plugins available that extend it to more use cases and integration scenarios.

Simplified overall RabbitMQ architecture. Source: http://kth.diva-portal.org/smash/get/diva2:813137/FULLTEXT01.pdf 

• Communication can be either synchronous or asynchronous as needed.
• Publishers send messages to exchanges, and consumers retrieve messages from queues.
• Decoupling producers from queues via exchanges ensures that producers aren't burdened with hardcoded routing decisions.
• RabbitMQ also offers a number of distributed deployment scenarios (and does require all nodes be able to resolve hostnames).
• It can be setup for multi-node clusters to cluster federation and does not have dependencies on external services (but some cluster formation plugins can use AWS APIs, DNS, Consul, etcd).  

Apache Kafka

• Apache Kafka is designed for high volume publish-subscribe messages and streams, meant to be durable, fast, and scalable.
• Kafka provides a durable message store, similar to a log, run in a server cluster, that stores streams of records in categories called topics.

Global Apache Kafka architecture (with 1 topic, 1 partition, replication factor 4). Source: http://kth.diva-portal.org/smash/get/diva2:813137/FULLTEXT01.pdf 

• Every message consists of a key, a value, and a timestamp.
• Kafka employs a dumb broker and uses smart consumers to read its buffer. Kafka does not attempt to track which messages were read by each consumer and only retain unread messages; rather, Kafka retains all messages for a set amount of time, and consumers are responsible to track their location in each log (consumer state).
• Consequently, with the right developer talent creating the consumer code, Kafka can support a large number of consumers and retain large amounts of data with very little overhead. 
• Kafka does require external services to run - in this case Apache Zookeeper, which is often regarded as non-trivial to understand, setup and operate.  

Use Cases

RabbitMQ

A general purpose messaging solution, often used to allow web servers to respond to requests quickly instead of being forced to perform resource-heavy procedures while the user waits for the result. It’s also good for distributing a message to multiple recipients for consumption or for balancing loads between workers under high load (20k+/sec).  When your requirements extend beyond throughput, RabbitMQ has a lot to offer: features for reliable delivery, routing, federation, HA, security, management tools and other features.  Let’s examine some scenarios best for RabbitMQ, like:

• Your application needs to work with any combination of existing protocols like AMQP 0-9-1, STOMP, MQTT, AMQP 1.0.
• You need a finer-grained consistency control/guarantees on a per-message basis (dead letter queues, etc.) However, Kafka has recently added better support for transactions. 
• Your application needs variety in point to point, request / reply, and publish/subscribe messaging
• Complex routing to consumers, integrate multiple services/apps with non-trivial routing logic

Apache Kafka

Includes the broker itself, which is actually the best known and the most popular part of it, and has been designed and prominently marketed towards stream processing scenarios. In addition to that, Apache Kafka has recently added Kafka Streams which positions itself as an alternative to streaming platforms such as Apache Spark, Apache Flink, Apache Beam/Google Cloud Data Flow and Spring Cloud Data Flow. The documentation does a good job of discussing popular use cases like Website Activity Tracking, Metrics, Log Aggregation, Stream Processing, Event Sourcing and Commit logs. One of those use cases it describes is messaging, which can generate some confusion.  So let’s unpack that a bit and get some clarity on which messaging scenarios are best for Kafka for, like:

• Stream from A to B without complex routing, with maximal throughput (100k/sec+), delivered in partitioned order at least once.
• When your application needs access to stream history, delivered in partitioned order at least once.  Kafka is a durable message store and clients can get a “replay” of the event stream on demand, as opposed to more traditional message brokers where once a message has been delivered, it is removed from the queue.
• Stream Processing
• Event Sourcing

Developer Friendliness 

RabbitMQ

• Officially supports Java, Spring, .NET, PHP, Python, Ruby, JavaScript, Go, Elixir, Objective-C, Swift - with many other clients and devtools via community plugins.
• The RabbitMQ client libraries are mature and well documented.

Apache Kafka

• Has made strides in this area, and while it only ships a Java client, there is a growing catalog of community open source clients, ecosystem projects, and well as an adapter SDK allowing you to build your own system integration.
• Much of the configuration is done via .properties files or programmatically.

Security and DevOps

RabbitMQ

• RabbitMQ management plugin provides an HTTP API, a browser-based UI for management and monitoring, plus CLI tools for operators.
• External tools like CollectD, Datadog, or New Relic are required for longer term monitoring data storage.
• RabbitMQ also provides API and tools for monitoring, audit and application troubleshooting.
• Besides support for TLS, RabbitMQ ships with RBAC backed by a built-in data store, LDAP or external HTTPS-based providers and supports authentication using x509 certificate instead of username/password pairs.
• Additional authentication methods can be fairly straightforwardly developed with plugins.

Apache Kafka

• TLS, JAAS role based access control and kerberos/plain/scram auth, using a CLI to manage security policy.  
• This made a substantial improvement on earlier versions where you could only lock down access at the network level, which didn’t work well for sharing or multi-tenancy.
• Uses a management CLI comprised of shell scripts, property files and specifically formatted JSON files.
• Kafka Brokers, Producers and Consumers emit metrics via Yammer/JMX but do not maintain any history, which pragmatically means using a 3rd party monitoring system.  

Performance

RabbitMQ

• 20K messages per second is easy to push through a single Rabbit queue, indeed rather more than that isn't hard, with not much demanded in the way of guarantees.
• The queue is backed by a single Erlang lightweight thread that gets cooperatively scheduled on a pool of native OS threads - so it  becomes a natural choke point or bottleneck as a single queue is never going to do more work than it can get CPU cycles to work in.
• Increasing the messages per second often comes down to properly exploiting the parallelism available in one's environment by doing such things as breaking traffic across multiple queues via clever routing (so that different queues can be running concurrently).
• When RabbitMQ achieved 1 million message per second , this use case basically came down entirely to doing that judiciously - but was achieved using lot of resources, around 30 RabbitMQ nodes.  Most RabbitMQ users enjoy excellent performance with clusters made up of anywhere from three to seven RabbitMQ nodes.

Apache Kafka

• 100k/sec performance is often a key driver for people choosing Apache Kafka.  
• Message per second rates are tricky to state and quantify since they depend on so much including your environment and hardware, the nature of your workload, which delivery guarantees are used (e.g. persistent is costly, mirroring even more so), etc.

Conclusion

This post is a summary of a post written by Pivotal named Understanding When to use RabbitMQ or Apache Kafka. They have based a Master's thesis by Nicolas Nannoni which has a lot more side by side comparison of the two products in depth. 

Java Pattern Matching Feature

Java 13 Pattern Matching Feature Explained

.

Read More

Developing Java Applications with Unicode Data

Developing Java Applications with Unicode Data

.

Read More

Corona Meter

Corona Meter is a DIY Project which uses an ESP 32 Dev Board and 0.96 inch OLED Module to Update Corona Virus Stats

.

Read More

Comments

Please login or register to post a comment.