RabbitMQ . understand How it works under the hood

in the new postion that i get as a junior software engineer . I found my self face to a great Platform that i felt in Love with . this platform use diffrent technolgies . RabbitMQ , Segement , gRPC , webSockets . HTTP 2 and a lot of other services . I known before about RabbitMQ from hussain nasser one of the best software engineers . today I am going to go deep and share all what i know about this techenolgy .

RabbitMQ is a Software Message broker . and before jumping into RabbitMQ . we need to understand what’s a Message broker .

Message broker

Message brokers are an inter-application communication technology to help build a common integration mechanism to support cloud native, microservices-based, serverless and hybrid cloud architectures.A message broker is software that enables applications, systems, and services to communicate with each other and exchange information. The message broker does this by translating messages between formal messaging protocols. This allows interdependent services to “talk” with one another directly, even if they were written in different languages or implemented on different platforms.

Message brokers are software modules within messaging middleware or message-oriented middleware (MOM) solutions. This type of middleware provides developers with a standardized means of handling the flow of data between an application’s components so that they can focus on its core logic. It can serve as a distributed communications layer that allows applications spanning multiple platforms to communicate internally.

MOM : This middleware layer allows software components that have been developed independently and that run on different networked platforms to interact with one another. Applications distributed on different network nodes use the application interface to communicate. In addition, by providing an administrative interface, this new, virtual system of interconnected applications can be made reliable and secure .

MOM provides software elements that reside in all communicating components of a client/server architecture and typically support asynchronous calls between the client and server applications. MOM reduces the involvement of application developers with the complexity of the master-slave nature of the client/server mechanism . advantage of messaging provider mediated messaging between clients is that by adding an administrative interface, you can monitor and tune performance.

Client applications are thus effectively relieved of every problem except that of sending, receiving, and processing messages. It is up to the code that implements the MOM system and up to the administrator to resolve issues like interoperability, reliability, security, scalability, and performance.Using a MOM system, a client makes an API call to send a message to a destination managed by the provider. The call invokes provider services to route and deliver the message. Once it has sent the message, the client can continue to do other work, confident that the provider retains the message until a receiving client retrieves it.

The message-based model, coupled with the mediation of the provider, makes it possible to create a system of loosely coupled components.Many message-oriented middleware implementations depend on a message queue system. Some implementations permit routing logic to be provided by the messaging layer itself, while others depend on client applications to provide routing information or allow for a mix of both paradigms. Some implementations make use of broadcast or multicast distribution paradigms.In a message-based middleware system, the message received at the destination need not be identical to the message originally sent.

let’s see this problem that message broker address to solve . let’s take youtube as an example youTube use a microservices architecutre and the videos get process by diffrent services . So example :

while trying to push a video on youtube you gonna notice that the first phase is importing your video and after the upload is done , youtube provide an Id for your video and gives you the URL even before publishing or adding it to your channel . this process is a good example to understand why we need Message Brokers .

in a parallel universe where we don’t have Message Borkers . before that youtube gives you the Id they should get done the full process . from the upload to the encoding pipeline which convert the the video into multiple formats and resolution then the catgorization using AI then the copyright service. after all of that you gonna get your response with Ok your video published or your video have been failed in one of those setps . youtube servers should keep the TCP connection with the client until the whole process end . and with the huge amount of videos that upload by the end-users every second , however the architecture or the server it gonna crush . this gonna make a huge cost on the backend and a bad user experience .

thanks to Message Broker . we wont get in this situation . because while using an MB like RabbitMQ or Kafka for example . we gonne give to the enduser the enrty point to the service of upload “the upload service is the entrypoint of service means the client can just interact with this service ” . after the upload get done. “our internal Communication start ” , we can send the id and the url as replay and tell the endUser we get your video we are processing it . and then we cut the tcp connection after that we gonna push the video in a MB. and the other services gonna recieve the video in a few seconds by every service after every phase of processing , and after the process is done we can send a notification to tell the endUser your video is published .

you can say : that instead of using Message Broker just use a webhook and share the same Id . the problem is you gonna make a full dependent services . so if the first service get down the second wont get the message and vice versa . For that you have to send the same request until the other service get alive . MB solve this problem .

the basic conecpt of message Broker “message Brokers have Storage” , means if app1 can send msg to MB and the broker can store it . when app2 comes to life . it can connect to the broker and get its msgs .

RabbitMQ :

from wiki : RabbitMQ is an open-source message-broker software that originally implemented the Advanced Message Queuing Protocol and has since been extended with a plug-in architecture to support Streaming Text Oriented Messaging Protocol, MQ Telemetry Transport, and other protocols

RabbitMQ is an implmentation of the AMQP model version O91. in this type of message model the producer of the message instead of producing to the message queue directly . it is going to produce to an exchange . the exchange gonna recieve all the messages and distribute them according to how they are addressed . an exchange can be connect to many queues . but the consumers are connected to the queues directly . the connection between the exchange and the queues called Binding . and those binding can be refrenced by the binding key . the great thing about this model of message brokers model is the flexibility . you can control the flow and the way the message move through the messaging system , mean : { exchange , binding keys ,and Queues } using the exchange types that RabbitMQ provide . and we gonna discus this now :

Fanout Exchange :

when the producer send the message to exchange , the exchnage duplicated it and brodacast it to all the queues that its known by the exchange .

Direct Exchange :

the produce send the message with a routing key . and the exchange gonna send the message to a specific queue by comparing the binding key : “the connection between the exchange and the queue ” and the routing key “metadata of the message ”, if there’s an exact matching then send it .

Topic Exchange :

the only diffrence between topic exchnage and difrect exchange is that we don’t search for exact matching but a partial matching , so the message can be send to more than just one queue .

Header Exchange :

the exchange ignore the routing key completly and the message move through the system according to the header of the message .

Default Exchange :

this is not a part of the AMQP model , but offred by RabbitMQ . in this type of exchange the routing key is the name of the queue itself .

at this time RabbitMQ support mutilple models and not only AMQP O91

so now we can say that in rabbitMQ the message meta deta defined a.k.a control the way the message gonna move in th system .

Advanced RabbitMQ Best Practices

to make sure that you use the full power of RabbitMQ . try to use and put those rules in your mind :

1-Don’t open and close connections or channels repeatedly.

2-Don’t use too many connections or channels

3-Don’t share channels between threads.

4-Don’t have queues that are too large or too long

5-Don’t use old RabbitMQ/Erlang versions or RabbitMQ clients/libraries

6-Don’t have an unlimited prefetch value

7-Don’t ignore lazy queues

8-Limit queue size with TTL or max-length, if possible.

9-Use multiple queues and consumers.

10-Persistent messages and durable queues for a message to survive a server restart

11-Split your queues over different cores.

12-Consume (push), don’t poll (pull) for messages.

13-Missing an HA policy while creating a new vhost on a cluster.

for more details read this article in cloudamqp.com.

i wish that this was helpful for you .

now we know how rabbitMQ works . in the next article about RabbitMQ we gonna understand how the system was implemented from the low level .

thanks for reading .