In this article, we will talk about our journey of creating Event Driven Architecture (EDA). We’ll cover the issues we faced before our EDA, things to keep in mind while designing it, and some technologies and/or solutions for EDA. To understand the rationale behind EDA, we must first understand how we reached there and the choices we made leading up to it.
Table Of Content
Story so far
Being a startup, in the early years of product development Ninjacart’s main aim was rapid iteration and fast releases. With a small development team, it wasn’t possible to focus on building a sprawling expanse of microservices because a lot of time would be unnecessarily spent on solving a problem that didn’t exist yet. Therefore, we went ahead with a monolith-first approach.
This approach helped us focus on understanding the domain without paying an extra premium for a complex system design. In a monolith, deployment is straightforward, there are no network calls between parts of code and most things can be bound by a transaction. This was important for a team that was just breaking into a domain like B2B agritech where the entire business and product model was unknown.
Trouble in paradise
The monolithic approach soon started to break. As our team size increased, more product flows had to be taken care of, and there were requirements that were not always completely comprehensible. Hence, it suffered from uncontrolled complexity growth.
As we started breaking down the complex system into simpler parts that were interacting with each other, the complexity was not reduced. It just moved to a ‘macro’ level, which created a ‘distributed monolith’. Fundamentally, systems must communicate with each other and that creates a degree of interdependence – coupling.
Moving further, ensuring a state of consistency in a distributed system was perhaps the most difficult aspect to get right. When parts of the distributed system eventually failed, all ephemeral states got lost, leaving the system in an inconsistent state that had somehow persisted, which could go wrong.
So, what do we do about it?
EDA is a paradigm promoting the production, detection, consumption of, and reaction to events. An event is any change in state that may be of interest to any participants or stakeholders in the system or outside the system.
EDA consists of 3 main parts:
- Emitters
- Consumers
- Channels
An emitter of an event is not made aware of any of the event’s downstream consumers. Every event raised should be like a radio or a newspaper-broadcast. Similarly, a consumer should react to an event without basing it on what particular emitter created it.
Another thing to keep in mind is that events are immutable. Once an event is raised, it cannot be altered. One can raise more events in reaction to it, but the original event remains the same.
Some use Cases
A way EDA can be used and one that we are moving towards is managing the entire system state through various events being raised by parts of the system.
For example —
- Every time status of an order placed by customer changes (placed, pending, processing, out-for-delivery, delivered), an event can be raised that triggers:
a. Finance module – to create invoice
b. Inventory management – to add or subtract stock
c. Communication modules – to send an email/SMS to customer - A delivery vehicle can send a location ping back to the system periodically which raises an event to alert the customer when order delivery is about to be made.
- The relational database can raise an event every time certain tables are written to, that trigger the analytics system to refresh dashboards or adjust projections.
Technologies
We use a mix of AWS SQS for simple use cases that need events raised as one-to-one fast messages. For publisher-subscriber use cases, a self-hosted Apache Kafka cluster is used.
Another popular tool is RabbitMQ that has fast, reliable queues with built-in user-friendly monitoring and management tools. It is a good fit if you don’t need a durable message broker to keep your events stored and the ability to replay them later.
To avoid the effort of doing it all in-house or to begin with EDA for small projects, it might make sense to use a fully managed Kafka cluster as a service. There are solutions available like AWS and Confluent, among others. In such cases, one needs to keep in mind the trade between ease of use and cost. Without proper care and management, such services can quickly run up to hundreds of dollars in monthly bills.
This was only a brief introduction to the vast expanse of EDA. For more information, feel free to connect back to us and we would be happy to help.
. . .
Written by
Saurabh Sharma
SDE-II (Logistics)
Tech Team – Ninjacart
No Comment! Be the first one.