Bite-Sized Serverless

General Icon

Introduction to Event-Driven Architectures

General - Foundational (100)
<!-- # Introduction to Event-Driven Architectures -->
Let me ask you three questions:
  1. What's easier to process - one task every minute, or no tasks for most of the day and 3600 processes at once, once a day?
  2. What's a better user experience when placing an order in a restaurant - waiting for the waiter to write it down, or waiting for the waiter to talk to the cook, come back and to confirm the cook started preparing your meal?
  3. What is more reliable - a system that retries a failed process, or a system that tells you a process has failed and asks you to restart it?
Of course the answers are pretty obvious. They also reflect the three main strengths of event-driven architectures. In this Bite we will explore and elaborate on these strengths, and learn about the benefits of event-driven versus request-response designs.

Components of Event-Driven Architectures

Every component in an event-driven architecture can be defined as an Event Producer, Event Router (also called Event Broker), or an Event Consumer. These components respectively generate events, route events to other components, and process events.
Some components might be both consumers of one type of event (say, "Order Placed"), as well as producers of other events (like "Invoice Sent").

What is an event?

When debating event-driven architectures you will commonly hear things like "sending out an event" and "processing events". Technically, this is a bad choice of words. An event is something that occurs, not something that is sent. An example of an event is "an order was placed". What was sent onto a message bus, message queue or other service decoupling technology is a message or a record. However, the terms events and messages are often used interchangeably in event-driven architectures.
An event (the actual event) can be defined as a "significant change in state" or an "action of significant interest". This event can be anything, from the placement of an order to the departure of an airplane, from the creation of an S3 bucket to the login on a website. Oftentimes the significance of the event is assumed. For example a "user logged in"-event might not have any significant value to the architecture when implemented, but might become very valuable in a security audit years down the road.
In an event-driven world two types of messages can be sent in response to events. The first type is a notification that the event occurred, the second type is a message that carries the state of the event. The notification type might require an event consumer to fetch additional details elsewhere, while the message type can be consumed as-is.
Which type you use depends on a number of factors: the capabilities of the technology, the amount of data being sent, security, performance, and business requirements.

Examples of Event-Driven Architectures

Real-time processing vs. batch processing

Let's say you're running a web shop. Customers place orders which need to be picked in a warehouse and sent to the customer by courier. In a batch processing system, all customer orders would be aggregated for a certain period of time (24 hours, for example), and then sent off to the warehouse. Similarly, all orders would be fetched from the database and invoices would be generated from them in a single batch. This implementation has two problems: 1) orders and invoices would be delayed by up to 24 hours, which is bad user experience, and 2) the processing systems (both the warehouse and the invoicing system) would be idling for most of the day, and have very high peak loads once a day. You can see the effect on unprocessed orders in the graph below.
In an event-driven system orders would be sent off to the warehouse as soon as they are placed, which allows the warehouse to process the orders continuously throughout the day. This removes both the backlog of orders and the peak hours at night, as we can see in the following chart. The amount of orders is the same as in the diagram above, but the amount of unprocessed orders stays close to zero. Note that the Y-axis now only goes to 600.

Reducing responsibilities and improving user experience

When placing an order in a restaurant you just want the waiter to confirm your order. You're not interested in what the waiter does to process your order, you only expect your dinner to arrive a reasonable amount of time later. It's the same for many web services: you just want quick confirmation of your order or any other input you gave. You do not care about the actual process of that event, as long as you get the expected results.
To achieve this, every step in an event-driven architecture - especially user facing processes - will need to perform the bare minimum of responsibilities, after which they send out an event for further processing. This results in the best user experience and the most robust event processing architecture, as we will see in the next paragraph.

Improving fault tolerance

In an non-event-driven architecture, a single application might be responsible for multiple steps in a process. In our order placement example, these steps might be:
  • verify user payment info
  • verify item stock
  • place order in database
  • send order details to warehouse
  • send invoice details to finance
When any of the steps in this singular process fail, the entire process fails. This might result in an error message for the user, and in a bad case it might lead to a half-processed order where the warehouse is picking the items, but finance will not send out an invoice.
An event-driven architecture solves this problem in two ways. The first is by decoupling the source (the event producer) from the processors (the event consumers). In a decoupled system a failure in a downstream system is invisible to the user, which results in a better user experience. Secondly, when a downstream process fails it can simply pick up the event at another time. When there are major issues in the downstream processor, processing events can be halted entirely, only to be restarted when the issues have been resolved.

History and future of event-driven architectures

From monolith to microservices to event-driven architectures

Event-driven architectures can be considered the third generation of systems architecture, the first and the second being monoliths and microservices. Let me take you through the improvements and insights that led to each iteration.
The monolith is one big application responsible for every detail of its processes: it receives input from users and administrators, generates invoices, sends out emails and analyzes and aggregates business intelligence. Maintaining this beast is very difficult, because a change in one part of the shared code base might have unforeseen consequences in another part. Additionally, scaling this application is very inefficient: because every component is contained in a single application, the entire monolith needs to scale when one component has a shortage of resources.
To solve these issues we can break up the monolith into separate microservices. Each of these can be maintained, scaled, tested and even replaced separately. A common mechanism for communication between these microservices is through APIs. This makes life a lot easier, but it also introduces new complexities. For example, the order service in our diagram needs to know the endpoints of all its downstream processors. It also needs to know every detail of their APIs to implement successful communication. To gracefully handle unavailability the order service also needs to implement retry logic for every downstream API. So even though these microservices are separated, they are still tightly coupled. And if the caller waits for a downstream service to complete, communication is still synchronous, and the architecture cannot be considered decoupled at all.
By putting a messaging mechanism between the microservices the services become loosely coupled. Now the event producer is only responsible for putting its messages onto a stream or queue, and it does not need to know about the consumers of that message at all. Some messaging mechanisms can only have a single consumer, others can have multiple pre-configured consumers, and some event buses can be read by many consumers.

Extensibility and future-readiness

Once you go event-driven you never go back, and extensibility is a big reason for that. When your architecture is producing events, it becomes very easy to add new functionality: maybe you want to alert on users who are logging in from many different locations - just build a new consumer that processes login events. Maybe you want to sent out a present to new customers - just consume the sign up event. Want to send out a mechanic when an elevator stops working? Respond to the elevator stuck event! By writing many events - even the ones you don't yet know the significance of - you're preparing your architecture for future use cases and success.

Conclusion

In this Bite we covered an introduction to event-driven architectures. We discussed use cases for event-driven designs, a short history of preceding systems, and many benefits unlocked by designing for events. For a deeper dive into messaging systems, see the Related Bites section below.