Bite-Sized Serverless

S3 Icon

Monitor Events from Multiple S3 Buckets with EventBridge

S3 - Intermediate (200)
S3 Buckets can be configured to stream their objects' events to the default EventBridge Bus. This is an on-or-off toggle per Bucket. When multiple buckets have EventBridge notifications enabled, they will all send their events to the same Event Bus. In this Bite, we will use this to respond to events across multiple S3 Buckets.

The history of event notifications

S3 Event Notifications have been around since 2014. These notifications are configured on an S3 Bucket itself and forward their events to a Lambda Function, SNS Topic or SQS Queue. A prefix or suffix can optionally be used to filter for which objects events are sent. Configuring these "classic" event notifications has always been a bit of a responsibility spaghetti. The major part of the configuration lies with the Bucket, so if a consuming Lambda Function requires additional events, they would need to update the Bucket. The Bucket, which is supposed to be a dumb storage component, now knows about it's consumers - a clear case of tight coupling. Additionally, an S3 Event Notification can only be configured on a single Bucket. If an event consumer would like to receive events for multiple buckets, Event Notifications would have to be configured and maintained on every Bucket. And to make matters worse, the Lambda Function needs to provide IAM access to every Bucket it receives events from. This is illustrated in the diagram below.

Introducing EventBridge support for S3 Notifications

At Re:Invent 2021, AWS introduced a much cleaner S3 Event Notification solution. Instead of configuring the targets on the S3 bucket, we can now simply enable EventBridge notifications. When turned on, all events for the Bucket are sent to the default Event Bus present in every AWS account.
With the EventBridge integration the S3 Buckets no longer know who consumes their events. This is a proper event driven implementation with isolated producers, event routers, and consumers. And through this isolation, S3 gets all the benefits of EventBridge, including event archival, replay and a growing ecosystem of supported consumers. Among other services, EventBridge can forward events to Step Functions State Machines, which in turn integrate with another 200 services and over 9000 API actions, allowing for functionless architectures.

Process events from multiple buckets

Because all events are routed over a single Event Bus, we can write Bucket agnostic event consumers. For example, the following CDK code deploys a Lambda Function that receives the Object Created events from any bucket. All code examples shown are included in a CDK project available for download at the bottom of this page.
1# Create an EventBridge Rule to match new objects in any bucket 2put_object_processor = LambdaFunction( 3 scope=self, 4 construct_id="PutObjectProcessor", 5 code=lambda_.Code.from_asset("lambda_functions/s3_event_processor"), 6) 7 8events.Rule( 9 self, 10 "PutObjectRule", 11 event_pattern=events.EventPattern( 12 source=["aws.s3"], detail_type=["Object Created"] 13 ), 14).add_target( 15 targets.LambdaFunction( 16 put_object_processor.function, 17 ) 18)
This Lambda Function could be used to build statistics or to perform tagging operations on every new object.
We could also write an EventBridge Rule matching objects above a certain size. These large objects might trigger an alert for an operations team to follow up on. As an example, the following Rule matches new objects larger than 1 MB.
1events.Rule( 2 self, 3 "LargeObjectsRule", 4 event_pattern=events.EventPattern( 5 source=["aws.s3"], 6 detail_type=["Object Created"], 7 detail={ 8 "object": { 9 "size": [{"numeric": [">", 1048576]}] # 1024 * 1024 bytes 10 } 11 }, 12 ), 13).add_target( 14 targets.LambdaFunction( 15 large_objects_processor.function, 16 ) 17)
Like the one before, this rule matches on every Bucket with EventBridgeEnabled set to True.
The third and final example involves an EventBridge Rule that triggers whenever the ACLs of an object in S3 are changed.
1events.Rule( 2 self, 3 "ChangeAclRule", 4 event_pattern=events.EventPattern( 5 source=["aws.s3"], detail_type=["Object ACL Updated"] 6 ), 7).add_target( 8 targets.LambdaFunction( 9 change_acl_processor.function, 10 ) 11)
Changing an ACL might be a security risk, so the function triggered by this rule might assess the change and decide to roll it back, to send out an alert, or both.

Enforce EventBridge Notifications on every bucket

Some of the rules described above, like the object size or ACL change rules, are most valuable when applied to every Bucket in a project. However, it can be cumbersome and failure-prone to apply the EventBridgeEnabled configuration on every single Bucket in your CDK application. Luckily we can use CDK Aspects to do it for us. The following code loops over every Bucket and sets EventBridgeEnabled to True for each of them.
1# app.py 2@jsii.implements(cdk.IAspect) 3class EventBridgeNotificationAdder: 4 """CDK Aspect to set EventBridgeEnabled on every CfnBucket.""" 5 6 def visit(self, node): 7 """Visit every node, check if it is a Bucket, update if it is.""" 8 if isinstance(node, s3.CfnBucket): 9 node.add_property_override( 10 "NotificationConfiguration.EventBridgeConfiguration", 11 {"EventBridgeEnabled": True}, 12 ) 13 14 15app = cdk.App() 16MonitorEventsFromMultipleS3BucketsWithEventbridgeStack( 17 scope=app, 18 construct_id="MonitorEventsFromMultipleS3BucketsWithEventbridgeStack", 19) 20cdk.Aspects.of(app).add(EventBridgeNotificationAdder()) 21 22app.synth()
With this code in place, S3 Event Notifications will be automatically sent for every Bucket, and we can reliably assess every change in our objects.

Cost

S3 Event Notifications are charged through EventBridge, but you only pay for events you consume. This is an important distinction. Let's take a look at the standard EventBridge pricing. It involves three components:
  1. AWS default service events: Free
  2. Custom events: $1.00/million custom events published
  3. $1.00/million custom events from opt-in AWS services (e.g., Amazon S3 Event Notifications)
Note the word published at the end of item 2: usually, EventBridge charges for the amount of events put onto an EventBus, regardless how often you consume those events. Even if you do not consume an event at all, you're charged for it.
To prevent extreme costs as soon as you enable EventBridge S3 Event Notifications, the S3 service reads the EventBridge rules and only publishes events matching a rule. So if you set EventBridgeEnabled on 50 Buckets processing millions of objects, but only write a single Rule matching a very specific prefix in one of those Buckets, S3 makes sure to only publish only those events.

When not to use EventBridge S3 Notifications

Generally speaking, EventBridge notifications should be the default implementation for any new S3 implementation. However, there are two exceptions:
  1. Implementations with very high event volumes. As described in the Cost section above, EventBridge notifications are charged per million events published. The old notification style only charges for the Lambda Functions invoked, which results in lower overall costs.
  2. Extensive suffix matching. The old notification style allows for suffix matching, for example only matching objects with a specific extension. EventBridge does not have support for suffix matching, which means you need to implement it yourself. Depending on your use case, it might be easier to use the old style S3 notifications instead.

Conclusion

S3 Event Notifications sent to EventBridge are a well-designed system for processing object changes in your S3 Buckets. Contrary to the old-style notifications, EventBridge makes it easy to monitor events across buckets, allowing for powerful new patterns.

CDK Project

The services and code described in this Bite are available as a Python AWS Cloud Development Kit (CDK) Project. Within the project, execute a cdk synth to generate CloudFormation templates. Then deploy these templates to your AWS account with a cdk deploy. For your convenience, ready-to-use CloudFormation templates are also available in the cdk.out folder. For further instructions how to use the CDK, see Getting started with the AWS CDK.

Click the Download button below for a Zip file containing the project.