S3 event notification
Hitachi Content Platform for cloud scale (HCP for cloud scale) lets you configure and manage S3 event notification.
A script is available to simplify the process of configuring S3 event notification.
About S3 event notification
HCP for cloud scale supports the AWS S3 methods PUT Bucket Notification Configuration
and GET Bucket Notification Configuration
. To enable notifications, an S3 user adds a notification configuration that identifies the events to be published and the destinations (notification target systems) where notifications are sent.
HCP for cloud scale supports overlapping notification rules. Unlike AWS, the same event can be sent to multiple queues.
Access to the event notification functions is controlled by role-based permissions to write or read (set and get) bucket configurations.
The list of supported events is:
s3:ObjectCreated:*
s3:ObjectCreated:Put
s3:ObjectCreated:Post
s3:ObjectCreated:Copy
s3:ObjectCreated:CompleteMultipartUpload
s3:ObjectRemoved:*
s3:ObjectRemoved:Delete
s3:ObjectRemoved:DeleteMarkerCreated
AWS S3 methods such as PUT, POST, and COPY can create an object. Using the ObjectCreated event types, you can enable notification when an object is created using a specific method, or you can use the s3:ObjectCreated:*
event type to request notification regardless of the method used to create an object. You do not receive an event notifications from failed operations.
Using the ObjectRemoved event types, you can enable notification when an object or a batch of objects is removed from a bucket. You can request notification when an object is deleted or a versioned object is permanently deleted by using the s3:ObjectRemoved:Delete
event type. This event is also sent when a delete marker is created. You can request notification when a delete marker is created for a versioned object by using the s3:ObjectRemoved:DeleteMarkerCreated
event. You can also use a wildcard (s3:ObjectRemoved:*
) to request notification any time an object is deleted. You do not receive an event notification from automatic deletions from lifecycle policies or from failed operations.
HCP for cloud scale fully supports notification configuration.
Configuration can include up to 100 rules. Each rule consists of:
- One or more event types (such as PUT, COPY, or DELETE)
- (Optional) A filter with zero or one prefix and zero or one suffix (tags are not supported)
- A notification target (an SQS queue)
Rules can overlap. That is, an HCP for cloud scale event notification can be sent to multiple targets. However, multiple rules can't send notification of the same event to the same target. A configuration containing rules that overlap in this way is blocked.
Script to generate S3 event notification configuration JSON
HCP for cloud scale includes a script to generate the JSON needed to configure S3 event notification.
The script is written in Python and located in the folder install_path/product/bin (for example, /opt/hcpcs/bin).
The script generates the JSON string that you can use for the element QueueArn
in the AWS S3 command put-bucket-notification-configuration
to configure the queue's Amazon Resource Name (ARN). Optionally, the script verifies whether the target AWS SQS queue exists, and if more than one matching SQS queue is found returns them all. If you omit the secret key, the script prompts you for it, which lets you create a script that calls this script without storing the secret key. You can mix the short and full form of arguments.
EventNotificationsJsonGenerator.py [--help] --queue queue --region region --accessKey access_key [--secretKey secret_key] [--jsonSample output_file.json] [--verifyTarget] [--insecure] [--eventType event_type[,...]] [--prefix prefix] [--suffix suffix] [--id queue_id] [--quietMode]
-h, --help
Optional. Displays a help message and exits.
--queue queue, -q queue
Name of the event notification queue.
--region region, -r region
Region of the event notification queue.
--accessKey access_key, -ak access_key
Access key for the event notification queue.
--secretKey secret_key, -sk secret_key
Secret key for the event notification queue. The script prompts for the key if you don't specify it.
--jsonSample output_file.json, -json output_file.json
Optional. Creates a file named output_file.json with a sample JSON structure for event notification configuration. If not specified, no sample file is created.
--verifyTarget, -verify
Optional. Verifies that the remote queue exists. SSL certificates aren't validated. This option requires python3 and boto3. If not specified, the queue's existence isn't verified.
NoteYou can't specify both--quietMode
and--verifyTarget
together.--insecure, -i
Optional. Suppresses Python warning messages.
--eventType event_type[,...], -et event_type[,...]
Optional. Event notification types. One or more comma-separated types.
s3:ObjectCreated:*
s3:ObjectCreated:Put
s3:ObjectCreated:Post
s3:ObjectCreated:Copy
s3:ObjectCreated:CompleteMultipartUpload
s3:ObjectRemoved:*
s3:ObjectRemoved:Delete
s3:ObjectRemoved:DeleteMarkerCreated
The default is
s3:ObjectCreated:*,s3:ObjectRemoved:*
.--prefix, -px
Optional. Filter prefix. If not specified, no prefix is used.
--suffix, -sx
Optional. Filter suffix. If not specified, no suffix is used.
--id queue_id, -id queue_id
Optional. The queue configuration ID. The default is
SampleEvenId
.--quietMode, -qm
Optional. Displays only JSON for the element QueueArn.
NoteYou can't specify both--quietMode
and--verifyTarget
together.
$ EventNotificationJsonGenerator.py -q queue1 -r us-east-2 -ak A1234567890 -sk S1234567890 -verify -json testqueue.json
This example can produce the following output:
Verifying that a remote notification queue with a prefix "queue1" exists...
Verification successfully completed: found "queue1" queue.
Generated a JSON string for QueueArn element for S3 Event Notifications configuration:
"{'type': 'AWS_SQS', 'queue': 'queue1', 'region': 'us-east-2',
'accessKey': 'QUtJQVNPS1cyRUkzQVlKSVZMTkY=', 'secretKey': 'bUtOQnUydUZaaFZqQTQ0eGs3M1NaRzZoMUdnVkt2MHpLOEFhOFdmUQ=='}"
Saved sample JSON file for event notification configuration in 'testqueue.json'
You can use 'testqueue.json' sample JSON file as an input to put-bucket-notification-configuration S3 API. For example, using aws s3api command:
aws s3api put-bucket-notification-configuration --no-verify-ssl --bucket cloudscale-bucket --notification-configuration file://testqueue.json