Skip to main content

We've Moved!

Product Documentation has moved to docs.hitachivantara.com
Hitachi Vantara Knowledge

S3 event notification

Hitachi Content Platform for cloud scale (HCP for cloud scale) lets you configure and manage S3 event notification.

A script is available to simplify the process of configuring S3 event notification.

About S3 event notification

HCP for cloud scale supports the AWS S3 methods PUT Bucket Notification Configuration and GET Bucket Notification Configuration. To enable notifications, an S3 user adds a notification configuration that identifies the events to be published and the destinations (notification target systems) where notifications are sent.

HCP for cloud scale supports overlapping notification rules. Unlike AWS, the same event can be sent to multiple queues.

Access to the event notification functions is controlled by role-based permissions to write or read (set and get) bucket configurations.

Supported events

The list of supported events is:

  • s3:ObjectCreated:*
  • s3:ObjectCreated:Put
  • s3:ObjectCreated:Post
  • s3:ObjectCreated:Copy
  • s3:ObjectCreated:CompleteMultipartUpload
  • s3:ObjectRemoved:*
  • s3:ObjectRemoved:Delete
  • s3:ObjectRemoved:DeleteMarkerCreated

AWS S3 methods such as PUT, POST, and COPY can create an object. Using the ObjectCreated event types, you can enable notification when an object is created using a specific method, or you can use the s3:ObjectCreated:* event type to request notification regardless of the method used to create an object. You do not receive an event notifications from failed operations.

Using the ObjectRemoved event types, you can enable notification when an object or a batch of objects is removed from a bucket. You can request notification when an object is deleted or a versioned object is permanently deleted by using the s3:ObjectRemoved:Delete event type. This event is also sent when a delete marker is created. You can request notification when a delete marker is created for a versioned object by using the s3:ObjectRemoved:DeleteMarkerCreated event. You can also use a wildcard (s3:ObjectRemoved:*) to request notification any time an object is deleted. You do not receive an event notification from automatic deletions from lifecycle policies or from failed operations.

Configuration

HCP for cloud scale fully supports notification configuration.

Configuration can include up to 100 rules. Each rule consists of:

  • One or more event types (such as PUT, COPY, or DELETE)
  • (Optional) A filter with zero or one prefix and zero or one suffix (tags are not supported)
  • A notification target (an SQS queue)

Rules can overlap. That is, an HCP for cloud scale event notification can be sent to multiple targets. However, multiple rules can't send notification of the same event to the same target. A configuration containing rules that overlap in this way is blocked.

Script to generate S3 event notification configuration JSON

HCP for cloud scale includes a script to generate the JSON needed to configure S3 event notification.

The script is written in Python and located in the folder install_path/product/bin (for example, /opt/hcpcs/bin).

The script generates the JSON string that you can use for the element QueueArn in the AWS S3 command put-bucket-notification-configuration to configure the queue's Amazon Resource Name (ARN). Optionally, the script verifies whether the target AWS SQS queue exists, and if more than one matching SQS queue is found returns them all. If you omit the secret key, the script prompts you for it, which lets you create a script that calls this script without storing the secret key. You can mix the short and full form of arguments.

NoteThe script produces JSON using single quotes.
Syntax
EventNotificationsJsonGenerator.py
  [--help]
  --queue queue
  --region region
  --accessKey access_key
  [--secretKey secret_key]
  [--jsonSample output_file.json]
  [--verifyTarget]
  [--insecure]
  [--eventType event_type[,...]]
  [--prefix prefix]
  [--suffix suffix]
  [--id queue_id]
  [--quietMode]
Options and parameters
  • -h, --help

    Optional. Displays a help message and exits.

  • --queue queue, -q queue

    Name of the event notification queue.

  • --region region, -r region

    Region of the event notification queue.

  • --accessKey access_key, -ak access_key

    Access key for the event notification queue.

  • --secretKey secret_key, -sk secret_key

    Secret key for the event notification queue. The script prompts for the key if you don't specify it.

  • --jsonSample output_file.json, -json output_file.json

    Optional. Creates a file named output_file.json with a sample JSON structure for event notification configuration. If not specified, no sample file is created.

  • --verifyTarget, -verify

    Optional. Verifies that the remote queue exists. SSL certificates aren't validated. This option requires python3 and boto3. If not specified, the queue's existence isn't verified.

    NoteYou can't specify both --quietMode and --verifyTarget together.
  • --insecure, -i

    Optional. Suppresses Python warning messages.

  • --eventType event_type[,...], -et event_type[,...]

    Optional. Event notification types. One or more comma-separated types.

    • s3:ObjectCreated:*
    • s3:ObjectCreated:Put
    • s3:ObjectCreated:Post
    • s3:ObjectCreated:Copy
    • s3:ObjectCreated:CompleteMultipartUpload
    • s3:ObjectRemoved:*
    • s3:ObjectRemoved:Delete
    • s3:ObjectRemoved:DeleteMarkerCreated

    The default is s3:ObjectCreated:*,s3:ObjectRemoved:*.

  • --prefix, -px

    Optional. Filter prefix. If not specified, no prefix is used.

  • --suffix, -sx

    Optional. Filter suffix. If not specified, no suffix is used.

  • --id queue_id, -id queue_id

    Optional. The queue configuration ID. The default is SampleEvenId.

  • --quietMode, -qm

    Optional. Displays only JSON for the element QueueArn.

    NoteYou can't specify both --quietMode and --verifyTarget together.
Example
$ EventNotificationJsonGenerator.py -q queue1 -r us-east-2 -ak A1234567890 -sk S1234567890 -verify -json testqueue.json 

This example can produce the following output:

Verifying that a remote notification queue with a prefix "queue1" exists...
Verification successfully completed: found "queue1" queue.

Generated a JSON string for QueueArn element for S3 Event Notifications configuration:
"{'type': 'AWS_SQS', 'queue': 'queue1', 'region': 'us-east-2', 
 'accessKey': 'QUtJQVNPS1cyRUkzQVlKSVZMTkY=', 'secretKey': 'bUtOQnUydUZaaFZqQTQ0eGs3M1NaRzZoMUdnVkt2MHpLOEFhOFdmUQ=='}"

Saved sample JSON file for event notification configuration in 'testqueue.json'

You can use 'testqueue.json' sample JSON file as an input to put-bucket-notification-configuration S3 API. For example, using aws s3api command:
aws s3api put-bucket-notification-configuration --no-verify-ssl --bucket cloudscale-bucket --notification-configuration file://testqueue.json