Bucket synchronization

Objects that existed before synchronization functions are configured are not synchronized. HCP for cloud scale verifies the rules that are valid at the time an object is synchronized, not at the time the object is ingested.

Encrypted objects, ACLs, and objects that are marked as deleted are also not synchronized.

Most system metadata is not synchronized, specifically:

• Owner ID and Name
• Timestamps (when last modified)
• Metadata returned in x-amz-grant-*
• Metadata returned in x-amz-acl
• Metadata returned in x-amz-storage-class
• Metadata returned in x-amz-replication-status
• Metadata returned in x-amz-server-side-encryption-*
• Metadata returned in x-amz-restore-*
• Metadata returned in x-amz-version-id-*
• Metadata returned in x-amz-website-redirect-location
• Metadata returned in x-amz-object-lock-*

Additionally, certain operations are not synchronized, specifically:

• DELETE Object
• Bulk DELETE Object
• PUT Object ACLs
• PUT Object tagging
• DELETE Object tagging

The bucket sync-from function only supports one rule for the same external SQS queue and external bucket. If a bucket has multiple sync-from rules for the same external queue, objects might not be synchronized. To use multiple rules for an external bucket, use one SQS queue for each rule.

In contrast with AWS replication, HCP for cloud scale does not synchronize the following:

• Access control lists (ACLs)
• Lock retention information
• Objects that are encrypted using Amazon S3 managed keys (SSE-S3) and AWS KMS managed keys (SSE-KMS)

If an object being synchronized has the same name as an object in the target bucket, the result depends on whether the target bucket uses versioning:

• If versioning is used, the old object is kept as an old version.
• If versioning is not used, the old object is replaced by the new object.

HCP for cloud scale buckets always use versioning. The best practice is to use versioning in all target buckets.

HCP for cloud scale guarantees that operations are applied in the order of their arrival (strong consistency). However, synchronizing multiple operations applied in a short period of time to the same object presents the following difficulties:

• In a distributed system, especially when many systems are involved, synchronizing all operations in correct order is complex.
• Even if HCP for cloud scale synchronizes all operations in correct order to an external storage component, that component might not guarantee that the operations are applied with strong consistency.
• For bucket sync-from, the external queue service might not guarantee that messages are provided in correct order. In particular, AWS Simple Queue Service (SQS) does not support first-in, first-out (FIFO) queues for S3 notifications.

Therefore, HCP for cloud scale makes its best effort to synchronize only the latest state of an object, not each version or operation for the object. For example:

• Assume that a client sends three operations to an object and that they are all committed: (1) PUT, (2) PUT, (3) DEL. The latest state of the object is (3) DEL. HCP for cloud scale only synchronizes DEL.
• Assume that a client sends three operations to an object and that they are all committed: (1) PUT, (2) DEL, (3) PUT. The latest state of the object is (3) PUT. HCP for cloud scale only synchronizes (3) PUT.

After they are created, buckets and objects are immediately visible.

HCP for cloud scale can apply multiple bucket synchronization rules to each new object, so long as the destination bucket has identical rules.

A rule collision is when two or more rules that apply to an object have the same destination (that is, the same external host, port, and bucket). HCP for cloud scale does not allow rule collisions, so PUT bucket replication requests are rejected if they contain rule collisions. To avoid rule collisions, you can define as many tags in a rule as necessary, so that multiple rules with the same destination are not needed.

After an object operation is performed, the policy engine asynchronously checks if that object needs to be copied according to the sync-to rules. When bucket synchronization rules are created, updated, or deleted, the changes only apply to new objects, object operations, and to objects that have not been yet processed by the policy engine. Objects that existed before the rules were configured are not synchronized. If an object exists in the PENDING state when a rule is created, updated, or deleted, the rule change might not be applied.

You cannot set up bucket synchronization with the same bucket as both the source and the destination.

aws --endpoint-url https://10.08.1019 s3api put-bucket-replication --bucket "hcpcs_bucket" --replication-configuration file://rules.json

A rule consists of up to 1000 prefixes and tag-value pairs. You can configure up to 1000 rules per bucket. Separate tag-value pairs in the rule using the keywords "And": or "Or":.

The content of the configuration JSON file is:

aws --endpoint-url https://company.com s3api put-bucket-replication --bucket "bucket name" --replication-configuration \
'{
    "Role": "",
    "Rules": [{
        "ID": "string",
        "Filter": {
            "Prefix": "string",
            "Tag": {
                "Key": "string",
                "Value": "string"
            }
        },
        "Status": "Enabled",
        "Destination": {
            "Bucket": "<a string with several parameters>",
            "Account": "<a string with several parameters>",
            "StorageClass": "<a string with several parameters>"
        }
      },
      {
        "ID": "string",
        "Filter": {
            "Prefix": "string",
            "Tag": {
                "Key": "string",
                "Value": "string"
            }
        },
        "Status": "Enabled",
        "Destination": {
            "Bucket": "<a string with several parameters>",
            "Account": "",
            "StorageClass": "<a string with several parameters>"
      }
    }]
}'

Bucket sync-to settings are defined by a set of parameters and passed in the value of Rules.Destination.Bucket as a Base64-encoded JSON structure.

The syntax inside the bucket parameter for the sync-to setting is:

{
  'version': 'version', 
  'action': 'sync-from', 
  'externalBucket': {
    'host': 'host', 
    'type': 'type', 
    'region': 'region', 
    'remoteBucketName': 'bucket_name', 
    'accessKey': 'B64_key', 
    'secretKey': 'B64_key', 
    'port': 'port', 
    'authVersion': 'auth_version', 
    'usePathStyleAlways': '[true|false]'
    },
  'notifications': {
    'type': 'type', 
    'region': 'region', 
    'queue': 'queue', 
    'accessKey': 'B64_key', 
    'secretKey': 'B64_key'
    }
}

Bucket sync-from settings include both a bucket address and a notification queue. The settings are defined by a set of parameters and passed in the value of Rules.Destination.Bucket as a Base64-encoded string.

The syntax inside the bucket parameter for sync-from setting is:

"{
  'version': 'version', 
  'action': 'sync-from', 
  'externalBucket': {
    'host': 'host', 
    'type': 'type', 
    'region': 'region', 
    'remoteBucketName': 'bucket_name', 
    'accessKey': 'B64_key', 
    'secretKey': 'B64_key', 
    'port': 'port', 
    'authVersion': 'auth_version', 
    'usePathStyleAlways': '[true|false]'
    }
}"

None.

Request example:

aws --endpoint-url https://10.08.1019 s3api put-bucket-replication --bucket "hcpcs_bucket" --replication-configuration file://rules.json

Configuration rules.json:

{
    "ID": "sync_rule2_for_music",
    "Filter": {
      "Prefix": "/music/october/",
      "Tag": {
        "Key": "target",
        "Value": "cloud"
        }
      }
    },
    "Status": "Enabled",
    "Destination": {
      "Bucket": "{
        'version' : '1.0',
        'action' : 'sync_from',
        'externalBucket' : {
          'type' : 'AMAZON_S3',
          'region' : 'us-east-1',
          'remoteBucketName' : 'bluebucket',
          'authVersion' : 'V4',
          'usePathStyleAlways' : 'true',
          'accessKey' : 'access_key',
          'secretKey' : 'secret_key'
          },
        "notifications" : {
          "type" : "AMAZON_SQS",
          "region" : "us-east-1",
          "queue" : "testQueue",
          "accessKey" : "access_key",
          "secretKey" : "secret_key"
          }
        },
      }
    }
  }]
}

aws --endpoint-url https://company.com s3api get-bucket-replication --bucket "hcpcs_bucket"

Not applicable.

This is the response body for sync-to:

{
    "ReplicationConfiguration": {
        "Role"": "",
        "Rules": [
            {
                "Filter": {
                    "And": {
                        "Prefix": "SQS",
                        "Tags": [
                            {
                                "Value": "string",
                                "Key": "string"
                            }
                        ]
                    }
                },
                "Status": "Enabled",
                "Destination": {
                    "Bucket": "arn::sync-to::1.0::s3.amazonaws.com:443::<AWS-Region>::<AWSBucketName>::V4::true"
                },
                "ID": "string"
            }
        ],
    }
}

This is the response body for sync-from:

{
    "ReplicationConfiguration": {
        "Role"": "",
        "Rules": [
            {
                "Filter": {
                    "And": {
                        "Prefix": "string",
                        "Tags": [
                            {
                                "Value": "string",
                                "Key": "string"
                            }
                        ]
                    }
                },
                "Status": "Enabled",
                "Destination": {
                     "Bucket": "arn::sync-from::1.0::s3.amazonaws.com:443::<AWS-Region>::<AWSBucketName>::V4::true::AWS_SQS::<SQS-Region>::<SQS-QUEUE-TopicName>",
                },
                "ID": "string"
            }
        ]
    }
}

Request example:

aws --endpoint-url https://10.08.1019 s3api get-bucket-replication --bucket "hcpcs_bucket"

JSON response:

{
  "ReplicationConfiguration": {
    "Role"": "",
    "Rules": [
      {
        "Filter": {
          "And": {
            "Prefix": "SQS",
            "Tags": [
              {
                "Value": "cloud",
                "Key": "target"
              }
            ]
          }
        },
        "Status": "Enabled",
        "Destination": {
          "Bucket": {
            'version': 'version', 
            'action': 'sync-from', 
            'externalBucket': {
              'host': 'host', 
              'type': 'type', 
              'region': 'region', 
              'remoteBucketName': 'bucket_name', 
              'port': 'port', 
              'authVersion': 'auth_version', 
              'usePathStyleAlways': '[true|false]'
              }
            }"
          },
        "ID": "mirrorBack_rule_for_images"
      }
    ]
  }
}

For a GET object or HEAD object request, the synchronization functions return a replication status header in addition to the standard response metadata. This information is useful before deletion from a source bucket to verify synchronization.

When an object is created, HCP for cloud scale evaluates the sync-to rules for the bucket. If the object matches the rules, it sets the object's sync state as either PENDING, COMPLETED, or FAILED. If the object does not match any of the rules, nothing changes.

Most of the time, this sync state is accurate. However, it is never definitive because users might change the sync-to rules for the bucket before the policy engine starts processing the object, which happens asynchronously. The policy engine evaluates the sync-to rules again when processing an object to act according to the latest sync rules.

For example:

• An object was ingested that matches the sync-to rules, so its sync state is set as PENDING. Then, a user changes the sync-to rules. The object does not match the rules anymore so the object is actually not synced and that sync state is removed.
• An object was ingested that does not match the sync-to rules, so its sync state is not set. Then, a user changes the sync rules. The object now matches the rules so the object is actually synced and the sync state is set to COMPLETED.

aws --endpoint -url https://host_ip s3api delete-bucket-replication --bucket "bucket"

None.

None.

Request example:

aws --endpoint-url https://company.com s3api delete-bucket-replication --bucket "hcpcs_bucket"