Bucket synchronization

Objects that existed before synchronization functions are configured are not synchronized.

HCP for cloud scale verifies the rules that are valid at the time an object is synchronized, not at the time the object is ingested.

Objects that are marked as deleted are not synchronized.

Most system metadata is not synchronized, specifically:

• Owner ID and Name
• Timestamps (when last modified)
• Metadata returned in x-amz-grant-*
• Metadata returned in x-amz-acl
• Metadata returned in x-amz-grant-*
• Metadata returned in x-amz-acl
• Metadata returned in x-amz-storage-class
• Metadata returned in x-amz-replication-status
• Metadata returned in x-amz-server-side-encryption-*
• Metadata returned in x-amz-restore-*
• Metadata returned in x-amz-version-id-*
• Metadata returned in x-amz-website-redirect-location
• Metadata returned in x-amz-object-lock-*

The bucket sync-from function only supports one rule for the same external SQS queue and external bucket. If a bucket has multiple sync-from rules for the same external queue, objects might not be synchronized. To use multiple rules for an external bucket, use one SQS queue for each rule.

Unlike AWS replication, HCP for cloud scale can synchronize with buckets on storage systems outside of AWS.

AWS determines the destination bucket using rules, but only applies one rule to each new object. In contrast, HCP for cloud scale can apply multiple rules to each new object so long as the destination buckets are different. This is how one-to-many synchronization is implemented.

AWS does not replicate, but HCP for cloud scale synchronizes, objects that the owner of the source bucket doesn't have permission to read.

In contrast with AWS replication, HCP for cloud scale does not synchronize the following:

• Access control lists (ACLs)
• Lock retention information
• Objects that are encrypted using Amazon S3 managed keys (SSE-S3) and AWS KMS managed keys (SSE-KMS)

If an object being synchronized has the same name as an object in the target bucket, the result depends on whether the target bucket uses versioning:

• If versioning is used, the old object is kept as an old version.
• If versioning is not used, the old object is replaced by the new object.

HCP for cloud scale buckets always use versioning. The best practice is to use versioning in all target buckets.

HCP for cloud scale guarantees that operations are applied in the order of their arrival (strong consistency). However, synchronizing multiple operations applied in a short period of time to the same object presents the following difficulties:

• In a distributed system, especially when many systems are involved, synchronizing all operations in correct order is complex.
• Even if HCP for cloud scale synchronizes all operations in correct order to an external storage component, that component might not guarantee that the operations are applied with strong consistency. In particular, AWS guarantees only "eventual consistency."
• For bucket sync-from, the external queue service might not guarantee that messages are provided in correct order. In particular, AWS Simple Queue Service (SQS) does not support first-in, first-out (FIFO) queues for S3 notifications.

Therefore, HCP for cloud scale makes its best effort to synchronize only the latest state of an object, not each version or operation for the object. For example:

• Assume that a client sends three operations to an object and that they are all committed: (1) PUT, (2) PUT, (3) DEL. The latest state of the object is (3) DEL. HCP for cloud scale only synchronizes DEL.
• Assume that a client sends three operations to an object and that they are all committed: (1) PUT, (2) DEL, (3) PUT. The latest state of the object is (3) PUT. HCP for cloud scale only synchronizes (3) PUT.

This approach does not guarantee that the latest state of an object will be in the external storage for all situations. Partly because of the "eventual consistency" offered by AWS S3 API, corner cases still exist.

After they are created, buckets and objects are not immediately visible. Some client applications (such as Cloudberry Explorer) immediately retrieve the list of buckets to display the new bucket or object, which is not visible. If you create a new bucket or object and it's not immediately visible, update the list manually.

HCP for cloud scale can apply multiple bucket synchronization rules to each new object so long as the destination buckets are different. This is how one-to-many synchronization is implemented.

A rule collision is when two or more rules that apply to an object have the same destination (that is, the same external host, port, and bucket). HCP for cloud scale does not allow rule collisions, so PUT bucket replication requests are rejected if they contain rule collisions. To avoid rule collisions, you can define as many tags in a rule as necessary, so that multiple rules with the same destination are not needed.

After an object operation is performed, the policy engine asynchronously checks if that object needs to be copied according to the sync-to rules. When bucket synchronization rules are created, updated, or deleted, the changes only apply to new objects, object operations, and to objects that have not been yet processed by the policy engine. Objects that existed before the rules were configured are not synchronized. If an object exists in the PENDING state when a rule is created, updated, or deleted, the rule change might not be applied.

You cannot set up bucket synchronization with the same bucket as both the source and the destination.

aws --endpoint-url https://10.08.1019 s3api put-bucket-replication --bucket "hcpcs_bucket" --replication-configuration file://rules.json

A rule consists of up to 1000 prefixes and tag-value pairs. You can configure up to 1000 rules per bucket. Separate tag-value pairs in the rule using the keywords "And": or "Or":.

The content of the configuration JSON file is:

{
  "Role": "",
  "Rules": [{
    "ID": "string",
    "Filter": {
       "Prefix": "string",
       "Tag": {
         "Key": "string",
         "Value": "string"
      }
    },
    "Status": "boolean",
    "Destination": {
      "Bucket": "json"
    }
   }
   .
   .
   .
  }]
}

Bucket sync-to settings are defined by a set of parameters and passed in the value of Rules.Destination.Bucket as a Base64-encoded JSON structure.

The syntax inside the bucket parameter for the sync-to setting is:

{
  'version': 'version', 
  'action': 'sync-from', 
  'externalBucket': {
    'host': 'host', 
    'type': 'type', 
    'region': 'region', 
    'remoteBucketName': 'bucket_name', 
    'accessKey': 'B64_key', 
    'secretKey': 'B64_key', 
    'port': 'port', 
    'authVersion': 'auth_version', 
    'usePathStyleAlways': '[true|false]'
    },
  'notifications': {
    'type': 'type', 
    'region': 'region', 
    'queue': 'queue', 
    'accessKey': 'B64_key', 
    'secretKey': 'B64_key'
    }
}

Bucket sync-from settings include both a bucket address and a notification queue. The settings are defined by a set of parameters and passed in the value of Rules.Destination.Bucket as a Base64-encoded string.

The syntax inside the bucket parameter for sync-from setting is:

"{
  'version': 'version', 
  'action': 'sync-from', 
  'externalBucket': {
    'host': 'host', 
    'type': 'type', 
    'region': 'region', 
    'remoteBucketName': 'bucket_name', 
    'accessKey': 'B64_key', 
    'secretKey': 'B64_key', 
    'port': 'port', 
    'authVersion': 'auth_version', 
    'usePathStyleAlways': '[true|false]'
    }
}"

None.

Request example:

aws --endpoint-url https://10.08.1019 s3api put-bucket-replication --bucket "hcpcs_bucket" --replication-configuration file://rules.json

Configuration rules.json:

{
    "ID": "sync_rule2_for_music",
    "Filter": {
      "Prefix": "/music/october/",
      "Tag": {
        "Key": "target",
        "Value": "cloud"
        }
      }
    },
    "Status": "Enabled",
    "Destination": {
      "Bucket": "{
        'version' : '1.0',
        'action' : 'sync_from',
        'externalBucket' : {
          'type' : 'AMAZON_S3',
          'region' : 'us-east-1',
          'remoteBucketName' : 'bluebucket',
          'authVersion' : 'V4',
          'usePathStyleAlways' : 'true',
          'accessKey' : 'access_key',
          'secretKey' : 'secret_key'
          },
        "notifications" : {
          "type" : "AMAZON_SQS",
          "region" : "us-east-1",
          "queue" : "testQueue",
          "accessKey" : "access_key",
          "secretKey" : "secret_key"
          }
        },
      }
    }
  }]
}

The script is written in Python and located in the folder install_path/product/bin (for example, /opt/hcpcs/bin).

The script generates the JSON string that you can use for the field destination.bucket in the AWS S3 command put-bucket-replication. Optionally, the script verifies whether the destination bucket exists. If you omit the secret key, the script prompts you for it, which lets you create a script that calls this script without storing the secret key. You can mix the short and full form of arguments.

SyncToBucketJsonGenerator.py
  [--help]
  --s3host host
  --region region
  --bucket bucket
  --accessKey access_key
  [--secretKey secret_key]
  [--s3type { GENERIC_S3 | AMAZON_S3 }]
  [--port port]
  [--authVersion { v2 | v4 }]
  [--usePathStyleAlways {true | false}]
  [--jsonSample output_file.json]
  [--verifyTarget]
  [--http]
  [--quietMode]

• -h, --help

  Optional. Displays a help message and exits.

• --s3host host, -s3 host

  Host name of the remote S3 storage component.

• --region region, -r region

  Region of the remote bucket.

• --bucket bucket, -b bucket

  Name of the remote bucket.

• --accessKey access_key, -ak access_key

  Access key for the remote bucket.

• --secretKey secret_key, -sk secret_key

  Secret key for the remote bucket. The script prompts for the key if you don't specify it.

• --s3type { GENERIC_S3 | AMAZON_S3 }, -s3t { GENERIC_S3 | AMAZON_S3 }

  Optional. The remote bucket type:

  • GENERIC_S3: An S3-compatible node
  • AMAZON_S3: An Amazon Web Services S3-compatible node
  If not specified, the default bucket type is AMAZON_S3.

• --port port, -p port

  Optional. Port of the remote bucket. If not specified, the default port is 443.

• --authVersion { v2 | v4 }, -av { v2 | v4 }

  Optional. The Auth Version of the remote bucket. If not specified, the default version is v4.

• --usePathStyleAlways {true | false}, -upsa {true | false}

  Optional. Sets the Use Path Style Always flag for the remote bucket. If not specified, the default is true.

• --jsonSample output_file.json, -json output_file.json

  Optional. Creates a file named output_file.json with a sample JSON structure for bucket replication configuration. If not specified, no sample file is created.

• --verifyTarget, -verify

  Optional. Verifies that the remote bucket exists. SSL certificates are not validated. This option requires python3 and boto3. If not specified, the bucket's existence isn't verified.

• --http, -http

  Optional. Use HTTP when verifying the remote bucket. If not specified, the default is to use HTTPS.

• --quietMode, -qm

  Optional. Displays only the Destination.Bucket element.

  NoteYou can't specify both --quietMode and --verifyTarget together.

The script is written in Python and located in the folder install_path/product/bin (for example, /opt/hcpcs/bin).

The script generates the JSON string that you can use for the field destination.bucket in the AWS S3 command put-bucket-replication. Optionally, the script verifies whether the destination bucket or the target AWS SQS queue exist. If you omit the secret key, the script prompts you for it, which lets you create a script that calls this script without storing the secret key. If you omit the access key for a queue, the script uses the access key and secret key for the bucket. You can mix the short and full form of arguments.

SyncFromBucketJsonGenerator.py
  [--help]
  --s3host host
  --region region
  --bucket bucket
  --accessKey access_key
  [--secretKey secret_key]
  [--s3type { GENERIC_S3 | AMAZON_S3 }]
  [--port port]
  [--authVersion { v2 | v4 }]
  [--usePathStyleAlways {true | false}]
  [--jsonSample output_file.json]
  [--verifyTarget]
  [--http]
  --notificationsQueue queue
  [--notificationsRegion region]
  [--notificationsAccessKey access_key]
  [--notificationsSecretKey secret_key]
  [--quietMode]

• -h, --help

  Optional. Displays a help message and exits.

• --s3host host, -s3 host

  Host name of the remote S3 storage component.

• --region region, -r region

  Region of the remote bucket.

• --bucket bucket, -b bucket

  Name of the remote bucket.

• --accessKey access_key, -ak access_key

  Access key for the remote bucket.

• --secretKey secret_key, -sk secret_key

  Secret key for the remote bucket. The script prompts for the key if you don't specify it.

• --s3type { GENERIC_S3 | AMAZON_S3 }, -s3t { GENERIC_S3 | AMAZON_S3 }

  Optional. The remote bucket type:

  • GENERIC_S3: An S3-compatible node
  • AMAZON_S3: An Amazon Web Services S3-compatible node
  If not specified, the default bucket type is AMAZON_S3.

• --port port, -p port

  Optional. Port of the remote bucket. If not specified, the default port is 443.

• --authVersion { v2 | v4 }, -av { v2 | v4 }

  Optional. The Auth Version of the remote bucket. If not specified, the default version is v4.

• --usePathStyleAlways {true | false}, -upsa {true | false}

  Optional. Sets the Use Path Style Always flag for the remote bucket. If not specified, the default is true.

• --jsonSample output_file.json, -json output_file.json

  Optional. Creates a file named output_file.json with a sample JSON structure for bucket replication configuration. If not specified, no sample file is created.

• --verifyTarget, -verify

  Optional. Verifies that the remote bucket exists. SSL certificates are not validated. This option requires python3 and boto3. If not specified, the bucket's existence isn't verified.

• --http, -http

  Optional. Use HTTP when verifying the remote bucket. If not specified, the default is to use HTTPS.

• --notificationsQueue queue, -nq queue

  Name of the notifications queue.

• --notificationsRegion region, -nq region

  Optional. Name of the notifications region. If not specified, the default is the region of the remote bucket.

• --notificationsAccessKey access_key, -nak access_key

  Optional. The notifications access key. If not specified, the default is the access key of the remote bucket.

• --notificationsSecretKey secret_key, -nsk secret_key

  Optional. The notifications secret key. If not specified, the default is the secret key of the remote bucket.

• --quietMode, -qm

  Optional. Displays only the JSON for QueueArn.

  NoteYou can't specify both --quietMode and --verifyTarget together.

aws --endpoint -url https://host_ip s3api get-bucket-replication --bucket "bucket"

Not applicable.

The response body is shown below:

{
  "ReplicationConfiguration": {
    "Role": "",
    "Rules": [
      {
        "Filter": {
          "And": {
            "Prefix": "string",
            "Tags": [
              {
              "Key": "string",
              "Value": "string"
        }
        .
        .
        .
      },
      "Status": "boolean",
      "Destination": {
        "Bucket": "access_settings",
      },
       "ID": "string",
     }
     ],
  }
}

Request example:

aws --endpoint-url https://10.08.1019 s3api get-bucket-replication --bucket "hcpcs_bucket"

JSON response:

{
  "ReplicationConfiguration": {
    "Role"": "",
    "Rules": [
      {
        "Filter": {
          "And": {
            "Prefix": "SQS",
            "Tags": [
              {
                "Value": "cloud",
                "Key": "target"
              }
            ]
          }
        },
        "Status": "Enabled",
        "Destination": {
          "Bucket": {
            'version': 'version', 
            'action': 'sync-from', 
            'externalBucket': {
              'host': 'host', 
              'type': 'type', 
              'region': 'region', 
              'remoteBucketName': 'bucket_name', 
              'port': 'port', 
              'authVersion': 'auth_version', 
              'usePathStyleAlways': '[true|false]'
              }
            }"
          },
        "ID": "mirrorBack_rule_for_images"
      }
    ]
  }
}

For a GET object or HEAD object request, the synchronization functions return a replication status header in addition to the standard response metadata. This information is useful before deletion from a source bucket to verify synchronization.

When an object is created, HCP for cloud scale evaluates the sync-to rules for the bucket. If the object matches the rules, it sets the object's sync state as PENDING. Most of the time, this sync state is accurate. However, it is never definitive because users may change the sync-to rules for the bucket before the policy engine starts processing the object, which happens asynchronously. The policy engine evaluates the sync-to rules again when processing an object to act according to the latest sync rules.

For example:

• An object was ingested that matches the sync-to rules, so its sync state is set as PENDING. Then, a user changes the sync-to rules. The object does not match the rules anymore so the object is actually not synced and that sync state is removed.
• An object was ingested that does not match the sync-to rules, so its sync state is not set. Then, a user changes the sync rules. The object now matches the rules so the object is actually synced and the sync state is set to COMPLETED.

aws --endpoint -url https://host_ip s3api delete-bucket-replication --bucket "bucket"

None.

None.

Request example:

aws --endpoint-url https://10.08.1019 s3api delete-bucket-replication --bucket "hcpcs_bucket"