Managing bucket synchronization

Unlike AWS replication, HCP for cloud scale can synchronize with buckets on storage systems outside of AWS.

AWS determines the destination bucket using rules, but only applies one rule to each new object. In contrast, HCP for cloud scale can apply multiple rules to each new object so long as the destination buckets are different. This is how one-to-many synchronization is implemented.

AWS does not replicate, but HCP for cloud scale synchronizes, objects that the owner of the source bucket doesn't have permission to read.

In contrast with AWS replication, HCP for cloud scale does not synchronize the following:

• Access control lists (ACLs)
• Lock retention information
• Objects that are encrypted using Amazon S3 managed keys (SSE-S3) and AWS KMS managed keys (SSE-KMS)

If an object being synchronized has the same name as an object in the target bucket, the result depends on whether or not the target bucket uses versioning:

• If versioning is used, the old object is kept as an old version.
• If versioning is not used, the old object is replaced by the new object.

HCP for cloud scale buckets always use versioning. The best practice is to use versioning in all target buckets.

HCP for cloud scale guarantees that operations are applied in the order of their arrival (strong consistency). However, synchronizing multiple operations applied in a short period of time to the same object presents the following difficulties:

• In a distributed system, especially when many systems are involved, synchronizing all operations in correct order is complex.
• Even if HCP for cloud scale synchronizes all operations in correct order to an external storage component, that component might not guarantee that the operations are applied with strong consistency. In particular, AWS guarantees only "eventual consistency."
• For bucket sync-from, the external queue service might not guarantee that messages are provided in correct order. In particular, AWS Simple Queue Service (SQS) does not support first-in, first-out (FIFO) queues for S3 notifications.

Therefore, HCP for cloud scale makes its best effort to synchronize only the latest state of an object, not each version or operation for the object. For example:

• Assume that a client sends three operations to an object, and that they are all committed: (1) PUT, (2) PUT, (3) DEL. The latest state of the object is (3) DEL. HCP for cloud scale only synchronizes DEL.
• Assume that a client sends three operations to an object, and that they are all committed: (1) PUT, (2) DEL, (3) PUT. The latest state of the object is (3) PUT. HCP for cloud scale only synchronizes (3) PUT.

This approach does not guarantee that the latest state of an object will be in the external storage for all situations. Partly because of the "eventual consistency" provided by AWS S3 API, corner cases still exist.

These are the high-level steps involved in setting up bucket synchronization:

1. If appropriate, the HCP for cloud scale administrator assigns a sync-to or sync-from role to tenant administrators.
2. The administrator creates buckets.
3. The administrator configures synchronization rules.
4. Users can now use synchronization when writing objects to buckets.

HCP for cloud scale can apply multiple bucket synchronization rules to each new object so long as the destination buckets are different. This is how one-to-many synchronization is implemented.

A rule collision is when two or more rules that apply to an object have the same destination (that is, the same external host, port, and bucket). HCP for cloud scale does not allow rule collisions, so PUT bucket replication requests are rejected if they contain rule collisions. To avoid rule collisions, you can define as many tags in a rule as necessary, so that multiple rules with the same destination are not needed.

When bucket synchronization rules are created, updated, or deleted, the changes only apply to new objects or new S3 API operations. Objects that existed before the rules are configured are not synchronized. If an object exists in the state PENDING when a rule is created, updated, or deleted, the rule might not be applied to it, because the object might be in the midst of copying.

aws --endpoint -url https://host_ip s3api put-bucket-replication --bucket "bucket" --replication-configuration '{body}'

A rule consists of up to 1000 prefixes and tag-value pairs. You can configure up to 1000 rules per bucket. Separate tag-value pairs in the rule using the keywords "And": or "Or":.

The request body is shown below:

'{
  "Role": "",
  "Rules": [{
    "ID": "string",
    "Filter": {
       "Prefix": "string",
       "Tag": {
         "Key": "string",
         "Value": "string"
      }
    },
    "Status": "boolean",
    "Destination": {
      "Bucket": "json",
      "Account": "B64_key, B64_key",
      "StorageClass": ""
    }
   }
   .
   .
   .
  }]
}'

Bucket sync-to settings are defined by a set of parameters and passed in the value of Destination.Bucket as a Base64-encoded string.

The syntax of a bucket sync-to setting is shown below:

arn::sync-to::version::host:port>::region::bucket_name::auth_version::path_style_always

Bucket sync-from settings include both a bucket address and a notification queue. The settings are defined by a set of parameters and passed in the value of Destination.Bucket as a Base64-encoded string.

The syntax of a bucket sync-from setting is shown below:

arn::sync-to::version::host:port>::S3_region::bucket_name::auth_version::path_style_always::AWS_SQS::SQS_region::SQS_queue::SQS_access_key::SQS_secret_key

None.

Request example:

aws --endpoint-url https://10.08.1019 s3api put-bucket-replication --bucket "hcpcs_bucket" --replication-configuration '{body}'

JSON request:

'{
  "Role": "",
  "Rules": [{
    "ID": "sync_rule1_for_images",
    "Filter": {
      "Prefix": "/images/september/",
      "Tag": {
        "Key": "target",
        "Value": "cloud"
        }
    },
    "Status": "Enabled",
    "Destination": {
      "Bucket": "arn::sync-to::1.0::s3.amazonaws.com:443::us-east-1::redbucket::v4::true",
      "Account": "access_key, secret_key",
      "StorageClass": "STANDARD_IA"
    }
  },
  {
    "ID": "sync_rule2_for_music",
    "Filter": {
      "Prefix": "/music/october/",
      "Tag": {
        "Key": "target",
        "Value": "cloud"
      }
    },
    "Status": "Enabled",
    "Destination": {
      "Bucket": "arn::sync-from::1.0::s3.amazonaws.com:443::us-east-1::bluebucket::v4::true::AWS_SQS::us-east-1::blackqueue::MTIzNA==::Njc4OQ==",
      "Account": "access_key, secret_key",
      "StorageClass": "STANDARD_IA"
    }
  }]
}'

aws --endpoint -url https://host_ip s3api get-bucket-replication --bucket "bucket"

Not applicable.

The response body is shown below:

{
  "ReplicationConfiguration": {
    "Role": "",
    "Rules": [
      {
        "Filter": {
          "And": {
            "Prefix": "string",
            "Tags": [
              {
              "Key": "string",
              "Value": "string"
        }
        .
        .
        .
      },
      "Status": "boolean",
      "Destination": {
        "Bucket": "access_settings",
      },
       "ID": "string",
     }
     ],
  }
}

Request example:

aws --endpoint-url https://10.08.1019 s3api get-bucket-replication --bucket "hcpcs_bucket"

JSON response:

{
  "ReplicationConfiguration": {
    "Role"": "",
    "Rules": [
      {
        "Filter": {
          "And": {
            "Prefix": "SQS",
            "Tags": [
              {
                "Value": "cloud",
                "Key": "target"
              }
            ]
          }
        },
        "Status": "Enabled",
        "Destination": {
          "Bucket": "arn::sync-from::1.0::s3.amazonaws.com:443::<AWS-Region>::hcpcs_bucket::V4::true::AWS_SQS::<SQS-Region>::<SQS-QUEUE-TopicName>",
        },
        "ID": "mirrorBack_rule_for_images"
      }
    ]
  }
}

For a GET object or HEAD object request, the synchronization functions return a replication status header in addition to the standard response metadata. This information is useful before deletion from a source bucket to verify synchronization.

aws --endpoint -url https://host_ip s3api delete-bucket-replication --bucket "bucket"

None.

Request example:

aws --endpoint-url https://10.08.1019 s3api delete-bucket-replication --bucket "hcpcs_bucket"