Skip to main content
Hitachi Vantara Knowledge

Replication

Replication is a process that supports configurations in which selected tenants and namespaces are maintained on two or more HCP systems and the objects in those namespaces are managed across those systems. This cross-system management helps ensure that data is well-protected against the unavailability or catastrophic failure of a system.

In addition to replicating objects, HCP replicates tenant and namespace configuration, user and group accounts, retention classes, content classes, all compliance log messages, and most other tenant log messages.

A replication topology is a configuration of HCP systems that are related to each other through replication. Typically, the systems in a replication topology are in separate geographic locations and are connected by a high-speed wide area network. This arrangement provides geographically distributed data protection (called geo-protection).

Clients can read from namespaces on all systems where those namespaces are replicated. The replication topology, which is configured at the system level, determines the systems on which clients can write to namespaces.

Replication benefits

Replication has several purposes:

  • If a system in a replication topology becomes unavailable (for example, due to network issues), another system in the topology can provide continued data availability.
  • If a system in a replication topology suffers irreparable damage, another system in the topology can serve as a source for disaster recovery.
  • If multiple HCP systems are widely separated geographically, each system may be able to provide faster data access for some applications than the other systems can, depending on where the applications are running.
  • If an enterprise has several satellite offices, an HCP system at a central facility can consolidate data from the HCP systems at those outlying locations.
  • If an object cannot be read from one system in a replication topology (for example, because a node is unavailable), HCP can try to read it from another system in the topology. HCP tries to do this only if:
    • The namespace that contains the object is being replicated.
    • The namespace has the read-from-remote-system feature enabled.
    • The object has already been replicated. Users can check object metadata to determine whether an object has been replicated.
  • If a system in a replication topology is unavailable, requests to that system that come through the REST, S3 compatible API can be automatically serviced by another system in the topology without the client needing to modify the target URL. The other system can service a request only if:
    • The namespace named in the URL is replicated on the other system
    • The namespace named in the URL is configured to accept requests redirected from other HCP systems
    • The HCP systems involved use DNS for system addressing

Replication implementation

Replication is implemented at the tenant and namespace level. When creating a tenant, the HCP system administrator specifies whether the tenant is eligible to be replicated. HCP system administrators can change this setting from not allowing replication to allowing it. However, they cannot do the reverse.

If the tenant is eligible for replication, you can choose whether to include each namespace in the replication of the tenant. You can change this setting for a namespace at any time.

If allowed by the tenant configuration, for each cloud-optimized namespace replicated with the tenant, you can specify whether HCP is allowed to use erasure coding to implement replication of the objects in that namespace. Erasure coding improves storage efficiency. However, read performance may be better for replicated namespaces that do not allow erasure coding.

System administrators can change a tenant from not being able to choose whether namespaces allow erasure coding to being able to do this. After this change occurs:

  • Preexisting cloud-optimized namespaces with replication enabled are automatically reconfigured to allow erasure coding.
  • When you enable cloud optimization for a preexisting namespace that had replication enabled but was not cloud optimized, the namespace is automatically configured to allow erasure coding.

The HCP system administrator selects tenants to be replicated from among those that are eligible. If a tenant has granted system-level administrative users access to itself, the system administrator can change the namespace selections for that tenant. HCP replicates the configuration of each selected tenant and the configuration and contents of all the namespaces selected for replication with the tenant.

Depending on the replication topology, you may not be able to make any configuration changes to the tenant or any of its namespaces on one or more systems in the topology. Clients cannot make any changes to namespace content on systems on which you cannot make configuration changes.

Replication is asynchronous with other HCP activity. If allowed by the system administrator, you can monitor replication progress in the Tenant Management Console.

Replication collision handling

If clients can write to multiple systems in a replication topology, collisions can occur when different changes are made to the same objects on different systems. Similarly, if you can make configuration changes to the tenant and its namespaces on multiple systems in a replication topology, configuration collisions can occur.

The way HCP handles collisions that occur due to replication depends on the type of collision. However, the general rule is that more recent changes have priority over conflicting less recent changes.

Object content collisions

An object content collision occurs when, for a namespace without versioning enabled, these events occur in the order shown:

  1. An object is created with the same name in that namespace on two systems in a replication topology, but the object has different content on the two systems.
  2. The object on one of the systems is replicated to the other system.

If versioning is enabled, no collision occurs. Instead, the less recently created of the two objects becomes an old version of the more recently created object.

When an object content collision occurs, the more recently created object keeps its name and location. The other object is either moved to the .lost+found directory in the same namespace or renamed, depending on the namespace configuration.

When HCP moves an object to the .lost+found directory, the full object path becomes .lost+found/replication/system-generated-directory/ old-object-path.

When renaming an object due to a content collision, HCP changes the object name to object-name.collision or object-name.version-id.collision, where version-id is the version ID of the object. HCP uses the second format only if versioning has ever been enabled for the namespace that contains the object but is not currently enabled.

If the new name is already in use, HCP changes the object name to object-name.1.collision or object-name.version-id.1.collision, as applicable. If that name is already in use, HCP successively increments the middle integer by one until a unique name is formed.

Objects that have been relocated or renamed due to content collisions are flagged as replication collisions in their system metadata. Clients can use the metadata query API to search for objects that are flagged as replication collisions.

If an object that’s flagged as a replication collision changes (for example, if its retention period is extended), its collision flag is removed. If a client creates a copy of a flagged object with a new name, the collision flag is not set on the copy.

You can configure namespaces to have the DIsposition service automatically delete objects that are flagged as replication collisions. When selecting this option for a namespace, you specify the number of days the DIsposition service should wait before deleting such an object. The days are counted from the time the collision flag is set. If the collision flag is removed from an object, the object is no longer eligible for deletion by the DIsposition service.

System metadata collisions

A system metadata collision occurs when these events occur in the order shown:

  1. Different changes are made to the system metadata for a given object on each of two systems in a replication topology.
  2. The changed system metadata on one of the systems is replicated to the other system.
For example, suppose a user on one system changes the shred setting for an object while a user on the other system changes the index setting for the same object. When the object on either system is replicated to the other system, a system metadata collision occurs.

If a collision occurs when changed system metadata for a given object is replicated from one system (system A) in a replication topology to another system (system B) in the topology:

  • For changed system metadata other than the retention setting and hold status:
    • If the last change made on system A is more recent than the last change made on system B, HCP changes the system metadata on system B to match the system metadata on system A.
    • If the last change on system B is more recent than the last change on system A, HCP does not change the system metadata on system B.
  • For a changed retention setting:
    • If the retention setting on system A specifies a longer retention period than does the retention setting on system B, HCP changes the retention setting on system B to match the retention setting on system A.
    • If the retention setting on system B specifies a longer retention period than does the retention setting on system A, HCP does not change the retention setting on system B.
  • For a changed hold status:
    • If the object is on hold on system A but not on system B, HCP places the object on hold on system B.
    • If the object is on hold on system B but not on system A, HCP leaves the object on hold on system B.
Here are some examples of how HCP handles collisions when changed system metadata for a given object is replicated from one system (system A) in a replication topology to another system (system B) in the topology.
Example 1

The object starts out on both system A and system B with these system metadata settings:

  • Shred: false
  • Index: false

The list below shows a sequence of events in which the system metadata for the object is changed and the changes are then replicated.

  1. On system A, a client changes the shred setting to true.
  2. On system B, a client changes the index setting to true.
  3. The changes on system A are replicated to system B. The resulting settings for the object on system B are:
    • Shred: false
    • Index: true
Example 2

The object starts out on both system A and system B with these system metadata settings:

  • Retention: Initial Unspecified
  • Shred: false
  • Index: false

The list below shows a sequence of events in which the system metadata for the object is changed and the changes are then replicated.

  1. On system A, a client changes the retention setting to Deletion Prohibited.
  2. On system B, a client changes the retention setting to Deletion Allowed.
  3. On system B, a client changes the index setting to true.
  4. On system A, a client changes the shred setting to true.
  5. The changes on system A are replicated to system B. The resulting settings for the object on system B are:
    • Retention: Deletion Prohibited
    • Shred: true
    • Index: false
Example 3

The object starts out on both system A and system B with these system metadata settings:

  • Retention: Initial Unspecified
  • Hold: true
  • Shred: false
  • Index: false

The list below shows a sequence of events in which the system metadata for the object is changed and the changes are then replicated.

  1. On system A, a client changes the retention setting to Deletion Allowed.
  2. On system B, a client changes the retention setting to Deletion Prohibited.
  3. On system B, a client changes the index setting to true.
  4. On system A, a client changes the shred setting to true.
  5. On system A, a client releases the object from hold.
  6. The changes on system A are replicated to system B. The resulting settings for the object on system B are:
    • Retention: Deletion Prohibited
    • Hold: true
    • Shred: true
    • Index: false
  7. The changes on system B are replicated to system A. The resulting settings for the object on system A are:
    • Retention: Deletion Prohibited
    • Hold: true
    • Shred: true
    • Index: false

Custom metadata collisions

A custom metadata collision occurs when these events occur in the order shown:

  1. One of these changes occurs:
    • An annotation is added with the same name to a given object on each of two systems in a replication topology, but the annotation has different content on the two systems.

      The addition of an annotation to a given object on only one of the systems does not result in a custom metadata collision if the object does not have an annotation with the same name on the other system. In this case, the new annotation is replicated without conflict.

    • Different changes are made to the content of a given annotation for a given object on each of the two systems in a replication topology.
    • A change is made to the content of a given annotation for a given object on one system in a replication topology, and the same annotation is deleted on another system in the topology.
  2. The change made on one of the systems is replicated to the other system.

If a collision occurs when a custom metadata change for a given object is replicated from one system (system A) in a replication topology to another system (system B) in the topology:

  • If the last change on system A is more recent than the last change on system B, HCP applies the change from system A to the custom metadata on system B
  • If the last change on system B is more recent than the last change on system A, HCP does not change the custom metadata on system B
Here are two examples of how HCP handles collisions when custom metadata changes for a given object are replicated from one system (system A) in a replication topology to another system (system B) in the topology.
Example 1

The object starts out with annotations named a1 and a2 on both system A and system B.

The list below shows a sequence of events in which the annotations for the object are changed and the changes are then replicated.

  1. On system B, a client changes the content of a1.
  2. On system A, a client makes a different change to the content of a1.
  3. On system A, a client adds annotation a3 to the object.
  4. On system B, a client adds annotation a3 with different content from the a3 added on system A.
  5. The changes on system A are replicated to system B. The resulting annotations for the object on system B are:
    • a1 with the changed content from system A
    • a2 (unchanged)
    • a3 with the content added on system B
  6. The changes on system B are replicated to system A. The resulting annotations for the object on system A are:
    • a1 with the changed content from system A
    • a2 (unchanged)
    • a3 with the content added on system B
Example 2

The object starts out with the annotations named a1, a2, and a3 on both system A and system B.

The list below shows a sequence of events in which the annotations for the object are changed and the changes are then replicated.

  1. On system B, a client changes the content of a1.
  2. On system A, a client deletes a1.
  3. On system A, a client changes the content of a2.
  4. On system B, a client changes the content of a2.
  5. On system A, a client deletes a3.
  6. On system B, a client changes the content of a3.
  7. The changes on system A are replicated to system B. The resulting annotations for the object on system B are:
    • a2 with the changed content from system B
    • a3 with the changed content from system B
  8. The changes on system B are replicated to system A, the resulting annotations for the object on system A are:
    • a2 with the changed content from system B
    • a3 with the changed content from system B

Access control list collisions

An ACL collision occurs when these events occur in the order shown:

  1. Different changes are made to the ACL for a given object on each of two systems in a replication topology.
  2. The changed ACL on one of the systems is replicated to the other system.

An ACL is treated as a single unit. If a collision occurs when a changed ACL for a given object is replicated from one system (system A) in a replication topology to another system (system B) in the topology:

  • If the last change to the ACL on system A is more recent than the last change to the ACL on system B, HCP changes the ACL on system B to match the changed ACL on system A
  • If the last change to the ACL on system B is more recent than the last change to the ACL on system A, HCP does not change the ACL on system B
For example, suppose the ACL for a given object starts out with these grants on both system A and system B:
  • All users: read
  • User lgreen: write
  • User mwhite: write, delete

The list below shows a sequence of events in which the ACL for the object is changed and the change is then replicated.

  1. On system B, a client changes the grants in the ACL to:
    • All users: read
    • User lgreen: write, delete
    • User pdgrey: write
  2. On system A, a client changes the grants in the ACL to:
    • All users: read
    • User mwhite: write
    • User pdgrey: write
  3. The changed ACL on system A is replicated to system B. The resulting ACL for the object on system B contains these grants:
    • All users: read
    • User mwhite: write
    • User pdgrey: write

Configuration collisions

A configuration collision occurs when these events occur in the order shown:

  1. Different changes are made to the same configuration property on each of two systems in a replication topology.
  2. The changed property on one of the systems is replicated to the other system.
Examples of configuration properties are:
  • The data access permission mask for a namespace
  • The default shred setting for a namespace
  • The HTTP protocol enabled setting for a namespace
  • The default versioning setting for new namespaces
  • The roles for a user account
  • The data access permissions a group account has for a namespace
  • The protocol optimization setting on a namespace

Certain groups of properties are treated as a single unit. Generally, these groups consist of properties that are updated by a single submission in the Tenant Management Console. Two notable exceptions to this rule are data access permissions for user accounts and content properties for content classes. In these cases, each set of data access permissions for a namespace and each content property is treated as an individual property.

If a collision occurs when a configuration change is replicated from one system (system A) in a replication topology to another system (system B) in the topology:

  • If the last change to the configuration on system A is more recent than the last change to the configuration on system B, HCP changes the configuration on system B to match the configuration on system A
  • If the last change to the configuration on system B is more recent than the last change to the configuration on system A, HCP does not change the configuration on system B

The rules above apply to all configuration collisions except collisions that occur when retention class properties are changed.

Here are two examples of how HCP handles collisions when configuration changes are replicated from one system (system A) in a replication topology to another system (system B) in the topology.

Example 1

A given namespace starts out on both system A and system B with these properties:

  • Read from remote system: enabled
  • Collision handling: move object

The list below shows a sequence of events in which the namespace configuration is changed and the change is then replicated.

  1. On system B, an administrator disables read from replica for the namespace.
  2. On system A, an administrator changes the collision handling option for the namespace to rename object.
  3. The change on system A is replicated to system B. Because read from remote system and collision handling are properties in the same submission group, the resulting properties for the namespace on system B are:
    • Read from remote system: enabled
    • Collision handling: rename object
Example 2

A given user account starts out on both system A and system B with these roles and data access permissions:

  • Roles: monitor, compliance
  • Namespace-1 permissions: browse, read, write, delete

The list below shows a sequence of events in which the user account is changed and the changes are then replicated.

  1. On system B, a security administrator adds the administrator role to the user account.
  2. On system A, a security administrator removes the monitor role from the account.
  3. On system B, an administrator removes the write permission from Namespace-1 and adds the purge permission.
  4. On system A, an administrator adds the privileged permission to Namespace-1.
  5. On system B, an administrator gives the user account browse and read permissions for Namespace-2.
  6. On system A, an administrator gives the user account browse, read, and write permissions for Namespace-3.
  7. The changes on system A are replicated to system B. Because roles and data access permissions are in separate submission groups, the resulting roles and data access permissions for the user account on system B are:
    • Roles: compliance
    • Namespace-1 permissions: browse, read, write, delete, privileged
    • Namespace-2 permissions: browse, read
    • Namespace-3 permissions: browse, read, write

Retention class collisions

A retention class collision occurs when these events occur in the order shown:

  1. Different changes are made to the same retention class on each of two systems in a replication topology.
  2. The changed retention class on one of the systems is replicated to the other system.

If a collision occurs when a change to a retention class is replicated from one system (system A) in a replication topology to another system (system B) involved in the topology:

  • If the last change to the retention class on system A is more recent than the last change to the class on system B and:
    • The value of the class on system A is greater than the value of the class on system B, HCP changes the value of the class on system B to the value of the class on system A
    • The value of the class on system A is less than the value of the class on system B and:
      • System B is in enterprise mode, HCP changes the value of the class on system B to the value of the class on system A
        NoteAn exception to this rule is when the value of the class on system A is -2 (Initial Unspecified) and the value of the class on system B is not 0 (Deletion Allowed). In this case, the value of the class on system B does not change.
      • System B is in compliance mode, HCP does not change the value of the class on system B
  • If the last change to the retention class on system B is more recent than the last change to the class on system A, HCP does not change the value of the class on system B
NoteA retention class value of -1 (Deletion Prohibited) is greater than a value that’s a specific duration. A retention class value of 0 (Deletion Allowed) or -2 (Initial Unspecified) is less than a value that’s a specific duration.

 

  • Was this article helpful?