Skip to main content
Hitachi Vantara Knowledge

About geo-protection

Geo-protection is implemented by the HCP Replication service. The Replication service is responsible for keeping tenants and namespaces on two or more HCP systems in sync with each other. Replication entails sending information from the system where the service is running to one or more other systems. The information sent includes tenant and namespace configuration information, metadata and data for newly created objects, metadata changes for existing objects, and instructions to perform delete operations.

The Replication service works on the systems in a replication topology. A replication topology is a configuration of HCP systems wherein each system is related to each other system through a path of one or more replication relationships. The HCP system from which you are viewing the replication topology is called the local system. Any other system in the replication topology is called a remote system.

You select the tenants to be replicated in a replication topology. For each tenant, the tenant administrator selects the namespaces to be replicated. If the tenant administrator has granted system-level users access to the tenant, you also can select namespaces for replication.

Geo-protection offers several benefits:

  • If a system in a replication topology becomes unavailable, a remote system can provide continued data availability. A system is unavailable if it is not running or is unreachable.
  • If a system in a replication topology suffers irreparable damage, a remote system can serve as a source for disaster recovery.
  • If the Content Verification or Protection service discovers objects it cannot repair on a system in a replication topology, the service can use object data and metadata from a remote system to make the needed repairs.
  • If an object cannot be read from a system in a replication topology, for example, because a node is unavailable, HCP can try to read it from a remote system. HCP tries to do this only if:
    • The namespace that contains the object is involved in replication.
    • The namespace has the read-from-remote-system feature enabled.
    • The object has already been replicated. Users can check object metadata to determine whether an object has been replicated.
    • The replication networks for the local and remote systems can both use the same IP mode (IPv4 or IPv6).
  • If a system in a replication topology is unavailable, requests to that system that come through the REST, S3 compatible, or HSwift namespace access protocol can be automatically serviced by a remote system without the client needing to modify the target URL. A remote system can service a request only if:
    • The namespace named in the URL is replicated to the remote system.
    • The namespace named in the URL is configured to accept requests redirected from other HCP systems.
    • The HCP systems involved use a shared DNS for system addressing.
    • Both the DNS and the unavailable system are configured to support DNS failover. (This condition is not a requirement if you’re using a load balancer that can target the requests to a remote system.)
  • If multiple HCP systems are widely separated geographically, each system may be able to provide faster data access for some applications than the other systems can, depending on where the applications are running.
  • If an enterprise has several satellite offices, an HCP system at a central facility can consolidate data from the HCP systems at those outlying locations.

Protection types

HCP supports two types of geo-protection: whole-object protection and erasure-coded protection.

Whole-object protectionWith whole-object protection, all the data for each object in a replicated namespace is maintained on each HCP system in a replication topology, except on systems where the object is on a metadata-only storage tier. In response to a client request for an object on a metadata-only storage tier on the target HCP system, HCP retrieves the object data from another system in the replication topology.

In case of system unavailability or loss, whole-object protection ensures continued data availability when at least one of the systems with the object data remains available. If an object is metadata-only on all the available systems, the object data is inaccessible.

The service plan associated with a namespace determines whether and when objects in that namespace move to a metadata-only storage tier.

Erasure-coded protection

With erasure-coded protection, the data for each object in a replicated namespace is subject to erasure coding. With erasure coding, the data is encoded and broken into multiple chunks that are then stored across multiple HCP systems. All but one chunk contains object data. The other chunk contains parity for the object data.

For the purpose of erasure coding, parts of multipart objects are each treated as an individual object.

To implement erasure-coded protection, you create an erasure coding topology. An erasure coding topology is a named replication topology in which object data is protected by erasure coding. The total number of chunks for an erasure-coded object is equal to the number of systems in the erasure coding topology, where each system stores one chunk.

An erasure-coded object can be read from any system in the erasure coding topology. In response to a client read request, HCP reconstructs the object data on the fly by retrieving the chunks stored on the other systems. If any one system in an erasure coding topology is unavailable, the data for an erasure-coded object can be completely reconstructed from the chunks on the remaining systems.

If more than one system in an erasure coding topology is unavailable, HCP cannot reconstruct the erasure-coded objects. In this case, HCP marks the erasure-coded objects as unavailable or irreparable, depending on the reasons why they cannot be reconstructed.

An erasure coding topology does not necessarily include all the HCP tenants that are replicated in the underlying replication topology. You need to explicitly select the tenants to be included. You cannot include the default tenant in an erasure coding topology.

An erasure coding topology has properties that control which objects are erasure coded and when those objects are erasure coded.

Protection types for namespaces

Protection types apply to individual namespaces. Only cloud-optimized namespaces support erasure-coded protection.

When you create a replication-enabled tenant, you choose between:

  • Allowing erasure-coded protection for all cloud-optimized namespaces owned by the tenant.
  • Allowing the tenant administrator to allow erasure-coded protection for selected cloud-optimized namespaces.

    If the tenant administrator has granted system-level users access to the tenant, you also can allow erasure-coded protection for selected namespaces.

Objects in a replicated namespace that allows erasure-coded protection are erasure coded only if the tenant that owns the namespace is included in an erasure coding topology. If the tenant is not in an erasure coding topology, the namespace uses whole-object protection.

Objects in a replicated namespace that does not allow erasure-coded protection use whole-object protection, even if the tenant that owns the namespace is included in an erasure coding topology.

NoteRegardless of protection type, namespaces obey the DPL specified by their associated service plans. That is, on any given system:
  • A namespace that uses whole-object protection maintains DPL copies of the complete data for each object
  • A namespace that uses erasure-coded protection maintains DPL copies of the chunk for each object

Distribution methods for erasure-coded protection

HCP supports two distribution methods for erasure-coded protection: chunk distribution and full-copy distribution.

Chunk distribution

With chunk distribution, after an object is ingested, the chunks for the object are calculated and distributed among the systems in the erasure coding topology. Only the system where the object was ingested has a full copy of the object data.

After all of the chunks have been successfully distributed and a specified amount of time has elapsed, the HCP Geo-Distributed Erasure Coding service reduces the object on the ingest system to a chunk. The amount of time that the service waits before reducing an object to a chunk is called the erasure coding delay.

Full-copy distribution

With full-copy distribution, after an object is ingested, full copies of the object data are distributed to the systems in the erasure coding topology. After the erasure coding delay expires, the Geo-Distributed Erasure Coding service on each system reduces the full copy of the object data on that system to a chunk.

Metadata-only considerations for objects subject to erasure coding

For an erasure-coded object to be protected properly, each system in the erasure coding topology must have a chunk for the object. Therefore, even if the object should be metadata-only on one or more systems, those systems must each store a chunk for the object:

  • With chunk distribution, each system receives a chunk for an erasure-coded object, regardless of whether the object should be metadata-only on that system.
  • With full-copy distribution, each system receives a full copy of the object data unless the object should be metadata-only on that system. After the erasure coding delay expires, each system where the object should be metadata-only receives a chunk for the object, and the Geo-Distributed Erasure Coding service on each system with a full copy of the object data reduces that full copy to a chunk.
Object counts

An HCP system keeps track of the total number of objects stored on it and, separately, the number of erasure-coded objects stored on it:

  • The total number of objects stored on a system includes whole objects, metadata-only objects, and objects that are subject to erasure coding, regardless of whether the system currently has a full copy of the object data or a chunk for the object.
  • The number of erasure-coded objects stored on a system includes only objects for which the system currently has a chunk.

HCP also tracks both of these object counts by tenant and namespace. You can view system-level and tenant-level object counts in the System Management Console, through the management API, and in system-level chargeback reports. For tenants to which you have administrative access, you can also view namespace-level object counts through the management API and in chargeback reports.

Tenant administrators can view total object counts for the applicable tenant and its namespaces in the Tenant Management Console, through the management API, and in tenant-level chargeback reports. Tenant administrators cannot see counts of erasure-coded objects.

Choosing a protection type

The two protection types, whole-object protection and erasure-coded protection, each have their own benefits and limitations. The following sections describe factors you should consider when choosing the protection type for a namespace.

Storage efficiency

With whole-object protection, all the data for an object in a replicated namespace is stored on each system that has that namespace, except on systems where the object is on a metadata-only storage tier. This means that the total storage used for the object data is at most the size of the data times the number of systems in the replication topology. (The storage used can be less due to compression/encryption, duplicate elimination, and metadata-only storage tiers.)

NoteThis discussion assumes a data protection level (DPL) of one. As the DPL increases, the amount of storage needed to store the data for an object also increases, regardless of the protection type.

For example, in a three-system replication topology in which the object is not metadata-only on any system, the storage used for a 1 MB object is, at most, 3 MB. Having the object be metadata-only on one or two of the systems increases storage efficiency but decreases the level of data protection.

With erasure-coded protection, each system in the erasure coding topology stores one data or parity chunk for any given erasure-coded object. The size of each chunk for an object is the size of the object data divided by the number of data chunks for the object. This means that the total storage used for an object in a replicated namespace is at most the size of a chunk times the total number of data and parity chunks for the object.

For example, each object in an erasure coding topology with three systems has two data chunks and one parity chunk, so the size of each chunk of a one-megabyte object is .5 MB. As a result, the total storage used for the object is, at most, 3 × 0.5 MB, which is 1.5 MB.

The greater the number of systems in an erasure coding topology, the greater the storage efficiency. The maximum number of systems that can be in an erasure coding topology is six. With six systems, the total storage used for a one-megabyte object is, at most, 1.2 megabytes.

Data availability

Whole-object protection can provide better protection from system unavailability than erasure-coded protection provides. With whole-object protection, clients can access objects in a replicated namespace if at least one of the systems where the object data is stored is available. (HCP does not store data for objects on metadata-only storage tiers.)

With erasure-coded protection, all but one of the systems in the erasure coding topology must be available for clients to access objects in a namespace that's replicated on those systems.

Disaster recovery

Both whole-object protection and erasure-coded protection provide support for disaster recovery in case of catastrophic system failure. Disaster recovery entails reprotecting objects in replicated namespaces to their prefailure protection level by recovering them to a replacement system.

With whole-object protection, if a single system fails, the object data can be recovered to the replacement system from any remaining system where the replicated objects are not metadata-only. If two or more systems fail concurrently, whole-object protection can still provide support for disaster recovery. However, as the level of protection increases, the number of systems that must store the complete object data also increases, thereby decreasing storage efficiency.

With erasure-coded protection, if a single system fails, object data can be reconstructed on the replacement system from the chunks on all the remaining systems in the erasure coding topology. If two or more systems fail concurrently, object data cannot be reconstructed and may be permanently lost.

Read performance

With whole-object protection, a client read request for a replicated object results in a single read operation when the request is issued against a system that has the object data. If the object is metadata-only on the target system, that system must retrieve the object data from another system in the replication topology, which may increase the response time.

With erasure-coded protection, a client read request for a single erasure-coded object causes the system that's servicing the request to reconstruct the object from multiple chunks. Only one of those chunks is stored on that system. Therefore, a client read request for an erasure-coded object results in multiple read operations. These operations can be concurrent or sequential, depending on the replication topology on which the erasure coding topology is based. If sequential reads are required, the response time may be longer than if all the reads are concurrent.

Rehydration with whole-object protection and restore periods with erasure-coded protection can decrease read response times.

These considerations apply to choosing a protection type for a replicated namespace when read performance is a factor:

  • If objects are likely to be read frequently throughout their lifespans and read performance is more important than storage efficiency, consider using whole-object protection for the namespace.
  • If objects are unlikely to be read and storage efficiency is important, consider using erasure-coded protection for the namespace.
  • If objects are likely to be read only for a limited amount of time in the short term and read performance is important, consider using erasure-coded protection with an erasure coding delay that's equal to or greater than the amount of time objects will be read for.
  • If objects are likely to be read frequently but only for limited periods throughout their lifespans and read performance is important, consider using erasure-coded protection with a restore period. With a restore period, after an object is read, a full copy of the object data is placed on the ingest tier for the containing namespace and is kept there for a configured amount of time.

    The ingest tier for a namespace is the storage tier where objects in the namespace are stored when the object data is first written to HCP. The ingest tier is determined by the service plan currently associated with the namespace.

Network bandwidth usage

For namespaces that use whole-object protection, replication entails sending the complete data for an object to each system in the replication topology, except to systems where the object should be metadata-only. Similarly, for namespaces that use erasure-coded protection with full-copy distribution, the complete object data is sent to each system in the erasure coding topology, except to systems where the object should be metadata-only.

For namespaces that use erasure-coded protection with chunk distribution, only object chunks are sent throughout the erasure coding topology. As a result, replication with this type of protection typically requires less network bandwidth than is required for whole-object protection and for erasure-coded protection with full-copy distribution.

On the other hand, with whole-object protection, a client read request against a system that has the object data results in a single data transfer operation. With erasure-coded protection, a client read request results in at least one data transfer operation for each required chunk that's not on the target system in addition to the data transfer operation between the system servicing the request and the client. Therefore, read requests with erasure-coded protection typically use more network bandwidth than is used for read requests with whole-object protection.

Additionally, the chunks for erasure-coded objects are transferred between systems on the replication network. Because servicing read requests takes precedence over replication processing, while any replicated namespaces are experiencing a high read rate, replication may be slow.

Rehydration with whole-object protection and restore periods with erasure-coded protection can affect network bandwidth usage for read operations.

PUT copy operations

For a PUT copy operation where the source object is erasure coded, the new object must be reconstructed from the chunks for the source object. For a large object, this reconstruction can take a long time, increasing the potential for the PUT copy operation to time out.

If large objects in a replicated namespace are likely to be sources for PUT copy operations, whole-object protection may be a better choice for the namespace than erasure-coded protection.

How replication works

To enable replication between two HCP systems, you create a replication link between the two systems. A replication link is a secure, trusted relationship between two systems. The link determines what is replicated and how data is transmitted between the systems.

Link types

A replication link can be active/active or active/passive:

  • With an active/active link, the HCP tenants and namespaces and default-namespace directories being replicated are read-write on both systems, and data is replicated in both directions between the two systems. Active/active links are designed for use in environments where applications need seamless access to namespaces regardless of where those applications are located. All links in an erasure coding topology must be active/active links.

    With an active/active link, HCP gives preference to synchronizing object metadata between the two systems, followed by data synchronization. This preferential treatment enables the fastest access to objects regardless of which system they were created on. If a full copy of the data for an object or a chunk for the object has not yet been replicated to the system targeted by a client request, HCP can read all the object data from a remote system. Additionally, synchronizing metadata first ensures that object names are reserved as quickly as possible.

    NoteUntil a full copy of the data for an object or a chunk for the object is replicated to a given system, the object is metadata-only on that system.
  • With an active/passive link, the HCP tenants and namespaces and default-namespace directories being replicated are read-write on one system, called the primary system, and read-only on the other system, called the replica. In this case, data is replicated in one direction only: from the primary system to the replica. Active/passive links are designed to support read-only content sharing and disaster recovery.

    With an active/passive link, HCP replicates object data and metadata together, thereby ensuring that objects are complete on the replica and can be recovered should that become necessary. However, this approach means that the replica cannot service requests for an object until that object is fully replicated.

To a primary system, an active/passive link is an outbound link; that is, the system is sending data through the link. To a replica, an active/passive link is an inbound link; that is, the system is receiving data through the link. To both systems involved in an active/active link, the link functions as both an outbound link and an inbound link.

Link security

Replication relies on SSL to ensure the integrity and security of transmitted data. Before you can create a replication link, the two systems involved in the link must each have a valid SSL server certificate. Each system must also have the server certificate from the other system installed as a trusted replication server certificate.

Replication priority

Replication is asynchronous with object ingest. An HCP system can, therefore, develop a backlog of objects to be replicated.

You can choose to have the Replication service work on objects with the oldest changes first, regardless of which namespaces they’re in. Alternatively, you can have the service balance its processing time evenly across the namespaces being replicated. In this case, the service may replicate recent changes in some namespaces before older changes in others.

What is replicated?

Replication works by tenant. For each replication link, you can select HCP tenants to be replicated on that link. If you have administrative access to the tenant, you can also select the namespaces to include in replication of that tenant. To replicate the default tenant, you select the default-namespace directories to be replicated.

Regardless of the system on which you create the link:

  • For an active/active link, you can select items to be replicated from both systems involved in the link
  • For an active/passive link, you can select items to be replicated only from the primary system

You can select the HCP tenants to include in an erasure coding topology from any system in the topology. Each tenant you select is automatically added to every link in the underlying replication topology. You cannot include the default tenant in an erasure coding topology.

What is replicated for each tenant differs between HCP tenants and the default tenant.

Replication of HCP tenants

When creating a tenant, you specify whether it can be replicated. After creating the tenant, you can change this setting from not allowing replication to allowing replication. You cannot do the reverse.

A replication link or erasure coding topology can include only HCP tenants that are configured to allow replication.

Replicated tenants and namespaces

When an HCP tenant is replicated, the Replication service works only on selected namespaces owned by that tenant. The tenant administrator can select and deselect namespaces for replication at any time. If the tenant is being replicated, the tenant and its selected namespaces are replicated on each link that includes the tenant. If the tenant is not being replicated, the selected namespaces are not replicated either.

If an HCP tenant has granted system-level users administrative access to itself, you can select and deselect its namespaces for replication when you create the replication link or by modifying the link at a later time. You should coordinate namespace selection with the tenant administrator to ensure that you both have the same understanding of what should be replicated.

Tenants can be replicated from one HCP system to another regardless of the number of tenants that already exist on the other system. However, you can create a new tenant on a system only if the total number of tenants on the system is less than 1,000.

Similarly, namespaces can be replicated from one HCP system to another regardless of the number of namespaces that already exist on the other system. However, you can create a new namespace on a system only if the total number of namespaces on the system is less than ten thousand.

HCP tenants created on different systems can have the same names as each other. However, every tenant has a unique internal ID, so HCP can distinguish tenants with the same name from one another. You cannot replicate a tenant from one system to another if the second system has a different tenant with the same name. To replicate the tenant in this case, you first need to change the name of the tenant on one of those systems.

Replicated items

The Replication service replicates these items for an HCP tenant:

  • The tenant configuration
  • The configuration of each namespace being replicated
  • Existing objects, object creations, object deletions, and metadata changes in the namespaces being replicated
  • Old versions of objects in the namespaces being replicated
  • Retention class creations, modifications, and deletions for the namespaces being replicated
  • Content class and content property creations, modifications, and deletions
  • All tenant-level log messages
  • All namespace-level log messages for the namespaces that are being replicated
  • User accounts defined for the tenant
  • Group accounts defined for the tenant
Replicated changes

With an active/active link, replicated HCP tenants are read-write on both systems involved in the link. Administrators can make changes to the configuration of the replicated tenants and namespaces on either system, and users and applications can make changes to the content of the replicated namespaces on either system. All changes made on each system are replicated to the other system.

With an active/passive link, replicated HCP tenants are read-write on the primary system and read-only on the replica. Administrators cannot make any configuration changes to the replicated tenants or to any of their namespaces on the replica, nor can users and applications make any changes to the content of the replicated namespaces on the replica. However, all changes made on the primary system are replicated to the replica.

NoteAlthough you cannot make any changes to the configuration of a replicated namespace on a replica, you can reindex the namespace.

Replication of the default tenant

For the default tenant, the Replication service works on one or more selected directories directly under fcfs_data in the default namespace. The service replicates the entire directory hierarchy under each selected directory.

Default namespace properties

Directories in the default namespace on one system involved in a replication link can be replicated only to the default namespace on the other system involved in the link. They cannot be replicated to HCP namespaces.

Additionally:

  • Before you can include any default-namespace directories in a replication link, the default namespace must exist on both systems involved in the link.
  • The default namespaces in the two systems must have the same cryptographic hash algorithm and retention mode.

The default namespace can be configured to prevent writes and deletes. This configuration can differ between the default namespace on the two systems involved in a link. However, even if the system receiving data is configured to prevent these actions, changes to namespace content on the sending system are replicated.

Replicated directories

You select directories for replication when you create the replication link. You can select additional directories or deselect directories by modifying the link at a later time. A replication link can include, at most, 256 directories.

Directories can be created in the default namespace on both of the systems involved in a replication link, so the two systems can have directories with the same name. You cannot replicate a directory created on one of the systems if a locally created directory with the same name already exists on the other system.

The one exception to this is the directory used for email archiving through SMTP. You can replicate the email directory if the email directory name is the same on both systems.

Replicated changes

With an active/active link, replicated directories are read-write on both systems involved in the link. All changes made to the content of those directories on either system are replicated to the other system.

With an active/passive link, the replicated directories are read-write on the primary system and read-only on the replica. Users and applications cannot make any changes to the content of the replicated directories on the replica. However, all changes made on the primary system are replicated to the replica.

Replicated items

The Replication service replicates these items for the default tenant:

  • Existing objects, object creations, object deletions, and metadata changes in the directories being replicated
  • Retention class creations, modifications, and deletions for the default namespace
  • Content class and content property creations, modifications, and deletions
  • All log messages relating to retention classes and privileged delete operations

Replication service processing

With an active/active link, the Replication service runs on both HCP systems involved in the link. With an active/passive link, the Replication service runs only on the primary system.

Each replication link has a schedule that specifies when the Replication service should run. By default, when you create a link, the link schedule has the service running 24 hours a day, seven days a week.

The Replication service starts sending information on a link the first time you add HCP tenants and/or default-namespace directories to that link, as long as the link schedule has the service running at that time. If the service is not scheduled to run at that time, the service starts sending information as soon as the next scheduled run period starts.

Configuration information first

When you first add HCP tenants, namespaces, and/or default-namespace directories to a replication link, the Replication service replicates the configuration of those items as soon as possible. The service then starts replicating objects in those namespaces and default-namespace directories. The service replicates objects in chronological order either across all namespaces or within each namespace in turn, depending on the link configuration. The chronological order is oldest-first based on the time of the last metadata change to the object.

Whenever you add a new item to a replication link or change the configuration of an item already included on the link, the Replication service replicates the configuration of the new item or the configuration change as soon as possible. This ensures the correct behavior of objects on the receiving system.

Whole-object protection with unavailable systems

With whole object protection, the Replication service on the system where an object is ingested sends a full copy of the data for the object to the available systems to which the ingest system is directly connected by a replication link and on which the object is not supposed to be metadata-only. If a system is unavailable, the Replication service sends the full copy of the object data to that system when the system becomes available.

Chunk distribution with unavailable systems

For erasure-coded protection with chunk distribution, the Replication service on the system where an object is ingested sends a chunk for the object to each available system in the erasure coding topology. When an unavailable system becomes available, the Replication service on the ingest system sends a chunk to that system. After all the systems that were unavailable have chunks and the erasure coding delay has expired, the Geo-distributed Erasure Coding service reduces the full copy of the object data on the ingest system to a chunk.

A system in an erasure coding topology can suffer a catastrophic failure after ingesting data while another system in the topology is unavailable. If this happens with a topology that's configured for chunk distribution, the data that was ingested on the failed system while the other system was unavailable is lost.

To minimize the risk of data loss, if an erasure coding topology is configured for chunk distribution:

  • Any upgrade of a system in the topology should be performed while the system is online
  • Planned system unavailability (due, for example, to a system shutdown for maintenance or to a loss of connectivity during a network upgrade) should be scheduled for times when data ingest is at a minimum
TipTo protect newly ingested objects during a period of planned system unavailability, take these actions before making the system unavailable:
  1. For each namespace that's owned by a tenant included in the erasure coding topology and that allows erasure coding, ensure that the service plan associated with that namespace does not include a metadata-only storage tier on at least two of the systems that will remain available. If an applicable service plan includes a metadata-only storage tier, you can temporarily either modify the service plan or associate the namespace with a different service plan.
  2. Disable the erasure coding service.
  3. Reconfigure the erasure coding topology to use full-copy distribution.

After making the system available again:

  1. Reconfigure the erasure coding topology to use chunk distribution.
  2. Enable the erasure coding service.
Full-copy distribution with unavailable systems

For erasure-coded protection with full-copy distribution, when an object is ingested, each available system in the erasure coding topology, except systems where the object should be metadata-only storage tier, receives a full-copy of the object data. When an unavailable system becomes available:

  • If the erasure coding delay has not expired and the object is not supposed to be metadata-only, the system receives a full copy of the object data. If the object should be metadata-only, the system does not receive either a full copy or a chunk.

    After all the systems in the topology are available, when the erasure coding delay expires, each system where the object should be metadata-only receives a chunk for the object, and all full copies of the object data are reduced to chunks.

  • If the erasure coding delay has expired, the system receives a chunk for the object regardless of whether the object should be metadata-only. After all the systems in the topology are available, full copies of the object data are reduced to chunks.

Geo-Distributed Erasure Coding service processing

The Geo-Distributed Erasure Coding service is responsible for ensuring that objects that are or have ever been subject to erasure coding are in the correct state at any given time.

On any given HCP system, the Geo-Distributed Erasure Coding service runs according to a system-specific schedule. Therefore, the service is not necessarily running at the same time on any set of systems. However, the service on one system can query other HCP systems for the state of the objects on those systems even when the service is not running on those systems.

While running, the Geo-Distributed Erasure Coding service examines each object individually. If the system in which the service is running has a full copy of the data for an object and should not, the service reduces the data to a chunk. If the system has a chunk for the object and should not, the service restores the chunk to a full copy. If the object is in the state it should be in, the service takes no action on the object.

To have sufficient information to restore a chunk for an object to a full copy of the object data, the Geo-Distributed Erasure Coding service can ask other systems to send the chunks they have for that object. Alternatively, if any of those systems has a full copy of the object data, the service can ask for the full copy.

During a scheduled run time, the Geo-Distributed Erasure Coding service actually runs only if the system is part of an active erasure coding topology or if the system contains erasure-coded objects. An active erasure coding topology is one that is currently being used for geo-protection. A system can participate in only one active erasure coding topology.

If the Geo-Distributed Erasure Coding service doesn't have sufficient run time on a system that's receiving full copies of object data, those full copies may not be reduced to chunks in a timely manner. When scheduling the service, keep in mind that full copies of object data use the full storage required by the object data until they are reduced to chunks.

The Geo-Distributed Erasure Coding service cannot restore a chunk for an object to a full copy of the object data if the system where the service is running doesn't have enough space for the complete object data. In this case, the service leaves the chunk on the system as is.

NoteThe Geo-Distributed Erasure Coding service is automatically added to the HCP Default Schedule service schedule when HCP is upgraded to release 8.x from a release earlier than 8.0. You need to schedule the service yourself in any user-created service schedules in which you want the service to have run time.
Determining what the state of an object should be

Several factors determine whether a system should have a chunk or a full copy of the data for a given object:

  1. First, the Geo-Distributed Erasure Coding service on the system must determine whether the namespace containing the object has erasure coding allowed.
  2. Then, if erasure coding is allowed, the service must determine whether the tenant that owns the namespace is currently included in an active erasure coding topology.
  3. Finally, if the tenant is included in an active erasure coding topology, the service must determine:
    • What the state of the object should be with respect to these properties of the erasure coding topology: distribution method, erasure coding delay, restore period, and minimum size for erasure coding.
    • What the state of the object is on the other systems in the erasure coding topology.
Keeping objects sufficiently protected

An object that is subject to erasure coding:

  • Is sufficiently protected as long as all the systems in the erasure coding topology are available or as long as at least two systems in the topology have a full copy of the object data
  • Is insufficiently protected if the object would become unavailable due to a system in the erasure coding topology becoming unavailable

The Geo-Distributed Erasure Coding service never reduces a full copy of the data for an object to a chunk if doing so would cause the object to be insufficiently protected.

Namespace configuration changes

Erasure coding can be disallowed for a namespace that previously had erasure coding allowed. Similarly, replication can be disabled for a namespace that was previously selected for replication.

If erasure coding is disallowed or replication is disabled for a namespace that contains erasure-coded objects, the Geo-Distributed Erasure Coding service on each system that contains those objects is responsible for:

  • Restoring the chunks for the objects to full copies of the object data on HCP systems where the object is not supposed to be metadata-only. The service can restore the object data as long as no more than one system in the erasure coding topology is unavailable or at least one system in the topology has a full copy of the object data.
  • Deleting chunks for the objects on HCP systems where the object should be metadata-only.

Before disallowing erasure coding or disabling replication for a namespace that contains erasure-coded objects, you need to ensure that each system where the chunks for the objects will be restored to full copies of the object data has sufficient storage to accommodate that data.

Replication Verification service processing

The HCP Replication Verification service is responsible for checking whether the Replication service missed replicating or was unable to replicate any objects that should have been replicated. If the Replication Verification service encounters such an object, the service tries to replicate the object.

If either the Replication service or the Replication Verification service encounters an object it cannot replicate, the service marks the object as nonreplicating. If the Replication Verification service replicates a nonreplicating object, the service removes the nonreplicating mark from the object.

The Replication Verification service is not a scheduled service. The service runs only when you:

  • Set the service to run continuously while the Replication service is running.
  • Set the service to run once. In this case, if the Replication service is running, the Replication Verification service starts immediately. Otherwise, the Replication Verification service waits to start until the next time the Replication service runs.

You can also disable the service so that it doesn't run at all.

While running, the Replication Verification service may degrade HCP system performance. Normally, you run the service only when directed to do so by your authorized HCP service provider.

The Overview panel for a replication link indicates whether any tenants on the link have namespaces with nonreplicating objects.

Tenant administrators can view the list of nonreplicating objects in a namespace in the Tenant Management Console.

Unavailable and irreparable erasure-coded objects

The status of an erasure-coded object can differ among the systems in an erasure coding topology:

  • An erasure-coded object is unavailable on a system if all of these are true:
    • One or more systems in the erasure coding topology are inaccessible.The object data cannot be reconstructed from the available chunks.
    • No accessible system has a good full copy of the object data.
  • An erasure-coded object is irreparable on a system if all of these are true:
    • All systems in the erasure coding topology are accessible.
    • The object data cannot be reconstructed from the available chunks.
    • No system in the topology has a good full copy of the object data.

 

  • Was this article helpful?