Skip to main content

We've Moved!

Product Documentation has moved to docs.hitachivantara.com
Hitachi Vantara Knowledge

About erasure coding topologies

An erasure coding topology consists of a set of HCP systems and active/active replication links that connect those systems. Within an erasure coding topology, each system must have at least two paths to each other system. Each path can consist of any number of systems and links, but the paths cannot have any systems or links in common.

When you create an erasure coding topology, you select the systems to include in it. The systems you select must be part of an existing active/active replication ring topology or fully connected active/active replication topology.

For each pair of selected systems that are connected to each other in the underlying replication topology, if the pair has only one active/active replication link between them, that link is automatically included in the erasure coding topology. If any pair of the selected systems is connected by more than one active/active link, you select which one of those links to include in the erasure coding topology.

An erasure coding topology does not automatically include the tenants on the links in the underlying replication topology. Instead, you select the tenants to include in an erasure coding topology from all the replication-enabled tenants defined on the systems in the topology. Each tenant you select is automatically added to each link in the erasure coding topology.

When you create an erasure coding topology, it's in the active state. If you want to stop using the active topology for erasure-coded protection, you need to retire the topology.

Retiring an erasure coding topology causes the Geo-distributed Erasure Coding service to restore object chunks to full-copies or, on systems where the objects are on metadata-only storage tiers, delete the chunks. While the chunks are being restored or deleted, the topology is in the retiring state.

When all the systems in a retiring erasure coding topology have no more chunks that are the result of that topology, the state of the topology changes to retired. Once a topology is retired, you can delete it. Deleting an erasure coding topology does not cause any systems, links, tenants, namespaces, or objects to be deleted.

A system can be in at most five erasure coding topologies concurrently. Only one of those topologies can be active. The rest must be either retiring or retired.

Erasure coding topology properties

An erasure coding topology has these properties:

  • A name and, optionally, a description.
  • The number of HCP systems in the erasure coding topology. A topology can have from three through six systems.
  • For an erasure coding topology with four or more systems, the type of the underlying replication topology: ring or fully connected. The topology type for three systems is always fully connected.
  • A distribution method — chunk or full copy.
  • A minimum size for erasure coding objects and object parts — 4 KB, 16 KB, 32 KB, 64 KB, 128 KB, 256 KB, 512 KB, or 1 MB. HCP erasure-codes only objects and object parts with a size greater than or equal to the minimum size you select. HCP uses whole-object protection for objects and parts below the minimum size, regardless of whether the containing namespace is using erasure-coded protection.
    NoteThe service plan associated with a namespace can include a metadata-only storage tier on all the systems in an erasure coding topology. With this configuration, objects and parts in the namespace that are smaller than the minimum size for erasure coding may have data on only one of those systems. As a result, those objects and parts are not protected against a single system failure.
  • An erasure coding delay. This is the amount of time the Geo-distributed Erasure Coding service waits before reducing a full copy of the data for an object or object part to a chunk. With full-copy distribution, this is also the amount of time the service waits on systems where the object or part is metadata-only before requesting a chunk for the object.
  • Optionally, a restore period. If an erasure coding topology has a restore period, after an object or object part that's subject to erasure coding is read from a system in the topology, a full copy of the object or part data is kept on that system at least until the restore period expires.

When setting the minimum size, erasure coding delay, and restore period, consider the tradeoffs with respect to storage efficiency and read performance.

Erasure coding delay

The erasure coding delay is counted from the time the ingest system receives the object or part data. You specify the delay as a number of days.

Even if the erasure coding delay has expired for an object, the Geo-distributed Erasure Coding service does not reduce a full copy of the data for the object or part to a chunk if doing so would leave the object or part insufficiently protected.

Restore period

You specify the restore period for an erasure coding topology as a number of days. If the specified number is zero, the topology does not have a restore period.

When an object or part that's subject to erasure coding is read from a given system, where that system is in an erasure coding topology with a restore period:

  • If the system currently has a full copy of the object or part data, the restore period applies to that full copy.
  • If the system currently has a chunk for the object or part on primary running storage, on an S Series Node, or on spindown storage and the chunk has never been tiered to extended storage, a full copy of the object or part data is restored on the current tier, and the chunk is deleted. The restore period applies to the restored full copy.
  • While the system has only a full copy of the object or part data, the object or part is not counted as an erasure-coded object on that system.
  • If the system currently has a chunk for the object or part and the chunk is or ever was on an extended storage tier, a full copy of the object or part data is restored on the highest storage tier that could have had a full copy of the data, and the chunk is deleted. The restore period applies to the restored full copy.
    NoteFor objects and parts that are subject to erasure coding, the storage tiers that can have a full copy of the object or part data are primary running storage, HCP S Series Node, and primary spindown storage. HCP never tiers a full copy of the object or part data to extended storage. Instead, when the Geo-distributed Erasure Coding service on a system reduces a full copy of object or part data to a chunk, if the object or part is due to be tiered, that service puts the chunk on the applicable storage tier.

The restore period for an object or part starts when the object or part is read. If the object or part is read again during the restore period, the restore period restarts from the time of that read operation.

The restore period for an object or part can restart only once on any given day. After the first restart on a day, subsequent reads on the same day do not cause the restore period to restart.

The restore period for an object or part can start before the erasure coding delay has expired. If the erasure coding delay then expires before the restore period expires, the full copy of the object or part data is not reduced to a chunk until the restore period expires. If the restore period expires before the erasure coding delay expires, the full copy of the object or part data is not reduced to a chunk until the erasure coding delay expires.

NoteInternal reads (for example, by an HCP service) do not cause object or part chunks to be restored to full copies of the object or part data and do not start or restart restore periods.

Supported replication topologies

An erasure coding topology can include from three through six HCP systems. An erasure coding topology with four, five, or six systems can be based on either an active/active replication ring topology or a fully connected active/active replication topology. A three-system erasure coding topology is always based on a fully connected active/active replication topology.

Before you can create an erasure coding topology, the underlying replication topology must already exist.

The figures below show all the replication topologies on which an erasure coding topology can be based.

GUID-D2684F0D-2263-4F57-BF25-27207AF94C50-low.png GUID-C6C1F186-FC5C-4F36-8F73-F7BDB581AA6B-low.png GUID-77E37803-8B8D-418F-8B05-BD6C484BFB57-low.png GUID-15F32C0A-FB34-44BF-9697-B76494A67650-low.png

Erasure coding topology retirement

When you retire an erasure coding topology, the protection type for any namespaces that were using erasure-coded protection changes to whole-object protection. This change causes the Geo-distributed Erasure Coding service to, for any given object:

  • Restore the chunk for the object to a full copy of the object data on HCP systems where the object is not supposed to be metadata-only. As the service restores chunks to full copies, the use of storage on those systems increases.
  • Delete the chunk for the object on HCP systems where the object should be metadata-only. The service does not delete the chunk for an object until a full copy of the object data exists either on at least two systems in the topology or on one system if the object should have data on only one system.

    For any given erasure-coded object, if the object should be metadata-only on every system in the topology, the Geo-distributed Erasure Coding service on one system restores the chunk to a full copy. The system where this occurs varies depending on when the service processes the object on each system in the topology.

The number of concurrent system failures HCP can tolerate for a replicated namespace with whole-object protection is one less than the number of systems where the namespace contains object data. Therefore, to maintain at least the same protection level with whole-object protection as the protection level maintained by erasure-coded protection, the underlying replication topology must include at least two systems where the replicated namespaces contain object data.

Before retiring an erasure coding topology, you need to ensure that any system where object chunks will be restored to full copies of the object data has sufficient storage to accommodate that data.

TipTo estimate the amount of storage required on a given system:
  1. For each tenant included in the retiring erasure coding topology, use the management API to get the value of the erasureCodedSize property of the /tenants/tenant-name/statistics resource. This value is the total amount of storage, in bytes, occupied by chunks for erasure-coded objects in all the tenant's namespaces.
  2. Add together the numbers of bytes occupied by chunks for erasure coded objects for all the tenants. The total is the number of bytes occupied by chunks for all erasure-coded objects on the system.
  3. Multiply the total number of bytes by the number of data chunks in the topology protection level. The result is an estimate of the required amount of storage, in bytes.

Also before retiring an erasure coding topology, if you don't want chunks for objects in a namespace restored to full copies of the object data on a given system, ensure that, on that system, the namespace is already associated with a service plan that keeps all objects in the namespace on a metadata-only storage tier. That way, chunks are not unnecessarily restored to full copies on systems where the objects should be metadata-only.

As the Geo-distributed Erasure Coding service on a system processes the chunks for a retiring erasure coding topology, the number of erasure-coded objects for that topology on that system decreases. When that number reaches zero on all the systems in the topology, the state of the topology changes to retired.

For any given system in an erasure coding topology, you can use the erasure coding topology details page in the HCP System Management Console to monitor the count of erasure-coded objects for that topology on that system.

You can also view counts of erasure-coded objects for all active and retiring erasure coding topologies:

  • The storage Overview page shows the total number of erasure-coded objects currently stored on the system.
  • The tenant Overview panel shows the total number of erasure-coded objects in all the tenant's namespaces.

These counts are not specific to an erasure coding topology. If an active erasure coding topology exists, these numbers do not go down to zero when an erasure coding topology has finished retiring.

Chargeback reports also contain erasure-coded object counts for the system, tenants, and, if you have administrative access to a tenant, the namespaces owned by that tenant.

Erasure coding topology replacement

You cannot add or remove HCP systems in an erasure coding topology. You also cannot add, remove, or replace replication links in an erasure coding topology. If you want to change a topology in any of these ways, you need to retire the topology and create a new one to replace it.

After you retire an erasure coding topology, you cannot make it active again. If you inadvertently retire a topology, you can replace it with a new topology that includes the same systems and links.

To replace a system in an active erasure coding topology, you don't need to create a new topology. Instead, you use the procedure for replacing a failed system. At the end of this procedure, replication restarts with the replacement system, and the replacement system receives full copies of object data or chunks for objects, as determined by the erasure coding topology.

NoteTo replace a system in an active erasure coding topology with a system that has a different domain name or different IP addresses, you need to reconfigure the replication links that connect other systems in the topology to the system you're replacing. When reconfiguring the links, ensure that they all connect to the same replacement system. If the links connect to different systems, the erasure coding topology will no longer be functional.

Erasure coding topology replacement process

When replacing an erasure coding topology, you don't need to wait for the old topology to finish retiring before you create the new topology. During the replacement process, the Geo-distributed Erasure Coding service on each system in the retiring topology determines what to do with each object affected by that topology based on:

  • The current state of the object on the system (full copy or chunk).
  • The state the object should be in with the new topology. (Until you create the new topology, the object is subject to whole-object protection.)
  • The requirement that the object always be protected against a single system failure.

During the topology replacement process, the Geo-distributed Erasure Coding service on any given system in the retiring topology can take any of these actions on an object:

  • If the system is part of the new topology:
    • Reduce a full copy of the object data to the chunk for the new topology
    • Replace an existing chunk for the object with the chunk for the new topology
    • Restore an existing chunk for the object to a full copy of the object data and, some time later, reduce the full copy to the chunk for the new topology
    • Restore an existing chunk for the object to a full copy of the object data and, if the object is smaller than the minimum size for erasure coding with the new topology, leave the full copy as is
    • Make no change if the existing full copy of the object data or chunk for the object is the same as it should be with the new topology
  • If the system is not part of the new topology but should contain data for the object, restore an existing chunk for the object to a full copy of the object data
  • Regardless of whether the system is part of the new topology:
    • Delete the existing chunk or full copy of the object data if the system should not contain data for the object (that is, the object should be metadata-only).
    • Completely delete the object (that is, delete both the data and metadata for the object).

      During the topology replacement process, the Geo-distributed Erasure Coding service deletes an object only on detecting that the object has been deleted on another system in the retiring topology. The service can delete the object even if replication is paused on the link over which the service detected the deletion.

For systems that are in both the retiring and new topologies, you cannot predict on which systems the service will try to restore chunks to full copies of object data. Therefore, to facilitate the topology replacement, you should ensure that each system in the topology you're retiring has significantly more free space than the space required for the addition of a full copy of the data for the largest object that was erasure coded.

Systems in the retiring topology that are not also in the new topology must have enough free space to accommodate full copies of the object data for all erasure-coded objects that are not supposed to be metadata-only.

During the topology replacement process, the number of erasure-coded objects for the retiring topology only decreases on each system. However, the total number of erasure-coded objects on each system can increase or decrease unpredictably until the topology replacement is complete.

Considerations for replacing an erasure coding topology

You can replace an erasure coding topology with a topology that includes some but not all of the HCP systems in the old topology. When you add a tenant in the old topology to the new topology, replication of that tenant is automatically paused on each link where:

  • The link is not in the new topology
  • One or both of the systems involved in the link are in the new topology

Although replication of the tenant is paused on those links, the tenant, its replicated namespaces, and the objects in those namespaces still exist on systems in the old topology that are not in the new topology.

In this situation, you may want to sever the relationship between the tenant on systems in the new topology and the tenant on systems that are not in the new topology. Severing this relationship lets you:

  • Delete the tenant either on systems in the new topology or on systems that are not in the new topology but not on both
  • Delete one or more of the tenant's replicated namespaces either on systems in the new topology or on systems that are not in the new topology but not on both
  • Delete one or more objects in the tenant's replicated namespaces either on systems in the new topology or on systems that are not in the old topology but not on both

Before you sever the tenant relationship, do not let clients access the tenant or its replicated namespaces on systems where you plan to delete that tenant or those namespaces . Disallowing access prevents situations such as this, where these events occur in the order shown:

  1. A client makes a change to an object on a system where the tenant or containing namespace will be deleted.
  2. You sever the tenant relationship before the change is replicated to a system where the tenant or namespace will not be deleted.

To sever the tenant relationship:

  1. Wait for each system in the old topology to have no objects that are erasure coded according to that topology.

    You can see the number of objects that are erasure coded according to a given topology on the details page for that topology in the HCP System Management Console.

  2. When the state of the old topology changes to retired, delete that topology.
    ImportantDo not resume replication of the tenant on the links where replication of the tenant is paused.
  3. Remove the tenant from each link between a system in the new topology and a system that's not in the new topology.

After severing a tenant relationship, you can add the tenant back to the links from which you removed that tenant only if the tenant is not in the erasure coding topology (either because you removed the tenant from the topology or because you deleted the tenant from the systems in the topology).

Erasure coding topology replacement procedure

When replacing an erasure coding topology, use this procedure to ensure that the replacement process occurs in the most efficient manner:

Procedure

  1. Disable the Geo-distributed Erasure Coding service on each system in the topology you're replacing.

    Doing this will prevent the service from restoring chunks for objects to full copies that will later need to be reduced to chunks.
    NoteServices can be disabled from the system Overview page in the System Management Console. To disable services, you need the service role.
  2. Disable the Storage Tiering service on each system in the topology you're replacing.

    Doing this will prevent the service from tiering full copies of object data that will later need to be reduced to chunks.
  3. If all of the following are true for any given namespace on any given system, ensure that the service plan associated with that namespace keeps all objects in the namespace on a metadata-only storage tier on that system:

    • The system will be included in the new erasure coding topology.
    • The namespace has erasure coding allowed, and the tenant that owns the namespace will be included in the new topology.
    • You want each object in the namespace to use whole-object protection until the erasure coding delay specified by the new topology expires.
    • You want objects in the namespace to be metadata-only on the system while they are subject to whole-object protection.

      With whole-object protection, HCP does not store data for objects on metadata-only storage tiers. However, if an object should be on a metadata-only storage tier on all the systems in the topology, one system in the topology will store data for that object.

    Keeping objects on a metadata-only storage tier on one or more systems will prevent full copies of object data from being distributed to those systems:

    • Between the retirement of the existing topology and the creation of the new topology
    • Before the expiration of the erasure coding delay for the new topology
    NoteTo keep objects protected during the replacement procedure, ensure that the service plan associated with each applicable namespace does not include a metadata-only storage tier on at least two of the systems in the topology you're replacing. To distribute the storage load, on any given system, configure the applicable service plans such that some include a metadata-only storage tier and some do not.
  4. Retire the existing erasure coding topology.

    Newly ingested objects are now replicated using whole-object protection. Objects that are already erasure coded remain erasure coded.
  5. Create any new replication links you need for the new erasure coding topology.

    Do not add tenants to the new links at this point. The tenants will be added to the links automatically when you add the tenants to the new topology.
  6. Create the new erasure coding topology and then add tenants to it.

    The tenants you add are automatically added to any replication links in the topology that do not already include them.The tenants you add to the new erasure coding topology may already be included on links not in the topology, where one of the systems involved in the link is in the topology. When this is the case, replication or recovery of those tenants is automatically paused on those links.
  7. Reenable the Geo-distributed Erasure Coding service.

    The service starts processing objects again.
  8. Reenable the Storage Tiering service.

    The service starts tiering object chunks that are the result of the new erasure coding topology.
  9. When the state of the retiring erasure coding topology changes to retired, delete that topology.

 

  • Was this article helpful?