UR disaster recovery operations
You can perform disaster recovery operations using Command Control Interface, including general recovery planning and procedures, recovery for 3DC configurations, and recovery for UR/ShadowImage shared configurations.
Overview of disaster recovery operations
The primary reason for using Universal Replicator and other remote replication software is disaster recovery. With copies of data at a secondary location, you can transfer operations to the secondary site, restore the lost or damaged information at the primary site, and then restore operations to the original configuration when recovery is complete.
Maintaining copies in multiple secondary locations increases the level of data security, but some recovery procedures can be complex.
Workflow for disaster recovery
Workflow for disaster recovery operations
- Prepare your data centers for disaster recovery, including identifying the data to be backed up, establishing the copy pairs, and installing and configuring the host failover software.
- Establish file and database recovery procedures before disaster or failure occurs, as part of the UR planning process.
- When a disaster or failure occurs at the primary site, switch host operations to the secondary site.
- While host operations continue at the secondary site, recover the primary
site, and then reverse the copy direction of the volume pairs to copy the
current host data from the secondary site back to the primary site.
See Reversing copy direction from secondary to primary site .
- When the copy pairs are again synchronized, resume host operations at the primary site, and re-establish the original copy direction of the volume pairs.
Preparing for disaster recovery
Perform the following tasks to prepare for disaster recovery:
- Identify the data volumes that you want to back up for disaster recovery.
- Pair the important volumes using Universal Replicator.
- Install and configure the required software for host failover between the primary and secondary sites.
Preparing for file and database recovery
File recovery procedures are necessary when the primary or secondary storage system suspends a pair due to a disaster or failure. When this occurs, the S-VOL might contain in-process data resulting from an open data set or transactions that could not complete. File recovery procedures in these circumstances are the same as when a data volume becomes inaccessible due to control unit failure.
You detect and re-create lost updates by checking database log files and other current information at the primary site.
Design your recovery plan so that detection and retrieval of lost updates is performed after the application is started at the secondary site. The detection and retrieval process can take some time.
Prepare for file and database recovery using files for file recovery (for example, database log files that have been verified as current).
Remote copy and disaster recovery procedures are inherently complex. Consult your Hitachi Vantara account team about recovery procedures.
Switching host operations to the secondary site
Use these instructions for pairs in one or more primary or secondary storage systems.
The first recovery task is to switch host operations to the secondary site using the CCI horctakeover command.
- The horctakeover command checks the pair status of S-VOLs and splits journals. Splitting the pairs in the journals ensures consistency and usability in the S-VOLs.
- The horctakeover command attempts to
resynchronize pairs to reverse P-VOLs and S-VOLs.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
- If the horctakeover command runs successfully, host operations are taken over by the secondary site using the S-VOLs. For detailed information about CCI and horctakeover, see Command Control Interface User and Reference Guide.
Reversing copy direction from secondary to primary site
When host operations have been switched to the secondary site, restore the primary site and re-establish the UR system from the secondary to the primary site.
Procedure
Restore the primary storage system and data paths.
Bring up the host servers at the primary site.
Make sure that all UR components are operational.
Do all of the following, as applicable:
At the primary site, locate P-VOLs whose status is COPY or PAIR. Locate corresponding S-VOLs at the secondary site whose status is SSWS, which indicates that data is being written to the S-VOLs. At the primary site, split these pairs.
At the primary site, locate P-VOLs whose status is other than unpaired. Locate corresponding S-VOLs at the secondary site whose status is unpaired. At the primary site, release the pairs.
At the primary site, locate pairs whose status is unpaired. At the secondary site, release the pairs.
On pairs that were split and in which S-VOL status is now SSWS, run the pairresync -swaps command. This reverses P-VOLs and S-VOLs and resynchronizes the pairs. The replication is from secondary site to original primary site.
NoteWhen you run the pairresync -swaps command, you can use the -d option to specify a data volume. However, the command is rejected if the restore journal where the data volume belongs is in Active, Halting, or Stopping status.For S-VOLs whose status is unpaired, re-create the UR pairs specifying the S-VOLs as P-VOLs. This creates pairs in which P-VOLs and S-VOLs are reversed.
Verify that pair status of all new S-VOLs (which were originally P-VOLs) changes from COPY to PAIR. When the pair status is changed to PAIR, initial copy operations are finished and consistency is maintained.
Results
Resuming host operations at the primary site
When UR pairs are established in the reverse direction between the secondary and primary sites, you can return host operations to the original configuration. This means resuming host operations at the primary site and reestablishing the original flow of data from primary to secondary storage systems. The following procedure explains how to resume normal operations at the primary site.
Procedure
At both sites, make sure that UR components are operational.
Make sure that pair status of all P-VOLs and S-VOLs in all UR pairs is PAIR.
Stop host applications at the secondary site.
Issue a request for splitting pairs to master journals (these were originally the restore journals on the secondary site): use the Flush option when splitting pairs. If an error occurs when splitting pairs, fix the error, resume host operations at the secondary site, and then go back to step 1.
If no errors occur, wait until suspension finishes. After suspension finishes, check for an S-VOL on the primary site whose status is not PSUS. If such a pair exists, fix the error and go back to step 1 after resuming your business task at the secondary site.
When all S-VOLs at the primary site are in PSUS status, data in P-VOLs and S-VOLs are the same. S-VOLs at the primary site are ready for host read/write activity. Resume applications at the primary site.
Run the pairresync -swaps command, which reverses P-VOLs and S-VOLs and resynchronizes the pairs. The replication is from the primary site to the secondary site (disaster recovery site).
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.NoteWhen you run the pairresync -swaps command, you can use the -d option to specify a data volume. However, the command is rejected if the restore journal where the data volume belongs is in Active, Halting or Stopping status.
Recovery for configurations with multiple primary systems and secondary systems
This topic provides recovery procedures for configurations with multiple primary systems and secondary systems.
Switching from the primary to secondary site (multiple storage systems)
If a disaster or a failure occurs in the primary site in a configuration with multiple primary and secondary systems, the operation continues at the secondary site, and data consistency within CCI CTG is maintained.
If a disaster or a failure occurs at the primary site, switch the operation from the primary site to the secondary site first.
Procedure
Switch the operation from the primary site to the secondary site.
From the secondary site, specify the swap option to split the pair (
pairsplit –RS).Check if the UR S-VOLs of all storage systems at the secondary site are in the SSWS status.
Resume the operation at the secondary site using the data volume in the SSWS status of storage systems at the secondary site.
Transferring operations back to the primary site (multiple storage systems)
After operations at the secondary site have resumed, if the other failures are corrected, operations at the primary site can be resumed.
Make sure that the status of the UR secondary volumes of all secondary systems are SSWS.
In the secondary site, specify the swap option to resynchronize the pairs (
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.pairresync -swaps).Switch operation from the secondary site to the primary site.
In the primary site, execute the
horctakeovercommand.
Recovery for 3DC configurations with three UR sites
This topic provides recovery procedures for 3DC configurations with three UR sites.
Recovery for 3 UR DC multi-target configuration (when the delta operation resync is performed)
Before performing the following procedures, refer to Workflow for 3DC multi-target configuration with three UR sites to confirm that the configuration is correct. If the correct configuration is not established, delete the delta resync UR pair, and perform the recovery operations according to Switching from the primary to secondary site (multiple storage systems).
Moving business operations to the secondary site
Procedure
Run the pairsplit -RS command on the pair at the alternative secondary site.
Check the execution result of the pairsplit -RS command.
Perform the delta resync operation by running the pairresync -swaps command on the delta resync pair.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.Check that the P-VOL of the pair for delta resync at the secondary site changes to the P-VOL of the pair at the primary site and that the status of the P-VOL for delta resync at the secondary site changes to PAIR.
You can use the P-VOL at the alternative secondary site to resume business operations after confirming these changes.
Moving business operations back to the primary site
You can return business operations from the secondary site back to the primary site after removing failures at the primary site and other sites.
Stop business operations at the secondary site.
If the secondary site to be reversed and the primary site pair are suspended, resynchronize the pair (pairresync -swaps), and then reverse the pair's primary and secondary volumes.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.Run the pairsplit -RS command for the pair between the secondary site to be reversed and the primary site.
Run the pairresync -swaps in the primary site, which reverses the primary and the secondary volumes to restore and resynchronize the pair.
Check the execution result of pairresync -swaps command.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.Resume business operations at the primary site.
Requirements for performing delta resync in 3DC multi-target configuration with three UR sites
To perform the delta resync operation successfully, the following requirements must be met. If a regular UR pair that does not meet the requirements exists in the journal, even if the UR delta resync pair meets all the requirements, the delta resync pair will be suspended by error.
- There is a UR pair with a mirror ID that is different from the UR delta resync pair in the same journal in the secondary site specified for the delta resync operation (the pairs with mirror ID P and M in the figure in 3DC multi-target configuration with three UR sites using delta resync).
- There is a UR pair with mirror ID that is different from the UR delta resync pair in the same journal in the UR delta resync pair's secondary site (the pairs with mirror ID P and N in the figure in 3DC multi-target configuration with three UR sites using delta resync).
- One of the two pairs is a UR delta resync pair (with the mirror ID P) in HOLD status, and the UR pair between the intermediate and secondary sites (with mirror ID N) is either PAIR, PSUS, or PSUE.
- All of the differential data of the P-VOL and S-VOL is stored in the master journal volume.
- The path for the UR delta resync pair between the primary and secondary sites is valid.
- In all three sites, remote command device is allocated to the two mirror IDs. You can confirm this status by using the raidcom get journal -key opt command.
- The capacity of both the volumes used in a UR delta resync pair (with the mirror ID P) of the two pairs is the same.
If even one of the above requirements is not met, an error or suspend by error occurs, and the delta resync operation will fail.
If the delta resync fails, check the following status. Especially in the following case, delta resync will not be performed because the journal data necessary for the delta resync does not exist.
- After the UR pair between the primary site and the reversed intermediate site is split, the restore journal volume data of the UR delta resync pair in the unreversed secondary site exceeds 70% of the volume capacity as a result of updating the volume in the reversed secondary site.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
If the specified UR delta resync pair status is not changed after performing the delta resync operation, the conditions for the delta resync operation described in this section might not be met. In this case, check the status of the UR pair and UR delta resync pair again, and verify whether all of the conditions for the delta resync operation are met. Then execute the delta resync again.
- When the UR delta resync pair is changed to HLDE, you cannot change it back to HOLD by pair resync. Delete the UR delta resync pair, and then create the UR pair again. After creating the UR pair, assign a remote command device to the mirror ID.
- If a suspend by error occurred in the pair
specified the delta resync right after performing the delta resync
operation, check the pair status and perform the following operation:
- For UR P-VOL, resync the UR pair that is in suspend by error.
- For UR S-VOL, delete the UR pair that is in suspend by error, and then create the UR pair again. After creating the UR pair, assign a remote command device to the mirror ID.
- When SOM 1242 is set to ON, SIM=dcf6xx is issued if the remote command devices for delta resync operations are not set up.
Recovering from failures in a 3 UR DC cascade configuration
Disasters can occur at any point in a 3 UR DC cascade configuration that can affect business operations or production data backup. How you maintain business operations or data backup and recover from the failure, depends on where the disaster occurs.
Recovering when a failure occurs in the primary site
Procedure
Transfer host operations to the intermediate site.
Run the horctakeover command on the primary-intermediate site pair.
Check the execution result.
Start host operations to the volume when the S-VOL status changes to an SSWS status or changes to a P-VOL status.
Remove the failure or failures from the primary site.
Transfer business operations back to the primary site (when the delta resync operation is performed).
Stop business tasks at the intermediate site.
If the primary-intermediate site pair is suspended, run the horctakeover command on the primary-intermediate site pair.
The intermediate-secondary site pair is automatically suspended.Resume business operations using the primary volume in the primary site.
Resynchronize the intermediate-secondary site pair.
Recovering when a failure occurs in the intermediate site
If a disaster occurs in the intermediate site, you can redirect production data backup to the secondary site. Then, you can recover from the failure and re-create the cascade configuration.
Procedure
Perform the delta resync operation between the primary site P-VOL and the secondary site S-VOL to redirect the production data backup.
Remove the failure or failures from the intermediate site.
Restore the intermediate site and cascade configuration (when the delta resync operation is performed).
If the primary-intermediate site pair is suspended, run the horctakeover command to resynchronize the pair.
After the resynchronization, the primary volume and secondary volume of the pair are reversed. The intermediate-secondary site pair is suspended automatically.Delete the primary-secondary site delta resync pair.
As a result, the intermediate-secondary site pair is also deleted.Create a pair between the intermediate and secondary sites, specifying Entire for Initial Copy.
Create a delta resync pair between the primary and secondary sites.
Recovering when a failure occurs in the primary-intermediate path
If the data path between the primary and intermediate sites fails, the status of journals in these sites might change to Stopping (with pair status changed to Suspending). You can recover by either resynchronizing the primary and intermediate sites or by performing a delta resync between primary-secondary sites.
Resynchronizing journals between the primary and secondary sites
You can recover from a primary-intermediate path failure by resynchronizing journals between the primary and secondary sites.
Procedure
Run the delta resync operation on the delta resync pair set up between the primary and secondary sites.
This changes the journal status between the primary and intermediate sites.
Resynchronizing journals in the primary and intermediate sites
You can recover from a primary-intermediate path failure by resynchronizing journals in the primary and intermediate sites.
Procedure
Delete the delta resync pair between the primary and secondary sites.
Confirm that the status of the journal between the primary and intermediate sites is changed to Stop.
Correct the reason for the data path failure.
Resync the pair between the primary and intermediate sites.
Recovering when a failure occurs in the secondary site
If the secondary site fails, business operations are not affected because the primary and intermediate sites are still operating; therefore, recovering from a secondary site failure simply re-creates the cascade pairs.
Procedure
Remove the failure or failures from the secondary site.
Set up the cascade pairs again.
Recovery for 3DC cascade configuration with three UR sites
Recovering from a primary site failure
When a failure or disaster occurs in the primary site, business tasks are transferred from the primary site to the intermediate site. After you correct the failure in the primary site, you can transfer the business tasks back to the primary site.
Procedure
Stop host operations at the intermediate site.
If the UR pair between the primary and intermediate sites are in suspend status, resync the pair by running the CCI horctakeover command.
After the resync, UR the P-VOL and S-VOL are switched. The UR pair between the intermediate and secondary sites are suspended automatically.Resume host operations by using the P-VOL in the primary site.
Resync the UR pair between the intermediate and secondary sites.
Recovering from an intermediate site failure for a 3DC cascade configuration with three UR sites
When a failure or disaster occurs in the intermediate site, data is duplicated by connecting the primary and secondary sites. After you correct the failure in the intermediate site, you can change back to the original cascade configuration.
Procedure
If the UR pair between the primary and intermediate sites is suspended, resync the pair.
Delete the pair between the primary and secondary sites.
The UR delta resync pair between the intermediate and secondary sites will be deleted automatically.Create a UR pair between the intermediate and secondary sites.
Specify Entire when performing the initial copy operation.To create the delta resync configuration, create a UR delta resync pair between the primary and the secondary sites.
Requirements for performing delta resync in 3DC cascade configuration with three UR sites
To perform delta resync operation successfully, the following requirements must be met. Note that the suspended by error will occur if a regular UR pair which does not meet the conditions exists in the journal, even if the UR delta resync pair meets all the requirements.
- There is a UR pair which has a different mirror ID than the UR delta resync pair in the same journal in the primary site (pairs with the mirror ID P and M shown in 3DC cascade configuration with three UR sites).
- There is a UR pair which has a mirror ID different than the UR delta resync pair in the same journal in the UR delta resync pair's secondary site (pairs with the mirror ID P and N shown in 3DC cascade configuration with three UR sites).
- Out of two pairs, one is UR delta resync pair (with the mirror ID P) and in HOLD status, and the other is the UR pair connecting the primary and intermediate sites and in PAIR or PSUE status.
- Out of two pairs, one is UR delta resync pair (with the mirror ID P) and in HOLD status, and the other is the UR pair connecting the intermediate and secondary sites (with the mirror ID P) and in PAIR, PSUS, or PSUE status.
- If you resynchronize the UR pair (with the mirror ID M) connecting the primary and intermediate sites, you must resynchronize the UR pair (with the mirror ID N) connecting the intermediate and secondary sites as well.
- P-VOL and S-VOL's entire differential data is stored in the master journal volume.
- The path between the primary and secondary sites of the UR delta resync pair is valid.
- Remote command devices are allocated to the two mirror IDs in all the three sites.
If even one of the above requirements is not met, an error occurs and the delta resync operation will fail. Especially in the following cases, delta resync will not be performed because the journal data necessary for the delta resync does not exist.
- After the UR pair between the intermediate site and secondary site is split, the journal volume data of the UR pair between the primary and intermediate sites in the secondary site exceeds 70% of the volume capacity.
- After updating the UR delta resync P-VOL, the journal volume data of the UR delta resync pair in the primary site exceeds 70% of the volume capacity.
Setting options can prevent the error from occurring in the delta resync operations with the cascade configuration with three UR sites, by copying the entire P-VOL data to the S-VOL when entire differential data of the P-VOL and S-VOL is not stored in the master journal. For setting the UR delta resync operation options for the UR delta resync pairs, see Changing options used by journals .
If the specified UR delta resync pair status is not changed after performing the delta resync operation, the conditions for the delta resync operation described in this section might not be met. In this case, check the status of the UR pair and UR delta resync pair again and verify whether all of the conditions for the delta resync operation are met. Then execute the delta resync again.
Recovery for 3DC UR/TC
Recovery procedures when UR P-VOLs and S-VOLs are shared with TrueCopy can be more complex than general procedures.
If your storage system is VSP 5100, VSP 5500, VSP G1x00, VSP F1500, VSP G900, VSP F900, VSP G800, VSP F800, or VSP, you can combine UR and TrueCopy for disaster recovery. The following topics provide recovery procedures for resuming host operations at a backup site and then restoring the original system and configurations.
- Recovery for 3DC UR/TC cascade configuration
- Recovery for 3DC UR/TC multi-target configuration
- Recovery in a delta resync configuration
For open systems, perform all procedures using CCI.
Recovery for 3DC UR/TC cascade configuration
Host operations are transferred to the TC secondary volume (S-VOL) at the intermediate site when a disaster or failure occurs in the primary site of a 3DC cascade configuration. The primary site failure is corrected and brought back online, and then either the cascade configuration is restored or the multi-target configuration is created.
See 3DC cascade configuration for information and illustrations on the configuration covered in this procedure.
Use the following procedure to recover a 3DC cascade configuration failure.
Procedure
Check consistency of the secondary volume at the intermediate site.
Run the horctakeover command on the secondary volume.
Wait until the secondary volume in the intermediate site is suspended (in SSWS status) or changes to a primary volume, and then use the volume to resume host operations.
Recovery for 3DC UR/TC multi-target configuration
The recovery procedure you perform in a 3DC multi-target configuration depends on the location of the failure:
- For failure in the primary site only, see Recovering from primary site failure.
- For failure in the primary and TrueCopy secondary sites, see Recovering from primary and secondary site failure (3DC multi-target)
See 3DC multi-target configuration for information and illustrations on the configuration covered in this procedure.
Recovering from primary site failure
Host operations are transferred to the secondary volume in the TC secondary site when disaster or failure occurs in the primary site in a 3DC multi-target configuration. The replication direction changes so that data flows from the TC secondary site to the primary site. Alternatively, a temporary UR system can be set up. Meanwhile, you correct the failure at the primary site and then transfer host operations back to it and return to a 3DC configuration.
Use the following procedure to recover a primary site failure.
Procedure
Release the UR pair.
Run the horctakeover command on the TC pair.
If a failure occurs after the one volume capacity of a TC pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the TC pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.Start host operations to the TC secondary site S-VOL.
Results
- The original copy flow of the TC pair reverses and flows from S-VOL to P-VOL.
- The original copy flow does not reverse because of failure in the primary site or data path.
Depending on the result, proceed as follows:
-
Re-create the UR pair from the TC primary site to the UR secondary site. This results in a 3DC cascade configuration, with the original TC S-VOL as the primary volume. See the lower right configuration in the following figure.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
-
Begin recovery of the original configuration by restoring the primary site or data path, or both. This must be done before proceeding.
-
Stop host operations to the TC secondary site.
-
Run the horctakeover command on the TC pair.
If a failure occurs after the one volume capacity of a TC pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the TC pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
-
Resume host operations to the TC P-VOL in the primary site.
-
To back up the data, create a new UR pair from the TC secondary site S-VOL to the UR secondary site S-VOL. See the upper-right configuration in the following figure.
If a failure occurs after the one volume capacity of a TC pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the TC pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
-
Begin recovery of the original configuration by restoring the primary site or data path, or both. This must be done before proceeding.
-
Release the UR pair.
-
Resynchronize the TC pair to start the replication to the primary site.
-
Release the current UR pair, which extends between the TC secondary site and the UR secondary site.
-
Stop host operations at the TC secondary site.
-
Run the horctakeover command on the TC pair.
If a failure occurs after the one volume capacity of a TC pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the TC pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
-
Resume host operations at the primary site TC volume.
-
Re-create the UR pair from the primary site to the secondary site. The system is now changed back to the original 3DC multi-target configuration.
Recovering from primary and secondary site failure (3DC multi-target)
Host operations are transferred to the Universal Replicator secondary site S-VOL when a disaster or failure occurs in both the primary and TC secondary sites in a 3DC multi-target configuration. Failures are corrected at the two sites, and then host operations are transferred back to the primary site.
Use the following procedure to recover a primary and TC secondary site failure.
Procedure
Run the horctakeover command on the UR pair.
Resume host operations at the UR secondary site.
Release the TC pair.
Make sure the UR pair is resynchronized so that the copy flow is from S-VOL to P-VOL.
Stop host operations at the UR secondary site.
Run the horctakeover command on the UR pair.
Resume host operations at the primary site.
Make sure the copy flow of the UR pair is from P-VOL to S-VOL. If it is not, resynchronize the pair.
Re-create the TC pair.
Results
Recovery in a delta resync configuration
Host operations are transferred to the TrueCopy secondary site S-VOL when a disaster or failure occurs in the primary site in a delta resync configuration. Then you run the delta resync operation. When the failure at the primary site is corrected, host operations are transferred back to the primary site and the delta resync configuration is reestablished.
Before you begin
- For information and illustrations on the configuration covered in this procedure, see Delta resync configuration .
- You can specify options for recovery in the event that the delta resync operation fails. See the Delta Resync Failure step in Changing options used by mirrors . This also provides important information if you share the delta resync S-VOL with ShadowImage, Thin Image, or a DP-VOL.
Procedure
Run the horctakeover command on the TC pair.
If a failure occurs after the one volume capacity of a TC pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the TC pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.Start host operations to the TC secondary site S-VOL.
Perform the delta resync operation on the TC secondary site S-VOL.
See Performing the delta resync operation. Note that pair status must be HOLD before the operation. When the operation completes, pair status is PAIR or COPY.NotePair status also changes for the original UR pair in the primary and UR secondary sites, from PAIR or COPY to HOLD.However, the status of the original pair may not change to the required state after the delta resync operation. If this occurs, host operations cannot be resumed in the primary site. See Problems with pair status during delta resync recovery and make the necessary corrections.
Resolve the problem at the primary site. This must be done before proceeding.
If the TC pair is suspended, resynchronize the pair and reverse the copy flow.
Stop host operations to the TC secondary site.
Run the horctakeover command on the TC pair.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.Resume host operations to the TC P-VOL in the primary site.
Perform the delta resync operation on the volume in the TC primary site. Pair status must be HOLD. When the operation completes, pair status is PAIR or COPY.
Also, pair status changes for the delta resync UR pair from the TC secondary to the UR secondary site, from PAIR, PSUS, or PSUE to HOLD.If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
Results
Problems with pair status during delta resync recovery
After performing the delta resync, it is necessary for the original UR pair from primary to UR secondary site to be in HOLD status, in order to resume operations at the primary site. However, the pair may not be in this status.
If pair status is not HOLD, match the pair’s actual statuses in the following table, and then perform the corrective action.
|
Current status |
Corrective action |
|
Primary site: PAIR or COPY UR secondary site: HOLD |
|
|
Primary site: PSUE or PSUS UR secondary site: HOLD |
|
|
Primary site: HLDE UR secondary site: HOLD |
Change the status of the HLDE pair back to HOLD using pairresync. |
|
Primary site: SMPL UR secondary site: HOLD |
|
Recovery for UR/SI configurations
See Configurations with SI secondary volumes for information and illustrations on the configuration covered in this procedure.
You resume host operations to the Universal Replicator secondary volume if a disaster or failure occurs at the primary site where the UR P-VOL is shared with a ShadowImage (SI) secondary volume. Then you recover the primary site and then resume host operations.
Use the following procedure to recover a UR P-VOL/SI S-VOL configuration.
Procedure
Release the SI pair using the pairsplit -S command.
Reverse the copy direction and resync the UR pair using the horctakeover command.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.Reverse the copy direction again on the UR pair using the horctakeover command.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.Delete the UR pair using the pairsplit -S command.
Create an SI pair from the SI S-VOL to perform copying in the reverse direction. Use the paircreate command.
If a failure occurs after the one volume capacity of a SI pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the SI pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.Release the SI pair using the pairsplit -S command.
Re-create the original SI pair from the original SI P-VOL to perform copying in the original direction. Use the paircreate command.
If a failure occurs after the one volume capacity of a SI pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the SI pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.Split the SI pair using the pairsplit command.
Re-create the UR pair using the paircreate command.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
Results
The system becomes the original configuration again.
Recovery from a failure at the primary site and TC secondary site
When a disaster or failure occurs both at the primary site and TC secondary site in the 3DC multi-target configuration, you can resume business by using the S-VOLs at the UR secondary site.
After transferring business tasks back to the UR secondary site, correct the failure at the primary site and TC secondary site so that business tasks can be transferred back to the primary site.
Switching from the primary to secondary site (TC secondary site)
Procedure
Run the CCI horctakeover command to the UR pairs.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.Resume business operations by using the volumes at the UR secondary site.
Transferring operations back to the primary site (TC secondary site)
After operations at the secondary site have resumed, if the other failures are corrected, operations at the primary site can be resumed.
Split the TC pairs by using the CCI pairsplit command.
If the pair resync to switch the UR P-VOLs and the S-VOLs is not performed (UR pairs are split by failure), resync the pairs by using the CCI pairresync command and switch the P-VOLs and the S-VOLs.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation. The original S-VOLs at the UR secondary site become the P-VOLs, and the original P-VOLs at the primary site become the S-VOLs.Stop the business process at the UR secondary site.
Run the CCI horctakeover command to the UR pairs.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation. The UR P-VOLs and the S-VOLs will be switched.Resume business operations at the primary site by using the P-VOLs at the primary site.
Resync the TC pairs by using the CCI pairresync command.
If a failure occurs after the one volume capacity of a UR pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the UR pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.The system configuration returns back to the 3DC multi-target.
Recovery for UR/GAD configurations
If a failure occurs in a UR/GAD configuration, the recovery procedure varies depending on the location of the failure. For details, see the Global-Active Device User Guide.