Disaster recovery in a GAD 3DC delta resync environment

Status before failure

The following figure shows the status in a GAD 3DC delta resync (GAD+UR) environment before a failure occurs. The data is copied from the GAD P-VOL to the UR S-VOL.

Disaster recovery procedures for GAD+UR failure conditions:

Recovering from primary site and P-VOL failures (GAD+UR)
Recovering from secondary site and S-VOL failures (GAD+UR)
Recovering from a failure at the primary and secondary sites (GAD+UR)
Recovering from a failure on the UR delta resync pair
Recovering from a quorum disk failure (GAD+UR)

Recovering from primary site and P-VOL failures (GAD+UR)

This section describes the procedures for recovering from a primary site failure or a P-VOL failure (LDEV blockade) at the primary site using examples.

GAD pairs might be created between the primary and secondary sites that configure GAD as follows:

A P-VOL of a GAD pair is at site 1 and a S-VOL is at site 2.
A P-VOL of a different GAD pair is at site 2 and a S-VOL is at site 1.

In this case, recover from a failure at the secondary site. For more information, see Recovering from secondary site and S-VOL failures (GAD+UR).

Recovering from a primary site failure (GAD+UR)

This subsection describes how to recover from a primary site failure. The example explains the case when a failure occurs in a storage system at the primary site.

When a failure occurs at a primary site, the status of the GAD pair changes to PSUE/SSWS. In this case, delta resync is automatically performed, and the pair statuses change as follows:

Before failure			After failure
Pair type	Pair status		Pair type	Pair status
Pair type	P-VOL	S-VOL	Pair type	P-VOL	S-VOL
GAD pair	PAIR	PAIR	GAD pair	PSUE¹	SSWS
UR pair	PAIR	PAIR	UR pair	PSUE^{1, 2}	-
UR pair	PAIR	PAIR	UR delta resync pair	-	SSUS
UR delta resync pair	PSUS	SSUS	UR pair	From COPY to PAIR	From COPY to PAIR
Notes: Pair status after the primary site is turned on. PSUE status of the UR pair. The UR pair does not change to a UR delta resync pair.

If a failure occurs, the UR delta resync pair changes to a UR pair, and copying from the GAD S-VOL to the UR S-VOL starts. When the UR pair copying is completed, the status of the P-VOL and the status of the S-VOL of the UR pair change to PAIR.

Overview of failure recovery

Procedure

Remove the failure on the P-VOL.
At the primary site for the UR delta resync pair, delete the UR delta resync pair.

Command example:
```
pairsplit -g oraREMOTE -S -IH0
```
At the primary site for the UR delta resync pair, create a UR delta resync pair to reconfigure a 3DC delta resync configuration.

Command example:
```
paircreate -g oraREMOTE -f async -vl -nocsus -jp 0 -js 0 -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
Reverse the P-VOL and the S-VOL, and then resynchronize the GAD pairs (swap resync) on the storage system at the GAD secondary site.

Command example:
```
pairresync -g oraHA -swaps -IH1
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

The volume on the primary storage system changes to an S-VOL, and the volume on the GAD secondary storage system changes to a P-VOL.

Confirm that the GAD P-VOL and S-VOL pair statuses change to PAIR.

Command example:

pairdisplay -g oraHA -fxce -IH0
Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
oraHA dev1(L) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIRNEVER , 100 4444 - - 0 - - - - L/M
oraHA dev1(R) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIRNEVER , 100 2222 - - 0 - - - - L/M

pairdisplay -g oraHA -fxce -IH1
Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
oraHA dev1(L) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIRNEVER , 100 2222 - - 0 - - - - L/M
oraHA dev1(R) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIRNEVER , 100 4444 - - 0 - - - - L/M

Keep updating I/O from the server to the P-VOL or S-VOL of the GAD pair for about two minutes.
Confirm that the delta UR P-VOL pair status is PSUS.

Command example:
```
pairdisplay -g oraREMOTE -fxce -IH0
Group PairVol(L/R) (Port#,TID, LU),Seq#, LDEV#.P/S,Status,Fence, %, P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
oraDELTA dev2(L) (CL1-A-1, 0, 1) 511111 2222. P-VOLPSUS ASYNC ,0 6666 - 0 0 - - - - -/-
oraDELTA dev2(R) (CL1-A-1, 0, 1) 544444 6666. S-VOLSSUS ASYNC ,0 2222 - 0 0 - - - -  -/-
```
NoteTo check the status of a pair in Device Manager - Storage Navigator, select Refresh All in the File menu to update the information displayed on Device Manager - Storage Navigator, and then view the pair status. The status of the UR delta resync pairs changes from HOLDING to HOLD.

Confirm that the mirror status of the journal of the UR delta resync pair is PJNS.

Command example:

pairdisplay -g oraREMOTE -v jnl -IH0
JID MU CTG JNLS AP U(%) Q-Marker Q-CNT D-SZ(BLK) Seq#Num LDEV#
000 1 1 PJNS 4 21 43216fde 30 512345 625001 39321

Confirm that no failure SIMs are displayed.
Reverse the GAD P-VOL and the S-VOL if necessary.
For instructions, see Reversing the GAD P-VOL and S-VOL (GAD+UR).If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering from a P-VOL failure (LDEV blockade) at the primary site (GAD+UR)

When a failure (LDEV blockade) occurs on a P-VOL at the primary site, the status of the GAD pair changes to PSUE/SSWS. In this case, delta resync is automatically performed, and the pair statuses change as follows:

Before failure			After failure
Pair type	Pair status		Pair type	Pair status
Pair type	P-VOL	S-VOL	Pair type	P-VOL	S-VOL
GAD pair	PAIR	PAIR	GAD pair	PSUE	SSWS
UR pair	PAIR	PAIR	UR delta resync pair	PSUE	SSUS
UR delta resync pair	PSUS	SSUS	UR pair	From COPY to PAIR	From COPY to PAIR

If a failure occurs, the UR delta resync pair changes to a UR pair, and copying from the GAD S-VOL to the UR S-VOL starts. When the UR pair copying is completed, the status of the P-VOL and the status of the S-VOL of the UR pair change to PAIR.

Overview of failure recovery

Procedure

Remove the failure (LDEV blockade) on the P-VOL.
If the P-VOL must be formatted for restoration, delete the GAD pair, and then format the P-VOL.
When deleting the GAD pair, first delete the S-VOL of which I/O mode is Local.
Depending on if you performed step 2, re-create the GAD pair, the delta resync UR pair, and the UR pair in that order, or reverse the P-VOL and the S-VOL in the GAD secondary storage system, and then resynchronize the GAD pair.
- If you performed step 2, re-create the GAD pair, the delta resync UR pair, and the UR pair in that order.
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
  For the GAD pair, first re-create the S-VOL.
  
  If necessary, after executing the command, reverse the P-VOL and S-VOL.
  
  Command example:
```
paircreate -g oraHA -fg never 2 -vl -jq 0 -IH1
paircreate -g oraDELTA -f async -vl -jp 0 -js 0 -nocsus -IH1
paircreate -g oraREMOTE -f async -vl -jp 0 -js 0 -IH0
```
- If you did not perform step 2, reverse the P-VOL and the S-VOL in the GAD secondary storage system, and then resynchronize the GAD pair (swap resync).
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
  Command example:
```
pairresync -g oraHA -swaps -IH1
```
  The primary storage system volume becomes the S-VOL, and the GAD secondary storage system volume becomes the P-VOL.

Confirm that the GAD P-VOL and S-VOL pair statuses change to PAIR.

Command example:

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M

Keep updating I/O from the server to the P-VOL or S-VOL of the GAD pair for about two minutes.

Confirm that the delta UR P-VOL pair status is PSUS.

Command example:

pairdisplay -g oraREMOTE -fxce -IH0
Group     PairVol(L/R)  (Port#,TID, LU),Seq#,  LDEV#.P/S,  
Status,Fence, %,  P-LDEV#  M  CTG  JID  AP  EM  E-Seq#  E-LDEV#  R/W
oraDELTA  dev2(L)       (CL1-A-1, 0, 1) 511111 2222. P-VOL 
PSUS   ASYNC ,0   6666     -  0    0    -   -   -       -       -/-
oraDELTA  dev2(R)       (CL1-A-1, 0, 1) 544444 6666. S-VOL 
SSUS   ASYNC ,0   2222     -  0    0    -   -   -       -       -/-

NoteTo check the status of a pair in Device Manager - Storage Navigator, select Refresh All in the File menu, update the information displayed on Device Manager - Storage Navigator, and then view the pair status. The status of the UR delta resync pairs changes from HOLDING to HOLD.

Confirm that the mirror status of the journal of the UR delta resync pair is PJNS.

Command example:

pairdisplay -g oraREMOTE -v jnl -IH0
JID  MU  CTG  JNLS  AP  U(%)  Q-Marker  Q-CNT  D-SZ(BLK)  Seq#   Num  LDEV#
000  1   1    PJNS  4   21    43216fde  30     512345     62500  1    39321

Confirm that no failure SIMs are displayed.
Reverse the GAD P-VOL and the S-VOL if necessary. For instructions, see Reversing the GAD P-VOL and S-VOL (GAD+UR).
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Reversing the GAD P-VOL and S-VOL (GAD+UR)

Use the following procedure to reverse the GAD P-VOL and S-VOL when sharing GAD volumes with UR in a GAD 3DC delta resync (GAD+UR) configuration.

Procedure

Suspend the GAD pair by specifying the S-VOL (swap suspend).
Command example: pairsplit -g oraHA -RS -IH0
Resynchronize the GAD pair by specifying the S-VOL (swap resync).
Command example: pairresync -g oraHA -swaps -IH0

If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
The volume on the primary storage system changes to a P-VOL, and the volume on the GAD secondary storage system changes to an S-VOL.
Keep updating I/O from the server to the P-VOL or S-VOL of the GAD pair for about two minutes.

Confirm that the delta UR P-VOL pair status is PSUS.

Command example:

pairdisplay -g oraDELTA -fxce -IH0
Group     PairVol(L/R)  (Port#,TID, LU),Seq#,  LDEV#.P/S,  
Status,Fence, %,  P-LDEV#  M  CTG  JID  AP  EM  E-Seq#  E-LDEV#  R/W
oraDELTA  dev2(L)       (CL1-A-1, 0, 1) 511111 2222. P-VOL 
PSUS   ASYNC ,0   6666     -  0    0    -   -   -       -       -/-
oraDELTA  dev2(R)       (CL1-A-1, 0, 1) 544444 6666. S-VOL 
SSUS   ASYNC ,0   2222     -  0    0    -   -   -       -       -/-

NoteTo check the status of a pair in Device Manager - Storage Navigator, select Refresh All in the File menu, update the information displayed on Device Manager - Storage Navigator, and then view the pair status. The status of the UR delta resync pairs changes from HOLDING to HOLD.

Confirm that the mirror status of the journal of the UR delta resync pair is PJNS.

Command example:

pairdisplay -g oraDELTA -v jnl -IH0
JID  MU  CTG  JNLS  AP  U(%)  Q-Marker  Q-CNT  D-SZ(BLK)  Seq#   Num  LDEV#
000  1   1    PJNS  4   21    43216fde  30     512345     62500  1    39321

Confirm that no failure SIMs are displayed.

Recovering from secondary site and S-VOL failures (GAD+UR)

This section describes the procedure for recovering from secondary site failures and failures (LDEV blockade) on an S-VOL at the secondary site using examples.

GAD pairs might be created between the primary and secondary sites that configure GAD as follows:

A P-VOL of a GAD pair is at site 1 and a S-VOL is at site 2.
A P-VOL of a different GAD pair is at site 2 and a S-VOL is at site 1.

In this case, recover from a failure at the primary site. For more information, see Recovering from primary site and P-VOL failures (GAD+UR).

Recovering from a secondary site failure (GAD+UR)

This subsection describes how to recover from a failure in a storage system at the secondary site. When a failure occurs at the secondary site, the GAD pair statuses at the primary site and the secondary site change to PSUE.

Overview of failure recovery

Procedure

Remove the failure on the S-VOL.
Resynchronize the GAD pair at the primary storage system.

Command example:
```
pairresync -g oraHA -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Confirm that the GAD P-VOL and S-VOL pair statuses change to PAIR.

Command example:

pairdisplay -g oraHA -fxce -IH0
Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
oraHA dev1(L) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIRNEVER , 100 4444 - - 0 - - - - L/M
oraHA dev1(R) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIRNEVER , 100 2222 - - 0 - - - - L/M

pairdisplay -g oraHA -fxce -IH1
Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
oraHA dev1(L) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIRNEVER , 100 2222 - - 0 - - - - L/M
oraHA dev1(R) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIRNEVER , 100 4444 - - 0 - - - - L/M

Keep updating I/O from the server to the P-VOL or S-VOL of the GAD pair for about two minutes.
Confirm that the pair status of the delta UR P-VOL is PSUS.

Command example:
```
pairdisplay -g oraDELTA -fxce -IH1
Group PairVol(L/R) (Port#,TID, LU),Seq#, LDEV#.P/S,Status,Fence, %, P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
oraDELTA dev3(L) (CL1-A-1, 0, 1) 522222 4444. P-VOLPSUS ASYNC ,0 6666 - 0 0 - - - - -/-
oraDELTA dev3(R) (CL1-A-1, 0, 1) 544444 6666. S-VOLSSUS ASYNC ,0 4444 - 0 0 - - - - -/-
```
NoteTo check the status of a pair in Device Manager - Storage Navigator, select Refresh All in the File menu, update the information displayed on Device Manager - Storage Navigator, and then view the pair status. The status of the UR delta resync pairs changes from HOLDING to HOLD.
Confirm that the mirror status of the journal of the UR delta resync pair is PJNS using the CCI instance for managing the GAD secondary storage system.

Command example:
```
pairdisplay -g oraDELTA -v jnl -IH1 
JID MU CTG JNLS AP U(%) Q-Marker Q-CNT D-SZ(BLK) Seq# Num LDEV# 
000 1 1 PJNS 4 21 43216fde 30 512345 62500 1 39321
```
Confirm that no failure SIMs are displayed.

Recovering from a failure (LDEV blockade) on an S-VOL at the secondary site (GAD+UR)

When a failure (LDEV blockade) occurs on an S-VOL at the secondary site, the GAD pair statuses of the P-VOL and S-VOL change to PSUE.

Overview of failure recovery

Procedure

Remove the failure (LDEV blockade) on the S-VOL.
If the S-VOL must be formatted for restoration, delete the GAD pair, and then format the S-VOL.
Depending on if you performed step 2, re-create the GAD pair, the delta resync UR pair, and the UR pair in that order, or resynchronize the GAD pair in the primary storage system.
- If you performed step 2, re-create the GAD pair, the delta resync UR pair, and the UR pair in that order.
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
  Command example:
```
paircreate -g oraHA -fg never 2 -vl -jq 0 -IH0
paircreate -g oraDELTA -f async -vl -jp 0 -js 0 -nocsus -IH1
paircreate -g oraREMOTE -f async -vl -jp 0 -js 0 -IH0
```
- If you did not perform step 2, resynchronize the GAD pair in the primary storage system.
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
  Command example:
```
pairresync -g oraHA -IH0
```

Confirm that the GAD P-VOL and S-VOL pair statuses change to PAIR.

Command example:

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M

Keep updating I/O from the server to the P-VOL or S-VOL of the GAD pair for about two minutes.

Confirm that the pair status of the delta UR P-VOL is PSUS.

Command example:

pairdisplay -g oraDELTA -fxce -IH1
Group     PairVol(L/R)  (Port#,TID, LU),Seq#,  LDEV#.P/S,  
Status,Fence, %,  P-LDEV#  M  CTG  JID  AP  EM  E-Seq#  E-LDEV#  R/W
oraDELTA  dev3(L)       (CL1-A-1, 0, 1) 522222 4444. P-VOL 
PSUS   ASYNC ,0   6666     -  0    0    -   -   -       -       -/-
oraDELTA  dev3(R)       (CL1-A-1, 0, 1) 544444 6666. S-VOL 
SSUS   ASYNC ,0   4444     -  0    0    -   -   -       -       -/-

NoteTo check the status of a pair in Device Manager - Storage Navigator, select Refresh All in the File menu, update the information displayed on Device Manager - Storage Navigator, and then view the pair status. The status of the UR delta resync pairs changes from HOLDING to HOLD.

Confirm that the mirror status of the journal of the UR delta resync pair is PJNS using the CCI instance for managing the GAD secondary storage system.

Command example:

pairdisplay -g oraDELTA -v jnl -IH1
JID  MU  CTG  JNLS  AP  U(%)  Q-Marker  Q-CNT  D-SZ(BLK)  Seq#   Num  LDEV#
000  1   1    PJNS  4   21    43216fde  30     512345     62500  1    39321

Confirm that no failure SIMs are displayed.

Recovering from a failure at the primary and secondary sites (GAD+UR)

Use the following procedure to recover from a failure at the primary and secondary sites when sharing GAD volumes with UR in a GAD 3DC delta resync (GAD+UR) configuration. You need to delete and re-create all pairs.

Procedure

Suspend the UR pair by specifying the S-VOL (swap suspend).

Command example: pairsplit -g oraREMOTE -RS -IH2

You can resume operations at this point by using the S-VOL at the UR secondary site.
Remove the failure at the primary and secondary sites.
Delete the UR pair.
Delete the UR delta resync pair.
Delete the GAD pair.

When the I/O mode of both the primary and secondary volumes is Block, forcibly delete the pair, as follows:
1. If you specify the S-VOL, delete the virtual LDEV ID at the same time.
```
pairsplit -g oraHA -RF -IH1
```
2. Confirm that the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff
(Omitted)
```
  Where VIR_LDEV : ffff indicates GAD reserve.
3. Delete the pair forcibly specifying the P-VOL.
  
  Do not delete the virtual LDEV ID.
```
pairsplit -g oraHA -SFV -IH0
```
4. Confirm that the virtual LDEV ID of the P-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
(Omitted)
```
  VIR_LDEV information is not displayed if it is same as the LDEV information.
To forcibly delete a pair when the I/O mode is not Block, call Hitachi Vantara LLC customer support.

CautionWhen you delete a GAD pair forcibly in Device Manager - Storage Navigator, select Force in Delete Mode in the Delete Pairs window.
Create a UR pair between the UR secondary and primary sites.

Command example:
```
paircreate -g oraREMOTE -f async -vl -jp 0 -js 0 -IH2
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
Suspend the UR pair by specifying the S-VOL (swap suspend).

Command example: pairsplit -g oraREMOTE -RS -IH0

You can resume operations at this point by using the S-VOL at the primary site.
Resynchronize the UR pair by specifying the S-VOL (swap resync).

Command example:
```
pairresync -g oraREMOTE -swaps -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

The volume on the primary storage system changes to a P-VOL, and the volume on the UR secondary storage system changes to an S-VOL.
Delete the UR pair.

Command example:
```
pairsplit -g oraREMOTE -S -IH0
```
Re-create the GAD pair.

Command example:
```
paircreate -g oraHA -fg never 2 -vl -jq 0 -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.
```
raidcom map resource -ldev_id 0x1111 
-virtual_ldev_id 0x1111 -IH1
```
The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.
Re-create the UR delta resync pair.

Command example:
```
paircreate -g oraDELTA -f async -vl -jp 0 -js 0 -nocsus -IH1
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
Re-create the UR pair.

Command example:
```
paircreate -g oraREMOTE -f async -vl -jp 0 -js 0 -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering from a failure on the UR delta resync pair

Use the following procedure to recover from a failure on the UR delta resync pair due to a journal volume failure on the storage system at the GAD secondary site.

When a failure occurs on the UR delta resync pair, a SIM is displayed.

Overview of failure recovery

Procedure

Set the system option mode 1182 to ON at the GAD secondary storage system.
Add a reserve journal volume.
After a reserve journal volume is added, the UR delta resync pair status changes to PSUE (HLDE for Device Manager - Storage Navigator), and the SIM code dc9xxx or 7ff103 is issued.
If a reserve journal volume has already been added, use CCI to delete the reserve journal volume from the journal, and then add a reserve journal volume again.
Delete the journal volume in which a failure occurred, from the journal.
Restore the UR delta resync pair.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
Verify that Return to standby is displayed in Resync Mode in the Resync Mirrors window of Device Manager - Storage Navigator, and then resynchronize the pair.

NoteYou cannot do this using CCI.

Confirm that the pair status of the delta UR P-VOL is PSUS (HOLD for Device Manager - Storage Navigator).

If the server has stopped I/Os, the pair status is PSUE (HOLDING for Device Manager - Storage Navigator)

Command example:

pairdisplay -g oraDELTA -fxce -IH1
Group     PairVol(L/R)  (Port#,TID, LU),Seq#,  LDEV#.P/S,  
Status,Fence, %,  P-LDEV#  M  CTG  JID  AP  EM  E-Seq#  E-LDEV#  R/W
oraDELTA  dev3(L)       (CL1-A-1, 0, 1) 522222 4444. P-VOL 
PSUS   ASYNC ,0   6666     -  0    0    -   -   -       -       -/-
oraDELTA  dev3(R)       (CL1-A-1, 0, 1) 544444 6666. S-VOL 
SSUS   ASYNC ,0   4444     -  0    0    -   -   -       -       -/-

Confirm that the mirror status of the journal of the UR delta resync pair is PJNS (for Device Manager - Storage Navigator, it is HOLD when the server issues I/Os, or HOLDING when the server stops I/Os).

Command example:
```
pairdisplay -g oraDELTA -v jnl -IH1
JID  MU  CTG  JNLS  AP  U(%)  Q-Marker  Q-CNT  D-SZ(BLK)  Seq#   Num  LDEV#
000  1   1    PJNS  4   21    43216fde  30     512345     62500  1    39321 
```
Set the system option mode 1182 to OFF at the GAD secondary storage system.
Confirm that no failure SIMs are displayed.

Recovering from a quorum disk failure (GAD+UR)

You can recover from a quorum disk failure when sharing GAD volumes with UR in a GAD 3DC delta resync (GAD+UR) configuration.

You can use a volume in an external storage system or a disk in a server for a quorum disk. Procedures are based on the assumption that a volume in an external storage system is used as the quorum disk and depend on either of the following statuses:

Able to access either volume of the GAD pair from the server
Not able to access either volume of the GAD pair from the server

To check whether you can access the volumes of a GAD pair from the server, use I/O mode for the GAD pair.

Command example

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.S-VOL PSUS 
NEVER ,  100  4444 -   -   0  -  -            -       - L/L
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.P-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - B/B

The server can access a volume whose I/O mode (R/W) is L/L (Local), and cannot access a volume whose I/O mode (R/W) is B/B (Block).

Recovering from a quorum disk failure when one GAD volume is accessible

Use the following procedure to recover a failure on the quorum disk when you can access either volume of a GAD pair from the server.

Overview of failure recovery

Procedure

Remove the failure on the quorum disk.
Resynchronize GAD pairs if they are suspended by a failure.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering from a quorum disk failure when neither GAD volume is accessible

When a failure occurs on the quorum disk and you cannot access either volume of a GAD pair from the server, you need to delete the UR and GAD pairs, then recover from the failure by using the volume at the UR secondary site, and then re-create the pairs.

Procedure

Delete the UR pair.

Command example: pairsplit -g oraREMOTE -S -IH0
Delete the UR delta resync pair.

Command example: pairsplit -g oraDELTA -S -IH1
Delete the GAD pair.

When the I/O mode of both the primary and secondary volumes is Block, you can forcibly delete the pair, as follows:
1. If you specify the S-VOL, delete the virtual LDEV ID at the same time.
```
pairsplit -g oraHA -RF -IH1
```
2. Confirm that the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff
(Omitted) 
```
  VIR_LDEV : ffff indicates GAD reserve.
3. Delete the pair forcibly by specifying the P-VOL.
  
  Do not delete the virtual LDEV ID.
```
pairsplit -g oraHA -SFV -IH0
```
4. Confirm that the virtual LDEV ID of the P-VOLis not deleted.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted) 
LDEV : 1111
(Omitted) 
```
  VIR_LDEV information is not displayed if it is same as LDEV information.
To forcibly delete a pair when the I/O mode is not Block, call Hitachi Vantara LLC customer support.

CautionWhen you delete a GAD pair forcibly in Device Manager - Storage Navigator, select Force in Delete Mode in the Delete Pairs window.
Create a UR pair between the UR secondary and primary sites.

Command example:
```
paircreate -g oraREMOTE -f async -vl -jp 0 -js 0 -IH2
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

You can resume operations at this point by using the P-VOL at the UR secondary site.
Remove the failure on the quorum disk.
Suspend the UR pair by specifying the S-VOL (swap suspend).

Command example: pairsplit -g oraREMOTE -RS -IH0

You can resume operations at this point by using the S-VOL at the primary site.
Resynchronize the UR pair by specifying the S-VOL (swap resync).

Command example:
```
pairresync -g oraREMOTE -swaps -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

The volume on the primary storage system changes to a P-VOL, and the volume on the UR secondary storage system changes to an S-VOL.
Delete the UR pair.

Command example: pairsplit -g oraREMOTE -S -IH0
Re-create the GAD pair.

Command example: paircreate -g oraHA -fg never 2 -vl -jq 0 -IH0

If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.
```
raidcom map resource -ldev_id 0x1111 
-virtual_ldev_id 0x1111 -IH1
```
The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.
Re-create the UR delta resync pair.

Command example:
```
paircreate -g oraDELTA -f async -vl -jp 0 -js 0 -nocsus -IH1
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
Re-create the UR pair.

Command example:
```
paircreate -g oraREMOTE -f async -vl -jp 0 -js 0 -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Quick Links

Status before failure

Recovering from primary site and P-VOL failures (GAD+UR)

Recovering from a primary site failure (GAD+UR)

Recovering from a P-VOL failure (LDEV blockade) at the primary site (GAD+UR)

Reversing the GAD P-VOL and S-VOL (GAD+UR)

Recovering from secondary site and S-VOL failures (GAD+UR)

Recovering from a secondary site failure (GAD+UR)

Recovering from a failure (LDEV blockade) on an S-VOL at the secondary site (GAD+UR)

Recovering from a failure at the primary and secondary sites (GAD+UR)

Recovering from a failure on the UR delta resync pair

Recovering from a quorum disk failure (GAD+UR)

Recovering from a quorum disk failure when one GAD volume is accessible

Recovering from a quorum disk failure when neither GAD volume is accessible