Disaster recovery of global-active device

Failure locations

The following figure and table describe the locations where GAD failures can occur, the SIMs that are issued, and whether the P-VOL and S-VOL are accessible. All GAD-related SIMs are described in SIMs related to GAD.

#	Failure location		SIM reference codes		GAD volume accessible?¹
#	Failure location		Primary storage system	Secondary storage system	P-VOL	S-VOL
1	Server		None (normal)	None (normal)	Yes	Yes
2	Path between the server and the storage system	Path between the server and the primary storage system	None (normal)	None (normal)	No	Yes²
3	Path between the server and the storage system	Path between the server and the secondary storage system	None (normal)	None (normal)	Yes³	No
4	GAD pair volume	P-VOL	3A0`xxx` DD1`xyy` DFA`xxx` DFB`xxx` EF9`xxx`	DD1`xyy`	No	Yes²
5	GAD pair volume	S-VOL	DD1`xyy`	3A0`xxx` DD1`xyy` DFA`xxx` DFB`xxx` EF9`xxx`	Yes³	No
6	Pool for GAD pair⁴	Pool for P-VOL	622`xxx` 62a`xxx` DD1`xyy`	DD1`xyy`	No	Yes²
7	Pool for GAD pair⁴	Pool for S-VOL	DD1`xyy`	622`xxx` 62a`xxx` DD1`xyy`	Yes³	No
8	Path between storage systems	Remote path from the primary to secondary storage system	2180`xx` DD0`xyy`	DD3`xyy`	Yes³	No
9	Path between storage systems	Remote path from the secondary to primary storage system	DD3`xyy`	2180`xx` DD0`xyy`	No	Yes²
10	Storage system	Primary storage system	Depends on the failure type⁵	2180`xx` DD0`xyy` DD3`xyy`	No	Yes²
11	Storage system	Secondary storage system	2180`xx` DD0`xyy` DD3`xyy`	Depends on the failure type⁵	Yes³	No
12	Quorum disk	Path between the primary storage system and quorum disk	21D0`xx` 21D2`xx` DD2`xyy` DEF0`zz` EF5`xyy` EFD000 FF5`xyy`	DD2`xyy`	Yes³	No
13		Path between the secondary storage system and quorum disk	DD2`xyy`	21D0`xx` 21D2`xx` DD2`xyy` DEF0`zz` EF5`xyy` EFD000 FF5`xyy`	Yes³	No
14		Quorum disk	21D0`xx` 21D2`xx` DD2`xyy` DEF0`zz` EF5`xyy` EFD000 FF5`xyy`	21D0`xx` 21D2`xx` DD2`xyy` DEF0`zz` EF5`xyy` EFD000 FF5`xyy`	Yes³	No
15		External storage system	21D0`xx` 21D2`xx` DD2`xyy` DEF0`zz` EF5`xyy` EFD000 FF5`xyy`	21D0`xx` 21D2`xx` DD2`xyy` DEF0`zz` EF5`xyy` EFD000 FF5`xyy`	Yes³	No
Notes: Pairs are not suspended and do not become inaccessible for: Failure in hardware used for redundancy in the storage system, such as drives, cache, front-end director (CHB), back-end director (BED), and MPU Failure in redundant physical paths The volume is not accessible if a failure occurs while the S-VOL pair status is COPY, SSUS, or PSUE. The volume is not accessible if a failure occurs while the P-VOL pair status is PSUS or PSUE and the I/O mode is BLOCK. A failure occurs due to a full pool for a GAD pair. The SIM might not be viewable, depending on the failure (for example, all cache failure, all MP failure, storage system failure).

SIMs related to GAD

The following table shows SIMs related to global-active device operations. All SIMs in the following table are reported to the service processor (SVP) of the storage system or recorded in the storage system, depending on your storage system.

SIM reference code	Description
2180`xx`	Logical path(s) on the remote copy connections was logically blocked (due to an error condition)
21D0`xx`	External storage system connection path blocking
21D2`xx`	Threshold over by external storage system connection path response time-out
3A0`xyy`	LDEV blockade (effect of microcode error)
622`xxx`	The DP POOL FULL
62a`xxx`	Actual DP pool use rate reaches upper limit
DD0`xyy`	GAD for this volume was suspended (due to an unrecoverable failure on the remote copy connections)
DD1`xyy`	GAD for this volume was suspended (due to a failure on the volume)
DD2`xyy`	GAD for this volume was suspended (due to an internal error condition detected)
DD3`xyy`	Status of the P-VOL was not consistent with the S-VOL
DEE0`zz`	Quorum disk restore
DEF0`xx`	Quorum disk blocked
DFA`xxx`	LDEV blockade (drive path: boundary 0/effect of drive port blockade)
DFB`xxx`	LDEV blockade (drive path: boundary 1/effect of drive port blockade)
EF5`xyy`	Abnormal end of write processing in external storage system
EF9`xxx`	LDEV blockade (effect of drive blockade)
EFD000	External storage system connection device blockade
FF5`xyy`	Abnormal end of read processing in external storage system

Pair condition before failure

The pair status and I/O mode of a GAD pair, the accessibility of the server, and the storage location of the latest data depend on the status before a failure occurs.

The following table shows pair status and I/O mode, the volumes accessible from the server, and the location of the latest data before a failure occurs. You can compare this information with the changes that take place after a failure occurs, as described in the following topics.

Pair status and I/O mode		Volume accessible from the server		Volume with latest data
P-VOL	S-VOL	P-VOL	S-VOL	Volume with latest data
PAIR (Mirror (RL))	PAIR (Mirror (RL))	OK	OK	Both P-VOL and S-VOL
PAIR (Mirror (RL))	PAIR (Block)	OK	NG	Both P-VOL and S-VOL
COPY (Mirror (RL))	COPY (Block)	OK	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	NG	OK	S-VOL

Pair condition and recovery: server failures

The following table shows transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when a server failure occurs.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server*
P-VOL	S-VOL	P-VOL	S-VOL	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror (RL))	PAIR (Mirror (RL))	PAIR (Mirror (RL))	PAIR (Mirror (RL))	OK	OK	Both P-VOL and S-VOL
PAIR (Mirror (RL))	PAIR (Block)	PAIR (Mirror (RL))	PAIR (Block)	OK	NG	Both P-VOL and S-VOL
COPY (Mirror (RL))	COPY (Block)	COPY (Mirror (RL))	COPY (Block)	OK	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
* If failures occur in all servers that access the P-VOL or S-VOL, then you cannot access either volume.

SIMs

Primary storage system: None
Secondary storage system: None

Procedure

Recover the server.
Recover the path from the server to the pair volumes.

Pair condition and recovery: path failure between the server and storage system

If a server cannot access a pair volume whose status is PAIR, though no SIM has been issued, a failure might have occurred between the server and the storage system. The following topics provide procedures for recovering of the physical path between the server and the storage systems.

The following table shows transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when you can no longer use a physical path between the server and a storage system.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server*
P-VOL	S-VOL	P-VOL	S-VOL	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror (RL))	PAIR (Mirror (RL))	PAIR (Mirror (RL))	PAIR (Mirror (RL))	OK	OK	Both P-VOL and S-VOL
PAIR (Mirror (RL))	PAIR (Block)	PAIR (Mirror (RL))	PAIR (Block)	OK	NG	Both P-VOL and S-VOL
COPY (Mirror (RL))	COPY (Block)	COPY (Mirror (RL))	COPY (Block)	OK	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
* If failures occur in all servers that access the P-VOL or S-VOL, then you cannot access either volume.

SIMs

Primary storage system: None
Secondary storage system: None

Procedure

Recover the path between the server and the storage system.
Recover the path from the server to the pair volume.

Recovering from a path failure: server to primary storage system

The following figure shows the failure area and recovery when the path between the server and the primary storage system fails.

Procedure

Recover the path.
1. Using the alternate path software and other tools, identify the path that cannot be accessed from the server.
2. Using the SAN management software, identify the failure location; for example, a host bus adapter, FC cable, switch, or other location.
3. Remove the cause of failure and recover the path.
Using the alternate path software, resume I/O from the server to the recovered path (I/O might resume automatically).

Recovering from a path failure: server to secondary storage system

The following figure shows the failure area and recovery when the path between the server and secondary storage system fails.

Procedure

Recover the path.
1. Using the alternate path software or other tools, identify the path that cannot be accessed from the server.
2. Using SAN management software, identify the failure location; for example, a host bus adapter, FC cable, switch, or other location.
3. Remove the cause of failure and recover the path.
Using the alternate path software, resume I/O from the server to the recovered path (I/O might resume automatically).

Pair condition and recovery: P-VOL failure (LDEV blockade)

The following table shows transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when you can no longer use the P-VOL due to LDEV blockade.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server¹
P-VOL	S-VOL	P-VOL	S-VOL	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror (RL))	PAIR (Mirror (RL))	PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
PAIR (Mirror (RL))	PAIR (Block)	PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
COPY (Mirror (RL))	COPY (Block)	PSUE (Local)	PSUE (Block)	NG	NG	None¹
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	NG	NG	None²
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
Notes: Recover the data from ShadowImage, Thin Image, or other backup data. Recover the data using the S-VOL data that is not the latest, ShadowImage, Thin Image, or other backup data.

SIMs

Primary storage system: 3A0xyy, DD1xyy, DFAxxx, DFBxxx, EF9xxx
Secondary storage system: DD1xyy

Procedure

Recover the P-VOL.
Re-create the pair.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering the P-VOL (DP-VOL) (pair status: PAIR)

The following figure shows the failure area and recovery when a pair is suspended due to a P-VOL failure and the P-VOL is a DP-VOL.

NoteIn this example no consistency group is specified.

Procedure

Delete the alternate path (logical path) to the volume that cannot be accessed from the server.
1. Using the alternate path software, identify the volume that cannot be accessed.
2. Confirm whether the volume (P-VOL) is blocked, and the pool ID (B_POOLID) of the pool to which the P-VOL is associated.
```
raidcom get ldev -ldev_id 0x2222 -IH0
(snip)
B_POOLID : 0
(snip)
STS : BLK
(snip)
```
3. Display the status of the volumes configuring the pool (pool volume) to identify the blocked volume.
```
raidcom get ldev -ldev_list pool -pool_id 0 -IH0
(snip)
LDEV : 16384
(snip) 
STS : BLK 
(snip) 
```
  For a blocked volume, BLK is indicated in the STS column.
4. Using the alternate path software, delete the alternate path to the volume that cannot be accessed from the server.
Go to the next step even if the alternate path cannot be deleted.

Delete the pair.

From the secondary storage system, delete the pair specifying the actual LDEV ID of the S-VOL.
```
pairsplit -g oraHA -R -d dev1 -IH1
```
NoteTo delete the pair specifying the S-VOL, use the -R option of the pairsplit command. Specify the actual LDEV ID (device name) of the S-VOL in the -d option.

Confirm that the pair is deleted.

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU), Seq#,   LDEV#.P/S,Status,Fence,   %,    P-LDEV# M   CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)      (CL1-C-1, 0,   0)522222  4444.SMPL  ---- ------,  ----- -----   -   -   -   -  -        -      -       -/-
oraHA   dev1(R)      (CL1-A-0, 0,   0)511111  2222.SMPL  ---- ------,  ----- -----   -   -   -   -  -        -      -       -/-

Remove the failure.

The following example shows recovery from a pool-volume failure.
1. Recover a pool volume that configures the P-VOL (DP-VOL).
2. Display the status of the pool volumes to confirm that the pool volume has been recovered.
```
raidcom get ldev -ldev_list pool -pool_id 0 -IH0
 (snip) 
LDEV : 16384 
(snip) 
STS : NML 
(snip)
```
  For a normal volume, NML is indicated in the STS column.
If the volume cannot be recovered, follow the procedure below to re-create the P-VOL:
1. At the primary storage system, delete the LU path to the P-VOL.
2. Delete the P-VOL.
3. Create a new volume.
4. Set an LU path to the new volume.
Confirm the quorum disk status. If the quorum disk is blocked, recover from the blockade. If you do not set a volume for the quorum disk, you can skip this step.
Re-create the pair.
1. If you created a volume in step 4, set the GAD reserve attribute to the created volume.
```
raidcom map resource -ldev_id 0x2222 -virtual_ldev_id reserve -IH0
```
2. From the secondary storage system, create the pair specifying the S-VOL's actual LDEV ID.
```
paircreate -g oraHA -f never -vl -jq 0 -d dev1 -IH1
```
  NoteTo create the pair specifying the S-VOL, specify the actual LDEV ID (device name) of the S-VOL in the -d option of the paircreate command.
  
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
  
  The volume of the primary storage system changes to an S-VOL, and the volume of the secondary storage system changes to a P-VOL.
3. Confirm that the P-VOL and S-VOL pair statuses change to PAIR (Mirror (RL)).
```
pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
```
```
pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
```
Using the alternate path software, add an alternate path from the server to the S-VOL (P-VOL before the failure).
Using the alternate path software, resume I/O from the server to the S-VOL (P-VOL before the failure).

NoteI/O from the server might resume automatically.
Reverse the P-VOL and the S-VOL if necessary.

Recovering the P-VOL (other than DP-VOL) (pair status: PAIR)

The following figure shows the failure area and recovery when a pair is suspended due to a P-VOL failure and the P-VOL is not a DP-VOL.

For details about storage system support (models, microcode) for volumes other than DP-VOLs, see Requirements and restrictions.

NoteIn this example no consistency group is specified.

Procedure

Delete the alternate path (logical path) to the volume that cannot be accessed from the server.
1. Using the alternate path software, identify the volume that cannot be accessed.
2. Confirm whether the volume (P-VOL) is blocked.
```
raidcom get ldev -ldev_id 0x2222 -IH0
(snip)
STS : BLK
(snip)
```
  For a blocked volume, BLK is indicated in the STS column.
3. Using the alternate path software, delete the alternate path to the volume that cannot be accessed from the server.
Go to the next step even if the alternate path cannot be deleted.

Delete the pair.

From the secondary storage system, delete the pair specifying the actual LDEV ID of the S-VOL.
```
pairsplit -g oraHA -R -d dev1 -IH1
```
NoteTo delete the pair specifying the S-VOL, use the -R option of the pairsplit command. Specify the actual LDEV ID (device name) of the S-VOL in the -d option.

Confirm that the pair is deleted.

pairdisplay -g oraHA -fxce -IH1Group PairVol(L/R) (Port#,TID, LU), Seq#, LDEV#.P/S,
Status,Fence, %, P-LDEV# M CTG JID AP EM ESeq# E-LDEV# R/W
oraHA dev1(L) (CL1-C-1, 0, 0)522222 4444.SMPL ---- ------, ----- ----- - - - - - - - -/-
oraHA dev1(R) (CL1-A-0, 0, 0)511111 2222.SMPL ---- ------, ----- ----- - - - - - - - -/-

Remove the failure. The following example shows recovery from a volume failure.
1. Recover the P-VOL.
2. Display the status of the P-VOL to confirm that the pool volume has been recovered.
```
raidcom get ldev -ldev_id 0x2222 -IH0
(snip)
STS : NML
(snip)
```
  For a normal volume, NML is indicated in the STS column.
If the volume cannot be recovered, follow the procedure below to re-create the P-VOL:
1. At the primary storage system, delete the LU path to the P-VOL.
2. Delete the P-VOL.
3. Create a new volume.
4. Set an LU path to the new volume.
Confirm the quorum disk status. If the quorum disk is blocked, recover from the blockade. If you do not set a volume for the quorum disk, you can skip this step.

Re-create the pair.

If you created a volume in step 4, set the GAD reserve attribute to the created volume.
```
raidcom map resource -ldev_id 0x2222 -virtual_ldev_id reserve -IH0
```
From the secondary storage system, create the pair specifying the S-VOL's actual LDEV ID.
```
paircreate -g oraHA -f never -vl -jq 0 -d dev1 -IH1
```
NoteTo create the pair specifying the S-VOL, specify the actual LDEV ID (device name) of the S-VOL in the -d option of the paircreate command. The volume in the primary storage system changes to an S-VOL, and the volume in the secondary storage system changes to a P-VOL.

Confirm that the P-VOL and S-VOL pair statuses have changed to PAIR (Mirror (RL)).

pairdisplay -g oraHA -fxce -IH0
Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status, Fence, 
%,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
oraHA dev1(L) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIR NEVER , 100 4444 - - 0 - - - - L/M
oraHA dev1(R) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIR NEVER , 100 2222 - - 0 - - - - L/M

pairdisplay -g oraHA -fxce -IH1
Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status, Fence, 
%,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
oraHA dev1(L) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIR NEVER , 100 2222 - - 0 - - - - L/M
oraHA dev1(R) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIR NEVER , 100 4444 - - 0 - - - - L/M

Using the alternate path software, add an alternate path from the server to the S-VOL (P-VOL before the failure).
Using the alternate path software, resume I/O from the server to the S-VOL (P-VOL before the failure).

NoteI/O from the server might resume automatically.
Reverse the P-VOL and the S-VOL if necessary.

Pair condition and recovery: S-VOL failure (LDEV blockade)

The following table shows the transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when you can no longer use the S-VOL due to LDEV blockade.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server
P-VOL	S-VOL	P-VOL	S-VOL	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror (RL))	PAIR (Mirror (RL))	PSUE (Local)	PSUE (Block)	OK	NG	P-VOL
PAIR (Mirror (RL))	PAIR (Block)	PSUE (Local)	PSUE (Block)	OK	NG	P-VOL
COPY (Mirror (RL))	COPY (Block)	PSUE (Local)	PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	NG	None*
* Recover data using the P-VOL data that is not the latest, ShadowImage, Thin Image, or other backup data.

SIMs

Primary storage system: DD1xyy
Secondary storage system: 3A0xyy, DD1xyy, DFAxxx, DFBxxx, EF9xxx

Procedure

Recover the S-VOL.
Re-create the pair.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering the S-VOL (DP-VOL) (pair status: PAIR)

The following figure shows the failure area and recovery when a pair is suspended due to an S-VOL failure and the S-VOL is a DP-VOL.

NoteIn this example no consistency group is specified.

Procedure

Delete the alternate path (logical path) to the volume that cannot be accessed from the server.
1. Using the alternate path software, identify the volume that cannot be accessed.
2. Confirm whether the volume (S-VOL) is blocked, and the pool ID (B_POOLID) of the pool to which the S-VOL is associated.
```
raidcom get ldev -ldev_id 0x4444 -IH1 
(snip) 
B_POOLID : 0 
(snip) 
STS : BLK 
(snip)
```
3. Display the status of the volumes configuring the pool (pool volume) to identify the blocked volume.
```
raidcom get ldev -ldev_list pool -pool_id 0 -IH1 
(snip) 
LDEV : 16384 
(snip) 
STS : BLK 
(snip)
```
  For the blocked volume, BLK is indicated in the STS column.
4. Using the alternate path software, delete the alternate path to the volume.
Go to the next step even if the alternate path cannot be deleted.

Delete the pair.

From the primary storage system, delete the pair specifying the P-VOL's actual LDEV ID.
```
pairsplit -g oraHA -S -d dev1 -IH0
```
NoteTo delete the pair specifying the P-VOL, use the -S option of the pairsplit command. Specify the actual LDEV ID (device name) of the P-VOL in the -d option.

Confirm that the pair is deleted.

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU), Seq#,   LDEV#.P/S,Status,
Fence,   %,    P-LDEV# M   CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)      (CL1-A-0, 0,   0)511111  2222.SMPL  ---- ------,  
----- -----   -   -   -   -  -        -      -       -/-
oraHA   dev1(R)      (CL1-C-1, 0,   0)522222  4444.SMPL  ---- ------,  
----- -----   -   -   -   -  -        -      -       -/-

Remove the failure.

The following example shows recovery from a pool-volume failure.
1. Recover a pool volume that configures the S-VOL (DP-VOL).
2. Display the status of the pool volumes to confirm that the pool volume has been recovered.
```
raidcom get ldev -ldev_list pool -pool_id 0 -IH1 
(snip) 
LDEV : 16384 
(snip) 
STS : NML 
(snip)
```
  For a normal volume, NML is indicated in the STS column.
If the volume cannot be recovered, follow the procedure below to create the S-VOL again:
1. At the secondary storage system, delete the LU path to the S-VOL.
2. Delete the S-VOL.
3. Create a new volume.
4. Set an LU path to the new volume.
Confirm the quorum disk status. If the quorum disk is blocked, recover from the blockade. If you do not set a volume for the quorum disk, you can skip this step.

Re-create the pair.

If you created a volume in step 4, set the GAD reserve attribute to the created volume.
```
raidcom map resource -ldev_id 0x4444 -virtual_ldev_id reserve -IH1
```
From the primary storage system, create the pair specifying the P-VOL's actual LDEV ID.
```
paircreate -g oraHA -f never -vl -jq 0 -d dev1 -IH0
```
NoteTo create the pair specifying the P-VOL, specify the actual LDEV ID (device name) of the P-VOL in the -d option of the paircreate command.

If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Confirm that the P-VOL and S-VOL pair statuses change to PAIR (Mirror (RL)).

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M

Using the alternate path software, add an alternate path from the server to the S-VOL.
Using the alternate path software, resume I/O from the server to the S-VOL.

NoteI/O from the server might resume automatically.

Recovering the S-VOL (other than DP-VOL) (pair status: PAIR)

The following figure shows the failure area and recovery when a pair is suspended due to an S-VOL failure and the S-VOL is not a DP-VOL.

For details about storage system support (models, microcode) for volumes other than DP-VOLs, see Requirements and restrictions.

NoteIn this example no consistency group is specified.

Procedure

Delete the alternate path (logical path) to the volume that cannot be accessed from the server.
1. Using the alternate path software, identify the volume that cannot be accessed.
2. Confirm whether the volume (S-VOL) is blocked.
```
raidcom get ldev -ldev_id 0x4444  – IH1
(snip)
STS : BLK
(snip)
```
  For a blocked volume, BLK is indicated in the STS column.
3. Using the alternate path software, delete the alternate path to the volume that cannot be accessed from the server.
Go to the next step even if the alternate path cannot be deleted.

Delete the pair.

From the primary storage system, delete the pair specifying the actual LDEV ID of the P-VOL.
```
pairsplit -g oraHA -S -d dev1 -IH1
```
NoteTo delete the pair specifying the P-VOL, use the -S option of the pairsplit command. Specify the actual LDEV ID (device name) of the P-VOL in the -d option.

Confirm that the pair is deleted.

pairdisplay -g oraHA -fxce  – IH0
Group PairVol(L/R) (Port#,TID, LU), Seq#, LDEV#.P/S,Status,
Fence, %, P-LDEV# M CTG JID AP EM ESeq# E-LDEV# R/W
oraHA dev1(L) (CL1-A-0, 0, 0)511111 2222.SMPL ---- ------, 
----- ----- - - - - - - - -/-
oraHA dev1(R) (CL1-C-1, 0, 0)522222 4444.SMPL ---- ------, 
----- ----- - - - - - - - -/-

Remove the failure. The following example shows recovery from a volume failure.
1. Recover an S-VOL.
2. Display the status of the P-VOL to confirm that the pool volume has been recovered.
```
raidcom get ldev -ldev_id 0x4444  – IH1
(snip)
STS : NML
(snip)
```
  For a normal volume, NML is indicated in the STS column.
If the volume cannot be recovered, follow the procedure below to re-create the S-VOL:
1. At the secondary storage system, delete the LU path to the S-VOL.
2. Delete the S-VOL.
3. Create a new volume.
4. Set an LU path to the new volume.
Confirm the quorum disk status. If the quorum disk is blocked, recover from the blockade. If you do not set a volume for the quorum disk, you can skip this step.

Re-create the pair.

If you created a volume in step 4, set the GAD reserve attribute to the created volume.
```
raidcom map resource -ldev_id 0x4444 -virtual_ldev_id reserve –IH1
```
From the primary storage system, create the pair specifying the P-VOL's actual LDEV ID.
```
paircreate -g oraHA -f never -vl -jq 0 -d dev1 -IH0
```
NoteTo create the pair specifying the P-VOL, specify the actual LDEV ID (device name) of the P-VOL in the -d option of the paircreate command.

Confirm that the P-VOL and S-VOL pair statuses have changed to PAIR (Mirror (RL)).

pairdisplay -g oraHA -fxce -IH0
Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status, 
Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
oraHA dev1(L) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIR NEVER , 
100 4444 - - 0 - - - - L/M
oraHA dev1(R) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIR NEVER , 
100 2222 - - 0 - - - - L/M
pairdisplay -g oraHA -fxce -IH1
Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status, 
Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
oraHA dev1(L) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIR NEVER , 
100 2222 - - 0 - - - - L/M
oraHA dev1(R) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIR NEVER , 
100 4444 - - 0 - - - - L/M

Using the alternate path software, add an alternate path from the server to the S-VOL.
Using the alternate path software, resume I/O from the server to the S-VOL.

NoteI/O from the server might resume automatically.

Pair condition and recovery: full pool for the P-VOL

When the P-VOL cannot be used due to a full pool, the GAD pair is suspended.

The following table shows the transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when you can no longer use the P-VOL due to full pool.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server
P-VOL	S-VOL	P-VOL	S-VOL	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror (RL))	PAIR (Mirror (RL))	PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
PAIR (Mirror (RL))	PAIR (Block)	PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
COPY (Mirror (RL))	COPY (Block)	PSUE (Local)	PSUE (Block)	NG	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	NG	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	OK	S-VOL

SIMs

Primary storage system: 622xxx, 62axxx, DD1xyy
Secondary storage system: DD1xyy

Procedure

Increase the available pool capacity to the P-VOL.
Resynchronize the pair.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering a full pool for the P-VOL (pair status: PAIR)

The following figure shows the failure area and recovery when a pair is suspended due to a full pool of the P-VOL.

Procedure

Increase the available capacity to the pool on which the full pool was detected.

For details on how to increase an available pool capacity, see the Provisioning Guide for the storage system.
Confirm the quorum disk status. If the quorum disk is blocked, recover from the blockade. If you do not set a volume for the quorum disk, you can skip this step.

Resynchronize a GAD pair.

Confirm that the I/O mode of the S-VOL is Local.

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL SSWS 
NEVER ,  100  2222 -   -   0  -  -            -       - L/L
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PSUE 
NEVER ,  100  4444 -   -   0  -  -            -       - B/B

At the secondary storage system, resynchronize the pair.
```
pairresync -g oraHA -swaps -IH1
```
The volume of the primary storage system changes to an S-VOL, and the volume of the secondary storage system changes to a P-VOL.

If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Confirm that the P-VOL and S-VOL pair statuses change to PAIR (Mirror (RL)).

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.S-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.P-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.P-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.S-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M

Using the alternate path software, resume I/Os to the S-VOL that was a P-VOL before the failure (I/O might resume automatically).
Reverse the P-VOL and the S-VOL if necessary.

Pair condition and recovery: full pool for the S-VOL

When the S-VOL cannot be used due to a full pool, the GAD pair is suspended.

The following table shows the transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when you can no longer use the S-VOL due to full pool.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server
P-VOL	S-VOL	P-VOL	S-VOL	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror (RL))	PAIR (Mirror (RL))	PSUE (Local)	PSUE (Block)	OK	NG	P-VOL
PAIR (Mirror (RL))	PAIR (Block)	PSUE (Local)	PSUE (Block)	OK	NG	P-VOL
COPY (Mirror (RL))	COPY (Block)	PSUE (Local)	PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	NG	S-VOL

SIMs

Primary storage system: DD1xyy
Secondary storage system: 622xxx, 62axxx, DD1xyy

Procedure

Increase an available pool capacity to the S-VOL.
Resynchronize the pair.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering a full pool for the S-VOL (pair status: PAIR)

The following figure shows the failure area and recovery when a pair is suspended due to a full pool of the S-VOL.

Procedure

Increase an available capacity to the pool on which the full pool was detected.

For details on how to increase an available pool capacity, see the Provisioning Guide for the storage system.
Confirm the quorum disk status. If the quorum disk is blocked, recover from the blockade. If you do not set a volume for the quorum disk, you can skip this step.

Resynchronize a GAD pair.

If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Confirm that the I/O mode of the P-VOL is Local.

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PSUE 
NEVER ,  100  4444 -   -   0  -  -            -       - L/L
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PSUE 
NEVER ,  100  2222 -   -   0  -  -            -       - B/B

At the primary storage system, resynchronize the pair.
```
pairresync -g oraHA -IH0
```

Confirm that the P-VOL and S-VOL pair statuses change to PAIR (Mirror (RL)).

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M

Using the alternate path software, resume I/O to the S-VOL (I/O might resume automatically).

Pair condition and recovery: path failure, primary to secondary storage system

If the statuses of storage systems in both the primary and secondary sites are normal, a failure might have occurred in a physical path or switch between the storage systems. You can correct the issue by recovering the paths and resynchronizing the pair.

GAD pairs might be created between the primary and secondary sites that configure GAD as follows:

A P-VOL of a GAD pair is at site 1 and a S-VOL is at site 2.
A P-VOL of a different GAD pair is at site 2 and a S-VOL is at site 1.

In this case, recover from a physical path failure at the secondary site. For more information, see Pair condition and recovery: path failure, secondary to primary storage system.

The following table shows transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when you can no longer use any physical path from the primary storage system to the secondary storage system.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server
P-VOL	S-VOL	P-VOL	S-VOL	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror(RL))¹	PAIR (Mirror(RL))¹	PSUE (Local)	PSUE (Block)	OK	NG	P-VOL
PAIR (Mirror(RL))	PAIR (Block)	PSUE (Local)	PAIR (Block)²	OK	NG	P-VOL
COPY (Mirror(RL))	COPY (Block)	PSUE (Local)	PSUE/COPY (Block)³	OK	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
Notes: The I/O modes for the P-VOL and S-VOL can change, even if I/Os are not issued from the server to the P-VOL. For example, a command that synchronizes the storage system at the primary site and the storage system at the secondary site might be issued to the P-VOL due to its health check (such as by ATS). When you execute this command, synchronization fails if a path between these storage systems has a failure. As a result, the I/O mode for the P-VOL to which I/Os were not issued from the server becomes Local, and the I/O mode for the S-VOL to which I/Os were issued becomes Block. This might cause a failure and suspend GAD pairs. For the recovery procedure, see Pair condition and recovery: quorum disk and primary-to-secondary path failure. If either of the following conditions is met, the S-VOL might become COPY (Block): No volume is set for the quorum disk. A quorum disk failure occurs.

SIMs

Primary storage system: DD0xyy, 2180xx
Secondary storage system: DD3xyy

Procedure

Recover the paths from the primary storage system to the secondary storage system.
Resynchronize the pair.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering paths, primary to secondary storage system (pair status: PAIR)

The following figure shows the failure area and recovery when a pair is suspended due to path failure from the primary storage system to the secondary storage system.

Procedure

Reconnect the physical path or reconfigure the SAN to recover the path failure.

When the path between the storage systems is recovered, the remote path is either automatically recovered or a manual recovery might be required. To verify the remote path status and perform any recommended action, see Troubleshooting related to remote path status. If the remote path failure persists even after following the recommended action, contact customer support.
Confirm the quorum disk status. If the quorum disk is blocked, recover from the blockade. If you do not set a volume for the quorum disk, you can skip this step.

Resynchronize the pair.

Confirm that the P-VOL I/O mode is Local.

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PSUE 
NEVER ,  100  4444 -   -   0  -  -            -       - L/L
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PSUE 
NEVER ,  100  2222 -   -   0  -  -            -       - B/B

At the primary storage system, resynchronize the pair.
```
pairresync -g oraHA -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Confirm that the P-VOL and S-VOL pair statuses change to PAIR (Mirror (RL)).

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M

Using the alternate path software, resume I/O to the volume that could not be accessed from the server (I/O might resume automatically).

Pair condition and recovery: path failure, secondary to primary storage system

If the statuses of the storage systems in both the primary and secondary sites are normal, a failure might have occurred in a physical path or switch between the storage systems.

GAD pairs might be created between the primary and secondary sites that configure GAD as follows:

A P-VOL of a GAD pair is at site 1 and a S-VOL is at site 2.
A P-VOL of a different GAD pair is at site 2 and a S-VOL is at site 1.

In this case, recover from a physical path failure at the primary site. For more information, see Pair condition and recovery: path failure, primary to secondary storage system.

The following table shows the transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when you can no longer use any physical path from the secondary storage system to the primary storage system.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server
P-VOL	S-VOL	P-VOL	S-VOL	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror(RL))¹	PAIR (Mirror(RL))¹	PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
PAIR (Mirror(RL))	PAIR (Mirror(RL))	PSUE (Local)²	PSUE (Block)²	OK	NG	P-VOL
PAIR (Mirror(RL))	PAIR (Block)	PAIR (Mirror(RL))	PAIR (Block)	OK	NG	P-VOL
COPY (Mirror(RL))	COPY (Block)	COPY (Mirror (RL))	COPY (Block)	OK	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
Notes: The I/O modes for the P-VOL and S-VOL can change, even if I/Os are not issued from the server to the S-VOL. For example, a command that synchronizes the storage system at the primary site and the storage system at the secondary site might be issued to the S-VOL due to its health check (such as by ATS). When you execute this command, synchronization fails if a path between these storage systems has a failure. As a result, the I/O mode for the S-VOL to which I/Os were not issued from the server becomes Local, and the I/O mode for the P-VOL to which I/Os were issued becomes Block. This might cause a failure and suspend GAD pairs. If either of the following conditions is met, the P-VOL might become PSUE (Local) and the S-VOL might become PSUE (Block): No volume is set for the quorum disk. A quorum disk failure occurs.

SIMs

Primary storage system: DD3xyy
Secondary storage system: DD0xyy, 2180xx

Procedure

Recover the paths from the secondary storage system to the primary storage system.
Resynchronize the pair.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering paths, secondary to primary storage system (pair status: PAIR when a volume is set for the quorum disk)

The following figure shows the failure area and recovery in a configuration with a volume set for the quorum disk when a pair is suspended due to path failure from the secondary storage system to the primary storage system.

Procedure

Reconnect the physical path or reconfigure the SAN to recover the path from the secondary storage system to the primary storage system.

After the path between the storage systems is recovered, the remote path is either automatically recovered or a manual recovery might be required. To verify the remote path status and perform any recommended action, see Troubleshooting related to remote path status. If the remote path failure persists even after following the recommended action, contact customer support.
Confirm the quorum disk status. If the quorum disk is blocked, recover from the blockade. If you do not set a volume for the quorum disk, you can skip this step.

Resynchronize the pair.

Confirm that the S-VOL I/O mode is Local.

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL SSWS 
NEVER ,  100  2222 -   -   0  -  -            -       - L/L
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PSUE 
NEVER ,  100  4444 -   -   0  -  -            -       - B/B

At the secondary storage system, resynchronize the pair.
```
pairresync -g oraHA -swaps -IH1
```
The volume on the primary storage system changes to an S-VOL, and the volume on the secondary storage system changes to a P-VOL.

If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Confirm that the P-VOL and S-VOL pair statuses change to PAIR (Mirror (RL)).

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.S-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.P-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.P-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.S-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M

Using the alternate path software, resume I/O to the S-VOL (P-VOL before the failure).

I/O from the server might resume automatically.
Reverse the P-VOL and the S-VOL if necessary.

Recovering paths, secondary to primary storage system (pair status: PAIR when no volume is set for the quorum disk)

When no volume is set for the quorum disk and the pair status is PAIR, if the GAD pair is suspended due to a path failure from the secondary storage system to the primary storage system, perform the following procedure.

Procedure

Recover the paths from the secondary storage system to the primary storage system.
Resynchronize the pair.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Pair condition and recovery: primary storage system failure

When the primary storage system fails, you can correct the issue by recovering the primary storage system, recovering the physical path between primary and secondary storage systems, and recovering the pair.

GAD pairs might be created between the primary and secondary sites that configure GAD as follows:

A P-VOL of a GAD pair is at site 1 and a S-VOL is at site 2.
A P-VOL of a different GAD pair is at site 2 and a S-VOL is at site 1.

In this case, recover from a failure at the secondary storage system. For more information, see Pair condition and recovery: secondary storage system failure.

The following table shows transitions for pair status and I/O mode, the volumes accessible from the server, and location of the latest data when you can no longer use the primary storage system due to failure.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server
P-VOL	S-VOL	P-VOL¹	S-VOL	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror (RL))	PAIR (Mirror (RL))	PSUE (Block)	SSWS (Local)²	NG	OK	S-VOL
PAIR (Mirror (RL))	PAIR (Mirror (RL))	PSUE (Block)³	PSUE (Block)³	NG	NG	Both P-VOL and S-VOL
PAIR (Mirror (RL))	PAIR (Block)	PSUE (Block)	PSUE (Block)	NG	NG	Both P-VOL and S-VOL
COPY (Mirror (RL))	COPY (Block)	PSUE (Local)	COPY (Block)	NG	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	NG	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
Notes: If shared memory in the primary storage system becomes volatilized, the P-VOL status changes to SMPL, and the GAD reserve attribute is set for the volume, which prevents host access to the volume. If the server does not issue write I/O, the pair status is PAIR (Mirror (RL)). If either of the following conditions is met, the P-VOL might become PSUE (Block) and the S-VOL might become PSUE (Block): No volume is set for the quorum disk. A quorum disk failure occurs.

SIMs

Primary storage system: SIM varies depending on the failure type
Secondary storage system: 2180xx, DD0xyy, DD3xyy

Procedure

When the primary storage system is powered off, delete an alternate path (logical path) to the P-VOL, and then turn on the power.
1. Using the alternate path software, distinguish the volumes which are not able to be accessed from the server.
2. Using the alternate path software, delete the alternate paths to the P-VOL.
  If you cannot delete the alternate paths, detach all physical paths which are connected to the server at the primary site.
Turn on the primary storage system.
Recover the primary storage system.

For details, contact customer support.
Recover the physical path between the primary storage system and the secondary storage system.
If S-VOL pair status is PAIR, suspend the pair specifying the S-VOL.
Confirm the quorum disk status. If the quorum disk is blocked, recover from the blockade. If you do not set a volume for the quorum disk, you can skip this step.

Resynchronize or re-create a pair using the following procedure whose pair status and I/O mode match your pair's status and I/O mode.

If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.If the failed primary site contains a P-VOL, recover from a failure in accordance with the following procedure. If the failed primary site also contains a S-VOL, recover from a failure at the secondary site. For more information, see Pair condition and recovery: secondary storage system failure.

Pair status		I/O mode
P-VOL	S-VOL	P-VOL	S-VOL
PSUE	COPY	Local	Block

Delete the pair forcibly from the S-VOL.
```
pairsplit -g oraHA -d dev1 -RF -IH2
```
When you perform this step, delete the virtual LDEV ID so that the volume cannot be accessed from the server.
Check if the virtual LDEV ID of the P-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV:1111
(Omitted)
```
The VIR_LDEV information is not displayed if it is same as the LDEV information. If the virtual LDEV ID is deleted, set a correct LDEV ID.
Delete the pair forcibly from the P-VOL.

When you perform this step, make sure not to delete the virtual LDEV ID, which allows the volume to be accessed from the server.
```
pairsplit -g oraHA -d dev1 -SFV -IH1
```
Check if the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff
(Omitted)
```
VIR_LDEV : ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
Re-create the GAD pair by specifying the P-VOL.
```
paircreate -g oraHA -d dev1 -f never -vl -jq 1 -IH1
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.

raidcom map resource -ldev_id 0x1111 
-virtual_ldev_id 0x1111 -IH1

The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.

Pair status		I/O mode
P-VOL	S-VOL	P-VOL	S-VOL
SMPL	COPY	Not applicable	Block

Delete the pair forcibly from the S-VOL.

When you perform this step, delete the virtual LDEV ID so that the volume cannot be accessed from the server.
```
pairsplit -g oraHA -d dev1 -RF -IH2
```
Check if the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -
fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff
(Omitted)
```
VIR_LDEV : ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
Check if the virtual LDEV ID of the P-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x1111 -
fx -IH1
(Omitted)
LDEV : 1111
(Omitted)
```
The VIR_LDEV information is not displayed if it is same as the LDEV information. If the virtual LDEV ID is deleted, set a correct LDEV ID.

NoteIf shared memory in the secondary storage system becomes volatilized, the S-VOL pair status changes to SMPL, and GAD reserve is assigned to the virtual attribute.
Re-create the GAD pair by specifying the P-VOL.
```
paircreate -g oraHA -d dev1 -f
never -vl -jq 1 -IH1
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.

raidcom map resource -ldev_id 0x2222 
-virtual_ldev_id 0x1111 -IH1

The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.

Pair status		I/O mode
P-VOL	S-VOL	P-VOL	S-VOL
PSUS/PSUE	SSWS	Block	Local

Resynchronize the pair specifying the S-VOL.
```
pairresync -g oraHA -d dev1 -swaps -IH2
```

If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Pair status		I/O mode
P-VOL	S-VOL	P-VOL	S-VOL
SMPL	SSWS	Not applicable	Local

Check if the virtual LDEV ID of the P-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
VIR_LDEV : ffff
(Omitted)
```
VGADIR_LDEV : ffff indicates reserve. If it shows another value, set a correct virtual LDEV ID.
Delete the pair forcibly from the S-VOL.

When you perform this step, make sure not to delete the virtual LDEV ID, which allows the volume to be accessed from the server.
```
pairsplit -g oraHA -d dev1 -RFV -IH2
```

Check if the virtual LDEV ID of the S-VOL is not deleted.

raidcom get ldev -ldev_id 0x2222 -
fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : 1111
(Omitted)

Re-create the pair specifying the S-VOL.
```
paircreate -g oraHA -d dev1 -f never -vl -jq 1 -IH2
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

If you accidentally delete the virtual LDEV ID of the S-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the S-VOL again using the raidcom map resource command.

raidcom map resource -ldev_id 0x2222 
-virtual_ldev_id 0x1111 -IH1

The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID of the P-VOL using the confirmation command.

Pair status		I/O mode
P-VOL	S-VOL	P-VOL	S-VOL
PSUS/PSUE	SSUS/PSUE	Local	Block

Resynchronize the pair specifying the P-VOL.
```
pairresync -g oraHA -d dev1 -IH1
```

If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Pair status		I/O mode
P-VOL	S-VOL	P-VOL	S-VOL
PSUE	PSUE	Block	Block

Delete the pair forcibly from the S-VOL.

When you perform this step, delete the virtual LDEV ID so that the volume cannot be accessed from the server.
```
pairsplit -g oraHA -d dev1 -RF -IH2
```
Check if the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -
fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff
(Omitted)
```
VIR_LDEV : ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
Delete the pair forcibly from the P-VOL.

When you perform this step, make sure not to delete the virtual LDEV ID, which allows the volume to be accessed from the server.
```
pairsplit -g oraHA -d dev1 -SFV -
IH1
```
Check if the virtual LDEV ID of the P-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x1111 -
fx -IH1
(Omitted)
LDEV : 1111
(Omitted)
```
The VIR_LDEV information is not displayed if it is same as the LDEV information. If the virtual LDEV ID is deleted, set a correct LDEV ID.
Re-create the pair specifying the P-VOL.
```
paircreate -g oraHA -d dev1 -f
never -vl -jq 1 -IH1
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.

raidcom map resource -ldev_id 0x1111 
-virtual_ldev_id 0x1111 -IH1

The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.

Pair status		I/O mode
P-VOL	S-VOL	P-VOL	S-VOL
SMPL	SSUS/PSUE	Not applicable	Block

Delete the pair forcibly from the S-VOL.
For VSP G/F350, G/F370, G/F700, G/F900, VSP E990, when you perform this step, make sure not to delete the virtual LDEV ID, which allows the volume to be accessed from the server.
For VSP 5000 series, when you perform this step, delete the virtual LDEV ID, which allows the volume to be accessed from the server.
```
pairsplit -g oraHA -d dev1 -RF -IH2
```
Check if the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -
fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff
(Omitted)
```
VIR_LDEV : ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
Check if the virtual LDEV ID of the P-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x1111 -
fx -IH1
(Omitted)
LDEV : 1111
(Omitted)
```
The VIR_LDEV information is not displayed if it is same as the LDEV information. If you cannot find the virtual LDEV ID, set a correct LDEV ID.

NoteIf shared memory in the secondary storage system becomes volatilized, the S-VOL pair status changes to SMPL, and GAD reserve is assigned to the virtual attribute.
Re-create the pair specifying the P-VOL.
```
paircreate -g oraHA -d dev1 -f never -vl -jq 1 -IH1
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.

raidcom map resource -ldev_id 0x2222 
-virtual_ldev_id 0x1111 -IH1

The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.

Pair status		I/O mode
P-VOL	S-VOL	P-VOL	S-VOL
PSUE	PSUE	Block	Block

These steps apply only to VSP 5000 series.

Delete the pair forcibly from the S-VOL.
When you perform this step, delete the virtual LDEV ID so that the volume cannot be accessed from the server.
```
pairsplit -g oraHA -d dev1 -RF -IH2
```
Check if the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff	
(Omitted)
```
VIR_LDEV: ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
Delete the pair forcibly from the P-VOL.
When you perform this step, make sure not to delete the virtual LDEV ID, which allows the volume to be accessed from the server.
```
pairsplit -g oraHA -d dev1 -SFV -IH1
```
Check if the virtual LDEV ID of the P-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
(Omitted)
```
The VIR_LDEV information is not displayed if it is same as the LDEV information. If the virtual LDEV ID is deleted, set a correct LDEV ID.
Re-create the GAD pair by specifying the P-VOL.
```
paircreate -g oraHA -d dev1 -f
never -vl -jq 1 -IH1
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Therefore, set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.
```
raidcom map resource -ldev_id 0x1111
-virtual_ldev_id 0x1111 -IH1
```
The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.

If the alternate path to the P-VOL has been deleted, add the alternate path.
1. If you have detached the physical paths of the primary site, restore all physical paths to their original status, and then add the alternate path.
2. Using the alternate path software, add the alternate path deleted at step 1 to the P-VOL.

Setting correct virtual LDEV ID values

If a virtual LDEV ID value is not correct, you can set the correct value.

Procedure

Check the virtual LDEV ID.

raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
VIR_LDEV : 2222
(Omitted)

Delete the virtual LDEV ID.
```
raidcom unmap resource -ldev_id 0x1111 -virtual_ldev_id 0x2222 -IH1
```
- If the command execution result shows VIR_LDEV is ffff, specify reserve for -virtual_ldev_id to delete the virtual LDEV ID.
```
raidcom unmap resource -ldev_id 0x1111 -virtual_ldev_id reserve -IH1
```
- If the command execution result does not show VIR_LDEV, specify the LDEV ID for -virtual_ldev_id to delete the virtual LDEV ID.
Confirm that the virtual LDEV ID is deleted.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
VIR_LDEV : fffe
(Omitted)
```
VIR_LDEV : fffe indicates that the virtual LDEV ID is deleted.
Set the LDEV ID as the virtual LDEV ID.
If you set a virtual LDEV ID which is different from the LDEV ID, set the virtual LDEV ID you set.
```
raidcom map resource -ldev_id 0x1111 -virtual_ldev_id 0x1111 -IH1
```
If you want to set GAD reserve, specify reserve for -virtual_ldev_id.
```
raidcom map resource -ldev_id 0x1111 -virtual_ldev_id reserve -IH1
```
Check the virtual LDEV ID.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
(Omitted)
```
VIR_LDEV information is not displayed if it is same as the LDEV information.

Pair condition and recovery: secondary storage system failure

When the secondary storage system fails, you can correct the issue by recovering the secondary storage system, recovering the physical path between primary and secondary storage systems, and recovering the pair.

GAD pairs might be created between the primary and secondary sites that configure GAD as follows:

A P-VOL of a GAD pair is at site 1 and a S-VOL is at site 2.
A P-VOL of a different GAD pair is at site 2 and a S-VOL is at site 1.

In this case, recover from a failure at the primary storage system. For more information, see Pair condition and recovery: primary storage system failure.

The following table shows transitions for pair status and I/O mode, the volumes accessible from the server, and location of the latest data when you can no longer use the secondary storage system due to failure.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server
P-VOL	S-VOL	P-VOL	S-VOL¹	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror (RL))	PAIR (Mirror (RL))	PSUE (Local)²	PSUE (Block)	OK	NG	P-VOL
PAIR (Mirror (RL))	PAIR (Block)	PSUE (Local)	PSUE (Block)	OK	NG	P-VOL
COPY (Mirror (RL))	COPY (Block)	PSUE (Local)	PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	NG	S-VOL
Notes: If shared memory in the secondary storage system becomes volatilized, the S-VOL pair status changes to SMPL, and the reserve attribute is set for the volume, which prevents host access to the volume. If the server does not issue write I/O, the pair status might be PAIR (Mirror (RL)).

SIMs

Primary storage system: 2180xx, DD0xyy, DD3xyy
Secondary storage system: SIM varies depending on the failure type

Procedure

When the secondary storage system is powered off, delete an alternate path (logical path) to the S-VOL, and then turn on the power.
1. Using the alternate path software, distinguish the volumes which are not able to be accessed from the server.
2. Using the alternate path software, delete the alternate paths to the S-VOL.
  If you cannot delete the alternate paths, detach all physical paths which are connected to the server at the secondary site.
Turn on the secondary storage system.
Recover the secondary storage system.

For details, contact customer support.
Recover the physical path between the primary storage system and the secondary storage system.
If P-VOL pair status is PAIR, suspend the pair specifying the P-VOL.
Confirm the quorum disk status. If the quorum disk is blocked, recover from the blockade. If you do not set a volume for the quorum disk, you can skip this step.

Resynchronize or re-create the pair using the following procedure whose pair status and I/O mode match your pair's status and I/O mode.

If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

If the failed secondary site contains a S-VOL, recover from a failure in accordance with the following procedure. If the failed secondary site also contains a P-VOL, recover from a failure at the primary site. For more information, see Pair condition and recovery: primary storage system failure.

Pair status		I/O mode
P-VOL	S-VOL	P-VOL	S-VOL
PSUS/PSUE	PSUS/PSUE	Local	Block

Resynchronize the pair specifying the P-VOL.
```
pairresync -g oraHA -d dev1 -IH1
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Pair status		I/O mode
P-VOL	S-VOL	P-VOL	S-VOL
PSUS/PSUE	SMPL	Local	Not applicable

Check if the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -
fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff
(Omitted)
```
VIR_LDEV : ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
Delete the pair forcibly from the P-VOL.

When you perform this step, make sure not to delete the virtual LDEV ID, which allows the volume to be accessed from the server.
```
pairsplit -g oraHA -d dev1 -SFV -IH1
```
Check if the virtual LDEV ID of the P-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x1111 -
fx -IH1
(Omitted)
LDEV : 1111
(Omitted)
```
The VIR_LDEV information is not displayed if it is same as the LDEV information. If virtual LDEV ID is deleted, set a correct virtual LDEV ID.
Re-create the pair specifying the P-VOL.
```
paircreate -g oraHA -d dev1 -f
never -vl -jq 1 -IH1
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.

raidcom map resource -ldev_id 0x2222 
-virtual_ldev_id 0x1111 -IH1

The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.

Pair status		I/O mode
P-VOL	S-VOL	P-VOL	S-VOL
PSUS/PSUE	SSWS	Block	Local

Resynchronize the pair specifying the S-VOL.
```
pairresync -g oraHA -d dev1 -swaps -IH2
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Pair status		I/O mode
P-VOL	S-VOL	P-VOL	S-VOL
PSUS/PSUE	SMPL	Block	Not applicable

Delete the pair forcibly from the P-VOL.

When you perform this step, ensure that you do not delete the virtual LDEV ID, which allows access to the volume from the server.
```
pairsplit -g oraHA -d dev1 -SF -IH1
```
Check if the virtual LDEV ID of the P-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x1111 -
fx -IH1
(Omitted)
LDEV : 1111
VIR_LDEV : ffff
(Omitted)
```
VIR_LDEV : ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
Check if the virtual LDEV ID of the S-VOL i s not deleted.
```
raidcom get ldev -ldev_id 0x2222 -
fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : 1111
(Omitted)
```
If the virtual LDEV ID is deleted, set a correct virtual LDEV ID.

NoteIf shared memory in the secondary storage system becomes volatized, the S-VOL pair status changes the SMPL and GAD reserve is assigned to the virtual attribute.
Re-create the pair specifying the S-VOL.
```
paircreate -g oraHA -d dev1 -f
never -vl -jq 1 -IH2
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

If you accidentally delete the virtual LDEV ID of the S-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the S-VOL again using the raidcom map resource command.

raidcom map resource -ldev_id 0x2222 
-virtual_ldev_id 0x1111 -IH1

The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID of the P-VOL using the confirmation command.

(VSP 5000 series)

Pair status		I/O mode
P-VOL	S-VOL	P-VOL	S-VOL
PSUE	PSUE	Block	Block

Delete the pair forcibly from the S-VOL.

When you perform this step, delete the virtual LDEV ID so that the volume cannot be accessed from the server.
```
pairsplit -g oraHA -d dev1 -RF -IH2
```
Check if the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff	
(Omitted)
```
VIR_LDEV : ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
Delete the pair forcibly from the P-VOL.
When you perform this step, make sure not to delete the virtual LDEV ID, which allows the volume to be accessed from the server.
```
pairsplit -g oraHA -d dev1 -SFV -IH1
```
Check if the virtual LDEV ID of the P-VOL i s not deleted.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
(Omitted)
```
The VIR_LDEV information is not displayed if it is same as the LDEV information. If the virtual LDEV ID is deleted, set a correct LDEV ID.

Re-create the pair specifying the P-VOL.

paircreate -g oraHA -d dev1 -f
never -vl -jq 1 -IH1

If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.

raidcom map resource -ldev_id 0x1111 
-virtual_ldev_id 0x1111 -IH1

The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.

If the alternate path to the S-VOL has been deleted, add the alternate path.
1. If you have detached the physical paths of the secondary site, restore all physical paths to their original status, and then add the alternate path.
2. Using the alternate path software, add the alternate path deleted at step 1 to the S-VOL.

Pair condition and recovery: path failure, primary to external storage system

If the status of the external storage system is normal, a failure might have occurred in a physical path from the primary or secondary storage system to the external storage system, or a switch. Recover from the failure that occurred in the physical path or switch.

The following table shows transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when you can no longer use any physical path from the primary storage system to the quorum disk's external storage system.

GAD pairs might be created between the primary and secondary sites that configure GAD as follows:

A P-VOL of a GAD pair is at site 1 and a S-VOL is at site 2.
A P-VOL of a different GAD pair is at site 2 and a S-VOL is at site 1.

In this case, recover from a physical path failure at the secondary site. For more information, see Pair condition and recovery: path failure, secondary to external storage system.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server
P-VOL	S-VOL	P-VOL	S-VOL	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror(RL))¹	PAIR (Mirror(RL))¹	PAIR (Mirror (RL))	PAIR (Block)	OK	NG	Both P-VOL and S-VOL
PAIR (Mirror(RL))	PAIR (Mirror(RL))	PAIR (Mirror(RL))	PAIR (Mirror(RL))	OK	OK	Both P-VOL and S-VOL
PAIR (Mirror (RL))	PAIR (Block)	PAIR (Mirror (RL))	PAIR (Block)	OK	NG	Both P-VOL and S-VOL
COPY (Mirror(RL))	COPY (Block)	COPY (Mirror(RL))²	COPY (Block)²	OK	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
Notes: The pair status and I/O mode after failure depends on the requirement of the pair. For details, see Server I/Os and data mirroring with blocked quorum disk or without quorum disk volumes. The P-VOL might change to PSUE (Local) and the S-VOL might change to PSUE (Block) if a failure occurred on a physical path to an external storage system for quorum disks immediately after the pair status changed to COPY.

SIMs

Primary storage system: 21D0xy, 21D2xx, DD2xyy, DEF0zz, EF5xyy, EFD000, FF5xyy
Secondary storage system: DD2xyy

Procedure

Recover the paths to the external storage system.
Resynchronize the pair suspended by a failure.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering the path: primary to external storage system (pair status: PAIR)

The following figure shows the failure area and recovery when the GAD status of a pair changes to Suspended or Quorum disk blocked due to path failure from the primary storage system to the external storage system.

When the GAD status of the pair changes to Suspended, the P-VOL I/O mode changes to Local and the S-VOL I/O mode changes to Block. Server I/O continues on the P-VOL. When the GAD status of the pair changes to Quorum disk blocked, the P-VOL I/O mode remains Mirror (RL), and the S-VOL I/O mode changes to Block. Server I/O continues on the P-VOL.

Procedure

Recover the path to the external storage system.
1. Reconnect the physical path or reconfigure the SAN to recover the path to the external storage system.
  After the path is recovered, the remote path is automatically recovered.
2. Confirm that the external storage system is connected correctly.
```
raidcom get path -path_grp 1 -IH0
PHG GROUP STS CM IF MP# PORT   WWN                 PR LUN PHS  Serial# PRODUCT_ID LB PM DM
  1 1-1   NML E  D    0 CL5-A  50060e8008000140     1   0 NML    55555 VSP 5000 series  N  M D
```
3. Confirm the LDEV ID of the quorum disk by obtaining the information of the external volume from the primary storage system.
```
raidcom get external_grp -external_grp_id 1-1 -IH0
T GROUP  P_NO  LDEV#   STS         LOC_LBA        SIZE_LBA            Serial#
E 1-1       0   9999   NML  0x000000000000  0x000003c00000             555555
```
4. Confirm that the primary storage system recognizes the external volume as a quorum disk by specifying the LDEV ID of the quorum disk.
```
raidcom get ldev -ldev_id 0x9999 -fx -IH0 
(snip) 
QRDID : 0 
QRP_Serial# : 522222 
QRP_ID : R9 
(snip)
```
  NoteVSP 5000 series is displayed as R9 in command output. VSP Fx00 models and VSP Gx00 models are displayed as M8 in command output.

If the GAD status of the pair is Quorum disk blocked: The pair changes to Mirrored status automatically.

If the GAD status of the pair is Suspended: Resynchronize the pair as follows.

Confirm that the I/O mode of the P-VOL is Local.

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PSUE 
NEVER ,  100  4444 -   -   0  -  -            -       - L/L
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PSUE 
NEVER ,  100  2222 -   -   0  -  -            -       - B/B

At the primary storage system, resynchronize the pair.
```
pairresync -g oraHA -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Confirm that the P-VOL and S-VOL pair statuses change to PAIR (Mirror (RL)).

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M

Using the alternate path software, resume I/O to the volume that could not be accessed from the server (I/O might resume automatically).

Pair condition and recovery: path failure, secondary to external storage system

If the status of the external storage system is normal, a failure might have occurred in a physical path from the primary or secondary storage system to the external storage system, or a switch. Recover from the failure that occurred in the physical path or switch.

The following table shows transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when you can no longer use any physical path from the secondary storage system to the quorum disk's external storage system.

GAD pairs might be created between the primary and secondary sites that configure GAD as follows:

A P-VOL of a GAD pair is at site 1 and a S-VOL is at site 2.
A P-VOL of a different GAD pair is at site 2 and a S-VOL is at site 1.

In this case, recover from a physical path failure at the primary site. For more information, see Pair condition and recovery: path failure, primary to external storage system.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server
P-VOL	S-VOL	P-VOL	S-VOL¹	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror(RL))¹	PAIR (Mirror(RL))¹	PAIR (Mirror(RL))	PAIR (Block)	OK	NG	Both P-VOL and S-VOL
PAIR (Mirror(RL))	PAIR (Mirror(RL))	PAIR (Mirror(RL))	PAIR (Mirror(RL))	OK	OK	Both P-VOL and S-VOL
PAIR (Mirror(RL))	PAIR (Block)	PAIR (Mirror(RL))	PAIR (Block)	OK	NG	Both P-VOL and S-VOL
COPY (Mirror(RL))	COPY (Block)	COPY (Mirror(RL))²	COPY (Block)²	OK	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
Notes: The pair status and I/O mode after failure depends on the requirement of the pair. For details, see Server I/Os and data mirroring with blocked quorum disk or without quorum disk volumes. The P-VOL might change to PSUE (Local) and the S-VOL might change to PSUE (Block) if a failure occurred on a physical path to an external storage system for quorum disks immediately after the pair status changed to COPY.

SIMs

Primary storage system: DD2xyy
Secondary storage system: 21D0xy, 21D2xx, DD2xyy, DEF0zz, EF5xyy, EFD000, FF5xyy

Procedure

Recover the paths to the external storage system.
Resynchronize the pair suspended by a failure.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering the path: secondary to external storage system (pair status: PAIR)

When a failure occurs on the path between the secondary storage system and the external storage system, you should know how to resolve the failure and restore the GAD pair. The following figure shows the failure area and recovery when the GAD status of a pair changes to Suspended or Quorum disk blocked due to path failure from the secondary storage system to the external storage system.

When the GAD status of the pair changes to Suspended, the P-VOL I/O mode changes to Local and the S-VOL I/O mode changes to Block. Server I/O continues on the P-VOL. When the GAD status of the pair changes to Quorum disk blocked, the P-VOL I/O mode remains Mirror (RL), and the S-VOL I/O mode changes to Block. Server I/O continues on the P-VOL.

Procedure

Recover the path to the external storage system.
1. Reconnect the physical path or reconfigure the SAN to recover the path to the external storage system.
  After the path is recovered, the remote path is automatically recovered.
2. Confirm that the external storage system is connected correctly.
  (VSP 5000 series)
```
raidcom get path -path_grp 1 -IH1
PHG GROUP STS CM IF MP# PORT   WWN                 PR LUN PHS  Serial# PRODUCT_ID LB PM DM
  1 1-1   NML E  D    0 CL5-C  50060e8008000160     1   0 NML   555555 VSP 5000 series  N  M D
```
  (VSP Gx00 models and VSP Fx00 models)
```
raidcom get path -path_grp 1 -IH1
PHG GROUP STS CM IF MP# PORT   WWN                 PR LUN PHS  Serial# PRODUCT_ID LB PM
1 1-2     NML E  D    0 CL5-C  50060e8007823521     1   0 NML   433333   VSP Gx00 N  M
```
3. Confirm the LDEV ID of the quorum disk by obtaining the information of the external volume from the secondary storage system.
```
raidcom get external_grp -external_grp_id 1-2 -IH1
T GROUP  P_NO  LDEV#   STS         LOC_LBA        SIZE_LBA            Serial#
E 1-2       0   9999   NML  0x000000000000  0x000003c00000             555555
```
4. Confirm that the secondary storage system recognizes the external volume as a quorum disk by specifying the LDEV ID of the quorum disk.
```
raidcom get ldev -ldev_id 0x8888 -fx -IH1
(snip)
QRDID : 0
QRP_Serial# : 511111
QRP_ID : R9
(snip)
```
  NoteVSP 5000 series is displayed as R9 in command output. VSP Fx00 models and VSP Gx00 models are displayed as M8 in command output.

If the GAD status of the pair is Quorum disk blocked: The pair changes to Mirrored status automatically.

If the GAD status of the pair is Suspended: Resynchronize the pair as follows.

Confirm that the P-VOL I/O mode is Local.

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PSUE 
NEVER ,  100  4444 -   -   0  -  -            -       - L/L
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PSUE 
NEVER ,  100  2222 -   -   0  -  -            -       - B/B

At the primary storage system, resynchronize the pair.
```
pairresync -g oraHA -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Confirm that the P-VOL and S-VOL pair statuses change to PAIR (Mirror (RL)).

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M

Using the alternate path software, resume I/O to the S-VOL (I/O might resume automatically).

Pair condition and recovery: quorum disk failure

GAD pairs might be created between the primary and secondary sites that configure GAD as follows:

A P-VOL of a GAD pair is at site 1 and a S-VOL is at site 2.
A P-VOL of a different GAD pair is at site 2 and a S-VOL is at site 1.

In this case, recover from a physical path failure.

The following table shows transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when you can no longer use the quorum disk volume.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server
P-VOL	S-VOL	P-VOL	S-VOL¹	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror(RL))¹	PAIR (Mirror(RL))¹	PAIR (Mirror(RL))	PAIR (Block)	OK	NG	Both P-VOL and S-VOL
PAIR (Mirror(RL))	PAIR (Mirror(RL))	PAIR (Mirror(RL))	PAIR (Mirror(RL))	OK	OK	Both P-VOL and S-VOL
PAIR (Mirror (RL))	PAIR (Block)	PAIR (Mirror(RL))	PAIR (Block)	OK	NG	Both P-VOL and S-VOL
COPY (Mirror(RL))	COPY (Block)	COPY (Mirror(RL))²	COPY (Block)²	OK	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
Notes: The pair status and I/O mode after failure depends on the requirement of the pair. For details, see Server I/Os and data mirroring with blocked quorum disk or without quorum disk volumes. The P-VOL might change to PSUE (Local) and the S-VOL might change to PSUE (Block) if a quorum disk failure occurred immediately after the pair status changed to COPY.

SIMs

Primary storage system: 21D0xy, 21D2xx, DD2xyy, DEF0zz, EF5xyy, EFD000, FF5xyy
Secondary storage system: 21D0xy, 21D2xx, DD2xyy, DEF0zz, EF5xyy, EFD000, FF5xyy

Procedure

Recover the quorum disk.
Resynchronize or re-create GAD pairs if they are suspended by a failure.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering the quorum disk (pair status: PAIR)

When the GAD status of the pair changes to Suspended, the P-VOL I/O mode changes to Local and the S-VOL I/O mode changes to Block. Server I/O continues on the P-VOL. When the GAD status of the pair changes to Quorum disk blocked, the P-VOL I/O mode remains Mirror (RL), and the S-VOL I/O mode changes to Block. Server I/O continues on the P-VOL.

The following figure shows the failure area and recovery when a pair is suspended due to quorum disk failure.

NoteIn this example no consistency group is specified.

NoteThe following procedure is also used for re-creating the quorum disk when it has been mistakenly reformatted.

NoteSteps 1 and 2 below describe the recovery procedure for an external storage system made by Hitachi, for example, VSP G1000. If you are using another vendor's storage system as external storage, follow the vendor's recovery procedure for the storage system. When you complete the recovery procedure for the external storage system, start the following procedure at step 3.

Procedure

On the external storage system, recover the quorum disk.
1. Block the quorum disk.
2. Format the quorum disk.
  If the quorum disk recovers after formatting, go to step h.
  If the quorum disk does not recover after formatting, continue to step c.
  
  NoteYou can recover the quorum disk by replacing its external storage system with a new one while keeping the GAD pair.
3. Confirm the following information about the quorum disk:
  - Vendor
  - Machine name
  
  - Volume properties
  
  - Device ID (if the information is valid)
  
  - Serial number
  
  - SSID
  
  - Product ID
  
  - LBA capacity (the capacity must be larger than the quorum disk before the failure occurred)
  
  - CVS attribute
  
  For details about confirming this information, see the Hitachi Universal Volume Manager User Guide.
  
  For details about confirming the CVS attribute, see the following table.
4. Delete the LU path to the quorum disk.
5. Delete the volume that is used as the quorum disk.
6. Create a new volume.
  For the LDEV ID, set the same value as the LDEV ID of the quorum disk that has been used since before the failure occurred. If you cannot set the same value, go to step 3.
  Also set the same values for the following information as the values that were used before the failure occurred. If you cannot set the same value, go to step 3.
  
  - Vendor
  
  - Machine name
  
  - Volume properties
  
  - Device ID (if the information is valid)
  
  - Serial number
  
  - SSID
  
  - Product ID
  
  - LBA capacity (the capacity must be larger than the quorum disk before the failure occurred)
  
  - CVS attribute
  
  For details about confirming this information, see the Hitachi Universal Volume Manager User Guide. For details about confirming the CVS attribute, see the following tables.
7. Set an LU path to the new volume.
  For the LU number, set the same value as the LU number of the quorum disk that was used since before the failure occurred. If you cannot set the same value, go to step 3.
8. Reconnect the external storage system or the quorum disk to the primary and secondary storage systems.

If the GAD status of the pair is Quorum disk blocked: The pair changes to the Mirrored status automatically.

If the GAD status of the pair is Suspended: Resynchronize the pair as follows.

Confirm that the P-VOL I/O mode is Local.

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PSUE 
NEVER ,  100  4444 -   -   0  -  -            -       - L/L
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PSUE 
NEVER ,  100  2222 -   -   0  -  -            -       - B/B

On the primary storage system, resynchronize the pair.
```
pairresync -g oraHA -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Confirm that the P-VOL and S-VOL pair status has changed to PAIR (Mirror (RL)). If so, go to step 4.

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M

Re-create the pairs.

On the primary storage system, delete all pairs that use the quorum disk where the failure occurred.
```
pairsplit -g oraHA -S -d dev1 -IH0
```

Confirm that the pairs were deleted.

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU), Seq#,   LDEV#.P/S,Status,Fence,   %,    P-LDEV# M   CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)      (CL1-A-0, 0,   0)511111  2222.SMPL  ---- ------,  ----- -----   -   -   -   -  -        -      -       -/-
oraHA   dev1(R)      (CL1-C-1, 0,   0)522222  4444.SMPL  ---- ------,  ----- -----   -   -   -   -  -        -      -       -/-

On the primary and secondary storage systems, delete the quorum disk.
On the primary and secondary storage systems, add a quorum disk.
On the primary storage system, create the pairs.
```
paircreate -g oraHA -f never -vl -jq 0 -d dev1 -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Confirm that the P-VOL and S-VOL pair statuses change to PAIR (Mirror (RL)).

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M

Using the alternate path software, resume I/O to the S-VOL (I/O might resume automatically).

Next steps

NoteWhen the external storage system is installed at the primary site, if a failure occurs in both the primary storage system and the external storage system, forcibly delete the pair on the secondary storage system, and then re-create the pair. For details, see Recovering the storage systems: primary site failure with external storage system.

Confirming the CVS attribute on the external storage system
Interface	To confirm the CVS attribute:
HCS	Open the Logical device window, and then confirm whether the CVS attribute is displayed in the Emulation type column for the LDEV that is being used as the quorum disk.
CCI	Execute the `raidcom get ldev` command from CCI to the LDEV which is used as quorum disk by external storage system, and then confirm whether the CVS attribute is output for `VOL_TYPE`. Details of the `raidcom get ldev` command; refer to Command Control Interface Command Reference.
Web Console*	Confirm whether the CVS attribute is displayed in the CVS column on the LUN Management window.
* Ask the maintenance personnel to operate the Web Console.

Conditions for the CVS attribute for volumes created in the external storage system
Interface	Condition			CVS attribute
HDvM - SN HCS CCI	Internal volume or external volume			Allowed
	HDP-VOL	VSP G1x00, VSP F1500, or VSP 5000 series VSP E990, or VSP Gx00 models and VSP Fx00 models or later		Allowed
		VSP or earlier HUS VM or earlier	Create LDEV of maximum size	Not allowed
		VSP or earlier HUS VM or earlier	Create LDEV less than maximum size	Allowed
Web Console*	The LDEV is created during the operation of the installation of Define Config & Install or ECC/LDEV, which remains the initial value of the Number of LDEVs on the Device Emulation Type Define window.			Not allowed
Web Console*	Other than above			Allowed
* Ask the maintenance personnel to operate the Web Console.

Recovering from quorum disk failures

You can recover from quorum disk failures without deleting GAD pairs.

Replacing a failed external storage system with a new one

By replacing a failed external storage system with a new one, you can recover from quorum disk failures without deleting GAD pairs.

CautionIf a GAD pair is not specified for the same quorum disk ID, delete the quorum disk first, and then re-create a quorum disk. When a GAD pair is not specified for the same quorum disk ID, if you replace the external volume for the quorum disk with a new one, the replacement might fail. When you re-create a quorum disk, create GAD pairs if necessary.

Procedure

Prepare a new quorum disk.
1. Format the disk of a new external storage system.
2. Map the formatted disk to the primary and secondary storage systems.
  Use the same procedure you use for creating a quorum disk. However, you do not need to set an external volume for the quorum disk.

Check the status of the quorum disk.

On the primary storage system, confirm that the status of the quorum disk is BLOCKED.

(VSP 5000 series)

raidcom get quorum -quorum_id 1 -IH0
QRDID : 1
LDEV : 2045
QRP_Serial# : 511111
QRP_ID : R9
Timeout(s) : 30
STS : BLOCKED

(VSP Gx00 models and VSP Fx00 models)

raidcom get quorum -quorum_id 1 -IH0
QRDID : 1
LDEV : 2045
QRP_Serial# : 411111
QRP_ID : M8
Timeout(s) : 30
STS : BLOCKED

On the secondary storage system, confirm that the status of the quorum disk is BLOCKED.

(VSP 5000 series)

raidcom get quorum -quorum_id 1 -IH1
QRDID : 1
LDEV : 2045
QRP_Serial# : 522222
QRP_ID : R9
Timeout(s) : 30
STS : BLOCKED

(VSP Gx00 models and VSP Fx00 models)

raidcom get quorum -quorum_id 1 -IH1
QRDID : 1
LDEV : 2045
QRP_Serial# : 422222
QRP_ID : M8
Timeout(s) : 30
STS : BLOCKED

Check the pair operation mode for the blocked quorum disk.
Depending on the check results, you might have to split the GAD pair.
1. If the QM column output with pairdisplay -fcxe command is AA, the GAD pair is split.
  Go to step 4 if this is the case.
  If the QM column is other than AA, the GAD pair is not split in most cases. Go to Step b.
```
# pairdisplay -g oraHA -fcxe -d dev0
Group   PairVol(L/R)(Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W QM
oraHA   dev0(L)     (CL1-C-0, 0,0)511111   400.P-VOL PAIR 
NEVER,  100   500  -   -   0  1  -      -       - L/M AA
oraHA   dev0(R)     (CL7-C-0,28,0)522222   500.S-VOL PAIR 
NEVER,  100   400  -   -   0  1  -      -       - L/M AA
```
2. Split the GAD pair if it is not already split.
```
pairsplit -g oraHA -IH0
```
Replace the external volume for the quorum disk.
1. On the primary storage system, replace the current external volume for the quorum disk with a new one.
```
raidcom replace quorum -quorum_id 1 -ldev_id 1234 -IH0
```
2. On the primary storage system, confirm that the status of the quorum disk is REPLACING.
  
  (VSP 5000 series)
```
raidcom get quorum -quorum_id 1 -IH0
QRDID : 1
LDEV : 1234
QRP_Serial# : 511111
QRP_ID : R9
Timeout(s) : 30
STS : REPLACING
```
  (VSP Gx00 models and VSP Fx00 models)
```
raidcom get quorum -quorum_id 1 -IH0
QRDID : 1
LDEV : 1234
QRP_Serial# : 411111
QRP_ID : M8
Timeout(s) : 30
STS : REPLACING
```
3. On the secondary storage system, replace the current external volume for the quorum disk with a new one.
```
 raidcom replace quorum -quorum_id 1 -ldev_id 1234 -IH1
```
4. On the secondary storage system, confirm that the status of the quorum disk is REPLACING.
  
  (VSP 5000 series)
```
raidcom get quorum -quorum_id 1 -IH1
QRDID : 1
LDEV : 1234
QRP_Serial# : 522222
QRP_ID : R9
Timeout(s) : 30
STS : REPLACING
```
  (VSP Gx00 models and VSP Fx00 models)
```
raidcom get quorum -quorum_id 1 -IH1
QRDID : 1
LDEV : 1234
QRP_Serial# : 422222
QRP_ID : M8
Timeout(s) : 30
STS : REPLACING
```
  Note If the raidcom replace quorum command is executed normally, the status of the quorum disk changes from BLOCKED to REPLACING in a few seconds. If the status does not change in a few minutes, contact customer support.
Resynchronize the GAD pair you previously split.
```
pairresync -g oraHA -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Confirm that the status of the quorum disk is NORMAL.

On the primary storage system, confirm that the status of the quorum disk is NORMAL.

(VSP 5000 series)

raidcom get quorum -quorum_id 1 -IH0
QRDID : 1
LDEV : 1234
QRP_Serial# : 511111
QRP_ID : R9
Timeout(s) : 30
STS : NORMAL

(VSP Gx00 models and VSP Fx00 models)

raidcom get quorum -quorum_id 1 -IH0
QRDID : 1
LDEV : 1234
QRP_Serial# : 411111
QRP_ID : M8
Timeout(s) : 30
STS : NORMAL

On the secondary storage system, confirm that the status of the quorum disk is NORMAL.

(VSP 5000 series)
```
raidcom get quorum -quorum_id 1 -IH1
QRDID : 1
LDEV : 1234
QRP_Serial# : 522222
QRP_ID : R9
Timeout(s) : 30
STS : NORMAL
```
(VSP Gx00 models and VSP Fx00 models)
```
raidcom get quorum -quorum_id 1 -IH1
QRDID : 1
LDEV : 1234
QRP_Serial# : 422222
QRP_ID : M8
Timeout(s) : 30
STS : NORMAL
```
If the raidcom replace quorum command is executed normally, the status of the quorum disk changes from REPLACING to NORMAL in a minute.

If the status does not change in five minutes, check whether remote paths between the storage systems are in Normal state.
Confirm that the GAD pair you resynchronized in step 5 is synchronized normally.
If the status of the replaced quorum disk is FAILED, the primary storage system and the secondary storage system might be connected to different quorum disks.
1. Specify the external volume so that the primary storage system and the secondary storage system are connected to the same quorum disk.
2. After specifying the correct external volume, perform steps 5 through 8.

Recovering from the FAILED status

When the primary storage system and the secondary storage system are connected to different quorum disks, the status of the quorum disk shows FAILED. If this happens, disconnect the storage systems from the quorum disk first, and then replace the external volume for the quorum disk with the new one.

Procedure

Check the status of the quorum disk.
1. On the primary site storage system, disconnect the connection to the quorum disk.
  
  (VSP 5000 series)
```
raidcom disconnect external_grp -ldev_id 0x2045 -IH0
```
  (VSP Gx00 models and VSP Fx00 models)
```
raidcom disconnect external_grp -ldev_id 0x2045 -IH0
```
2. On the secondary storage system, disconnect the connection to the quorum disk.
  
  (VSP 5000 series)
```
raidcom disconnect external_grp -ldev_id 0x2045 -IH1
```
  (VSP Gx00 models and VSP Fx00 models)
```
raidcom disconnect external_grp -ldev_id 0x2045 -IH1
```

Confirm that the primary storage system and the secondary storage system are disconnected from the quorum disk.

On the primary storage system, confirm that the connection with the quorum disk is disconnected.

(VSP 5000 series)

raidcom get path -path_grp 1 -IH0
PHG GROUP STS CM IF MP# PORT WWN PR LUN PHS
Serial# PRODUCT_ID LB PM DM
1 1-1 DSC E D 0 CL5-A 50060e8008000140 1 0 NML
55555 VSP 5000 series N M D

(VSP Gx00 models and VSP Fx00 models)

raidcom get path -path_grp 1 -IH0
PHG GROUP STS CM IF MP# PORT WWN PR LUN PHS
Serial# PRODUCT_ID LB PM
1 1-1 DSC E D 0 CL5-A 50060e8007823520 1 0 NML
433333 VSP Gx00 N M

On the secondary storage system, confirm that the connection with the quorum disk is disconnected.

(VSP 5000 series)

raidcom get path -path_grp 1 -IH1
PHG GROUP STS CM IF MP# PORT WWN PR LUN PHS
Serial# PRODUCT_ID LB PM
1 1-2 DSC E D 0 CL5-C 50060e8008000140 1 0 NML
55555 VSP 5000 series N M

(VSP Gx00 models and VSP Fx00 models)

raidcom get path -path_grp 1 -IH1
PHG GROUP STS CM IF MP# PORT WWN PR LUN PHS
Serial# PRODUCT_ID LB PM
1 1-2 DSC E D 0 CL5-C 50060e8007823521 1 0 NML
433333 VSP Gx00 N M

Replace the external volume for the quorum disk with a new one.
1. On the primary storage system, replace the current external volume for the quorum disk with a new one.
  
  (VSP 5000 series)
```
raidcom replace quorum -quorum_id 1 -ldev_id 1234 -IH0
```
  (VSP Gx00 models and VSP Fx00 models)
```
raidcom replace quorum -quorum_id 1 -ldev_id 1234 -IH0
```
2. On the primary storage system, confirm that the status of the quorum disk is REPLACING.
  
  (VSP 5000 series)
```
raidcom get quorum -quorum_id 1 -IH0
QRDID : 1
LDEV : 1234
QRP_Serial# : 511111
QRP_ID : R9
Timeout(s) : 30
STS : REPLACING
```
  (VSP Gx00 models and VSP Fx00 models)
```
raidcom get quorum -quorum_id 1 -IH0
QRDID : 1
LDEV : 1234
QRP_Serial# : 411111
QRP_ID : M8
Timeout(s) : 30
STS : REPLACING
```
3. On the secondary storage system, replace the current external volume for the quorum disk with a new one.
  
  (VSP 5000 series)
```
raidcom replace quorum -quorum_id 1 -ldev_id 1234 -IH1
```
  (VSP Gx00 models and VSP Fx00 models)
```
raidcom replace quorum -quorum_id 1 -ldev_id 1234 -IH1
```
4. On the secondary storage system, confirm that the status of the quorum disk is REPLACING.
  
  (VSP 5000 series)
```
raidcom get quorum -quorum_id 1 -IH1
QRDID : 1
LDEV : 1234
QRP_Serial# : 522222
QRP_ID : R9
Timeout(s) : 30
STS : REPLACING
```
  (VSP Gx00 models and VSP Fx00 models)
```
raidcom get quorum -quorum_id 1 -IH1
QRDID : 1
LDEV : 1234
QRP_Serial# : 422222
QRP_ID : M8
Timeout(s) : 30
STS : REPLACING
```
  Note When the raidcom replace quorum command is executed normally, the quorum disk status changes to REPLACING in a few seconds. If it does not change from FAILED in a few minutes, contact customer support.

Pair condition and recovery: external storage system failure

The following table shows transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when you can no longer use the external storage system.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server
P-VOL	S-VOL	P-VOL	S-VOL	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror(RL))¹	PAIR (Mirror(RL))¹	PAIR (Mirror(RL))	PAIR (Block)	OK	NG	Both P-VOL and S-VOL
PAIR (Mirror(RL))	PAIR (Mirror(RL))	PAIR (Mirror(RL))	PAIR (Mirror(RL))	OK	OK	Both P-VOL and S-VOL
PAIR (Mirror(RL))	PAIR (Block)	PAIR (Mirror(RL))	PAIR (Block)	OK	NG	Both P-VOL and S-VOL
COPY (Mirror(RL))	COPY (Block)	COPY (Mirror(RL))²	COPY (Block)²	OK	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
Notes: The pair status and I/O mode after failure depends on the requirement of the pair. For details, see Server I/Os and data mirroring with blocked quorum disk or without quorum disk volumes. The P-VOL might change to PSUE (Local) and the S-VOL might change to PSUE (Block) if a failure occurred in an external storage system immediately after the pair status changed to COPY.

SIMs

Primary storage system: 21D0xy, 21D2xx, DD2xyy, DEF0zz, EF5xyy, EFD000, FF5xyy
Secondary storage system: 21D0xy, 21D2xx, DD2xyy, DEF0zz, EF5xyy, EFD000, FF5xyy

Procedure

Recover the external storage system. For details, contact the vendor.
Resynchronize or re-create GAD pairs if they are suspended by a failure.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Pair condition and recovery: other failures

Generally, you can correct failures by recovering the paths and resynchronizing the pair. The following table shows transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when a failure other than explained above occurs.

Before failure		After failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server
P-VOL	S-VOL	P-VOL	S-VOL	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror (RL))¹	PAIR (Mirror (RL))¹	PSUE (Local)	PSUE (Block)	OK	NG	P-VOL
		PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
		PAIR (Mirror (RL))	PAIR (Block)	OK	NG	Both P-VOL and S-VOL
PAIR (Mirror (RL))	PAIR (Block)	PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
COPY (Mirror (RL))	COPY (Block)	PSUE (Local)	PSUE/COPY (Block)	OK	NG	P-VOL
COPY (Mirror (RL))	COPY (Block)	PSUE (Local)	PSUE/COPY (Block)	NG²	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	OK	NG	P-VOL
PSUS/PSUE (Local)	SSUS/PSUE (Block)	PSUS/PSUE (Local)	SSUS/PSUE (Block)	NG²	NG	P-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	OK	S-VOL
PSUS/PSUE (Block)	SSWS (Local)	PSUS/PSUE (Block)	SSWS (Local)	NG	NG³	S-VOL
Notes: The pair status and I/O mode after failure depends on the requirement of the pair. For details, see Server I/Os and data mirroring with blocked quorum disk or without quorum disk volumes. Depending on the failure factor, if you cannot access the P-VOL, you cannot access the P-VOL or the S-VOL. Depending on the failure factor, if you cannot access the S-VOL, you cannot access the P-VOL or the S-VOL.

SIMs

Primary storage system: SIM varies depending on the failure type
Secondary storage system: SIM varies depending on the failure type

Procedure

Recover the system.
Resynchronize the pair.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovery procedure for GAD pair suspension due to other failures

A GAD pair might be suspended due to a failure other than those described in this chapter. Use the following procedure to recover a suspended pair from other types of failure.

If you are not able to restore the GAD volumes using this procedure, contact customer support.

Procedure

Recover from the failure.
1. Verify that a failure, such as a suspended GAD pair, has occurred, for example, by checking for SIMs issued by the primary or secondary storage system.
2. When a failure has occurred, identify the failure and perform troubleshooting according to the failure type to remove the cause of the failure.
Confirm the quorum disk status. If the quorum disk is blocked, recover from the blockade. If you do not set a volume for the quorum disk, you can skip this step.

Resynchronize the GAD pair.

Check the I/O mode of the P-VOL and the S-VOL of the suspended GAD pair.

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PSUE 
NEVER ,  100  4444 -   -   0  -  -            -       - L/L
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PSUE 
NEVER ,  100  2222 -   -   0  -  -            -       - B/B

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PSUE 
NEVER ,  100  2222 -   -   0  -  -            -       - B/B
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PSUE 
NEVER ,  100  4444 -   -   0  -  -            -       - L/L

If the I/O mode of the P-VOL is Local, resynchronize the GAD pair at the primary storage system.
```
pairresync -g oraHA -IH0
```
If the I/O mode of the S-VOL is Local, resynchronize the GAD pair at the secondary storage system.
```
pairresync -g oraHA -swaps -IH1
```
The volume in the primary storage system changes to an S-VOL, and the volume in the secondary storage system changes to a P-VOL.

Confirm that the pair status of the P-VOL and the S-VOL of the GAD pair has changed to PAIR (Mirror (RL)).

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M

Using the alternate path software, resume I/O to the S-VOL.
If necessary, reverse the P-VOL and the S-VOL.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering the storage systems: primary site failure with external storage system

If a failure occurs at the primary site in a configuration with the external storage system for the quorum disk located at the primary site, the failure might affect the primary storage system and the external storage system simultaneously. In this case, the GAD pair is suspended, and access to the GAD volumes stops.

Failure at the primary site (external storage system at the primary site)

Failure locations	Reference codes of SIMs that might be issued		Can the volume access to the GAD volumes?¹
Failure locations	Primary storage system	Secondary storage system	P-VOL	S-VOL
Both the primary storage system and the external storage system for the quorum disk	Depends on the failure type²	DD0`xyy` DD2`xyy` DD3`xyy` 2180`xx` 21D0`xx` 21D2`xx` EF5`xyy` EFD000 FF5`xyy` DEF0`zz`	No	No³
Notes: Hardware such as drives, cache, front-end director (CHB), back-end director (BED), and MPU is redundant in the storage system configuration. Even if a failure occurs in a part of redundant hardware, the failure does not cause a GAD pair being suspended, or an inaccessible GAD volume. The failure does not cause the GAD pair suspended, or the inaccessible GAD volume even if a failure occurs in a part of hardware, if the following physical paths are redundant. Between a server and a storage systems of the primary and secondary sites Between an external storage system and storage systems of the primary and secondary sites Between storage systems of the primary and secondary sites A SIM that corresponds to the failure type is issued. You might not be able to view SIMs according to the failure type. You can access the S-VOL, if the pair status of the S-VOL is SSWS, even if a failure occurs.

Procedure

Using the alternate path software, delete the alternate path to the GAD P-VOL.
At the secondary storage system, delete the GAD pair forcibly.

When deleting the pair forcibly, do not delete the virtual ID, which allows the volume to be accessed from the server.
```
pairsplit -g oraHA -d dev1 -RFV -IH2
```

Confirm that the virtual LDEV ID is not deleted.

raidcom get ldev -ldev_id 0x2222 -fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : 1111
(Omitted)

Confirm that the GAD pair is deleted.
Using the alternate path software, resume I/Os from the server to the GAD S-VOL.
Restore the primary storage system from the failure.
At the primary storage system, delete the GAD pair forcibly.

When deleting the pair forcibly, delete the LDEV ID so that the volume cannot be accessed from the server.

Depending on the failure type of the primary storage system, after the primary storage system is restored from a failure, the pair status of the P-VOL might change to SMPL, and the GAD reserve attribute might be set. In this case, you do not need to delete the GAD pair forcibly.
```
pairsplit -g oraHA -d dev1 -SF -IH1
```
Confirm that the virtual LDEV ID indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
VIR_LDEV : ffff
(Omitted)
```
VIR_LDEV : ffff indicates GAD reserve.
Confirm that the GAD pair is deleted.
Restore the external storage system from a failure.
From the primary and secondary storage systems, delete the quorum disk.

Depending on the failure type of the external storage system, after the external storage system is restored from a failure, a quorum disk can be deleted. In this case, you do not need to delete the quorum disk.
From the primary and secondary storage systems, add a quorum disk.
From the secondary storage system, re-create a GAD pair.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.If you accidentally delete the virtual LDEV ID of the S-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the S-VOL again using the raidcom map resource command.
```
raidcom map resource -ldev_id 0x2222 
-virtual_ldev_id 0x1111 -IH1
```
The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID of the P-VOL using the confirmation command.
Using the alternate path software, add a path to the GAD P-VOL, and then resume I/Os.
Reverse the P-VOL and the S-VOL if necessary.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Reversing the P-VOL and S-VOL

During disaster recovery operations, P-VOLs are changed to S-VOLs and S-VOLs to P-VOLs to reverse the flow of data from the secondary site to the primary site to restore the primary site. When normal operations are resumed at the primary site, the direction of copy is changed again so that the original P-VOLs become primary volumes again and the original S-VOLs become secondary volumes again with data flowing from the primary site to the secondary site.

Procedure

Using the alternate path software, stop I/O from the server to P-VOLs in the secondary storage system.

Continue to the next step even if the alternate path cannot be deleted.

Confirm that the P-VOL and the S-VOL have been reversed.

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.S-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.P-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

At the primary storage system, change the pair statuses of the S-VOLs to SSWS to suspend the pairs (swap suspension).
```
pairsplit -g oraHA -d dev1 -RS -IH0
```
At the primary storage system, reverse the P-VOL and the S-VOL, and then resynchronize the pairs (swap resync).
```
pairresync -g oraHA -d dev1 -swaps -IH0
```
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Confirm that the P-VOL and the S-VOL pair statuses change to PAIR (Mirror (RL)).

pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M

pairdisplay -g oraHA -fxce -IH1
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - L/M
oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
NEVER ,  100  4444 -   -   0  -  -            -       - L/M

Using the alternate path software, restart I/Os from the server to S-VOLs in the secondary storage system.

Creating GAD pairs when virtual LDEV IDs are deleted from the P-VOL and S-VOL

Some failure recovery operations, such as a primary storage system failure or secondary storage system failure, require you to delete GAD pairs. You might not be able to create them again until you assign a virtual LDEV ID.

Procedure

Confirm that the GAD reserve attribute is assigned to the P-VOL and the S-VOL by using the raidcom get ldev command.
raidcom get ldev -ldev_id 0x4444 -fx -IH0
```
LDEV : 4444 VIR_LDEV : ffff
```
raidcom get ldev -ldev_id 0x5555 -fx -IH1
```
LDEV : 5555 VIR_LDEV : ffff
```
If you execute the raidcom get ldev command for a volume that has the GAD reserve attribute, ffff is displayed for VIR_LDEV (virtual LDEV ID).
Delete all of the LU paths to the P-VOL.
Release the GAD reserve attribute of the P-VOL (LDEV ID: 0x4444) by using the raidcom unmap resource command.
raidcom unmap resource -ldev_id 0x4444 -virtual_ldev_id reserve -IH0
Display the information about the P-VOL by using the raidcom get ldev command.
raidcom get ldev -ldev_id 0x4444 -fx -IH0
```
LDEV : 4444 VIR_LDEV : fffe
```
For the volume whose GAD reserve attribute was released, a virtual LDEV ID is not assigned. If you execute the raidcom get ldev command for a volume to which a virtual LDEV ID is not assigned, fffe is displayed for VIR_LDEV (virtual LDEV ID).
Set a virtual LDEV ID for the P-VOL (LDEV ID: 0x4444) by using the raidcom map resource command.
raidcom map resource -ldev_id 0x4444 -virtual_ldev_id 0x4444 -IH0
Display the information about the P-VOL by using the raidcom get ldev command.
raidcom get ldev -ldev_id 0x4444 -fx -IH0
```
LDEV : 4444 VIR_LDEV : 4444
```
Check the virtual attributes of the P-VOL and the S-VOL by using the raidcom get ldev command.
raidcom get ldev -ldev_id 0x4444 -fx -IH0
```
LDEV : 4444
```
raidcom get ldev -ldev_id 0x5555 -fx -IH1
```
LDEV : 5555 VIR_LDEV : ffff
```
The virtual LDEV ID (0x4444) is assigned to the P-VOL (LDEV ID: 0x4444) and the GAD reserve attribute ( VIR_LDEV : ffff) is assigned to the S-VOL (LDEV ID: 0x5555).
Specify a port and host group for the P-VOL, and set the LU path again.
Create GAD pairs again.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Creating GAD pairs when virtual LDEV IDs are set for the P-VOL and S-VOL

Some failure recovery operations, such as a primary storage system failure or secondary storage system failure, require you to delete GAD pairs. You might not be able to create them again until you set a GAD reserve attribute as the virtual attribute of the S-VOL.

Procedure

Check the virtual attributes of the P-VOL and the S-VOL by using the raidcom get ldev command.
raidcom get ldev -ldev_id 0x4444 -fx -IH0
```
LDEV : 4444 
```
raidcom get ldev -ldev_id 0x5555 -fx -IH1
```
LDEV : 5555 
```
Delete all of the LU paths to the S-VOL.
Delete the virtual LDEV ID (0x5555) of the S-VOL (LDEV ID: 0x5555) by using the raidcom unmap resource command.
raidcom unmap resource -ldev_id 0x5555 -virtual_ldev_id 0x5555 -IH1
Display the information about the S-VOL (LDEV ID: 0x5555) by using the raidcom get ldev command.
raidcom get ldev -ldev_id 0x5555 -fx -IH1
```
LDEV : 5555 VIR_LDEV : fffe
```
If you execute the raidcom get ldev command for a volume to which a virtual LDEV ID is not assigned, fffe is displayed for VIR_LDEV (virtual LDEV ID).
Set the GAD reserve attribute as the virtual attribute of the S-VOL (LDEV ID: 0x5555) by using the raidcom map resource command.
raidcom map resource -ldev_id 0x5555 -virtual_ldev_id reserve -IH1
Display the information about the S-VOL by using the raidcom get ldev command.
raidcom get ldev -ldev_id 0x5555 -fx -IH1
```
LDEV : 5555 VIR_LDEV : ffff
```
The GAD reserve attribute (VIR_LDEV: ffff) is assigned to the S-VOL (LDEV ID: 0x5555).
Check the reserve attributes of the P-VOL and the S-VOL by using the raidcom get ldev command.
raidcom get ldev -ldev_id 0x4444 -fx -IH0
```
LDEV : 4444
```
raidcom get ldev -ldev_id 0x5555 -fx -IH1
```
LDEV : 5555 VIR_LDEV : ffff
```
The virtual LDEV ID (0x4444) is assigned to the P-VOL (LDEV ID: 0x4444) and the GAD reserve attribute ( VIR_LDEV : ffff) is assigned to the S-VOL (LDEV ID: 0x5555).
Specify a port and host group for the S-VOL, and set the LU path again.
Create GAD pairs again.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Resolving failures in multiple locations

If failures occur in multiple locations, use the following recovery procedure:

Procedure

Identify the failure locations from the SIMs issued by the primary and secondary storage systems and using the SAN management software, and then recover from the failures.
If data has been lost from both volumes, recover from the backup data using ShadowImage or Thin Image volumes, or backup software.
If I/O is stopped, resume I/O from the server.
If GAD pairs are suspended, resynchronize the pairs.
If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
If the pairs cannot be resynchronized, perform the following recovery procedures depending on the pair status and I/O mode.

Pair status I/O mode
P-VOL S-VOL P-VOL S-VOL
PSUE COPY Local Block
1. Delete the pair forcibly from the S-VOL.
  When you perform this step, delete the virtual LDEV ID so that the volume cannot be accessed from the server.
```
pairsplit -g oraHA -d dev1 -RF -IH2
```
2. Check if the virtual LDEV ID of the P-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
(Omitted)
```
  The VIR_LDEV information is not displayed if it is the same as the LDEV information. If the virtual LDEV ID is deleted, set a correct virtual LDEV ID.
3. Delete the pair forcibly from the P-VOL.
  When you perform this step, make sure not to delete the virtual LDEV ID, which allows the volume to be accessed from the server.
```
pairsplit -g oraHA -d dev1 -SFV -IH1
```
4. Check if the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff	
(Omitted)
```
  VIR_LDEV: ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
5. Re-create the pair by specifying the P-VOL.
```
paircreate -g oraHA -d dev1 -f never -vl -jq 1 -IH1
```
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.
```
raidcom map resource -ldev_id 0x1111 
-virtual_ldev_id 0x1111 -IH1
```
The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.

Pair status I/O mode
P-VOL S-VOL P-VOL S-VOL
SMPL COPY Not applicable Block
1. Delete the pair forcibly from the S-VOL.
  When you perform this step, delete the virtual LDEV ID so that the volume cannot be accessed from the server.
```
pairsplit -g oraHA -d dev1 -RF -IH2
```
2. Check if the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff	
(Omitted)
```
  VIR_LDEV: ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
3. Check if the virtual LDEV ID of the P-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
(Omitted)
```
  The VIR_LDEV information is not displayed if it is the same as the LDEV information. If the virtual LDEV ID is deleted, set a correct virtual LDEV ID.
  
  Note If shared memory in the secondary storage system becomes volatilized, the S-VOL pair status changes to SMPL, and GAD reserve is assigned to the virtual attribute.
4. Re-create the pair by specifying the P-VOL.
```
paircreate -g oraHA -d dev1 -f never -vl -jq 1 -IH1
```
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.
```
raidcom map resource -ldev_id 0x1111 
-virtual_ldev_id 0x1111 -IH1
```
The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.

Pair status I/O mode
P-VOL S-VOL P-VOL S-VOL
PSUS/PSUE SSWS Block Local
1. Resynchronize the pair by specifying the S-VOL.
```
pairresync -g oraHA -d dev1 -swaps -IH2
```
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
Pair status I/O mode
P-VOL S-VOL P-VOL S-VOL
SMPL SSWS Not applicable Local
1. Check if the virtual LDEV ID of the P-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
VIR_LDEV : ffff	
(Omitted)
```
  VIR_LDEV: ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
2. Delete the pair forcibly from the S-VOL.
  When you perform this step, make sure not to delete the virtual LDEV ID, which allows the volume to be accessed from the server.
```
pairsplit -g oraHA -d dev1 -RFV -IH2
```
3. Check if the virtual LDEV ID of the S-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x2222 -fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : 1111	
(Omitted)
```
4. Re-create the pair by specifying the S-VOL.
```
paircreate -g oraHA -d dev1 -f never -vl -jq 1 -IH2
```
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
If you accidentally delete the virtual LDEV ID of the S-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the S-VOL again using the raidcom map resource command.
```
raidcom map resource -ldev_id 0x1111 
-virtual_ldev_id 0x1111 -IH1
```
The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID of the P-VOL using the confirmation command.

Pair status I/O mode
P-VOL S-VOL P-VOL S-VOL
PSUS/PSUE SSUS/PSUE Local Block
1. Resynchronize the pair by specifying the P-VOL.
```
pairresync -g oraHA -d dev1 -IH1
```
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
Pair status I/O mode
P-VOL S-VOL P-VOL S-VOL
SMPL SSUS/PSUE Not applicable Block
1. Delete the pair forcibly from the S-VOL.
  When you perform this step, delete the virtual LDEV ID so that the volume can be accessed from the server.
```
pairsplit -g oraHA -d dev1 -RF -IH2
```
2. Check if the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff	
(Omitted)
```
  VIR_LDEV: ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
3. Check if the virtual LDEV ID of the P-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
(Omitted)
```
  The VIR_LDEV information is not displayed if it is the same as the LDEV information. If the virtual LDEV ID is deleted, set a correct virtual LDEV ID.
  
  Note If shared memory in the secondary storage system becomes volatilized, the S-VOL pair status changes to SMPL, and GAD reserve is assigned to the virtual attribute.
4. Re-create the pair by specifying the P-VOL.
```
paircreate -g oraHA -d dev1 -f never -vl -jq 1 -IH1
```
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.
```
raidcom map resource -ldev_id 0x1111 
-virtual_ldev_id 0x1111 -IH1
```
The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.

Pair status I/O mode
P-VOL S-VOL P-VOL S-VOL
PSUE PSUE Block Block
1. Delete the pair forcibly from the S-VOL.
  When you perform this step, delete the virtual LDEV ID so that the volume cannot be accessed from the server.
```
pairsplit -g oraHA -d dev1 -RF -IH2
```
2. Check if the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff	
(Omitted)
```
  VIR_LDEV: ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
3. Delete the pair forcibly from the P-VOL.
  When you perform this step, make sure not to delete the virtual LDEV ID, which allows the volume to be accessed from the server.
```
pairsplit -g oraHA -d dev1 -SFV -IH1
```
4. Check if the virtual LDEV ID of the P-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
(Omitted)
```
  The VIR_LDEV information is not displayed if it is the same as the LDEV information. If the virtual LDEV ID is deleted, set a correct virtual LDEV ID.
5. Re-create the pair by specifying the P-VOL.
```
paircreate -g oraHA -d dev1 -f never -vl -jq 1 -IH1
```
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.
```
raidcom map resource -ldev_id 0x1111 
-virtual_ldev_id 0x1111 -IH1
```
The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.

Pair status I/O mode
P-VOL S-VOL P-VOL S-VOL
PSUS/PSUE PSUS/PSUE Local Block
1. Resynchronize the pair by specifying the P-VOL.
```
pairresync -g oraHA -d dev1 -IH1
```
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
Pair status I/O mode
P-VOL S-VOL P-VOL S-VOL
PSUS/PSUE SMPL Local Not applicable
1. Check if the virtual LDEV ID of the S-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x2222 -fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : ffff	
(Omitted)
```
  VIR_LDEV: ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
2. Delete the pair forcibly from the P-VOL.
  When you perform this step, make sure not to delete the virtual LDEV ID, which allows the volume to be accessed from the server.
```
pairsplit -g oraHA -d dev1 -SFV -IH1
```
3. Check if the virtual LDEV ID of the P-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
(Omitted)
```
  The VIR_LDEV information is not displayed if it is the same as the LDEV information. If the virtual LDEV ID is deleted, set a correct virtual LDEV ID.
4. Re-create the pair by specifying the P-VOL.
```
paircreate -g oraHA -d dev1 -f never -vl -jq 1 -IH1
```
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.
```
raidcom map resource -ldev_id 0x1111 
-virtual_ldev_id 0x1111 -IH1
```
The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.

Pair status I/O mode
P-VOL S-VOL P-VOL S-VOL
PSUS/PSUE SMPL Block Not applicable
1. Delete the pair forcibly from the P-VOL.
  When you perform this step, make sure not to delete the virtual LDEV ID, which allows the volume to be accessed from the server.
```
pairsplit -g oraHA -d dev1 -SF -IH1
```
2. Check if the virtual LDEV ID of the P-VOL indicates GAD reserve.
```
raidcom get ldev -ldev_id 0x1111 -fx -IH1
(Omitted)
LDEV : 1111
VIR_LDEV : ffff	
(Omitted)
```
  VIR_LDEV: ffff indicates GAD reserve. If it shows another value, set a correct virtual LDEV ID.
3. Check if the virtual LDEV ID of the S-VOL is not deleted.
```
raidcom get ldev -ldev_id 0x2222 -fx -IH2
(Omitted)
LDEV : 2222
VIR_LDEV : 1111
(Omitted)
```
  If the virtual LDEV ID is deleted, set a correct virtual LDEV ID.
  
  Note If shared memory in the secondary storage system becomes volatilized, the S-VOL pair status changes to SMPL, and GAD reserve is assigned to the virtual attribute.
4. Re-create the pair by specifying the S-VOL.
```
paircreate -g oraHA -d dev1 -f never -vl -jq 1 -IH2
```
  If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
If you accidentally delete the virtual LDEV ID of the S-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the S-VOL again using the raidcom map resource command.
```
raidcom map resource -ldev_id 0x1111 
-virtual_ldev_id 0x1111 -IH1
```
The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID of the P-VOL using the confirmation command.

Pair status	I/O mode
PSUE	COPY	Local	Block

Pair status	I/O mode
SMPL	COPY	Not applicable	Block

Pair status	I/O mode
PSUS/PSUE	SSWS	Block	Local

Pair status	I/O mode
SMPL	SSWS	Not applicable	Local

Pair status	I/O mode
PSUS/PSUE	SSUS/PSUE	Local	Block

Pair status	I/O mode
SMPL	SSUS/PSUE	Not applicable	Block

Pair status	I/O mode
PSUE	PSUE	Block	Block

Pair status	I/O mode
PSUS/PSUE	PSUS/PSUE	Local	Block

Pair status	I/O mode
PSUS/PSUE	SMPL	Local	Not applicable

Pair status	I/O mode
PSUS/PSUE	SMPL	Block	Not applicable

Pair condition and recovery: quorum disk and primary-to-secondary path failure

If a failure occurs in the quorum disk, the pair status of the P-VOL and S-VOL does not change from PAIR (Mirror(RL). However, if a failure occurs on the physical path from the storage system at the primary site to the storage system at the secondary site, the status of the P-VOL changes from PAIR (Mirror(RL)) to PSUE (Local), and the status of the S-VOL changes from PAIR (Mirror(RL)) to PSUE (Block).

The following table shows transitions for pair status and I/O mode, the volumes that are accessible from the server, and the location of the latest data when you can no longer use any physical path from the primary storage system to the secondary storage system after the quorum disk failure.

After quorum disk failure		After primary-to-secondary path failure
Pair status and I/O mode		Pair status and I/O mode		Volume with latest data	Volume accessible from the server
P-VOL	S-VOL	P-VOL	S-VOL	Volume with latest data	P-VOL	S-VOL
PAIR (Mirror(RL))	PAIR (Mirror(RL))	PSUE (Local)	PAIR (Block)	OK	NG	P-VOL

SIMs

Primary storage system: 21D0xy, 21D2xx, DD2xyy, DEF0zz, EF5xyy, EFD000, FF5xyy, DD0xyy, 2180xx
Secondary storage system: 21D0xy, 21D2xx, DD2xyy, DEF0zz, EF5xyy, EFD000, FF5xyy, DD3xyy, DD0xyy, DD1xyy

Procedure

Recover the quorum disk failure and the path to the external storage system.
Recover the path from the primary storage system to the secondary storage system.
Resynchronize the GAD pair suspended due to the failure.
Confirm the pair status.

When the pair status of the P-VOL and S-VOL is PAIR (Mirror(RL)), the recovery is completed.
Suspend the GAD pair by specifying the S-VOL (swap suspend).

The pair suspension operation fails, but the S-VOL pair status changes to PSUE (Block).
Resynchronize the GAD pair by specifying the P-VOL.

The pair status of the P-VOL and S-VOL changes to PAIR (Mirror(RL)).

Recovering the quorum disk and primary-to-secondary path failure (VSP 5000 series)

The following figure shows the failure area and recovery from the path failure from the primary storage system to the secondary storage system after the GAD status changes to Quorum disk blocked.

Procedure

Recover the quorum disk failure and the path to the external storage system.
1. Recover the quorum disk.
2. Reconnect the physical path or reconfigure the SAN to recover the path to the external storage system. When the path is recovered, the external path is automatically recovered.
3. Confirm that the external storage system is connected correctly.
```
raidcom get path -path_grp 1 -IH0
PHG GROUP STS CM IF MP# PORT WWN PR LUN PHS Serial# PRODUCT_ID LB PM
1 1-1 NML E D 0 CL5-A 50060e8008000140 1 0 NML 55555 VSP 5000 series  N M
```
4. Confirm the LDEV ID of the quorum disk by obtaining the information of the external volume from the primary storage system.
```
raidcom get external_grp -external_grp_id 1-1 -IH0
T GROUP P_NO LDEV# STS LOC_LBA SIZE_LBA Serial#
E 1-1 0 9999 NML 0x000000000000 0x000003c00000 55555
```
5. Confirm that the primary storage system recognizes the external volume as a quorum disk by specifying the LDEV ID of the quorum disk.
```
raidcom get ldev -ldev_id 0x9999 -fx -IH0
(snip)
QRDID : 0
QRP_Serial# : 522222
QRP_ID : R9
(snip)
```
  NoteThe VSP 5000 series is displayed as R9 in command output. VSP Fx00 models and VSP Gx00 models are displayed as M8 in command output.
Reconnect the physical path or reconfigure the SAN to recover the path failure from the primary to secondary storage system.

When the path between the storage systems is recovered, the remote path is either automatically recovered or a manual recovery might be required. To verify the remote path status and perform any recommended action, see Troubleshooting related to remote path status. If the remote path failure persists even after following the recommended action, contact customer support.
Resynchronize the GAD pair whose GAD status is Suspended.
1. Confirm that the P-VOL I/O mode is Local.
```
pairdisplay -g oraHA -fxce -IH0
Group PairVol(L/R) (Port#,TID, LU), Seq#,LDEV#.P/S,Status,
Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
oraHA dev1(L) (CL1-A-0, 0, 0)511111 2222.P-VOL PSUE NEVER , 100 4444 - - 0 - - - - L/L 
oraHA dev1(R) (CL1-C-1, 0, 0)522222 4444.S-VOL PSUE NEVER , 100 2222 - - 0 - - - - B/B
```
2. At the primary storage system, resynchronize the pair.
```
pairresync -g oraHA -IH0
```
  NoteWhen the P-VOL pair status is PSUE (Local) and the S-VOL pair status is PAIR(Block), the pair resynchronization fails. The result of the pair resynchronization depends on whether the GAD pair is registered to the consistency group.
  - When the GAD pair is registered to the consistency group, the pair resynchronization operation fails.
  - When the GAD pair is not registered to the consistency group, the pair resynchronization operation succeeds, but the pair resynchronization process fails. The pair status of the P-VOL after the pair resynchronization remains PSUE.
3. Confirm that the pair status of the P-VOL and S-VOL changes to PAIR (Mirror (RL)).
```
pairdisplay -g oraHA -fxce -IH0
Group PairVol(L/R) (Port#,TID, LU), Seq#, LDEV#.P/S,Status,
Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
oraHA dev1(L) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIR NEVER , 100 4444 - - 0 - - - - L/M
oraHA dev1(R) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIR NEVER , 100 2222 - - 0 - - - - L/M
pairdisplay -g oraHA -fxce -IH1
Group PairVol(L/R) (Port#,TID, LU), Seq#, LDEV#.P/S,Status,
Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
oraHA dev1(L) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIR NEVER , 100 2222 - - 0 - - - - L/M
oraHA dev1(R) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIR NEVER , 100 4444 - - 0 - - - - L/M
```
  NoteWhen the pair whose P-VOL pair status is PSUE (Local) and S-VOL pair status is PAIR (Block) exists, go to step 4. When no pairs meet this condition, go to step 6.
Suspend all GAD pairs whose P-VOL pair status is PSUE (Local) and S-VOL pair status is PAIR (Block) by specifying the S-VOL (swap suspend).
```
pairsplit -g oraHA -RS -d dev1 -IH0
```
The pair suspension operation fails, but the S-VOL pair status changes to PSUE.
Note
- Even if the pairs are registered to a consistency group, swap-suspend the pairs by pair.
- The following SIMs might be issued, but you do not need to address these SIMs: DD0xyy, DD1xyy, DD2xyy, DD3xyy
At the primary storage system, resynchronize the GAD pair.
```
pairresync -g oraHA -IH0
```
Using the alternate path software, resume I/O to the S-VOL (I/O might resume automatically).

Quick Links

Failure locations

SIMs related to GAD

Pair condition before failure

Pair condition and recovery: server failures

Pair condition and recovery: path failure between the server and storage system

Recovering from a path failure: server to primary storage system

Recovering from a path failure: server to secondary storage system

Pair condition and recovery: P-VOL failure (LDEV blockade)

Recovering the P-VOL (DP-VOL) (pair status: PAIR)

Recovering the P-VOL (other than DP-VOL) (pair status: PAIR)

Pair condition and recovery: S-VOL failure (LDEV blockade)

Recovering the S-VOL (DP-VOL) (pair status: PAIR)

Recovering the S-VOL (other than DP-VOL) (pair status: PAIR)

Pair condition and recovery: full pool for the P-VOL

Recovering a full pool for the P-VOL (pair status: PAIR)

Pair condition and recovery: full pool for the S-VOL

Recovering a full pool for the S-VOL (pair status: PAIR)

Pair condition and recovery: path failure, primary to secondary storage system

Recovering paths, primary to secondary storage system (pair status: PAIR)

Pair condition and recovery: path failure, secondary to primary storage system

Recovering paths, secondary to primary storage system (pair status: PAIR when a volume is set for the quorum disk)

Recovering paths, secondary to primary storage system (pair status: PAIR when no volume is set for the quorum disk)

Pair condition and recovery: primary storage system failure

Setting correct virtual LDEV ID values

Pair condition and recovery: secondary storage system failure

Pair condition and recovery: path failure, primary to external storage system

Recovering the path: primary to external storage system (pair status: PAIR)

Pair condition and recovery: path failure, secondary to external storage system

Recovering the path: secondary to external storage system (pair status: PAIR)

Pair condition and recovery: quorum disk failure

Recovering the quorum disk (pair status: PAIR)

Recovering from quorum disk failures

Replacing a failed external storage system with a new one

Recovering from the FAILED status

Pair condition and recovery: external storage system failure

Pair condition and recovery: other failures

Recovery procedure for GAD pair suspension due to other failures

Recovering the storage systems: primary site failure with external storage system

Reversing the P-VOL and S-VOL

Creating GAD pairs when virtual LDEV IDs are deleted from the P-VOL and S-VOL

Creating GAD pairs when virtual LDEV IDs are set for the P-VOL and S-VOL

Resolving failures in multiple locations

Pair condition and recovery: quorum disk and primary-to-secondary path failure

Recovering the quorum disk and primary-to-secondary path failure (VSP 5000 series)