Skip to main content

We've Moved!

Product Documentation has moved to docs.hitachivantara.com
Hitachi Vantara Knowledge

Disaster recovery in a GAD 3DC delta resync environment

This chapter provides information and instructions specifically for disaster recovery operations in a GAD 3DC delta resync (GAD+UR) environment requires specific instruction and information.

Recovering from primary site and P-VOL failures (GAD+UR)

This section describes the procedures for recovering from a primary site failure or a P-VOL failure (LDEV blockade) at the primary site using examples.

GAD pairs might be created between the primary and secondary sites that configure GAD as follows:

  • A P-VOL of a GAD pair is at site 1 and a S-VOL is at site 2.
  • A P-VOL of a different GAD pair is at site 2 and a S-VOL is at site 1.

In this case, recover from a failure at the secondary site. For more information, see Recovering from secondary site and S-VOL failures (GAD+UR).

Recovering from a primary site failure (GAD+UR)

This subsection describes how to recover from a primary site failure. The example explains the case when a failure occurs in a storage system at the primary site.

When a failure occurs at a primary site, the status of the GAD pair changes to PSUE/SSWS. In this case, delta resync is automatically performed, and the pair statuses change as follows:

Before failure

After failure

Pair type

Pair status

Pair type

Pair status

P-VOL

S-VOL

P-VOL

S-VOL

GAD pair

PAIR

PAIR

GAD pair

PSUE1

SSWS

UR pair

PAIR

PAIR

UR pair

PSUE1, 2

-

UR delta resync pair

-

SSUS

UR delta resync pair

PSUS

SSUS

UR pair

From COPY to PAIR

From COPY to PAIR

Notes:

  1. Pair status after the primary site is turned on.
  2. PSUE status of the UR pair. The UR pair does not change to a UR delta resync pair. If your microcode version is 90-07-01-00/00 or later for VSP 5000 series, or 93-05-02-x0/00 or later for VSP E series, the UR pair changes to a UR delta resync pair.

If a failure occurs, the UR delta resync pair changes to a UR pair, and copying from the GAD S-VOL to the UR S-VOL starts. When the UR pair copying is completed, the status of the P-VOL and the status of the S-VOL of the UR pair change to PAIR.

Overview of failure recovery GUID-8B641BF9-3E5B-4AFC-99E6-1C4B01158AB8-low.png

Procedure

  1. Remove the failure on the P-VOL.

    If your microcode version is 90-07-01-00/00 or later for VSP 5000 series, or 93-05-02-x0/00 or later for VSP E series, go to step 4. In this case, you can skip steps 2 and 3 because a 3DC delta resync configuration is automatically re-created.
  2. At the primary site for the UR delta resync pair, delete the UR delta resync pair.

    Command example:

    pairsplit -g oraREMOTE -S -IH0
  3. At the primary site for the UR delta resync pair, create a UR delta resync pair to reconfigure a 3DC delta resync configuration.

    Command example:

    paircreate -g oraREMOTE -f async -vl -nocsus -jp 0 -js 0 -IH0
    If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
  4. Reverse the P-VOL and the S-VOL, and then resynchronize the GAD pairs (swap resync) on the storage system at the GAD secondary site.

    Command example:

    pairresync -g oraHA -swaps -IH1

    If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

    The volume on the primary storage system changes to an S-VOL, and the volume on the GAD secondary storage system changes to a P-VOL.

  5. Confirm that the GAD P-VOL and S-VOL pair statuses change to PAIR.

    Command example:

    pairdisplay -g oraHA -fxce -IH0
    Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
    oraHA dev1(L) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIRNEVER , 100 4444 - - 0 - - - - L/M
    oraHA dev1(R) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIRNEVER , 100 2222 - - 0 - - - - L/M
    pairdisplay -g oraHA -fxce -IH1
    Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
    oraHA dev1(L) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIRNEVER , 100 2222 - - 0 - - - - L/M
    oraHA dev1(R) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIRNEVER , 100 4444 - - 0 - - - - L/M
  6. Keep updating I/O from the server to the P-VOL or S-VOL of the GAD pair for about two minutes.

  7. Confirm that the delta UR P-VOL pair status is PSUS.

    Command example:

    pairdisplay -g oraREMOTE -fxce -IH0
    Group PairVol(L/R) (Port#,TID, LU),Seq#, LDEV#.P/S,Status,Fence, %, P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
    oraDELTA dev2(L) (CL1-A-1, 0, 1) 511111 2222. P-VOLPSUS ASYNC ,0 6666 - 0 0 - - - - -/-
    oraDELTA dev2(R) (CL1-A-1, 0, 1) 544444 6666. S-VOLSSUS ASYNC ,0 2222 - 0 0 - - - -  -/-
    NoteTo check the status of a pair in Device Manager - Storage Navigator, select Refresh All in the File menu to update the information displayed on Device Manager - Storage Navigator, and then view the pair status. The status of the UR delta resync pairs changes from HOLDING to HOLD.
  8. Confirm that the mirror status of the journal of the UR delta resync pair is PJNS.

    Command example:

    pairdisplay -g oraREMOTE -v jnl -IH0
    JID MU CTG JNLS AP U(%) Q-Marker Q-CNT D-SZ(BLK) Seq#Num LDEV#
    000 1 1 PJNS 4 21 43216fde 30 512345 625001 39321
  9. Confirm that no failure SIMs are displayed.

  10. Reverse the GAD P-VOL and the S-VOL if necessary.

    For instructions, see Reversing the GAD P-VOL and S-VOL (GAD+UR).If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering from a P-VOL failure (LDEV blockade) at the primary site (GAD+UR)

When a failure (LDEV blockade) occurs on a P-VOL at the primary site, the status of the GAD pair changes to PSUE/SSWS. In this case, delta resync is automatically performed, and the pair statuses change as follows:

Before failure

After failure

Pair type

Pair status

Pair type

Pair status

P-VOL

S-VOL

P-VOL

S-VOL

GAD pair

PAIR

PAIR

GAD pair

PSUE

SSWS

UR pair

PAIR

PAIR

UR delta resync pair

PSUE

SSUS

UR delta resync pair

PSUS

SSUS

UR pair

From COPY to PAIR

From COPY to PAIR

If a failure occurs, the UR delta resync pair changes to a UR pair, and copying from the GAD S-VOL to the UR S-VOL starts. When the UR pair copying is completed, the status of the P-VOL and the status of the S-VOL of the UR pair change to PAIR.

Overview of failure recovery GUID-A95A110E-A289-4525-A8C6-E34819D6D9F7-low.png

Procedure

  1. Remove the failure (LDEV blockade) on the P-VOL.

  2. If the P-VOL must be formatted for restoration, delete the GAD pair, and then format the P-VOL.

    When deleting the GAD pair, first delete the S-VOL of which I/O mode is Local.
  3. Depending on if you performed step 2, re-create the GAD pair, the delta resync UR pair, and the UR pair in that order, or reverse the P-VOL and the S-VOL in the GAD secondary storage system, and then resynchronize the GAD pair.

    • If you performed step 2, re-create the GAD pair, the delta resync UR pair, and the UR pair in that order.

      If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

      For the GAD pair, first re-create the S-VOL.

      If necessary, after executing the command, reverse the P-VOL and S-VOL.

      Command example:

      paircreate -g oraHA -fg never 2 -vl -jq 0 -IH1
      paircreate -g oraDELTA -f async -vl -jp 0 -js 0 -nocsus -IH1
      paircreate -g oraREMOTE -f async -vl -jp 0 -js 0 -IH0
    • If you did not perform step 2, reverse the P-VOL and the S-VOL in the GAD secondary storage system, and then resynchronize the GAD pair (swap resync).

      If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

      Command example:

      pairresync -g oraHA -swaps -IH1

      The primary storage system volume becomes the S-VOL, and the GAD secondary storage system volume becomes the P-VOL.

  4. Confirm that the GAD P-VOL and S-VOL pair statuses change to PAIR.

    Command example:

    pairdisplay -g oraHA -fxce -IH0
    Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
    Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
    oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
    NEVER ,  100  4444 -   -   0  -  -            -       - L/M
    oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
    NEVER ,  100  2222 -   -   0  -  -            -       - L/M
    pairdisplay -g oraHA -fxce -IH1
    Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
    Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
    oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
    NEVER ,  100  2222 -   -   0  -  -            -       - L/M
    oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
    NEVER ,  100  4444 -   -   0  -  -            -       - L/M
  5. Keep updating I/O from the server to the P-VOL or S-VOL of the GAD pair for about two minutes.

  6. Confirm that the delta UR P-VOL pair status is PSUS.

    Command example:

    pairdisplay -g oraREMOTE -fxce -IH0
    Group     PairVol(L/R)  (Port#,TID, LU),Seq#,  LDEV#.P/S,  
    Status,Fence, %,  P-LDEV#  M  CTG  JID  AP  EM  E-Seq#  E-LDEV#  R/W
    oraDELTA  dev2(L)       (CL1-A-1, 0, 1) 511111 2222. P-VOL 
    PSUS   ASYNC ,0   6666     -  0    0    -   -   -       -       -/-
    oraDELTA  dev2(R)       (CL1-A-1, 0, 1) 544444 6666. S-VOL 
    SSUS   ASYNC ,0   2222     -  0    0    -   -   -       -       -/-
    NoteTo check the status of a pair in Device Manager - Storage Navigator, select Refresh All in the File menu, update the information displayed on Device Manager - Storage Navigator, and then view the pair status. The status of the UR delta resync pairs changes from HOLDING to HOLD.
  7. Confirm that the mirror status of the journal of the UR delta resync pair is PJNS.

    Command example:

    pairdisplay -g oraREMOTE -v jnl -IH0
    JID  MU  CTG  JNLS  AP  U(%)  Q-Marker  Q-CNT  D-SZ(BLK)  Seq#   Num  LDEV#
    000  1   1    PJNS  4   21    43216fde  30     512345     62500  1    39321 
  8. Confirm that no failure SIMs are displayed.

  9. Reverse the GAD P-VOL and the S-VOL if necessary. For instructions, see Reversing the GAD P-VOL and S-VOL (GAD+UR).

    If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Reversing the GAD P-VOL and S-VOL (GAD+UR)

Use the following procedure to reverse the GAD P-VOL and S-VOL when sharing GAD volumes with UR in a GAD 3DC delta resync (GAD+UR) configuration.

Procedure

  1. Suspend the GAD pair by specifying the S-VOL (swap suspend).

    Command example: pairsplit -g oraHA -RS -IH0

  2. Resynchronize the GAD pair by specifying the S-VOL (swap resync).

    Command example: pairresync -g oraHA -swaps -IH0

    If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

    The volume on the primary storage system changes to a P-VOL, and the volume on the GAD secondary storage system changes to an S-VOL.

  3. Keep updating I/O from the server to the P-VOL or S-VOL of the GAD pair for about two minutes.

  4. Confirm that the delta UR P-VOL pair status is PSUS.

    Command example:

    pairdisplay -g oraDELTA -fxce -IH0
    Group     PairVol(L/R)  (Port#,TID, LU),Seq#,  LDEV#.P/S,  
    Status,Fence, %,  P-LDEV#  M  CTG  JID  AP  EM  E-Seq#  E-LDEV#  R/W
    oraDELTA  dev2(L)       (CL1-A-1, 0, 1) 511111 2222. P-VOL 
    PSUS   ASYNC ,0   6666     -  0    0    -   -   -       -       -/-
    oraDELTA  dev2(R)       (CL1-A-1, 0, 1) 544444 6666. S-VOL 
    SSUS   ASYNC ,0   2222     -  0    0    -   -   -       -       -/-
    NoteTo check the status of a pair in Device Manager - Storage Navigator, select Refresh All in the File menu, update the information displayed on Device Manager - Storage Navigator, and then view the pair status. The status of the UR delta resync pairs changes from HOLDING to HOLD.
  5. Confirm that the mirror status of the journal of the UR delta resync pair is PJNS.

    Command example:

    pairdisplay -g oraDELTA -v jnl -IH0
    JID  MU  CTG  JNLS  AP  U(%)  Q-Marker  Q-CNT  D-SZ(BLK)  Seq#   Num  LDEV#
    000  1   1    PJNS  4   21    43216fde  30     512345     62500  1    39321 
  6. Confirm that no failure SIMs are displayed.

Recovering from secondary site and S-VOL failures (GAD+UR)

This section describes the procedure for recovering from secondary site failures and failures (LDEV blockade) on an S-VOL at the secondary site using examples.

GAD pairs might be created between the primary and secondary sites that configure GAD as follows:

  • A P-VOL of a GAD pair is at site 1 and a S-VOL is at site 2.
  • A P-VOL of a different GAD pair is at site 2 and a S-VOL is at site 1.

In this case, recover from a failure at the primary site. For more information, see Recovering from primary site and P-VOL failures (GAD+UR).

Recovering from a secondary site failure (GAD+UR)

This subsection describes how to recover from a failure in a storage system at the secondary site. When a failure occurs at the secondary site, the GAD pair statuses at the primary site and the secondary site change to PSUE.

Overview of failure recovery GUID-8825DDDB-9D59-4597-8A90-CF5A9C023791-low.png

Procedure

  1. Remove the failure on the S-VOL.

  2. Resynchronize the GAD pair at the primary storage system.

    Command example:

    pairresync -g oraHA -IH0
    If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.
  3. Confirm that the GAD P-VOL and S-VOL pair statuses change to PAIR.

    Command example:

    pairdisplay -g oraHA -fxce -IH0
    Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
    oraHA dev1(L) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIRNEVER , 100 4444 - - 0 - - - - L/M
    oraHA dev1(R) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIRNEVER , 100 2222 - - 0 - - - - L/M
    pairdisplay -g oraHA -fxce -IH1
    Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
    oraHA dev1(L) (CL1-C-1, 0, 0)522222 4444.S-VOL PAIRNEVER , 100 2222 - - 0 - - - - L/M
    oraHA dev1(R) (CL1-A-0, 0, 0)511111 2222.P-VOL PAIRNEVER , 100 4444 - - 0 - - - - L/M
  4. Keep updating I/O from the server to the P-VOL or S-VOL of the GAD pair for about two minutes.

  5. Confirm that the pair status of the delta UR P-VOL is PSUS.

    Command example:

    pairdisplay -g oraDELTA -fxce -IH1
    Group PairVol(L/R) (Port#,TID, LU),Seq#, LDEV#.P/S,Status,Fence, %, P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
    oraDELTA dev3(L) (CL1-A-1, 0, 1) 522222 4444. P-VOLPSUS ASYNC ,0 6666 - 0 0 - - - - -/-
    oraDELTA dev3(R) (CL1-A-1, 0, 1) 544444 6666. S-VOLSSUS ASYNC ,0 4444 - 0 0 - - - - -/-
    NoteTo check the status of a pair in Device Manager - Storage Navigator, select Refresh All in the File menu, update the information displayed on Device Manager - Storage Navigator, and then view the pair status. The status of the UR delta resync pairs changes from HOLDING to HOLD.
  6. Confirm that the mirror status of the journal of the UR delta resync pair is PJNS using the CCI instance for managing the GAD secondary storage system.

    Command example:

    pairdisplay -g oraDELTA -v jnl -IH1 
    JID MU CTG JNLS AP U(%) Q-Marker Q-CNT D-SZ(BLK) Seq# Num LDEV# 
    000 1 1 PJNS 4 21 43216fde 30 512345 62500 1 39321
  7. Confirm that no failure SIMs are displayed.

Recovering from a failure (LDEV blockade) on an S-VOL at the secondary site (GAD+UR)

When a failure (LDEV blockade) occurs on an S-VOL at the secondary site, the GAD pair statuses of the P-VOL and S-VOL change to PSUE.

Overview of failure recovery GUID-BCB4FB97-9784-4854-A5D8-8F7BACF3FDB3-low.png

Procedure

  1. Remove the failure (LDEV blockade) on the S-VOL.

  2. If the S-VOL must be formatted for restoration, delete the GAD pair, and then format the S-VOL.

  3. Depending on if you performed step 2, re-create the GAD pair, the delta resync UR pair, and the UR pair in that order, or resynchronize the GAD pair in the primary storage system.

    • If you performed step 2, re-create the GAD pair, the delta resync UR pair, and the UR pair in that order.

      If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

      Command example:

      paircreate -g oraHA -fg never 2 -vl -jq 0 -IH0
      paircreate -g oraDELTA -f async -vl -jp 0 -js 0 -nocsus -IH1
      paircreate -g oraREMOTE -f async -vl -jp 0 -js 0 -IH0
    • If you did not perform step 2, resynchronize the GAD pair in the primary storage system.

      If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

      Command example:

      pairresync -g oraHA -IH0
  4. Confirm that the GAD P-VOL and S-VOL pair statuses change to PAIR.

    Command example:

    pairdisplay -g oraHA -fxce -IH0
    Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
    Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
    oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
    NEVER ,  100  4444 -   -   0  -  -            -       - L/M
    oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
    NEVER ,  100  2222 -   -   0  -  -            -       - L/M
    pairdisplay -g oraHA -fxce -IH1
    Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
    Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
    oraHA   dev1(L)     (CL1-C-1, 0,   0)522222  4444.S-VOL PAIR 
    NEVER ,  100  2222 -   -   0  -  -            -       - L/M
    oraHA   dev1(R)     (CL1-A-0, 0,   0)511111  2222.P-VOL PAIR 
    NEVER ,  100  4444 -   -   0  -  -            -       - L/M
  5. Keep updating I/O from the server to the P-VOL or S-VOL of the GAD pair for about two minutes.

  6. Confirm that the pair status of the delta UR P-VOL is PSUS.

    Command example:

    pairdisplay -g oraDELTA -fxce -IH1
    Group     PairVol(L/R)  (Port#,TID, LU),Seq#,  LDEV#.P/S,  
    Status,Fence, %,  P-LDEV#  M  CTG  JID  AP  EM  E-Seq#  E-LDEV#  R/W
    oraDELTA  dev3(L)       (CL1-A-1, 0, 1) 522222 4444. P-VOL 
    PSUS   ASYNC ,0   6666     -  0    0    -   -   -       -       -/-
    oraDELTA  dev3(R)       (CL1-A-1, 0, 1) 544444 6666. S-VOL 
    SSUS   ASYNC ,0   4444     -  0    0    -   -   -       -       -/-
    NoteTo check the status of a pair in Device Manager - Storage Navigator, select Refresh All in the File menu, update the information displayed on Device Manager - Storage Navigator, and then view the pair status. The status of the UR delta resync pairs changes from HOLDING to HOLD.
  7. Confirm that the mirror status of the journal of the UR delta resync pair is PJNS using the CCI instance for managing the GAD secondary storage system.

    Command example:

    pairdisplay -g oraDELTA -v jnl -IH1
    JID  MU  CTG  JNLS  AP  U(%)  Q-Marker  Q-CNT  D-SZ(BLK)  Seq#   Num  LDEV#
    000  1   1    PJNS  4   21    43216fde  30     512345     62500  1    39321 
  8. Confirm that no failure SIMs are displayed.

Recovering from a failure at the primary and secondary sites (GAD+UR)

Use the following procedure to recover from a failure at the primary and secondary sites when sharing GAD volumes with UR in a GAD 3DC delta resync (GAD+UR) configuration and your microcode version is 90-07-01-00/00 or later for VSP 5000 series, or 93-05-02-x0/00 or later for VSP E series.

This procedure is different from the procedure for the same topic in earlier versions of the Global-Active Device User Guide. If your microcode is earlier than that listed, refer to the SVOS RF 9.6.1 version of the Global-Active Device User Guide.

Procedure

  1. Suspend the UR pair by specifying the S-VOL (swap suspend).

    Command example:

    pairsplit -g oraREMOTE -RS -IH2

    You can resume operations at this point by using the S-VOL at the UR secondary site.

    GUID-6314FD21-DD03-4C87-8461-7CA7EF6E3F0D-low.png
  2. Remove the failure at the primary and secondary sites.

  3. Change the I/O mode of the P-VOL to Local.

    Command example:
    pairsplit -g oraHA -iomd local -IH0
  4. Resynchronize the UR pair by specifying the S-VOL (swap resync).

    Command example:
    pairresync -g oraREMOTE -swaps -IH2
  5. Swap the P-VOL and S-VOL of the UR pair.

    Command example:
    pairresync -g oraREMOTE -swaps -IH0
  6. Resynchronize the GAD pair by specifying the P-VOL.

    Command example:
    pairresync -g oraHA -IH0

Recovering from a failure on the UR delta resync pair (VSP 5000 series)

Use the following procedure to recover from a failure on the UR delta resync pair due to a journal volume failure on the storage system at the GAD secondary site.

When a failure occurs on the UR delta resync pair, a SIM is displayed.

Overview of failure recovery GUID-C7775138-AE42-4926-8EE7-A981A1ECAE68-low.png

Procedure

  1. Set the system option mode 1182 to ON at the GAD secondary storage system.

  2. Add a reserve journal volume.

    After a reserve journal volume is added, the UR delta resync pair status changes to PSUE (HLDE for Device Manager - Storage Navigator), and the SIM code dc9xxx or 7ff103 is issued.

    If a reserve journal volume has already been added, use CCI to delete the reserve journal volume from the journal, and then add a reserve journal volume again.

  3. Delete the journal volume in which a failure occurred, from the journal.

  4. Restore the UR delta resync pair.

    If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

    Verify that Return to standby is displayed in Resync Mode in the Resync Mirrors window of Device Manager - Storage Navigator, and then resynchronize the pair.

    NoteYou cannot do this using CCI.
  5. Confirm that the pair status of the delta UR P-VOL is PSUS (HOLD for Device Manager - Storage Navigator).

    If the server has stopped I/Os, the pair status is PSUE (HOLDING for Device Manager - Storage Navigator)

    Command example:

    pairdisplay -g oraDELTA -fxce -IH1
    Group     PairVol(L/R)  (Port#,TID, LU),Seq#,  LDEV#.P/S,  
    Status,Fence, %,  P-LDEV#  M  CTG  JID  AP  EM  E-Seq#  E-LDEV#  R/W
    oraDELTA  dev3(L)       (CL1-A-1, 0, 1) 522222 4444. P-VOL 
    PSUS   ASYNC ,0   6666     -  0    0    -   -   -       -       -/-
    oraDELTA  dev3(R)       (CL1-A-1, 0, 1) 544444 6666. S-VOL 
    SSUS   ASYNC ,0   4444     -  0    0    -   -   -       -       -/-
  6. Confirm that the mirror status of the journal of the UR delta resync pair is PJNS (for Device Manager - Storage Navigator, it is HOLD when the server issues I/Os, or HOLDING when the server stops I/Os).

    Command example:

    pairdisplay -g oraDELTA -v jnl -IH1
    JID  MU  CTG  JNLS  AP  U(%)  Q-Marker  Q-CNT  D-SZ(BLK)  Seq#   Num  LDEV#
    000  1   1    PJNS  4   21    43216fde  30     512345     62500  1    39321 
  7. Set the system option mode 1182 to OFF at the GAD secondary storage system.

  8. Confirm that no failure SIMs are displayed.

Recovering from a failure on the UR delta resync pair (VSP E series)

Use the following procedure to recover from a failure on the UR delta resync pair due to a journal volume failure on the storage system at the GAD secondary site.

When a failure occurs on the UR delta resync pair, a SIM is displayed.

Overview of failure recovery GUID-C2BCBD2E-5F30-462A-8A3A-EC4F335848D1-low.png

Procedure

  1. Increase the available capacity of the pool used by the journal at the GAD secondary storage system.

  2. Restore the UR delta resync pair.

    Verify that Return to standby is displayed in Resync Mode in the Resync Mirrors window of Device Manager - Storage Navigator), and then resynchronize the pair.
    NoteYou cannot do this using CCI.
  3. Confirm that the pair status of the delta UR P-VOL is PSUS.

    Command example:
    pairdisplay -g oraDELTA -fxce -IH1
    Group PairVol(L/R) (Port#,TID, LU),Seq#, LDEV#.P/S,
    Status,Fence, %, P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W
    oraDELTA dev3(L) (CL1-A-1, 0, 1) 422222 4444. P-VOL
    PSUS ASYNC ,0 6666 - 0 0 - - - - -/-
    oraDELTA dev3(R) (CL1-A-1, 0, 1) 444444 6666. S-VOL
    SSUS ASYNC ,0 4444 - 0 0 - - - - -/-
  4. Confirm that the mirror status of the journal of the UR delta resync pair is PJNS using the CCI instance for managing the GAD secondary storage system.

    Command example:
    pairdisplay -g oraDELTA -v jnl -IH1
    JID MU CTG JNLS AP U(%) Q-Marker Q-CNT D-SZ(BLK) Seq# Num LDEV#
    000 1 1 PJNS 4 21 43216fde 30 512345 62500 1 39321
    
  5. Confirm that no failure SIMs are displayed.

Recovering from a quorum disk failure (GAD+UR)

You can recover from a quorum disk failure when sharing GAD volumes with UR in a GAD 3DC delta resync (GAD+UR) configuration.

You can use a volume in an external storage system or a disk in a server for a quorum disk. Procedures are based on the assumption that a volume in an external storage system is used as the quorum disk and depend on either of the following statuses:

  • Able to access either volume of the GAD pair from the server
  • Not able to access either volume of the GAD pair from the server

To check whether you can access the volumes of a GAD pair from the server, use I/O mode for the GAD pair.

Command example
pairdisplay -g oraHA -fxce -IH0
Group   PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,
Fence,   %,P-LDEV# M CTG JID AP EM       E-Seq# E-LDEV# R/W
oraHA   dev1(L)     (CL1-A-0, 0,   0)511111  2222.S-VOL PSUS 
NEVER ,  100  4444 -   -   0  -  -            -       - L/L
oraHA   dev1(R)     (CL1-C-1, 0,   0)522222  4444.P-VOL PAIR 
NEVER ,  100  2222 -   -   0  -  -            -       - B/B

The server can access a volume whose I/O mode (R/W) is L/L (Local), and cannot access a volume whose I/O mode (R/W) is B/B (Block).

Recovering from a quorum disk failure when one GAD volume is accessible

Use the following procedure to recover a failure on the quorum disk when you can access either volume of a GAD pair from the server.

Overview of failure recovery GUID-81911ECB-6801-4A8C-95A5-A12B2FE79017-low.png

Procedure

  1. Remove the failure on the quorum disk.

  2. Resynchronize GAD pairs if they are suspended by a failure.

    If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

Recovering from a quorum disk failure when neither GAD volume is accessible

When a failure occurs on the quorum disk and you cannot access either volume of a GAD pair from the server, you need to delete the UR and GAD pairs, then recover from the failure by using the volume at the UR secondary site, and then re-create the pairs.

Procedure

  1. Delete the UR pair.

    Command example: pairsplit -g oraREMOTE -S -IH0

  2. Delete the UR delta resync pair.

    Command example: pairsplit -g oraDELTA -S -IH1

  3. Delete the GAD pair.

    When the I/O mode of both the primary and secondary volumes is Block, you can forcibly delete the pair, as follows:

    1. If you specify the S-VOL, delete the virtual LDEV ID at the same time.

      pairsplit -g oraHA -RF -IH1
    2. Confirm that the virtual LDEV ID of the S-VOL indicates GAD reserve.

      raidcom get ldev -ldev_id 0x2222 -fx -IH2
      (Omitted)
      LDEV : 2222
      VIR_LDEV : ffff
      (Omitted) 

      VIR_LDEV : ffff indicates GAD reserve.

    3. Delete the pair forcibly by specifying the P-VOL.

      Do not delete the virtual LDEV ID.

      pairsplit -g oraHA -SFV -IH0
    4. Confirm that the virtual LDEV ID of the P-VOLis not deleted.

      raidcom get ldev -ldev_id 0x1111 -fx -IH1
      (Omitted) 
      LDEV : 1111
      (Omitted) 

      VIR_LDEV information is not displayed if it is same as LDEV information.

    To forcibly delete a pair when the I/O mode is not Block, call Hitachi Vantara LLC customer support.

    CautionWhen you delete a GAD pair forcibly in Device Manager - Storage Navigator, select Force in Delete Mode in the Delete Pairs window.
  4. Create a UR pair between the UR secondary and primary sites.

    Command example:

    paircreate -g oraREMOTE -f async -vl -jp 0 -js 0 -IH2

    If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

    You can resume operations at this point by using the P-VOL at the UR secondary site.

    GUID-868ED394-A8FF-42F4-890B-C25A551A74BE-low.png
  5. Remove the failure on the quorum disk.

  6. Suspend the UR pair by specifying the S-VOL (swap suspend).

    Command example: pairsplit -g oraREMOTE -RS -IH0

    You can resume operations at this point by using the S-VOL at the primary site.

    GUID-1AA3416F-D61A-41B5-BA29-93ED57C4765C-low.png
  7. Resynchronize the UR pair by specifying the S-VOL (swap resync).

    Command example:

    pairresync -g oraREMOTE -swaps -IH0

    If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

    The volume on the primary storage system changes to a P-VOL, and the volume on the UR secondary storage system changes to an S-VOL.

    GUID-4CB6FB62-2EFA-4D4A-836F-AE85611840D7-low.png
  8. Delete the UR pair.

    Command example: pairsplit -g oraREMOTE -S -IH0

  9. Re-create the GAD pair.

    Command example: paircreate -g oraHA -fg never 2 -vl -jq 0 -IH0

    If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

    If you accidentally delete the virtual LDEV ID of the P-VOL, you cannot create a GAD pair again. Set the virtual LDEV ID to the P-VOL again using the raidcom map resource command.

    raidcom map resource -ldev_id 0x1111 
    -virtual_ldev_id 0x1111 -IH1

    The default virtual LDEV ID is the same as the actual LDEV ID. After setting the virtual LDEV ID again, verify that the virtual LDEV ID is the same as the actual LDEV ID using the confirmation command.

  10. Re-create the UR delta resync pair.

    Command example:

    paircreate -g oraDELTA -f async -vl -jp 0 -js 0 -nocsus -IH1

    If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.

  11. Re-create the UR pair.

    Command example:

    paircreate -g oraREMOTE -f async -vl -jp 0 -js 0 -IH0

    If a failure occurs after the one volume capacity of a GAD pair can be expanded, the creation, resync, swap resync, and horctakeover operations of the GAD pair cannot be performed because the capacity of both the volumes is not the same. Make sure to expand the other volume capacity so that the capacity of both the volumes is the same, and then retry the operation.