Skip to main content

We've Moved!

Product Documentation has moved to docs.hitachivantara.com
Hitachi Vantara Knowledge

Monitoring the UR system

You can monitor how well Universal Replicator is performing, which allows you to detect and correct issues.

Monitoring pair activity, status, license capacity

Monitoring pairs, journals, data paths, I/O, licensed capacity, and hardware performance is essential for ensuring that Universal Replicator pairs continue to function correctly.

  • Before performing a pair operation, check the pair status to verify that the desired operation can be performed. For details about the pair statuses, see Pair status definitions.
  • The pair status changes when an operation is performed. If the pair status is PAIR, data update is executed correctly from the P-VOL to the S-VOL. If the pair status is PSUS/PSUE, the pair is split or suspended, and therefore differential data is managed.

Frequent monitoring of the system is recommended but optional. You can set up automatic e-mail notification from Device Manager - Storage Navigator for problems found during monitoring.

To perform this task in CCI, use the raidcom get license command to check the license capacity.

Procedure

  1. In the Explorer pane, expand the Storage Systems tree.

  2. Expand the target storage system tree, expand Replication, and click Remote Replication.

  3. In the UR Pairs tab, select the pair that you want to monitor.

  4. Click More Actions View Pair Properties.

  5. In the View Pair Properties window, you can display only the columns you want to see by clicking Filter. Click File/Refresh as needed to display the most current information.

  6. To review license capacity, see the Summary area in the Replication window.

Suspend conditions for Universal Replicator pairs

The following table lists the suspend conditions for Universal Replicator pairs, the storage system that detects the condition, and the volume pairs that are suspended.

Suspend condition Detecting storage system Suspended pair

Logical error was detected in secondary system while selecting journal data that will be restored.

Secondary system

All S-VOLs in journal or affected S-VOL.

Journal data could not be restored in secondary system due to HW failure, track status, or logical error.

Secondary system

Differential bitmap for each P-VOL is stored in shared memory in the primary system. The differential bitmap for each S-VOL is stored in shared memory in the secondary system. A UR pair that uses a DP-VOL V-VOL whose size is more than 4,194,304 MB (8,589,934,592 blocks) does not use shared memory. For tracks that include the following journal data, it will be marked within differential bitmap (since it needs to be copied during pair resync):

  • Journal data that was created at the primary system and not yet sent to the secondary system. The primary system will mark these P-VOL tracks as "changed" and then delete the journal data.
  • Journal data that was sent to the secondary system but not recognized yet. The primary system will mark these P-VOL tracks as "changed" and then delete the journal data. By this, it can be marked as confirmed for journal data that was lost while being sent to the secondary system.
  • Journal data that was delivered to the secondary system but not yet restored. The primary system will mark these P-VOL tracks as "changed" and then delete the journal data. By this, it can be marked as confirmed for journal data that was lost while being sent to the secondary system. The primary system will not delete the target journal data from the master journal volume until receiving the restored journal's sequence number that is added to the read journal command from the secondary system.
  • P-VOL that was updated by write I/O from the host after the pair was suspended.

Monitoring UR pair synchronization rate

You can check the percentage of synchronized data between the P-VOL and S-VOL. To perform this task in CCI, use the pairdisplay -fc command.

The synchronization rate is displayed in HDvM - SN as follows.

  • When the volume on the local storage system is a P-VOL, the initial copy progress is displayed.
  • When the volume on the local storage system is an S-VOL, the synchronization rate depends on whether the pair is split:
    • If the pair is not split, a hyphen (-) is displayed.
    • If the pair is split, the S-VOL synchronization rate between before and after the split is displayed. For example, if the S-VOL content before and after the split is the same, 100 is displayed.
    CautionIf the pair is split due to initial copy failure, a hyphen (-) is displayed. If initial copy fails, the status Initial copy failed is displayed in the View Pair Properties window.
  • For the following cases, a hyphen (-) is displayed:
    • When the volume in the local storage system is neither a P-VOL nor an S-VOL.
    • When the volume in the local storage system has one of the following pair statuses: HOLD, HOLDING, or HLDE.

Procedure

  1. In the Explorer pane, expand the Storage Systems tree.

  2. Expand the target storage system tree, expand Replication, and click Remote Replication.

  3. In the UR Pairs tab, select the pair whose synchronization rate you want to monitor.

  4. Click More Actions View Pair Synchronization Rate.

    Clicking Refresh View displays the latest synchronization rate.

Monitoring UR operations history

In HDvM - SN, you can review the history of operations for each pair, including a description of each operation, the date and time the operation took place, primary and secondary storage system information, and other details.

Note
  • The most recent operations are displayed up to a maximum of 524,288 (VSP 5000 series) or 8,192 (VSP E series) within the last seven days. If the history information of more than the maximum operations is generated within seven days, the excess information will be deleted in the order from oldest to newest. In addition, whenever the date changes, the information on the oldest date is deleted from the stored information, and the information older than seven days is not shown.
  • The LDEV ID displayed in the LDEV ID column is the physical LDEV ID in the storage system, not the virtual LDEV ID used in a virtual storage machine.
  • The displayed operations include user-requested operations (for example, pair creation) and internal operations (for example, completion of pair creation, pair suspension due to failure).
  • If you use 1,000 or more pairs concurrently, some operation history might not be recorded.

Procedure

  1. In the Explorer pane, expand the Storage Systems tree.

  2. Expand the target storage system tree, and click Replication.

  3. Click View History Remote Replication.

  4. In the History window, select UR as the Copy Type.

Pair operations history

The following table describes the operations displayed in the History window.

Operation

Description

Pair definition

A pair was defined.

Paircreate Start

Creation of the pair was started. When you use UR with TC and if you specify Entire Copy for the Delta Resync Failure option (Edit Mirror Options window), Paircreate Start is displayed because all P-VOL data is copied to the S-VOL if delta resync is not executed. When you use UR with GAD, specifying Entire Copy for the Delta Resync Failure option is invalid.

Paircreate Complete

Creation of the pair was finished. When you use UR with TC and if you specify Entire Copy for the Delta Resync Failure option (Edit Mirror Options window), Paircreate Complete is displayed because all P-VOL data is copied to the S-VOL if delta resync is not executed. When you use UR with GAD, specifying Entire Copy for the Delta Resync Failure option is invalid.

Pairresync Start

Restoring of the pair was started.

Pairresync Complete

Restoring of the pair was finished.

Pairsplit-r Start

Splitting (Suspending) of the pair was started.

Pairsplit-r Complete

Splitting (Suspending) of the pair was finished.

Pairsplit-r(Failure)

The pair was split (suspended) because of a failure.

Pairsplit-S Start

Release of the pair was started.

Pairsplit-S Complete

Release of the pair was finished.

Status Change by MCU(SMPL to COPY)

The status of the pair was changed from SMPL to COPY because of an operation from the primary storage system.

Status Change by MCU(SMPL to PAIR)

The status of the pair was changed from SMPL to PAIR because of an operation from the primary storage system.

Status Change by MCU(COPY to PAIR)

The status of the pair was changed from COPY to PAIR because of an operation from the primary storage system.

Status Change by MCU(COPY to PSUS/PSUE)

The status of the pair was changed from COPY to PSUS or PSUE because of an operation from the primary storage system.

Status Change by MCU(PAIR to PSUS/PSUE)

The status of the pair was changed from PAIR to PSUS or PSUE because of an operation from the primary storage system.

Status Change by MCU(PAIR to SMPL)

The status of the pair was changed from PAIR to SMPL because of an operation from the primary storage system.

Status Change by MCU(COPY to SMPL)

The status of the pair was changed from COPY to SMPL because of an operation from the primary storage system.

Status Change by MCU(PSUS/PSUE to SMPL)

The status of the pair was changed from PSUS or PSUE to SMPL because of an operation from the primary storage system.

Status Change by MCU(PSUS/PSUE to COPY)

The status of the pair was changed from PSUS or PSUE to COPY because of an operation from the primary storage system.

Status Change by RCU(Pairsplit-r Start)

The status of the pair was changed because an operation for splitting a pair started at the secondary storage system.

Status Change by RCU(Pairsplit-r Complete)

The status of the pair was changed because an operation for splitting a pair finished at the secondary storage system.

Status Change by RCU(PSUS/PSUE to SMPL; Pairsplit-S Start)

An operation for releasing a pair has been started at the secondary storage system. The status of the pair will change from PSUS or PSUE to unpaired (SMPL).

Status Change by RCU(COPY to SMPL; Pairsplit-S Start)

An operation for releasing a pair has been started at the secondary storage system. The status of the pair will change from COPY to unpaired (SMPL).

Status Change by RCU(PAIR to SMPL; Pairsplit-S Start)

An operation for releasing a pair has been started at the secondary storage system. The status of the pair will change from PAIR to unpaired (SMPL).

Status Change by RCU(Pairsplit-S Complete)

The status of the pair was changed because an operation for releasing a pair finished at the secondary storage system.

Ready for Delta resync

A pair became ready for delta resync.

Ready for Delta resync(Failure)

The failure occurred with the pair which was ready for delta resync.

Status Change for Delta resync

The status of the P-VOL was changed to HOLD because of a delta resync operation.

Status Change by MCU(SMPL to HOLD)

The status of the pair was changed from unpaired (SMPL) to HOLD because of an operation from the primary storage system.

Status Change by MCU(HOLD to PAIR)

The status of the pair was changed from HOLD to PAIR because of an operation from the primary storage system.

Status Change by MCU(HOLD to COPY)

The status of the pair was changed from HOLD to COPY because of an operation from the primary storage system.

Status Change by MCU(HOLD to SMPL)

The status of the pair was changed from HOLD to unpaired (SMPL) because of an operation from the primary storage system.

Status Change by RCU(HOLD to SMPL; Pairsplit-S Start)

An operation for releasing a pair has been started at the secondary storage system. The status of the pair will change from HOLD to unpaired (SMPL).

Status Change to HOLD

The status of the S-VOL was changed to HOLD because of a delta resync operation.

Saving pair information to a text file

In HDvM - SN, you can save pair status and other information to a tab-delimited TSV file.

Procedure

  1. In the Explorer pane, expand the Storage Systems tree.

  2. Expand the target storage system tree, expand Replication, and click Remote Replication.

  3. In the UR Pairs tab, select the pair that you want to monitor.

  4. Click More Actions Export and follow on-screen instructions.

Monitoring copy operations and I/O statistics

You can monitor copy operations and I/O statistics to determine the replication performance.

For more information about monitoring performance, see the Performance Guide.

For the primary site in a 3DC multi-target configuration combining three UR sites, two or more mirrors are set in the primary system. The following table describes the data displayed in Logical Device (UR/URMF) and Journal (UR/URMF) for Performance Monitor in the primary storage system.

Object of monitoring1

Monitoring data

Description

Logical Device (UR/URMF)

Initial copy cache hit (%)

Average of all mirror’s cache hit rates in primary system.

Initial copy data transfer (MB/s)

Total of all mirror’s data transfer amount in primary system during initial copy.

Journal (UR/URMF)

Master journal throughput (IOPS)

Total throughput of all mirrors in the primary system.

Master journal (count/sec)

Total of all mirror’s journal responses in primary system.

Master journal data transfer (MB/s)

Total of all mirror’s data transfer amount in primary system.

Master journal response time (ms)

Average of all mirror’s response time in primary system.

Notes:

  1. The journal information per mirror is displayed in Journal (UR/URMF). Note that it is impossible to reference the initial copy information per mirror.

If the journal group attribute is Master/Restore, only the restore journal information is displayed in Journal (UR/URMF), as shown in the following table.

NoteIn a cascade configuration in which three UR sites are combined, a restore journal value might be greater than the corresponding master journal value.

Object of monitoring

Monitoring data

Description

Journal (UR/URMF)

Restore journal throughput (IOPS)

Restore journal throughput per second.

Restore journal (count/sec)

Number of count for restore journal per second.

Restore journal data transfer (MB/s)

Restore journal data transfer amount.

Restore journal response time (ms)

Restore journal response time.

Restore journal usage data (%)

Restore journal usage data.

Restore journal metadata usage rate (%)

Restore journal metadata usage rate.

Monitoring I/O and hardware performance with Performance Monitor

The Performance Monitor software provides detailed information on I/O activity and hardware performance.

Storage system usage and performance data that is collected and displayed by Performance Monitor enables you to:

  • Identify the optimum timing for performing Universal Replicator copy operations.
  • Determine the best locations for the Universal Replicator S-VOLs (for example, parity groups with less frequently accessed volumes to avoid bottlenecks of back-end activity).
  • Monitor system performance during Universal Replicator operations and during testing activities.

Use the following procedure to lessen the impact on Universal Replicator operations while Performance Monitor is collecting data for one or more storage systems on the same internal LAN:

  1. If Performance Monitor is collecting high amounts of LDEV data, disabling this activity, if possible, for one or more systems.
  2. If Performance Monitor is collecting data for more than three storage systems on the internal LAN, disconnect Performance Monitor, if possible, from one or more systems before using Universal Replicator.
  3. After you have disabled LDEV data collection and/or disconnected Performance Monitor wherever possible, then connect to the system using Device Manager - Storage Navigator and launch Universal Replicator.
  4. When Universal Replicator operations are completed, exit Universal Replicator and Device Manager - Storage Navigator.
  5. Re-enable Performance Monitor data collection.

For further information on Performance Monitor, see the Performance Guide for your storage system.

Monitoring journal (mirror) status

The status of a mirror associated with a journal relates to the pair operations that have been performed on the journal. Monitoring these statuses can help to resolve problems and maintain the Universal Replicator system.

To perform this task in CCI, use the raidcom get journal command.

Procedure

  1. In the Explorer pane, expand the Storage Systems tree.

  2. Expand the target storage system tree, expand Replication, and click Journals.

  3. In the Journals window, select the Journal ID for the journal you want to monitor. The Mirror tab is displayed.

  4. In the Mirror tab, view the journal’s status.

Device Manager - Storage Navigator mirror status definitions

Status

Description

GUID-FFEAABA5-9E03-4947-ACBF-819C6F8A8CE3-low.pngInitial

Data volumes are not registered in the mirror.

GUID-AB4E9070-4852-46FA-AFD0-5F1F78E1C2AE-low.pngActive

The mirror has only pairs that are not synchronized with the initial copy operation in progress, or pairs that are synchronized with the initial copy complete.

If a mirror in Active status contains pairs that are split, Active (Warning) is displayed. The split pairs must be resynchronized to restore the mirror to the Active status.

GUID-095B60F0-DE34-4B77-9E7F-7F3E217EC0DA-low.pngActive (Warning)

The mirror status is Active; some pairs in the mirror are split.

GUID-D4EB3A24-3953-46F1-BD25-57A0DA67AAA5-low.pngHaltAccept

The status at the beginning of a mirror-split operation. The status changes immediately to Halting. HaltAccept indicates status of restore journals only.

GUID-D4EB3A24-3953-46F1-BD25-57A0DA67AAA5-low.pngHalting

An operation for splitting or deleting the mirror is in progress.

  • Splitting a mirror changes the status in the following order: Halting, Halt, Stopping, and finally Stopped.
  • Deleting a mirror changes the status in the following order: Halting, Halt, Stopping, Stopped, and finally Initial.

GUID-D4EB3A24-3953-46F1-BD25-57A0DA67AAA5-low.pngHalt

An operation for splitting or deleting the mirror is in progress.

GUID-D4EB3A24-3953-46F1-BD25-57A0DA67AAA5-low.pngStopping

An operation for splitting or deleting the mirror is in progress.

GUID-D4EB3A24-3953-46F1-BD25-57A0DA67AAA5-low.pngStopped

An operation for splitting or deleting the mirror is finished.

GUID-D4EB3A24-3953-46F1-BD25-57A0DA67AAA5-low.pngHold

When you use UR with TC, a delta resync pair has been created. It is ready for the delta resync operation. Delta resync might not be executed depending on the internal process even when the creation of the delta resync pair is completed. When you use UR with GAD, it is ready for the delta resync operation.

GUID-D4EB3A24-3953-46F1-BD25-57A0DA67AAA5-low.pngHolding

When you use UR with TC, a delta resync pair is being created. When you use UR with GAD, it is preparing for the delta resync operation. Delta resync execution will fail if the status is HOLDING.

GUID-7BA176B1-F8AA-4E73-AF8F-A93098EEDE7C-low.pngHold(Failure)

An error occurred with the UR pair during the delta resync operation.

CCI mirror status definitions

CCI mirror statuses appear beside Device Manager - Storage Navigator statuses in the HDvM - SN/CCI format. The following describe CCI statuses.

Status

Description

SMPL

The mirror is not used.

PJNN

The master journal is normal.

SJNN

The restore journal is normal.

PJSN

The master journal is in normal split status.

SJSN

The restore journal is in normal split status.

PJNS

The master journal is in normal split status (delta resync configuration).

SJNS

The restore journal is in normal split status (delta resync configuration).

PJNF

The master journal is full.

SJNF

The restore journal is full.

PJSF

The master journal is full and split.

SJSF

The restore journal is full and split.

PJSE

The master journal is split due to an error (including link errors).

SJSE

The restore journal is split due to an error (including link errors).

PJES

The master journal is in split status (delta resync configuration) due to an error.

SJES

The restore journal is in split status (delta resync configuration) due to an error.

Monitoring UR remote connections and path status

You can navigate to remote connection and remote path status to determine any issues with the remote paths.

To perform this task in CCI, use the raidcom get rcu command.

Procedure

  1. In the Explorer pane, expand the Storage Systems tree.

  2. Expand the target storage system tree, expand Replication, and click Remote Connections.

  3. Check the Status field.

  4. To view more information, select a connection and click View Remote Connection Properties.