Replication and disaster recovery
Object replication provides replication of file systems, including the replication of related access points, such as SMB shares and NFS exports, as well as tools to automate disaster recovery.
Object replication has several important concepts:
- Primary file system: The "primary file system" is the file system that network clients access. The primary file system is the "live" file system and the source of the replication.
- Direction of replication: When the primary file system from site "A" is replicated to another server at site "B," the direction of the replication determines which file system is designated the "source" and the "target." The primary file system from site "A" is always the replication source. The target file system is the replica on the server at site "B" (which may located at the same physical site as "A" or at a remote location). Note that the target file system at site "B" may also be used as a replication source to a third site (site "C").
- Swapping roles: Moving the network client access from the file system at site "A" to the replicated file system at site "B." Roles are swapped when the replication either stops (for example, if site "A" goes offline for some reason) or the direction of the replication is intentionally reversed (a planned role swap).
- During the role swap, the file system at site "B" is promoted to primary, and the file system at site "A" is demoted. (If site "A" is accessible, the file system at site "A" typically becomes the target file system.)
- As a part of the role swap, access point (SMB share or NFS export) settings are deleted from the server at site "A" and, along with other configuration settings, are applied to the server at site "B" so that network clients now access the primary file system, which is now physically located at site "B." Clients accessing the file system now communicate with the server at site "B" and read from and write to the file system at that site, which has become the primary file system.
- Note that a single server can host many file systems, and could be providing "primary" access to several file systems while other file systems hosted by the same server could be target file systems. Primary access for any file system can be moved independently of any other file system on the same server.
Object replication is most often used in the following situations:
- A planned promotion of the file system at site "B" to primary. In this case, it is possible to ensure that the file systems at sites "A" and "B" are exact replicas (though this would require a period of read-only access at site "A"). If both sites are functional, it may be possible for the server at site "B" to access the server at site "A" to retrieve information such as configuration settings.
In the case of a planned promotion, the administrator puts the primary file system into syslocked-mode, then schedules an incremental replication to the target to ensure both file systems are synchronized. Once the replication is complete, the promotion can proceed, and after the transfer of primary access, the clients access the newly promoted primary file system.
- An unplanned promotion of the file system at site "B" to primary (also known as "disaster recovery". If, for any reason, the primary file system at site "A" becomes inaccessible, the file system at site "B" is promoted (becomes primary). In this case it is unlikely that the file system at site "B" will be an exact copy of the file system at A at the time of the outage (because the replication is asynchronous). Server B must already have access to all the information necessary to function as the primary.
Replication process for planned promotions
Procedure
On the primary file system, create a replication policy to synchronize the primary and target file systems.
Create and enable a replication schedule to perform the replication.
Synchronize the source and target file systems.
Syslock the primary (source) file system and perform the final replication.
Swap roles, which promotes the target file system to become the primary file system.
Verify that all access points have been created.
Redirect network clients to the new primary file system.
Verify that clients can access the newly promoted primary file system.
Demote the original source file system to become a replication target.
If the replication policy schedule was disabled, reactivate it.
Verify that the replication runs successfully.
Allow user access to the new primary file system.
Recovering a file system
Procedure
Navigate to to display the File System Versions page.
The following table describes the fields in this page:
Field/Item Description File System Details This section displays the name of the EVS hosting the file system, and the currently selected file system that can be recovered from the snapshots listed in the Versions section. EVS/File System Displays the name of the currently selected EVS and file system. Click change to select a different file system. Status Displays the current mount status of the file system. The file system status may be Syslocked, Checking, Unmounted, Mounted, Mounted as read only or Mounted as a replication target. Object Replication Details For Latest Version If the file system is a replication target, this section displays the status of the most recent replication and information about the replication source. If the currently selected file system is not a replication target, or the replication information cannot be retrieved (if the source server is not known to the replication process), the Source File System, Source Server, and Source File System Status fields are not displayed. Status If the currently selected file system is a replication target, this field displays a status indicator and a message about the most recent replication. If the file system is not a replication target, the status indicator is greyed out and the message reads "Latest version is not an object replication target". The status indicator is green if the most recent object replication completed successfully. If the first full replication is currently running, this status indicator is grey.
If the currently selected file system is not involved in a replication policy (neither the source or the target), then the message reads "No status found".
If a replication associated with this file system has not yet run, the light is grayed out, and the message reads, "No status found".
Source File System If the currently selected file system is a replication target, this field displays the name of the source EVS and file system. If the currently selected file system is not a replication target, this field is not displayed.
Source Server If the currently selected file system is a replication target, this field displays the name of the server hosting the EVS/replication source file system. If the currently selected file system is not a replication target, this field is not displayed.
Source File System Status If the currently selected file system is a replication target, this field displays the current mount status of the replication source file system. The file system status may be Not mounted, Mounted, Mounted as read-only, Syslocked, or Checking. If the currently selected file system is not a replication target, this field is not displayed.
Versions This section lists versions of this file system that are in available snapshots, and identifies the snapshot copied to the replication target and the replication source snapshot. Time of Version The date and time the object replication policy last ran. "Time" refers to when the snapshot was taken. Version Identifies the specific snapshot copied to the replication target. Replicated From Snapshot Identifies the replication's source snapshot. Recover File System To Version Opens the File System Recovery Selection page, on which you choose the type of file system recovery you want. Recover Multiple File Systems To Version Opens the Recover File Systems page, on which you choose options for recovering multiple file systems. File System Recovery Reports Opens the file system File System Reports page. Object Replication Policies & Schedules Opens the main Object Replication page. Click Recover File System to Version.
Use this page to begin the recovery process and either:
- Promote the file system to a normal file system (and optionally, mount it as read-write or read-only).
- Demote the file system to an object replication target (and mount it as an object replication target).
In order to promote or demote a file system, the process rolls back the file system to the last successful replicated snapshot. You must check the available snapshots so that you can choose the most recent (in order to minimise data loss due to promoting a previous version of the file system). Using your browser, go back to the File System Versions page and note the time and version of the snapshots, so you can choose the most recent successful replication snapshot.
In a disaster recovery scenario, your primary system will probably be unavailable, so you must access the file system version using the File System Versions page of the backup system at the recovery site. Note the time of the replication snapshot and the versions of the source and target. You will use the latest version when promoting the file system to a normal file system.
Recovering and promoting a file system
Procedure
From the File System Versions page, select the "Recover File System To Version" option and then click "Promote the file system to a normal file system (and, optionally, mount as read-write or read-only)" to display the Recovery File System page.
In step 2, Recover file system to version created at, on the Recover File System page, verify that the file system version, the snapshot name, and the snapshot source are correct.
In step 3, Promote file system 'name' and, select one of the following mount options for the recovered file system:
- mount read write
- mount read only
- do not mount (selecting this option disables the "Recover access points" option - no access points are moved)
In step 4, Recover access points, fill in the check boxes to specify which types of file system access points to recover. Leave the check boxes empty to specify not to recover SMB shares or NFS exports for the recovered file system.
- shares
- exports
For NFS exports, when promoting a replication target file system on the same EVS as the original source file system, there are two additional options to determine which file system the NFS clients will access after the recovery process:
- Clients will continue to access source file system
Enable this option so that the exports that are recovered on the target file system are named
<export name>_<task id>to avoid any conflicts with<export name>on the source. Both file systems will have their own set of exports. - Clients will access target file system without interruption
Enable this option so that exports are moved from the source file system to the target without the need to remount NFS v2/3 clients. The source file system will no longer have the recovered exports assigned to it.
- Clients will continue to access source file system
Click next to display the Recover File System Confirmation page.
Verify the file system recovery settings, and click OK to proceed with the file system recovery, and display the File System Recovery Report page.
Monitor the file system recovery.
The following table describes the fields in this page:
Field/Item Description File System Details Display the name of the currently selected EVS and file system, along with its status. Recovery Details Displays the progress of the current or last run recovery, and status of the recovery. Progress Displays the progress of the current or last run recovery, including start and end times. Request Summary Displays the recovery options in use, including the name of the snapshot used for rollback and the share/export handling options. Source File System "Transfer Access Point" Setting After Object Replication completes, it promotes the file system to a normal file system on the target EVS. That is, Object Replication transfers the access point to the target file system. By default, it uses the source file system. Recovery Statistics Displays NFS Exports and SMB Shares import statistics. abort Stops the active recovery operation. View Log Opens the File System Recovery Report Log page. Use your browser’s back button to return to the previous page, or click abort to abort an active recovery operation.
Results
Recovering and demoting a file system
From the File System Versions page, select the "Recover File System To Version" option and then click "Demote the file system to an Object Replication Target (and mount as an Object Replication Target)" to display the Demote File System To Object Replication Target page.
Specify recovery options.
NoteAfter you have selected all the options on this page, it will take a few minutes for the server to roll back the file system and remove all the access points. You can monitor the server's progress on the File System Recovery Progress" page.Field/Item Description File System Details Displays the name of the EVS hosting the file system, and the currently selected file system that can be recovered from the snapshots listed in the Versions section. EVS/File System Displays the name of the currently selected EVS and file system. Status Displays the current mount status of the file system. The file system status might be unmounted, mounted, or mounted as a replication target. Object Replication Details For Latest Version If the file system is a replication target, this section displays the status of the most recent replication and information about the replication source. If the currently selected file system is not a replication target, or the replication information cannot be retrieved (if the source server is not known to the replication process), the Source File System, Source Server, and Source Server fields are not displayed. Status If the currently selected file system is a replication target, this field displays a status indicator and a message about the most recent replication. If the file system is not a replication target, the status indicator is greyed out and the message "Not an object replication target" is displayed. The status indicator is green if a replication is currently running, or if the most recent object replication completed successfully.
If a replication associated with this file system has not yet run, the light is grayed out, and the message reads, "Not an object replication target".
Source File System If the currently selected file system is a replication target, this field displays the name of the source EVS and file system. If the currently selected file system is not a replication target, this field is not displayed.
Source Server If the currently selected file system is a replication target, this field displays the name of the server hosting the EVS/replication source file system. If the currently selected file system is not a replication target, this field is not displayed.
Source File System Status If the currently selected file system is a replication target, this field displays the current mount status of the replication source file system. The file system status might be unmounted or mounted. If the currently selected file system is not a replication target, this field is not displayed.
The following steps will be taken This section lists the recovery steps, and allows you to specify recovery options. - Displays the file system to be recovered.
- Displays the date and time the snapshot to be used to recover the file system was taken. You can use the drop-down list to select a different snapshot by the date and time the snapshot was taken.
For the selected snapshot, the name of the corresponding snapshot on the target file system and the name of the snapshot on the source file system are also displayed.
- Displays the recovery goal to confirm that the file system you have chosen will be demoted to an object replication target.
- Check boxes allow you to specify if you want the access points (SMB shares and NFS exports) of the demoted file system to be removed. Fill the check boxes of the access points you want to remove. Leave the check boxes empty to keep the access points for the demoted file system.
NoteIn general, when demoting a file system as a part of disaster recovery, you should remove both SMB shares and NFS exports.
Click next to display the Demote File System to Object Replication Target Confirmation page.
Verify the file system recovery settings and proceed with the file system recovery.
- Click back to return to the Demote File System To Object Replication Target page.
- Click OK to begin the recovery, and display the File System Recovery Report page.
- Click cancel to return to the File System Versions page.
Monitor the file system recovery.
Field/Item Description File System Details Display the name of the currently selected EVS and file system, along with its status. Recovery Details Displays the progress of the current or last run recovery, and status of the recovery. Progress Displays the progress of the current or last run recovery, including start and end times. Request Summary Displays the recovery options in use, including the name of the snapshot used for rollback and the share/export handling options. Recovery Statistics Displays NFS Exports and SMB Shares import statistics. abort Stops the active recovery operation. View Log Opens the File System Recovery Report Log page. Use your browser's back button to return to the previous page, or click abort to abort an active recovery operation.
Recovering multiple file systems
Procedure
Navigate to to display the File System Versions page.
Click Recover Multiple File System to Version to display the Recover File Systems page.
Select the file systems to recover from the Available File Systems list, and click the right arrow to the add the file systems to the Selected File Systems list.
Specify how the recovered file systems are to be mounted:
- mount read write
- mount read only
- do not mount (selecting this option disables the 'Recover access points' options)
In step 4, Recover access points, select the check boxes to specify which types of file system access points to recover. Leave the check boxes empty to specify not to recover SMB shares or NFS exports for the recovered file systems.
- shares
- exports
Click next to display the Recover File Systems Confirmation page.
Verify the file system recovery settings, and click OK to proceed with the file system recovery, and display the File System Recovery Reports page.
Monitor the file system recovery.
Results