Solr index backup and restore
As an added layer of preparedness on your HCI system, you can backup your Solr indexes and restore them in the event of a catastrophic failure.
- An external, dedicated NFS mount point for each HCI cluster.
- A directory created on each node in the HCI cluster called
solrBackups
, located at the following path:install_path/solrBackups
- The file system from step 1 mounted on each node of the directory listed in step 2.NoteThe mechanism used to mount the file system needs to be able to persist through node reboot.
- Sufficient disk space available on the mounted file system in order to successfully backup your HCI index.
- Sufficient disk space reserved on your HCI nodes in order to successfully restore your HCI index.
install_path/solrBackups
after starting HCI.service, then the Index and SearchApp services will need to be restarted.Index backup API
This API allows you to incrementally back up each HCI index individually. When the operation is initiated, HCI will automatically create a backup for the HCI index in the mounted NFS share. Subsequent backups will result with a new backup ID being created that corresponds with the new state of the HCI Index
POST /recovery/backup/indexes/{indexName}/start
HCI Index Name
- Once a backup for an HCI index has successfully completed, future backup operations will be done incrementally, where only the changes in the HCI index from the last backup are recorded.
- Workflows leveraging the index being backed up will continue to run without error.
- An attempt to update the system will fail a precheck if a backup is in progress.
- Once a backup has been created for an index, you will no longer be able to increase the desired shard count of that HCI index.
- If the desired shard count is in the process of being increased on an HCI index, backup operations cannot be initiated on the HCI cluster of any HCI index.
- Deleting an HCI index does not delete its backups
Index restore API
This API allows you to restore a backup or a specific backup number of an HCI index. If a backup number is not provided, the latest backup is restored by default.
While a restore is in progress, alerts and events are generated that inform you of its progress, success, or failure.
POST /recovery/restore/indexes/{indexName}/start
- Source HCI Index Name
- The source HCI index must be on HCI before this operation is performed. Additionally, both Source HCI Index Name and Destination HCI Index Name can be the same value.
- Destination HCI Index Name
- If the Destination HCI Index Name doesn't exist, the source HCI index will be cloned and its name used instead. The collection under the existing index will also be overwritten.
- backupID (optional)
- To restore a Solr backup to a cluster other than the one it was originally set up on, you need to copy the backup information stored under the NFS mount point for the original cluster to the NFS mount location of the new cluster.
- You will be able to initiate one restore at a time per Destination HCI Index Name. You may, however, issue multiple restore operations using different Destination HCI Index Names.
- If the Destination HCI Index is currently in a running workflow, the restore operation will fail.
- Workflows leveraging the Source HCI Index Name will continue to run without any errors.
- An attempt to update the system will fail a precheck when a restore operation is in progress.
- If an HCI index has been backed up, it can only be restored by one of its own backups.
- Attempts to restore to a non-existent Destination HCI Index Name that previously had backups will fail unless all backups related to the HCI index are deleted from the NFS mountpoint.
- The existing index settings will be maintained. The settings that the index receives from Solr for the collection are intended to reflect the settings of the backup.
- Metrics will be invalid after recovering from a Solr backup.
- If you are using an MQE connector:
- Enable Customize the Query Range.
- Set Custom Query Start Time (millisec) to the epoch timestamp of when the restored backup was taken.
- Click Workflow task > Actions > Start Over. This will start the workflow from the timestamp specified in step 2.
- If you are not using an MQE connector:
- Remove the Destination Index from your workflow.
- Add a new HCI index to the workflow.
- Click Workflow task > Actions > Start Over.
- Use the Destination Index for all query operations until the new HCI index from step 2 has caught up to the Destination Index.
- The schema will be the equivalent to the schema of the source index.
- The IPL of the index is not maintained by the restore.
- If an index is restored to a non-existent index, or if a restore needed to delete the underlying collection to succeed in processing, the IPL will be set to 1. This is the default.
- If an index is successfully restored over an existing collection, the IPL of that index is inherited.
- You will not be able to increase the desired shard count on a restored HCI index. A restore operation will always result in an index with a shard count equal to that of the index which ran the backup.
- If the shard count of the destination index is lower than the backup shard count, the restore will complete successfully and will have a shard count equal to the backup shard count.
- If the shard count of the destination index is higher than the backup shard count, the restore operation will fail in error due to the shard counts not matching.
Backup and restore status API
This API provides the status of the latest backup or restore of an HCI index.
GET /recovery/backup/indexes/{indexName}/status
GET /recovery/restore/indexes/{indexName}/status
- HCI Index Name
- Destination HCI Index Name (if restoring)
- JSON object of the backup/restore state:
IN PROGRESS
COMPLETED
FAILED
NONE
- Solr message regarding the backup/restore operation
- The full backup/restore status, provided as a JSON response from Solr
List backups API
This API provides information about each stored backup for the index name provided. The index does not need to exist in HCI.
GET /recovery/backup/indexes/{indexName}/list
HCI Index Name
JSON objects representing the completed backup:
startTime
backupID
Delete backup API
purgeUnused
parameter, as a known bug can delete files in use and cause data loss on your system.POST /recovery/backup/indexes/{indexName}/delete
- HCI Index Name
- Your choice of one of the following JSON values:
backupID
: Deletes a backup by its specific IDmaxNumBackups
: Specifies how many recent backups should be retained, deleting all othersdeleteAllBackups
: When set toTrue
, all backups are deleted