Instances
A system is made up of one or more instances of the software. This section includes information on adding and removing instances to the system.
About master and worker instances
Master instances are special instances that run an essential set of services, including:
- Admin-App service
- Cluster-Coordination service
- Synchronization service
- Service-Deployment service
Non-master instances are called workers. Workers can run any services except for those listed previously.
Single-instance systems have one master instance while multi-instance systems have either one or three master instances.
Single-instance systems vs. multi-instance systems
A system can have a single instance or can have multiple instances (four or more).
- Every instance must meet the minimum RAM, CPU, and disk space requirements.
- Three instances are sufficient to perform leader election for distributing work. However, a multi-instance system needs a minimum of four instances because, with the minimum hardware requirements, three instances are not sufficient for running all HCI services at their recommended distributions.
- Hitachi Vantara has qualified HCI systems with up to 16 instances.
A single-instance system is useful for testing and demonstration purposes. It needs only a single server or virtual machine and can perform all product functionality.
However, a single-instance system has these drawbacks:
-
Only a single point of failure. If the instance hardware fails, you lose access to the system.
-
With no additional instances, you cannot choose where to run services. All services run on the single instance.
A multi-instance system is suitable for use in a production environment because it offers these advantages over a single-instance system:
- You can control how services are distributed across the multiple instances, providing improved service redundancy, scale out, and availability.
- A multi-instance system can survive instance outages. For example, with a four-instance
system running the default distribution of services, the system can lose one instance and
still remain available.Note For a search index to survive an instance outage:
- The system must have at least two instances running the Index service.
- The Index Protection Level for the index must be at least 2.
For more information, see the HCI Administrator Help, which is available in the Admin App.
-
Performance is improved as work can be performed in parallel across instances.
-
You can add additional instances to the system at any time.
By adding additional instances to a single-instance system, your system still has only one master instance, meaning there is still a single point of failure for the essential services that only a master instance can run.
For information about adding instances to an existing HCI system, see the Content Intelligence Administrator Help, which is available from the Admin App.
Two-instance systems are a viable option for the HCM use case, but not recommended for Hitachi Content Search.
Three-instance systems should have only a single master instance. If you deploy a three-instance system where all three instances are masters, the system might not have enough resources to do much beyond running the master services.
Requirements for running system instances
This section lists the hardware and operating system requirements for running system instances.
Hardware resources
This table lists:
- Minimum: The minimum amounts of RAM, CPU, and available disk space needed to run a system instance.
Every server or virtual machine you provide as an instance must meet these minimums.
- Better: The amounts of RAM, CPU, and available disk space that is better for system instances.
A system in which all instances meet or exceed these values can index more documents and can process documents faster than a system whose instances meet only the minimum requirements.
Resource | Minimum | Better |
RAM | 16 GB | 32 GB |
CPU | 4-core | 8-core |
Available disk space | 50 GB | 500 GB |
Depending on how you use your system, you might require additional hardware resources to index all the documents you want and at the rate you require. See Detailed sizing.
For more information on sizing your Hitachi Content Intelligence system, see the Administrator Help, which is available from the Admin App.
Sizing guidance for Hitachi Content Search
Simple sizing
This table shows the minimum and recommended hardware requirements for each instance in an HCI running Hitachi Content Search.
Resource |
Minimum |
Recommended |
RAM |
16 GB |
32 GB |
CPU |
4-core |
8-core |
Available disk space |
50 GB |
500 GB |
- A large number of factors determine how many documents your system can index and how fast it can process them, including: the number of documents to be indexed; the contents of those documents; what search features (such as sorting) the index supports; the number of fields in the index; the number of users querying the system; and so on.
Depending on how you use your system, you might require additional hardware resources to index all the documents you want and at the rate you require.
- Each instance uses all available RAM and CPU resources on the server or virtual machine on which it's installed.
Detailed sizing
To determine the system size that you need:
Procedure
Determine how many documents you need to index.
Based on the number of documents you want to index, use the following tables to determine:
- How many instances you need
- How much RAM each instance needs
- The Index service configuration needed to support indexing the number of documents you want
Total documents to be indexed System configuration 15 million 25 million 50 milliona Total instances required: 1b
Instances running the Index service: 1
Index service configuration required:
- Shards per index: 1
- Index Protection Level per index: 1
- Container memory: 200MB greater than Heap settings
- Heap settings: Depends on instance RAM.
Instance RAM Heap setting 16 GB 1800m 32 GB 9800m 64 GB 25800m
16 GB 32 GB 64 GB Instance RAM needed (for each instance running the Index service) a Contact Hitachi Vantara for guidance before trying to index this many documents on this number of instances. At this scale, your documents and required configuration settings can greatly affect the number of documents you can index.
b Single-instance systems are suitable for testing and development, but not for production use.
Total documents to be indexed System configuration 45 million 75 million 150 milliona Total instances required: 4
Instances running the Index service: 3
Index service configuration required:
- Shards per index: 3
- Index Protection Level per index: 1
- Container memory: 200MB greater than Heap settings
- Heap settings: Depends on instance RAM.
Instance RAM Heap setting 16 GB 1800m 32 GB 9800m 64 GB 25800m
16 GB 32 GB 64 GB Instance RAM needed (for each instance running the Index service) a Contact Hitachi Vantara for guidance before trying to index this many documents on this number of instances. At this scale, your documents and required configuration settings can greatly affect the number of documents you can index.
Total documents to be indexed System configuration 75 million 125 million 250 milliona Total instances required: 8
Instances running the Index service: 5
Index service configuration required:
- Shards per index: 5
- Index Protection Level per index: 1
- Container memory: 200MB greater than Heap settings
- Heapb settings: Depends on instance RAM.
Instance RAM Heap setting 16 GB 7800m 32 GB 15800m 64 GB 31000m
16 GB 32 GB 64 GB Instance RAM needed (for each instance running the Index service) a Contact Hitachi Vantara for guidance before trying to index this many documents on this number of instances. At this scale, your documents and required configuration settings can greatly affect the number of documents you can index.
b With an 8-instance system, the Index service should be the only service running on each of its 5 instances. With the Index service isolated this way, you can allocate more heap space to the service than you can on a single or 4-instance system.
Total documents to be indexed System configuration 195 million 325 million 650 milliona Total instances required: 16
Instances running the Index service: 13
Index service configuration required:
- Shards per index: 13
- Index Protection Level per index: 1
- Container memory: 200MB greater than Heap settings
- Heapb settings: Depends on instance RAM.
Instance RAM Heap setting 16 GB 7800m 32 GB 15800m 64 GB 31000m
16 GB 32 GB 64 GB Instance RAM needed (for each instance running the Index service) a Contact Hitachi Vantara for guidance before trying to index this many documents on this number of instances. At this scale, your documents and required configuration settings can greatly affect the number of documents you can index.
b With a 16-instance system, the Index service should be the only service running on each of its 13 instances. With the Index service isolated this way, you can allocate more heap space to the service than you can on a single or 4-instance system.
For example, if you need to index up to 150 million documents, you need at minimum a 4-instance system with 64 GB RAM per instance.
Determine how fast you need to index documents, in documents per second.
For example:
- To index 100 million documents in 2 days, you need an indexing rate of 578 documents per second.
- To continuously index 1 million documents every day, you need an indexing rate of 12 documents per second.
Determine the base indexing rate for your particular dataset and processing pipelines:
- Install a single-instance HCI system with that has the minimum required hardware resources.
- Run a workflow with the pipelines you want and on a representative subset of your data.
- Use the workflow task details to determine the rate of documents processed per second.
To determine the number of cores you need per instance, replace Base rate in this table with the rate you determined in step 4.
Number of instances you need Cores per instance 4 (minimum required) 8 (recommended) 1 Base rate 70% Base rate 4 300% Base rate 500% Base rate 8 600% Base rate 900% Base rate More than 8 Contact Hitachi Vantara for guidance For example, if you had previously determined that:
- You need a 4-instance system.
- You need to process 500 documents per second.
- The base processing rate for your data and pipelines is 100 documents per second.
You need 8 cores per instance.
Multiply the number of instances you need times the number of cores per instances to determine the total number of cores that you need for your system.
After your system is installed, configure it with the index settings you determined in step 2.
For information on index shards, Index Protection Level, and moving the Index service, see the Administrator Help, which is available from the Admin App.
Sizing guidance for HCM
Minimum hardware requirements
If you are installing HCI to run HCM, each instance in the system must meet these minimum hardware requirements:
Documents per second | Cores | RAM (GB) | Disk (GB) |
Up to 1200 | 8 | 28 | 600 |
1200-1600 | 12 | 32 | 800 |
1600-2000 | 16 | 40 | 1000 |
2000-2400 | 18 | 48 | 1400 |
2400-2800 | 20 | 56 | 1700 |
2800-3200 | 24 | 64 | 2000 |
Determining number of instances
- Whether you need the system to remain highly available.
- The number of documents being produced by the HCP system you want to monitor. In this case, each document represents a single piece of data about the HCP system. A more active HCP system will produce more documents than a less active one.
- The total number of documents you want HCM to store.
Number of instances: simple procedure
If you're monitoring a typically-active HCP system (roughly 75 operations per second per node), you can use this table to determine the number of HCM instances you need. This table lists the number of HCM instances you need based on the number of nodes in your HCP system and the number of days that you want your HCM system to retain the data it receives from HCP.
If your system is more active, see Number of instances: detailed procedure.
HCP nodes | Data retention time on HCM | Instances needed |
Up to 8 | Up to 30 days |
1* |
Up to 8 | Up to 60 days |
3* |
Up to 16 | Up to 30 days | 4 |
Up to 24 | Up to 60 days | 8 |
*An HCM system must have a minimum of 4 instances to maintain high system availability. |
Number of instances: detailed procedure
Determine whether you need your HCM system to maintain high availability. If so, you need a minimum of 4 instances. For more information, see Single-instance systems versus multi-instance systems.
Determine the number of documents per second being produced by the HCP system you want to monitor. You can easily do this if you already have an HCM system up and running:
Go to the Monitor App:
https://system-hostname:6162
Add the HCP system as a source. For information, see the help that's available from the Monitor App.
Go to the HCI Admin App:
https://system-hostname:8000
Go to Workflows > Monitor App Workflow > Task > Metrics.
View the value for the Average DPS field.
TipLet the workflow run for a while to get a more accurate measure for the Average DPS field.
Select a time period.
Download the HCP Internal Logs for this time period. For more information, see the help that's accessible from the HCP System Management Console.
In the downloaded logs for each node, count the number lines logged during the selected time period.
Add the line value for each node and then divide the sum by the number of seconds in the time window you selected.
Use this table to determine the number of instances needed based on the number of documents per second produced by your HCP system.
Documents per second Instances needed ≤ 3,200
1
3,201 to 7,200
3
7,201-10,500*
4
*This is the maximum documents per second that HCM currently supports. Based on your data availability requirements, determine the number of instances you need.
Data availability requirement Index replicas needed Instances needed Impact on total documents stored No failure tolerance
1
1 None Survive 2 failed replicas
3
3 3x
Survive 3 failed replicas
4 4 4x
An index with multiple copies remains available in the event of an instance outage. For example, if an index has two copies stored on two instances and one of those instances fails, one copy of the index remains available for servicing requests.
Use this formula to determine the total number of documents your HCM system must be able to store:
documents per second from step 2.
x 3600 seconds in an hour
x 24 hours in a day
x number of days you want to store data (default is 30)
x Impact from the data availability table in step 4
.= Total document count
For example, if your HCP system produces 1500 documents per second, you want to store data for 30 days, and you want to maintain two copies of each index containing the stored data, your system must have enough instances to be able to store roughly 8 billion documents:
1500
x 3600
x 24
x 30
x 2
= 7,776,000,000
Use this table to determine the number of instances needed based on the total number of documents your HCM must store.
Total document count Instances needed 2 billion or less
1
6 billion or less
3
8 billion or less
4
Take the highest number of instances from steps 2, 3, and 6. That's the number instances you need.
Operating system and Docker minimum requirements
Each server or virtual machine you provide must have the following:
- 64-bit Linux distribution
- Docker version 1.13.1 or later installed
- IP and DNS addresses configured
Additionally, you should install all relevant patches on the operating system and perform appropriate security hardening tasks.
Operating system and Docker qualified versions
This table shows the operating systems, Docker and SELinux configurations with which the HCI system has been qualified.
Operating system | Docker version | Docker storage configuration | SELinux setting |
Fedora 27 | Docker 18.09.0-ce | direct-lvm | Enforcing |
Red Hat Enterprise Linux 7.4 | Docker 18.09.0-ce | direct-lvm | Enforcing |
Ubuntu 16.04-LTS | Docker 17.03.0-ce | aufs | N/A |
CentOS 7.4 | Docker 18.09.1-ce | overlay2 | Enforcing |
Docker considerations
The Docker installation folder on each instance must have at least 20 GB available for storing the Docker images.
Make sure that the Docker storage driver is configured correctly on each instance before installing the product. After you install the product, to change the Docker storage driver you must reinstall the product. To view the current Docker storage driver on an instance, run:
docker info
Core dumps can fill a host's file system, which can result in host or container instability. Also, if your system uses the data at rest encryption (DARE) feature, encryption keys are written to the dump file. It's best to disable core dumps.
To enable SELinux on the system instances, you need to use a Docker storage driver that SELinux supports. The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.
If you are using the Docker devicemapper
storage driver:
- Make sure that there's at least 40 GB of Docker metadata storage space available on each instance. The product needs 20 GB to install successfully and an additional 20 GB to successfully update to a later version.
To view Docker metadata storage usage on an instance, run:
docker info
- On a production system, do not run
devicemapper
inloop-lvm
mode. This can cause slow performance or, on certain Linux distributions, the product might not have enough space to run.
SELinux considerations
- You should decide whether you want to run SELinux on system instances and enable or disable it before installing additional software on the instance.
Enabling or disabling SELinux on an instance needs a restart of the instance.
To view whether SELinux is enabled on an instance, run:
sestatus
- To enable SELinux on the system instances, you need to use a Docker storage driver that SELinux supports.
The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.
Time source
If you are installing a multi-instance system, each instance should run NTP (network time protocol) and use the same external time source. For information, see support.ntp.org.
File ownership considerations
Within some of the Docker containers on each system instance, file ownership is assigned to this user and group:
- User: hci, UID: 10001
- Group: hci, GID: 10001
When you view such files in the instance operating system
(for example, by running ls -l
), the files appear to be owned by an
unknown or undefined user and group. Typically, this causes no issues.
However, if you run applications on the system instances that change file ownership (for example, security hardening scripts), changing the ownership of files owned by the hci user and group can cause the system to become unresponsive.
To avoid these issues:
- Create the expected user and group on each instance:
sudo groupadd hci -g 10001
sudo useradd hci -u 10001 -g 10001
- Configure your applications to not change the ownership of files owned by the hci user and group.
Adding new instances
You might want to add additional instances to the system if:
- You want to improve system performance.
- You are running out of disk space on one or more instances.
However, these situations might also be improved by adding additional CPU, RAM, or disks to the instances you already have. For guidance, see Sizing guidance for HCM or Sizing guidance for Hitachi Content Search.
When adding HCM instances to your HCI system, there are several guidelines to consider to ensure the best performance.
- If you add only a single worker node to an existing HCM instance, you will begin to receive alerts that your system services are running below their minimum recommended requirements. It is strongly advised that at least two instances are added.
- After adding new instances, you need to pause and resume the workflow in order for them to be parsed by the workflow-agent job.
- If you do not have acess to the Workflow Designer app, editing the description of the signal source will trigger the workflow to pause and resume, which will allow the instances to be correctly parsed.
Before adding a new instance
- Obtain the Hitachi Content Intelligence-<version-number>.tgz installation file. When adding an instance, you unpack and deploy this file on a bare-metal server or a pre-existing Linux virtual machine.
- Record the IP address of at least one of the master instances in the system.
If your system uses internal and external networks, you need to record both the internal and external IP addresses for the master instances.
You can view instance IP addresses on the Instances page in the Admin App.
- Ensure that the new instances you are adding meet the minimum hardware, OS, and networking requirements. For information, see Requirements for running system instances.
- Record the Docker volume drivers currently used by services and jobs across all existing instances. You need to install all of these volume drivers on the new instance that you're adding.
To find the volume drivers currently in use by your system, run this command on each system instance:
docker volume ls
Take note of each value for the DRIVER field.
Install Docker on each server or virtual machine
On each server or virtual machine that is to be an HCI instance:
Procedure
In a terminal window, verify whether Docker 1.13.1 or later is installed:
docker --version
If Docker is not installed or if you have a version before 1.13.1, install the current Docker version suggested by your operating system.
The installation method you use depends on your operating system. See the Docker website for instructions.
Configure Docker on each server or virtual machine
Procedure
Ensure that the Docker installation folder on each instance has at least 20 GB available for storing the product Docker images.
Ensure that the Docker storage driver is configured correctly on each instance. After installation, changing the Docker storage driver needs reinstallation of the product.
To view the current Docker storage driver on an instance, run: docker info .To enable SELinux on the system instances, use a Docker storage driver that SELinux supports.
The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.If you are using the Docker
The product needs 20 GB to install successfully and an additional 20 GB to successfully update to a later version.To view Docker metadata storage usage on an instance, run: docker infodevicemapper
storage driver, ensure that there's at least 40 GB of Docker metadata storage space available on each instance.
Next steps
devicemapper
in loop-lvm
mode. This can cause slow performance or, on certain Linux distributions, the product might not have enough space to run.(Optional) Configure Docker volume drivers
If any services or jobs on your system are using Docker volume drivers (that is, not the bind-mountsetting) for storing data, you need to install those volume drivers on the new instance that you are adding. If you don't, jobs and services might fail to run on the new instance.
Volume drivers are provided by Docker and other third-party developers, not by the system itself. For information on volume drivers, their capabilities, and their valid configuration settings, see the applicable Docker or third-party developer's documentation.
Configure maximum map count setting
sysctl.conf
.Procedure
On each server or virtual machine that is to be a system instance, open the file /etc/sysctl.conf.
Append this line:
If the line already exists, ensure that the value is greater than or equal tovm.max_map_count = 262144
262144
.Save and close the file.
Optional: Enable or disable SELinux on each server or virtual machine
You should decide whether you want to run SELinux on system instances before installation.
Procedure
Enable or disable SELinux on each instance.
Restart the instance.
Configure the firewall rules on each server or virtual machine
Before you begin
install_path/config/network.config
.Procedure
Edit the firewall rules to allow communication over all network ports that you want your system to use. You do this using a firewall management tool such as
firewalld
.Restart the server or virtual machine.
Install and configure NTP
Install NTP (Network Time Protocol) on the new server or virtual machine and configure it to use the same time source as the other system instances. For information, see http://support.ntp.org.
Run Docker on each server or virtual machine
On each server or virtual machine that is to be a system instance, you need to start Docker and keep it running. You can use whatever tools you typically use for keeping services running in your environment.
For example, to run Docker using systemd:
Procedure
Verify that Docker is running:
systemctl status dockerIf Docker is not running, start the
sudo systemctl start dockerdocker
service:(Optional) Configure the Docker service to start automatically when you restart the server or virtual machine:
sudo systemctl enable docker
Unpack the installation package
Procedure
Download the installation package
hci-version_number.tgz
and store it in a folder on the server or virtual machine.In the largest disk partition on the server or virtual machine, create a folder named install_path/hci:
mkdir install_path/hciMove the installation package from the folder where you stored it to the folder install_path/hci:
mv hci-version_number.tgz install_path/hci/hci-version_number.tgzNavigate to the installation folder:
cd install_path/hciUnpack the installation package:
tar -zxf hci-version_number.tgzMultiple folders are created upon unpacking in the installation folder.NoteIf you encounter problems unpacking the installation file (for example, the error message "tar: This does not look like a tar archive"), the file might have been packed multiple times during download. Use the following commands to fully extract the file:
$ gunzip hci-version_number.tgz
$ mv hci-version_number.tar hci-version_number.tgz
$ tar -zxf hci-version_number.tgz
Run the installation script
sudo ./cluster/sys_ver_num/bin/installinstall
, located within a folder matching the version number of system software used by the HCI software.Note- Don't change directories after running the installation script. The following tasks are performed in your current folder.
-
The
install
script can be run only one time on each instance. You cannot rerun this script to try to repair or upgrade a system instance.
Set up networking
On each server or virtual machine that is to be a system instance, edit the file installation-folder/config/network.config
file to be identical to the copies of the same file on the existing system instances.
Run the setup script on each server or virtual machine
Before you begin
- When installing a multi-instance system, make sure you specify the same list of master instance IP addresses on every instance that you are installing.
- When entering IP address lists, do not separate IP addresses with spaces. For example, the following is correct:
sudo install_path/hci/bin/setup ‑i 192.0.2.4 ‑m 192.0.2.0,192.0.2.1,192.0.2.3
Procedure
Run the script setup with the applicable options:
Use the following table to determine which options to use:Option Description -i The external network IP address for the instance on which you're running the script. -I The internal network IP address for the instance on which you're running the script. -m Comma-separated list of external network IP addresses of each master instance. -M Comma-separated list of internal network IP addresses of each master instance. -i IPADDRESS Displays the external instance IP address. If not specified, this value is discovered automatically. -I IPADDRESS Displays the internal instance IP address. If not specified, this value is the same as the external IP address. -d Attempts to automatically discover the real master list from the provided masters. –-hci_uid UID Allows you to set the desired user ID (UID) for the HCI USER at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.--hci_gid GID Allows you to set the desired group ID (GID) for the HCI GROUP at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.--mesos_uid UID Allows you to set the desired user UID for the MESOS USER at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.--mesos_gid GID Allows you to set the desired GID for the MESOS GROUP at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.--haproxy_uid UID Allows you to set the desired UID for the HAPROXY USER at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.--haproxy_gid GID Allows you to set the desired GID for the HAPROXY GROUP at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.--zk_uid UID Allows you to set the desired UID for the ZOOKEEPER USER at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.--zk_gid GID Allows you to set the desired GID for the ZOOKEEPER GROUP at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.Number of instances in the system Network type usage Options to use Multiple Single network type for all services Either: -i and -m
or -I and -M
Multiple Internal for some services, external for others All of these: -i, -I, -m, -M
Single Single network type for all services Either -i or -I Single Internal for some services, external for others Both -i and -I
Results
setup
script, ensure that Docker is running. sudo install_path/hci/bin/setup -i 192.0.2.4
To set up a multi-instance system that uses both internal and external networks, type the command in this format:
sudo install_path/hci/bin/setup ‑i external_instance_ip ‑I internal_instance_ip ‑m external_master_ips_list ‑M internal_master_ips_list
For example:
sudo install_path/hci/bin/setup ‑i 192.0.2.4 ‑I 10.236.1.0 ‑m 192.0.2.0,192.0.2.1,192.0.2.3 ‑M 10.236.1.1,10.236.1.2,10.236.1.3
The following table shows sample commands to create a four-instance system. Each command is entered on a different server or virtual machine that is to be a system instance. The resulting system contains three master instances and one worker instance and uses both internal and external networks.
Instance internal IP | Instance external IP | Master or worker | Command |
192.0.2.1 | 10.236.1.1 | Master | sudo install_path/hci/bin/setup ‑I 192.0.2.1 ‑i 10.236.1.1 ‑M 192.0.2.1,192.0.2.2,192.0.2.3 ‑m 10.236.1.1,10.236.1.2,10.236.1.3 |
192.0.2.2 | 10.236.1.2 | Master | sudo install_path/hci/bin/setup ‑I 192.0.2.2 ‑i 10.236.1.2 ‑M 192.0.2.1,192.0.2.2,192.0.2.3 ‑m 10.236.1.1,10.236.1.2,10.236.1.3 |
192.0.2.3 | 10.236.1.3 | Master | sudo install_path/hci/bin/setup ‑I 192.0.2.3 ‑i 10.236.1.3 ‑M 192.0.2.1,192.0.2.2,192.0.2.3 ‑m 10.236.1.1,10.236.1.2,10.236.1.3 |
192.0.2.4 | 10.236.1.4 | Worker | sudo install_path/hci/bin/setup ‑I 192.0.2.4 ‑i 10.236.1.4 ‑M 192.0.2.1,192.0.2.2,192.0.2.3 ‑m 10.236.1.1,10.236.1.2,10.236.1.3 |
Start the application on each server or virtual machine
Procedure
Start the application script
run
using whatever methods you usually use to run scripts.ImportantEnsure that the method you use can keep therun
script running and can automatically restart it in the event of a server restart or other availability event.
Results
Here are some examples of how you can start the script:
- You can run the script in the foreground:
sudo /install_path/hci/bin/run
When you run therun
script this way, the script does not automatically complete, but instead remains running in the foreground. - You can run the script as a service using
systemd
:- Copy the HCI.service file to the appropriate location for your OS. For example:
cp /install_path/bin/HCI.service /etc/systemd/system
- Enable and start the HCI.service service:
sudo systemctl enable HCI.service sudo systemctl start HCI.service
- Copy the HCI.service file to the appropriate location for your OS. For example:
systemctl
might display this message:The unit files have no [Install] section. They are not meant to be enabled using systemctl.
Possible reasons for having this kind of units are:
1) A unit may be statically enabled by being symlinked from another unit's .wants/ or .requires/ directory.
2) A unit's purpose may be to act as a helper for some other unit which has a requirement dependency on it.
3) A unit may be started when needed via activation (socket, path, timer, D-Bus, udev, scripted systemctl call, ...).
Depending on your OS, the search service might or might not have successfully been enabled.
To avoid this, make sure that you moved the HCI.service file to the appropriate location, typically /etc/systemd/system.
Configure services and jobs on the new instances
Hitachi Content Intelligence does not automatically begin running services on the instances you've added. You need to manually configure services to run on those new instances. For information, see Moving and scaling services.
Hitachi Content Intelligence also does not automatically configure the new instance to run workflows. You need to manually configure the new instance to run Workflow-Agent jobs. For information, see Configuring where jobs run.
Viewing instances
You can use the Admin App, CLI, and REST API to view a list of all instances in the system.
Viewing all instances
To view all instances, in the Admin App, click Dashboard > Instances.
The page shows all instances in the system. Each instance is identified by its IP address.
This table describes the information shown for each instance.
Property | Description |
State |
|
Services | The number of services running on the instance. |
Service Units |
The total number of service units for all services and job types running on the instance, out of the best-practice service unit limit for the instance. An instance with a higher number of service units is likely to be more heavily used by the system than an instance with a lower number of service units. The Instances page displays a blue bar for instances running less than the best-practice service unit limit. The Instances page displays a red bar for instances running more than the best-practice service unit limit. |
Load Average | The load averages for the instance for the past one, five, and ten minutes. |
CPU | The sum of the percentage utilization for each CPU core in the instance. |
Memory Allocated |
This section shows both:
|
Memory Total | The total amount of RAM for the instance. |
Disk Used | The current amount of disk space that your system is using in the partition on which it is installed. |
Disk Free | The amount of free disk space in the partition in which your system is installed. |
Viewing the services running on an instance
To view the services running on an individual instance, in the Admin App:
Procedure
Click Dashboard > Instances.
Select the instance you want.
The page lists all services running on the instance.
For each service, the page shows:
- The service name
- The service state:
- Healthy: The service is running normally.
- Unconfigured: The service has yet to be configured and deployed.
- Deploying: The system is currently starting or restarting the service. This can happen when:
- You move the service to run on a completely different set of instances.
- You repair a service.
- Balancing: The service is running normally, but performing background maintenance.
- Under-protected: In a multi-instance system, one or more of the instances on which a service is configured to run are offline.
- Failed: The service is not running or the system cannot communicate with the service.
- CPU Usage: The current percentage CPU usage for the service across all instances on which it's running.
- Memory: The current RAM usage for the service across all instances on which it's running.
- Disk Used: The current total amount of disk space that the service is using across all instances on which it's running.
Related CLI commands
getInstance
listInstances
Related REST API methods
GET /instances
GET /instances/{uuid}
You can get help on specific REST API methods for the Admin App at REST API - Admin.
Removing instances
You typically remove an instance from your system in these situations:
- You are retiring the hardware on which the instance runs.
- The instance is in the Down state and cannot be recovered.
- You want to run a system with fewer instances.
(Optional) Shut down the instance you want to remove
If the instance has already shut down due to failure, the instance is in the Down state. Your system automatically tries to move all services from that instance to other instances in the system. After all services have been moved, the instance is eligible for removal. Continue to the next step, Remove the shut down instance from the system.
If the instance that you want to remove is in the Up state, you need to shut it down yourself before you can remove it from the system.
Procedure
Move all the services that the instance is currently running to the other instances in the system.
ImportantShutting down an instance without first moving its services can cause data loss.If the system has jobs configured to run on only the failed instance, configure those jobs to run on other instances.
Stop the
run
script from running. You do this using whatever method you're currently using to run the script.For example, if you used systemd to run HCI.service, you can run these commands to stop and then disable it:
sudo systemctl stop HCI.service sudo systemctl disable HCI.service
Additionally, if you used systemd to run Hitachi Content Intelligence, delete the HCI.service file from where you copied it when you installed Hitachi Content Intelligence on the instance. Typically, the HCI.service is copied to /etc/systemd/system.
Run this command to stop all system Docker containers on the instance:
sudo <installation-directory>/bin/stop
Delete the Hitachi Content Intelligence Docker containers:
List all Docker containers:
sudo docker ps
Note the container IDs for all containers that use a com.hds.ensemble image.
Delete each of those containers:
sudo docker rm <container-id>
Delete the system Docker images:
List all Docker images:
sudo docker images
Note the image IDs for all images that use a com.hds.ensemble repository.
Delete each of those images:
sudo docker rmi <image-id>
Delete the Hitachi Content Intelligence installation folder:
rm -rf /<installation-directory>
Remove the shut down instance from the system
Procedure
Select the Instances window.
Click the instance you want to remove.
Click Remove Instance.
Related CLI commands
deleteInstance
Related REST API methods
DELETE /instances/{uuid}
You can get help on specific REST API methods for the Admin App at REST API - Admin.
Replacing a failed instance
If an instance suffers an unrecoverable failure, you need to replace that instance with a new one.
Procedure
In the Admin App, view the Instances page to determine whether the failed instance was a master instance.
Select a new server or virtual machine to add as a new instance to the system.
Remove the failed instance from the system.
WARNINGIf the failed instance was a master, after you remove the instance, you have only two master instances remaining. If any other instance fails while you are in this state, the system becomes completely unavailable until you add a third master back to the system by completing this procedure.Add the replacement instance to the system.
ImportantIf the instance you are replacing was a master instance, when you run setup on the replacement instance, the list of masters that you specify for the -m option needs to include:- The IP addresses of the two remaining healthy master instances.
- The IP address of the new instance that you're adding.
For example, in a system with master instance IPs ranging from 192.0.2.1 to 192.0.2.3 and you are replacing instance 192.0.2.3 with 192.0.2.5, run setup with these options:
sudo bin/setup -i 192.0.2.5 -m 192.0.2.1,192.0.2.2,192.0.2.5
This does not apply when you're replacing a worker instance. In that case, specify the IP addresses of the three existing masters.