Instances

Last updated
Save as PDF

A system is made up of one or more instances of the software. This section includes information on adding and removing instances to the system.

About master and worker instances

Master instances are special instances that run an essential set of services, including:

Admin-App service
Cluster-Coordination service
Synchronization service
Service-Deployment service

Non-master instances are called workers. Workers can run any services except for those listed previously.

Single-instance systems have one master instance while multi-instance systems have either one or three master instances.

ImportantYou cannot add master instances to a system after it's installed. You can, however, add any number of worker instances.

Single-instance systems vs. multi-instance systems

A system can have a single instance or can have multiple instances (four or more).

Note

Every instance must meet the minimum RAM, CPU, and disk space requirements.
Three instances are sufficient to perform leader election for distributing work. However, a multi-instance system needs a minimum of four instances because, with the minimum hardware requirements, three instances are not sufficient for running all HCI services at their recommended distributions.
Hitachi Vantara has qualified HCI systems with up to 16 instances.

One instance

A single-instance system is useful for testing and demonstration purposes. It needs only a single server or virtual machine and can perform all product functionality.

However, a single-instance system has these drawbacks:

Only a single point of failure. If the instance hardware fails, you lose access to the system.
With no additional instances, you cannot choose where to run services. All services run on the single instance.

Multiple instances

A multi-instance system is suitable for use in a production environment because it offers these advantages over a single-instance system:

You can control how services are distributed across the multiple instances, providing improved service redundancy, scale out, and availability.
A multi-instance system can survive instance outages. For example, with a four-instance system running the default distribution of services, the system can lose one instance and still remain available.
Note For a search index to survive an instance outage:
- The system must have at least two instances running the Index service.
- The Index Protection Level for the index must be at least 2.
For more information, see the HCI Administrator Help, which is available in the Admin App.
Performance is improved as work can be performed in parallel across instances.
You can add additional instances to the system at any time.

NoteYou cannot change a single-instance system into a production-ready multi-instance system by adding new instances. This is because you cannot add master instances. Master instances are special instances that run a particular set of Content Intelligence services. Single-instance systems have one master instance. Multi-instance systems have at least three.

By adding additional instances to a single-instance system, your system still has only one master instance, meaning there is still a single point of failure for the essential services that only a master instance can run.

For information about adding instances to an existing HCI system, see the Content Intelligence Administrator Help, which is available from the Admin App.

Two-instance system considerations

Two-instance systems are a viable option for the HCM use case, but not recommended for Hitachi Content Search.

Three-instance system considerations

Three-instance systems should have only a single master instance. If you deploy a three-instance system where all three instances are masters, the system might not have enough resources to do much beyond running the master services.

Requirements for running system instances

This section lists the hardware and operating system requirements for running system instances.

Hardware resources

This table lists:

Minimum: The minimum amounts of RAM, CPU, and available disk space needed to run a system instance.
Every server or virtual machine you provide as an instance must meet these minimums.
Better: The amounts of RAM, CPU, and available disk space that is better for system instances.
A system in which all instances meet or exceed these values can index more documents and can process documents faster than a system whose instances meet only the minimum requirements.

Resource	Minimum	Better
RAM	16 GB	32 GB
CPU	4-core	8-core
Available disk space	50 GB	500 GB

ImportantEach instance uses all available RAM and CPU resources on the server or virtual machine on which it's installed.

ImportantA large number of factors determine how many documents your system can index and how fast it can process them, including: the number of documents to be indexed; the contents of those documents; what search features (such as sorting) the index supports; the number of fields in the index; the number of users querying the system.

Depending on how you use your system, you might require additional hardware resources to index all the documents you want and at the rate you require. See Detailed sizing.

For more information on sizing your Hitachi Content Intelligence system, see the Administrator Help, which is available from the Admin App.

Sizing guidance for Hitachi Content Search

Simple sizing

This table shows the minimum and recommended hardware requirements for each instance in an HCI running Hitachi Content Search.

Resource	Minimum	Recommended
RAM	16 GB	32 GB
CPU	4-core	8-core
Available disk space	50 GB	500 GB

Important

A large number of factors determine how many documents your system can index and how fast it can process them, including: the number of documents to be indexed; the contents of those documents; what search features (such as sorting) the index supports; the number of fields in the index; the number of users querying the system; and so on.
Depending on how you use your system, you might require additional hardware resources to index all the documents you want and at the rate you require.
Each instance uses all available RAM and CPU resources on the server or virtual machine on which it's installed.

Detailed sizing

If you are installing HCI to run Hitachi Content Search, you should size your system based on the number of documents you need to index and the rate at which you need documents to be processed and indexed.

ImportantThis sizing guide details the resources required for a system with a single Index Protection Level (IPL). To scale your system accordingly, you will need to double the recommended values to accommodate IPL 2, triple the recommended values to accommodate IPL 3, etc.

To determine the system size that you need:

Procedure

Determine how many documents you need to index.

Based on the number of documents you want to index, use the following tables to determine:

How many instances you need
How much RAM each instance needs
The Index service configuration needed to support indexing the number of documents you want

Total documents to be indexed

System configuration

15 million

25 million

50 million^a

Total instances required: 1^b

Instances running the Index service: 1

Index service configuration required:

Shards per index: 1
Index Protection Level per index: 1
Container memory: 200MB greater than Heap settings
Heap settings: Depends on instance RAM.
Instance RAM Heap setting
16 GB 1800m
32 GB 9800m
64 GB 25800m

16 GB

32 GB

64 GB

Instance RAM needed (for each instance running the Index service)

^a Contact Hitachi Vantara for guidance before trying to index this many documents on this number of instances. At this scale, your documents and required configuration settings can greatly affect the number of documents you can index.

^b Single-instance systems are suitable for testing and development, but not for production use.

Total documents to be indexed

System configuration

45 million

75 million

150 million^a

Total instances required: 4

Instances running the Index service: 3

Index service configuration required:

Shards per index: 3
Index Protection Level per index: 1
Container memory: 200MB greater than Heap settings
Heap settings: Depends on instance RAM.
Instance RAM Heap setting
16 GB 1800m
32 GB 9800m
64 GB 25800m

16 GB

32 GB

64 GB

Instance RAM needed (for each instance running the Index service)

Total documents to be indexed

System configuration

75 million

125 million

250 million^a

Total instances required: 8

Instances running the Index service: 5

Index service configuration required:

Shards per index: 5
Index Protection Level per index: 1
Container memory: 200MB greater than Heap settings
Heap^b settings: Depends on instance RAM.
Instance RAM Heap setting
16 GB 7800m
32 GB 15800m
64 GB 31000m

16 GB

32 GB

64 GB

Instance RAM needed (for each instance running the Index service)

^b With an 8-instance system, the Index service should be the only service running on each of its 5 instances. With the Index service isolated this way, you can allocate more heap space to the service than you can on a single or 4-instance system.

Total documents to be indexed

System configuration

195 million

325 million

650 million^a

Total instances required: 16

Instances running the Index service: 13

Index service configuration required:

Shards per index: 13
Index Protection Level per index: 1
Container memory: 200MB greater than Heap settings
Heap^b settings: Depends on instance RAM.
Instance RAM Heap setting
16 GB 7800m
32 GB 15800m
64 GB 31000m

16 GB

32 GB

64 GB

Instance RAM needed (for each instance running the Index service)

^b With a 16-instance system, the Index service should be the only service running on each of its 13 instances. With the Index service isolated this way, you can allocate more heap space to the service than you can on a single or 4-instance system.

For example, if you need to index up to 150 million documents, you need at minimum a 4-instance system with 64 GB RAM per instance.

Determine how fast you need to index documents, in documents per second.

For example:
- To index 100 million documents in 2 days, you need an indexing rate of 578 documents per second.
- To continuously index 1 million documents every day, you need an indexing rate of 12 documents per second.
Determine the base indexing rate for your particular dataset and processing pipelines:
1. Install a single-instance HCI system with that has the minimum required hardware resources.
2. Run a workflow with the pipelines you want and on a representative subset of your data.
3. Use the workflow task details to determine the rate of documents processed per second.

To determine the number of cores you need per instance, replace Base rate in this table with the rate you determined in step 4.

Number of instances you need	Cores per instance
Number of instances you need	4 (minimum required)	8 (recommended)
1	Base rate	70% Base rate
4	300% Base rate	500% Base rate
8	600% Base rate	900% Base rate
More than 8	Contact Hitachi Vantara for guidance

For example, if you had previously determined that:

You need a 4-instance system.
You need to process 500 documents per second.
The base processing rate for your data and pipelines is 100 documents per second.

You need 8 cores per instance.

Multiply the number of instances you need times the number of cores per instances to determine the total number of cores that you need for your system.
After your system is installed, configure it with the index settings you determined in step 2.

For information on index shards, Index Protection Level, and moving the Index service, see the Administrator Help, which is available from the Admin App.

Sizing guidance for HCM

Minimum hardware requirements

If you are installing HCI to run HCM, each instance in the system must meet these minimum hardware requirements:

Documents per second	Cores	RAM (GB)	Disk (GB)
Up to 1200	8	28	600
1200-1600	12	32	800
1600-2000	16	40	1000
2000-2400	18	48	1400
2400-2800	20	56	1700
2800-3200	24	64	2000

Determining number of instances

The number of instances that your HCM system needs is based on:

Whether you need the system to remain highly available.
The number of documents being produced by the HCP system you want to monitor. In this case, each document represents a single piece of data about the HCP system. A more active HCP system will produce more documents than a less active one.
The total number of documents you want HCM to store.

Number of instances: simple procedure

If you're monitoring a typically-active HCP system (roughly 75 operations per second per node), you can use this table to determine the number of HCM instances you need. This table lists the number of HCM instances you need based on the number of nodes in your HCP system and the number of days that you want your HCM system to retain the data it receives from HCP.

If your system is more active, see Number of instances: detailed procedure.

HCP nodes	Data retention time on HCM	Instances needed
Up to 8	Up to 30 days	1*
Up to 8	Up to 60 days	3*
Up to 16	Up to 30 days	4
Up to 24	Up to 60 days	8
*An HCM system must have a minimum of 4 instances to maintain high system availability.

Number of instances: detailed procedure

Determine whether you need your HCM system to maintain high availability. If so, you need a minimum of 4 instances. For more information, see Single-instance systems versus multi-instance systems.
Determine the number of documents per second being produced by the HCP system you want to monitor. You can easily do this if you already have an HCM system up and running:
1. Go to the Monitor App: https://system-hostname:6162
2. Add the HCP system as a source. For information, see the help that's available from the Monitor App.
3. Go to the HCI Admin App: https://system-hostname:8000
4. Go to Workflows > Monitor App Workflow > Task > Metrics.
5. View the value for the Average DPS field.
  TipLet the workflow run for a while to get a more accurate measure for the Average DPS field.
Otherwise, you can get an average documents per second value by doing this:
1. Select a time period.
2. Download the HCP Internal Logs for this time period. For more information, see the help that's accessible from the HCP System Management Console.
3. In the downloaded logs for each node, count the number lines logged during the selected time period.
4. Add the line value for each node and then divide the sum by the number of seconds in the time window you selected.
Use this table to determine the number of instances needed based on the number of documents per second produced by your HCP system.

Documents per second Instances needed

≤ 3,200

1

3,201 to 7,200

3

7,201-10,500*

4

*This is the maximum documents per second that HCM currently supports.

Documents per second	Instances needed
≤ 3,200	1
3,201 to 7,200	3
7,201-10,500*	4
*This is the maximum documents per second that HCM currently supports.

Based on your data availability requirements, determine the number of instances you need.

Data availability requirement	Index replicas needed	Instances needed	Impact on total documents stored
No failure tolerance	1	1	None
Survive 2 failed replicas	3	3	3x
Survive 3 failed replicas	4	4	4x

An index with multiple copies remains available in the event of an instance outage. For example, if an index has two copies stored on two instances and one of those instances fails, one copy of the index remains available for servicing requests.

Use this formula to determine the total number of documents your HCM system must be able to store:

documents per second from step 2.

x 3600 seconds in an hour

x 24 hours in a day

x number of days you want to store data (default is 30)

x Impact from the data availability table in step 4.

= Total document count

For example, if your HCP system produces 1500 documents per second, you want to store data for 30 days, and you want to maintain two copies of each index containing the stored data, your system must have enough instances to be able to store roughly 8 billion documents:

1500

x 3600

x 24

x 30

x 2

= 7,776,000,000
Use this table to determine the number of instances needed based on the total number of documents your HCM must store.

Total document count Instances needed

2 billion or less

1

6 billion or less

3

8 billion or less

4
Take the highest number of instances from steps 2, 3, and 6. That's the number instances you need.

Total document count	Instances needed
2 billion or less	1
6 billion or less	3
8 billion or less	4

Operating system and Docker minimum requirements

Each server or virtual machine you provide must have the following:

64-bit Linux distribution
Docker version 1.13.1 or later installed
IP and DNS addresses configured

Additionally, you should install all relevant patches on the operating system and perform appropriate security hardening tasks.

ImportantInstall the current Docker version suggested by your operating system, unless that version is earlier than 1.13.1. The system cannot run with Docker versions before 1.13.1.

Operating system and Docker qualified versions

This table shows the operating systems, Docker and SELinux configurations with which the HCI system has been qualified.

ImportantDocker versions after 18.09 are not supported.

Operating system	Docker version	Docker storage configuration	SELinux setting
Fedora 27	Docker 18.09.0-ce	direct-lvm	Enforcing
Red Hat Enterprise Linux 7.4	Docker 18.09.0-ce	direct-lvm	Enforcing
Ubuntu 16.04-LTS	Docker 17.03.0-ce	aufs	N/A
CentOS 7.4	Docker 18.09.1-ce	overlay2	Enforcing

Docker considerations

The Docker installation folder on each instance must have at least 20 GB available for storing the Docker images.

Make sure that the Docker storage driver is configured correctly on each instance before installing the product. After you install the product, to change the Docker storage driver you must reinstall the product. To view the current Docker storage driver on an instance, run:

docker info

Core dumps can fill a host's file system, which can result in host or container instability. Also, if your system uses the data at rest encryption (DARE) feature, encryption keys are written to the dump file. It's best to disable core dumps.

To enable SELinux on the system instances, you need to use a Docker storage driver that SELinux supports. The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.

If you are using the Docker devicemapper storage driver:

Make sure that there's at least 40 GB of Docker metadata storage space available on each instance. The product needs 20 GB to install successfully and an additional 20 GB to successfully update to a later version.
To view Docker metadata storage usage on an instance, run:
docker info
On a production system, do not run devicemapper in loop-lvm mode. This can cause slow performance or, on certain Linux distributions, the product might not have enough space to run.

SELinux considerations

You should decide whether you want to run SELinux on system instances and enable or disable it before installing additional software on the instance.
Enabling or disabling SELinux on an instance needs a restart of the instance.
To view whether SELinux is enabled on an instance, run: sestatus
To enable SELinux on the system instances, you need to use a Docker storage driver that SELinux supports.
The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.

Time source

If you are installing a multi-instance system, each instance should run NTP (network time protocol) and use the same external time source. For information, see support.ntp.org.

File ownership considerations

Within some of the Docker containers on each system instance, file ownership is assigned to this user and group:

User: hci, UID: 10001
Group: hci, GID: 10001

When you view such files in the instance operating system (for example, by running ls -l), the files appear to be owned by an unknown or undefined user and group. Typically, this causes no issues.

However, if you run applications on the system instances that change file ownership (for example, security hardening scripts), changing the ownership of files owned by the hci user and group can cause the system to become unresponsive.

To avoid these issues:

Create the expected user and group on each instance:
sudo groupadd hci -g 10001
sudo useradd hci -u 10001 -g 10001
Configure your applications to not change the ownership of files owned by the hci user and group.

Adding new instances

You might want to add additional instances to the system if:

You want to improve system performance.
You are running out of disk space on one or more instances.

ImportantEach instance must run a number of essential services. Because of this, each new instance you add counts against your licensed service unit limit.

However, these situations might also be improved by adding additional CPU, RAM, or disks to the instances you already have. For guidance, see Sizing guidance for HCM or Sizing guidance for Hitachi Content Search.

Guidance for adding new HCM instances

When adding HCM instances to your HCI system, there are several guidelines to consider to ensure the best performance.

If you add only a single worker node to an existing HCM instance, you will begin to receive alerts that your system services are running below their minimum recommended requirements. It is strongly advised that at least two instances are added.
After adding new instances, you need to pause and resume the workflow in order for them to be parsed by the workflow-agent job.
If you do not have acess to the Workflow Designer app, editing the description of the signal source will trigger the workflow to pause and resume, which will allow the instances to be correctly parsed.

Before adding a new instance

Obtain the Hitachi Content Intelligence-<version-number>.tgz installation file. When adding an instance, you unpack and deploy this file on a bare-metal server or a pre-existing Linux virtual machine.
Record the IP address of at least one of the master instances in the system.
If your system uses internal and external networks, you need to record both the internal and external IP addresses for the master instances.
You can view instance IP addresses on the Instances page in the Admin App.
Ensure that the new instances you are adding meet the minimum hardware, OS, and networking requirements. For information, see Requirements for running system instances.
Record the Docker volume drivers currently used by services and jobs across all existing instances. You need to install all of these volume drivers on the new instance that you're adding.
To find the volume drivers currently in use by your system, run this command on each system instance:
```
docker volume ls
```
Take note of each value for the DRIVER field.

Install Docker on each server or virtual machine

On each server or virtual machine that is to be an HCI instance:

Procedure

In a terminal window, verify whether Docker 1.13.1 or later is installed:
docker --version
If Docker is not installed or if you have a version before 1.13.1, install the current Docker version suggested by your operating system.
The installation method you use depends on your operating system. See the Docker website for instructions.

Configure Docker on each server or virtual machine

Before installing the product, configure Docker with settings suitable for your environment. For guidance on configuring and running Docker, see the applicable Docker documentation.

Procedure

Ensure that the Docker installation folder on each instance has at least 20 GB available for storing the product Docker images.
Ensure that the Docker storage driver is configured correctly on each instance. After installation, changing the Docker storage driver needs reinstallation of the product.
To view the current Docker storage driver on an instance, run: docker info .
To enable SELinux on the system instances, use a Docker storage driver that SELinux supports.
The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.
If you are using the Docker devicemapper storage driver, ensure that there's at least 40 GB of Docker metadata storage space available on each instance.
The product needs 20 GB to install successfully and an additional 20 GB to successfully update to a later version.To view Docker metadata storage usage on an instance, run: docker info

Next steps

On a production system, do not run devicemapper in loop-lvm mode. This can cause slow performance or, on certain Linux distributions, the product might not have enough space to run.

(Optional) Configure Docker volume drivers

If any services or jobs on your system are using Docker volume drivers (that is, not the bind-mountsetting) for storing data, you need to install those volume drivers on the new instance that you are adding. If you don't, jobs and services might fail to run on the new instance.

Volume drivers are provided by Docker and other third-party developers, not by the system itself. For information on volume drivers, their capabilities, and their valid configuration settings, see the applicable Docker or third-party developer's documentation.

Configure maximum map count setting

You need to configure a value in the file sysctl.conf.

Procedure

On each server or virtual machine that is to be a system instance, open the file /etc/sysctl.conf.
Append this line: vm.max_map_count = 262144
If the line already exists, ensure that the value is greater than or equal to 262144.
Save and close the file.

Optional: Enable or disable SELinux on each server or virtual machine

You should decide whether you want to run SELinux on system instances before installation.

Procedure

Enable or disable SELinux on each instance.
Restart the instance.

Configure the firewall rules on each server or virtual machine

Before you begin

Determine the port values currently used by your system. To do this, on any instance, view the file install_path/config/network.config.

On each server or virtual machine that is to be a system instance:

Procedure

Edit the firewall rules to allow communication over all network ports that you want your system to use. You do this using a firewall management tool such as firewalld.
Restart the server or virtual machine.

Install and configure NTP

Install NTP (Network Time Protocol) on the new server or virtual machine and configure it to use the same time source as the other system instances. For information, see http://support.ntp.org.

Run Docker on each server or virtual machine

On each server or virtual machine that is to be a system instance, you need to start Docker and keep it running. You can use whatever tools you typically use for keeping services running in your environment.

For example, to run Docker using systemd:

Procedure

Verify that Docker is running:
systemctl status docker
If Docker is not running, start the docker service:
sudo systemctl start docker
(Optional) Configure the Docker service to start automatically when you restart the server or virtual machine:
sudo systemctl enable docker

Unpack the installation package

On each server or virtual machine that is to be a system instance:

Procedure

Download the installation package hci-version_number.tgz and store it in a folder on the server or virtual machine.
In the largest disk partition on the server or virtual machine, create a folder named install_path/hci:
mkdir install_path/hci
Move the installation package from the folder where you stored it to the folder install_path/hci:
mv hci-version_number.tgz install_path/hci/hci-version_number.tgz
Navigate to the installation folder:
cd install_path/hci
Unpack the installation package:
tar -zxf hci-version_number.tgzMultiple folders are created upon unpacking in the installation folder.
Note
If you encounter problems unpacking the installation file (for example, the error message "tar: This does not look like a tar archive"), the file might have been packed multiple times during download. Use the following commands to fully extract the file:

$ gunzip hci-version_number.tgz

$ mv hci-version_number.tar hci-version_number.tgz

$ tar -zxf hci-version_number.tgz
Run the installation script install, located within a folder matching the version number of system software used by the HCI software.
sudo ./cluster/sys_ver_num/bin/install
Note
- Don't change directories after running the installation script. The following tasks are performed in your current folder.
- The install script can be run only one time on each instance. You cannot rerun this script to try to repair or upgrade a system instance.

Set up networking

On each server or virtual machine that is to be a system instance, edit the file installation-folder/config/network.config file to be identical to the copies of the same file on the existing system instances.

Run the setup script on each server or virtual machine

Before you begin

Note

When installing a multi-instance system, make sure you specify the same list of master instance IP addresses on every instance that you are installing.
When entering IP address lists, do not separate IP addresses with spaces. For example, the following is correct:
sudo install_path/hci/bin/setup ‑i 192.0.2.4 ‑m 192.0.2.0,192.0.2.1,192.0.2.3

On each server or virtual machine that is to be a system instance:

Procedure

Run the script setup with the applicable options:

Option	Description
`-i`	The external network IP address for the instance on which you're running the script.
`-I`	The internal network IP address for the instance on which you're running the script.
`-m`	Comma-separated list of external network IP addresses of each master instance.
`-M`	Comma-separated list of internal network IP addresses of each master instance.
`-i IPADDRESS`	Displays the external instance IP address. If not specified, this value is discovered automatically.
`-I IPADDRESS`	Displays the internal instance IP address. If not specified, this value is the same as the external IP address.
`-d`	Attempts to automatically discover the real master list from the provided masters.
`–-hci_uid UID`	Allows you to set the desired user ID (UID) for the HCI USER at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.
`--hci_gid GID`	Allows you to set the desired group ID (GID) for the HCI GROUP at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.
`--mesos_uid UID`	Allows you to set the desired user UID for the MESOS USER at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.
`--mesos_gid GID`	Allows you to set the desired GID for the MESOS GROUP at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.
`--haproxy_uid UID`	Allows you to set the desired UID for the HAPROXY USER at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.
`--haproxy_gid GID`	Allows you to set the desired GID for the HAPROXY GROUP at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.
`--zk_uid UID`	Allows you to set the desired UID for the ZOOKEEPER USER at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.
`--zk_gid GID`	Allows you to set the desired GID for the ZOOKEEPER GROUP at install time only. ImportantThis value needs to be greater than 1000, less than or equal to 65533, and the same on all nodes in a cluster.

Use the following table to determine which options to use:

Number of instances in the system	Network type usage	Options to use
Multiple	Single network type for all services	Either: `-i` and `-m` or `-I` and `-M`
Multiple	Internal for some services, external for others	All of these: `-i`, `-I`, `-m`, `-M`
Single	Single network type for all services	Either `-i` or `-I`
Single	Internal for some services, external for others	Both `-i` and `-I`

Results

NoteIf the terminal displays Docker errors when you run the setup script, ensure that Docker is running.

The following example sets up a single-instance system that uses only one network type for all services:

sudo install_path/hci/bin/setup -i 192.0.2.4

To set up a multi-instance system that uses both internal and external networks, type the command in this format:

sudo install_path/hci/bin/setup ‑i external_instance_ip ‑I internal_instance_ip ‑m external_master_ips_list ‑M internal_master_ips_list

For example:

sudo install_path/hci/bin/setup ‑i 192.0.2.4 ‑I 10.236.1.0 ‑m 192.0.2.0,192.0.2.1,192.0.2.3 ‑M 10.236.1.1,10.236.1.2,10.236.1.3

The following table shows sample commands to create a four-instance system. Each command is entered on a different server or virtual machine that is to be a system instance. The resulting system contains three master instances and one worker instance and uses both internal and external networks.

Instance internal IP	Instance external IP	Master or worker	Command
192.0.2.1	10.236.1.1	Master	`sudo install_path/hci/bin/setup ‑I 192.0.2.1 ‑i 10.236.1.1 ‑M 192.0.2.1,192.0.2.2,192.0.2.3 ‑m 10.236.1.1,10.236.1.2,10.236.1.3`
192.0.2.2	10.236.1.2	Master	`sudo install_path/hci/bin/setup ‑I 192.0.2.2 ‑i 10.236.1.2 ‑M 192.0.2.1,192.0.2.2,192.0.2.3 ‑m 10.236.1.1,10.236.1.2,10.236.1.3`
192.0.2.3	10.236.1.3	Master	`sudo install_path/hci/bin/setup ‑I 192.0.2.3 ‑i 10.236.1.3 ‑M 192.0.2.1,192.0.2.2,192.0.2.3 ‑m 10.236.1.1,10.236.1.2,10.236.1.3`
192.0.2.4	10.236.1.4	Worker	`sudo install_path/hci/bin/setup ‑I 192.0.2.4 ‑i 10.236.1.4 ‑M 192.0.2.1,192.0.2.2,192.0.2.3 ‑m 10.236.1.1,10.236.1.2,10.236.1.3`

Start the application on each server or virtual machine

On each server or virtual machine that is to be a system instance:

Procedure

Start the application script run using whatever methods you usually use to run scripts.

ImportantEnsure that the method you use can keep the run script running and can automatically restart it in the event of a server restart or other availability event.

Results

When the service starts, the server or virtual machine automatically joins the system as a new instance.

Here are some examples of how you can start the script:

You can run the script in the foreground:
sudo /install_path/hci/bin/run
When you run the run script this way, the script does not automatically complete, but instead remains running in the foreground.
You can run the script as a service using systemd:
1. Copy the HCI.service file to the appropriate location for your OS. For example:
  cp /install_path/bin/HCI.service /etc/systemd/system
2. Enable and start the HCI.service service:
```
sudo systemctl enable HCI.service
sudo systemctl start HCI.service
```

NoteWhen you enable the search service, systemctl might display this message:

The unit files have no [Install] section. They are not meant to be enabled using systemctl.

Possible reasons for having this kind of units are:

1) A unit may be statically enabled by being symlinked from another unit's .wants/ or .requires/ directory.

2) A unit's purpose may be to act as a helper for some other unit which has a requirement dependency on it.

3) A unit may be started when needed via activation (socket, path, timer, D-Bus, udev, scripted systemctl call, ...).

Depending on your OS, the search service might or might not have successfully been enabled.

To avoid this, make sure that you moved the HCI.service file to the appropriate location, typically /etc/systemd/system.

Configure services and jobs on the new instances

Hitachi Content Intelligence does not automatically begin running services on the instances you've added. You need to manually configure services to run on those new instances. For information, see Moving and scaling services.

Hitachi Content Intelligence also does not automatically configure the new instance to run workflows. You need to manually configure the new instance to run Workflow-Agent jobs. For information, see Configuring where jobs run.

Viewing instances

You can use the Admin App, CLI, and REST API to view a list of all instances in the system.

Viewing all instances

To view all instances, in the Admin App, click Dashboard > Instances.

The page shows all instances in the system. Each instance is identified by its IP address.

This table describes the information shown for each instance.

Property	Description
State	Up: The instance is reachable by other instances in the system. Down: The instance cannot be reached by other instances in the system.
Services	The number of services running on the instance.
Service Units	The total number of service units for all services and job types running on the instance, out of the best-practice service unit limit for the instance. An instance with a higher number of service units is likely to be more heavily used by the system than an instance with a lower number of service units. The Instances page displays a blue bar for instances running less than the best-practice service unit limit. The Instances page displays a red bar for instances running more than the best-practice service unit limit.
Load Average	The load averages for the instance for the past one, five, and ten minutes.
CPU	The sum of the percentage utilization for each CPU core in the instance.
Memory Allocated	This section shows both: The amount of RAM on the instance that's allocated to all services running on that instance. The percentage of this allocated RAM to the total RAM for the instance.
Memory Total	The total amount of RAM for the instance.
Disk Used	The current amount of disk space that your system is using in the partition on which it is installed.
Disk Free	The amount of free disk space in the partition in which your system is installed.

Viewing the services running on an instance

To view the services running on an individual instance, in the Admin App:

Procedure

Click Dashboard > Instances.
Select the instance you want.

The page lists all services running on the instance.

For each service, the page shows:
- The service name
- The service state:
  - Healthy: The service is running normally.
  - Unconfigured: The service has yet to be configured and deployed.
  - Deploying: The system is currently starting or restarting the service. This can happen when:
    - You move the service to run on a completely different set of instances.
    - You repair a service.
  - Balancing: The service is running normally, but performing background maintenance.
  - Under-protected: In a multi-instance system, one or more of the instances on which a service is configured to run are offline.
  - Failed: The service is not running or the system cannot communicate with the service.
- CPU Usage: The current percentage CPU usage for the service across all instances on which it's running.
- Memory: The current RAM usage for the service across all instances on which it's running.
- Disk Used: The current total amount of disk space that the service is using across all instances on which it's running.

Related CLI commands

getInstance

listInstances

Related REST API methods

GET /instances

GET /instances/{uuid}

You can get help on specific REST API methods for the Admin App at REST API - Admin.

Removing instances

You typically remove an instance from your system in these situations:

You are retiring the hardware on which the instance runs.
The instance is in the Down state and cannot be recovered.
You want to run a system with fewer instances.

(Optional) Shut down the instance you want to remove

If the instance has already shut down due to failure, the instance is in the Down state. Your system automatically tries to move all services from that instance to other instances in the system. After all services have been moved, the instance is eligible for removal. Continue to the next step, Remove the shut down instance from the system.

If the instance that you want to remove is in the Up state, you need to shut it down yourself before you can remove it from the system.

Procedure

Move all the services that the instance is currently running to the other instances in the system.

ImportantShutting down an instance without first moving its services can cause data loss.
If the system has jobs configured to run on only the failed instance, configure those jobs to run on other instances.
Stop the run script from running. You do this using whatever method you're currently using to run the script.

For example, if you used systemd to run HCI.service, you can run these commands to stop and then disable it:
```
sudo systemctl stop HCI.service
sudo systemctl disable HCI.service
```
Additionally, if you used systemd to run Hitachi Content Intelligence, delete the HCI.service file from where you copied it when you installed Hitachi Content Intelligence on the instance. Typically, the HCI.service is copied to /etc/systemd/system.
Run this command to stop all system Docker containers on the instance:
sudo <installation-directory>/bin/stop
Delete the Hitachi Content Intelligence Docker containers:
1. List all Docker containers:
  sudo docker ps
2. Note the container IDs for all containers that use a com.hds.ensemble image.
3. Delete each of those containers:
  sudo docker rm <container-id>
Delete the system Docker images:
1. List all Docker images:
  sudo docker images
2. Note the image IDs for all images that use a com.hds.ensemble repository.
3. Delete each of those images:
  sudo docker rmi <image-id>
Delete the Hitachi Content Intelligence installation folder:
rm -rf /<installation-directory>

Remove the shut down instance from the system

Admin App instructions

Procedure

Select the Instances window.
Click the instance you want to remove.
Click Remove Instance.

Related CLI commands

deleteInstance

Related REST API methods

DELETE /instances/{uuid}

You can get help on specific REST API methods for the Admin App at REST API - Admin.

Replacing a failed instance

If an instance suffers an unrecoverable failure, you need to replace that instance with a new one.

Procedure

In the Admin App, view the Instances page to determine whether the failed instance was a master instance.
Select a new server or virtual machine to add as a new instance to the system.
Remove the failed instance from the system.

WARNINGIf the failed instance was a master, after you remove the instance, you have only two master instances remaining. If any other instance fails while you are in this state, the system becomes completely unavailable until you add a third master back to the system by completing this procedure.
Add the replacement instance to the system.
ImportantIf the instance you are replacing was a master instance, when you run setup on the replacement instance, the list of masters that you specify for the -m option needs to include:
- The IP addresses of the two remaining healthy master instances.
- The IP address of the new instance that you're adding.
For example, in a system with master instance IPs ranging from 192.0.2.1 to 192.0.2.3 and you are replacing instance 192.0.2.3 with 192.0.2.5, run setup with these options:
sudo bin/setup -i 192.0.2.5 -m 192.0.2.1,192.0.2.2,192.0.2.5
This does not apply when you're replacing a worker instance. In that case, specify the IP addresses of the three existing masters.

Quick Links

About master and worker instances

Single-instance systems vs. multi-instance systems

Requirements for running system instances

Hardware resources

Sizing guidance for Hitachi Content Search

Simple sizing

Detailed sizing

Sizing guidance for HCM

Minimum hardware requirements

Determining number of instances

Number of instances: simple procedure

Number of instances: detailed procedure

Operating system and Docker minimum requirements

Operating system and Docker qualified versions

Docker considerations

SELinux considerations

Time source

File ownership considerations

Adding new instances

Before adding a new instance

Install Docker on each server or virtual machine

Configure Docker on each server or virtual machine

(Optional) Configure Docker volume drivers

Configure maximum map count setting

Optional: Enable or disable SELinux on each server or virtual machine

Configure the firewall rules on each server or virtual machine

Install and configure NTP

Run Docker on each server or virtual machine

Unpack the installation package

Set up networking

Run the setup script on each server or virtual machine

Start the application on each server or virtual machine

Configure services and jobs on the new instances

Viewing instances

Viewing all instances

Viewing the services running on an instance

Related CLI commands

Related REST API methods

Removing instances

(Optional) Shut down the instance you want to remove

Remove the shut down instance from the system

Related CLI commands

Related REST API methods

Replacing a failed instance