Instances

Last updated
Save as PDF

You are here:

A system is made up of one or more instances of the software. This section includes information on adding and removing instances to the system.

For more information on instances, see System scaling.

Sub-topics:

Trademarks, Legal disclaimer, Third-party software in this documentation

About master and worker instances

You are here:

Master instances are special instances that run an essential set of services, which includes:

•Admin-App service

•Cluster-Coordination service

•Synchronization service

•Service-Deployment service

Non-master instances are called workers. Workers can run any services besides those listed above. For information on services, see Services perform functions essential to the health or functionality of the system. For example, the Metrics service stores and manages system events, while the Watchdog service ensures that other services remain running..

Note: The Watchdog service runs on every instance in the system, both masters and workers.

Single-instance systems have one master instance while multi-instance systems have either one or three master instances.

Important: You cannot add master instances to a system after it's installed. You can, however, add any number of worker instances.

Trademarks, Legal disclaimer, Third-party software in this documentation

Single-instance systems versus multi-instance systems

You are here:

A system can have a single instance or can have multiple instances (four or more).

Notes:

•Every instance must meet the minimum RAM, CPU, and disk space requirements. For information, see Hardware resources.

•Three instances are sufficient to perform leader election for distributing work. However, a multi-instance system requires a minimum of four instances because, with the minimum hardware requirements, three instances are not sufficient for running all system services at their recommended distributions.

•Hitachi Vantara has qualified system systems with up to 16 instances.

One instance

A single-instance system is useful for testing and demonstration purposes. It requires only a single server or virtual machine and can perform all product functionality.

However, a single-instance system has these drawbacks:

•It has a single point of failure. If the instance hardware fails, you lose access to the system.

•With no additional instances, you cannot choose where to run services. All services run on that one instance.

Multiple instances

A multi-instance system is suitable for use in a production environment because it offers these advantages over a single-instance system:

•You can control how services are distributed across the multiple instances, providing improved service redundancy, scale out, and availability.

For more information, see Service list.

• A multi-instance system can survive instance outages. For example, with a four-instance system running the default distribution of services, the system can lose one instance and still remain available.

•Performance is improved as work can be performed in parallel across instances.

•You can add additional instances to the system at any time.

Note: You cannot change a single-instance system into a production-ready multi-instance system by adding new instances. This is because you cannot add master instances. Master instances are special instances that run a particular set of system services. Single instance systems have one master instance. Multi-instance systems have three.

By adding additional instances to a single-instance system, your system still has only one master instance, meaning there is a single point of failure for the essential services that only a master instance can run.

For more information, see Adding new instances.

Trademarks, Legal disclaimer, Third-party software in this documentation

Requirements for running system instances

You are here:

This section lists the hardware and operating system requirements for running system instances. Also see Networking for information on network requirements for both instances and services.

Sub-topics:

Trademarks, Legal disclaimer, Third-party software in this documentation

Hardware resources

You are here:

This table lists:

•Minimum —The minimum amounts of RAM, CPU, and available disk space required to run a system instance.

Every server or virtual machine you provide as an instance must meet these minimums.

•Recommended — The amounts of RAM, CPU, and available disk space that Hitachi Vantara recommends for system instances.

A system in which all instances meet or exceed these recommendations can index more documents and can process documents faster than a system whose instances meet only the minimum requirements.

Resource	Minimum	Recommended
RAM	16 GB	32 GB
CPU	4-core	8-core
Available disk space	50 GB	500 GB

Important:

•Each instance uses all available RAM and CPU resources on the server or virtual machine on which it's installed.

•A large number of factors determine how many documents your system can index and how fast it can process them, including: the number of documents to be indexed; the contents of those documents; what search features (such as sorting) the index supports; the number of fields in the index; the number of users querying the system; and so on.

Depending on how you use your system, you may require additional hardware resources to index all the documents you want and at the rate you require.

Trademarks, Legal disclaimer, Third-party software in this documentation

Operating system and Docker requirements

You are here:

To be a system instance, each server or virtual machine you provide:

•Must run a 64-bit Linux distribution

•Must have Docker version 1.10.3 or later installed

Important: Install the current Docker version suggested by your operating system, unless that version is earlier than 1.10.3. The system cannot run with Docker versions prior to 1.10.3.

This table shows the operating systems and Docker and SELinux configurations that system has been qualified with. For more information, see Docker considerations and SELinux considerations.

Operating System	Docker Version	Docker Storage Configuration	SELinux setting
Fedora 24	Version: 1.10.3 API version: 1.22 Package version: docker-common-1.10.3-55.gite03ddb8.fc24.x86_64 Go version: go1.6.3 Git commit: e03ddb8/1.10.3 OS/Arch: linux/amd64	direct-lvm	Enforcing
Red Hat Enterprise Linux 7.3	Version: 1.12.6 API version: 1.24 Package version: docker-1.12.6-48.git0fdc778.el7.x86_64 Go version: go1.8.3 Git commit: 0fdc778/1.12.6 Built: Thu Jul 20 00:06:39 2017 OS/Arch: linux/amd64	direct-lvm	Enforcing
Ubuntu 16.04	Version: 17.03.0-ce API version: 1.26 Go version: go1.7.5 Git commit: 60ccb22 Built: Thu Feb 23 11:02:43 2017 OS/Arch: linux/amd64	aufs	N/A
CentOS 7.3	Version: 17.06.0-ce API version: 1.30 (minimum version 1.12) Go version: go1.8.3 Git commit: 02c1d87 Built: Fri Jun 23 21:21:56 2017 OS/Arch: linux/amd64 Experimental: false	overlay	Enforcing

Trademarks, Legal disclaimer, Third-party software in this documentation

Docker considerations

You are here:

•The Docker installation directory on each instance must have at least 20 GB available for storing the system Docker images.

•Make sure that the Docker storage driver is configured correctly on each instance before installing the system.

After installing the system, changing the Docker storage driver requires a reinstallation of the system.

To view the current Docker storage driver on an instance, run:

docker info

•If you want to enable SELinux on your system instances, you need to use a Docker storage driver that supports it. The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.

•If you are using the Docker devicemapper storage driver:

oMake sure that there's at least 40 MB of Docker metadata storage space available on each instance. system requires 20 MB to install successfully and an additional 20 MB to successfully update to a later version.

To view Docker metadata storage usage on an instance, run:

docker info

oOn a production system, do not run devicemapper in loop-lvm mode. This can cause slow performance or, on certain Linux distributions, the system may not have enough space to run.

Trademarks, Legal disclaimer, Third-party software in this documentation

SELinux considerations

You are here:

•You should decide whether you want to run SELinux on the new instance. Then, enable or disable it before installing system on the instance.

Enabling or disabling SELinux on an instance requires you to reboot the instance.

To view whether SELinux is enabled on an instance, run:

sestatus

•If you want to enable SELinux on your system instances, you need to use a Docker storage driver that supports it.

The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.

Trademarks, Legal disclaimer, Third-party software in this documentation

Time source requirements

You are here:

If you are installing a multi-instance system, each instance should run NTP (network time protocol) and use the same external time source. For information, see support.ntp.org.

Trademarks, Legal disclaimer, Third-party software in this documentation

File ownership considerations

You are here:

Within some of the Docker containers on each system instance, file ownership is assigned to this user and group:

•User: hci, UID: 10001

•Group: hci, GID: 10001

When you view such files in the instance operating system (for example, by running ls -l), the files appear to be owned by an unknown or undefined user and group. Typically, this causes no issues.

However, if you run applications on the system instances that change file ownership (for example, security hardening scripts), changing the ownership of files owned by the hci user and group can cause the system to become unresponsive.

To avoid these issues:

1.Create the expected user and group on each instance:

sudo groupadd hci -g 10001

sudo useradd hci -u 10001 -g 10001

2.Configure your applications to not change the ownership of files owned by the hci user and group.

Trademarks, Legal disclaimer, Third-party software in this documentation

Adding new instances

You are here:

You may want to add additional instances to the system if:

•You want to improve system performance.

•You are running out of disk space on one or more instances.

Important:

•You cannot add new master instances, only new worker instances.

However, these situations may also be improved by adding additional CPU, RAM, or disks to the instances you already have. For guidance, see Hardware resources.

Trademarks, Legal disclaimer, Third-party software in this documentation

Viewing instances

You are here:

You can use the Administration App, CLI, and REST API to view a list of all system instances in the system.

Administration App instructions

Viewing all instances

To view all instances, in the Administration App, click on Monitoring > Dashboard > Instances.

The page shows all instances in the system separated into two sections, Masters and Workers. Each instance is identified by its IP address.

If your system uses both internal and external networks, the page shows the internal and external IP addresses for each instance.

This table describes the information shown for each instance.

Property	Description
State	One of these: •Up — The instance is reachable by other instances in the system. •Down — The instance cannot be reached by other instances in the system.
Services	The number of services running on the instance.
Service Units	The total number of service units for all services running on the instance out of the recommended service unit limit for the instance. An instance with a higher number of service units is likely to be more heavily utilized by the system than an instance with a lower number of service units. The Instances page displays a blue bar for instances running less than the recommended service unit limit. The Instances page displays an orange bar and warning icon for instances running more than the recommended service unit limit. For more information on service units and service unit recommendations, see Service units.
Load Average	The load averages for the instance for the past one, five, and ten minutes.
CPU	The sum of the percentage utilization for each CPU core in the instance.
Memory Allocated	This section shows both: •The amount of RAM on the instance that's allocated to all services running on that instance. •The percentage of this allocated RAM to the total RAM for the instance.
Memory Total	The total amount of RAM for the instance.
Disk Used	The current amount of disk space that your system is using in the partition on which it is installed.
Disk Free	The amount of free disk space in the partition in which your system is installed.

Viewing the services running on an instance

To view the services running on an individual instance, in the Administration App:

1.Click on Monitoring > Dashboard > Instances.

2.Click on the instance you want.

The page lists all services running on the instance.

For each service, the page shows:

•The service name

•The service state. One of these:

oHealthy — The service is running normally.

oUnconfigured — The service has yet to be configured and deployed.

oDeploying — The system is currently starting or restarting the service. This can happen when:

–You move the service to run on a completely different set of instances.

–You repair a service.

For information on viewing the status service operations, see Monitoring service operations.

oBalancing — The service is running normally, but performing some background maintenance operations.

oUnder-protected — In a multi-instance system:

–One or more of the instances on which a service is configured to run are offline.

–The service is configured to run on fewer than the recommended number of instances. For information, see Service list.

oFailed — The service is not running or the system cannot communicate with the service.

•Service Units — The total number of service units currently being spent to run this service. This value is equal to the service's service unit cost times the number of instances on which the service is running. For more information, see Service units.

•Avg CPU Usage — The current percentage CPU usage for the service across all instances on which it's running.

•Memory — The current RAM usage for the service across all instances on which it's running.

•Disk Used — The current total amount of disk space that the service is using across all instances on which it's running.

Related CLI command(s)

getInstance

listInstances

For information on running CLI commands, see CLI reference.

Related REST API method(s)

GET /instances

GET /instances/{uuid}

For information on specific REST API methods, in the Administration App, click on the help icon (). Then:

•To view the administrative REST API methods, click on Admin API.

•To view the API methods used for performing searches, click on Search API.

For general information about the administrative REST API, see REST API reference.

Trademarks, Legal disclaimer, Third-party software in this documentation

Removing instances

You are here:

You would typically remove an instance from your system in these situations:

•You are retiring the hardware on which the instance runs

•The instance is in the Down state and cannot be recovered

•You want to run a system with fewer instances

Step 1: Conditionally, shut down the instance you want to remove

If the instance has already shut down as a result of a failure, the instance is in the Down state. Your system automatically attempts to move all services from that instance to other instances in the system. After all services have been moved, the instance is eligible for removal. Continue to Step 2: Remove the shut down instance from the system.

If the instance that you want to remove is in the Up state, you need to shut it down yourself before you can remove it from the system.

Procedure:

1.Move all the services that the instance is currently running to the other instances in the system. For information, see Moving and scaling services.

Important: Shutting down an instance without first moving its services can cause data loss.

2.Stop the run script from running. You do this using whatever method you're currently using to run the script.

3.Run this command to stop all system Docker containers on the instance:

sudo <installation-directory>/bin/stop

4.Delete the system Docker containers:

a.List all Docker containers:

sudo docker ps

b.Note the container IDs for all containers that use a com.hds.ensemble image.

c.Delete each of those containers:

sudo docker rm <container-id>

5.Delete the system Docker images:

a.List all Docker images:

sudo docker images

b.Note the image IDs for all images that use a com.hds.ensemble repository.

c.Delete each of those images:

sudo docker rmi <image-id>

6.Delete the system installation directory:

rm -rf /<installation-directory>

Step 2: Remove the shut down instance from the system

Administration App instructions

1.Click on System Configuration.

2.Click on the Instances panel.

The Removable Instances tab lists only the instances eligible for removal.

3.Click on the remove icon () for the instance you want to remove.

Related CLI command(s)

deleteInstance

For information on running CLI commands, see CLI reference.

Related REST API method(s)

DELETE /instances/{uuid}

For information on specific REST API methods, in the Administration App, click on the help icon (). Then:

•To view the administrative REST API methods, click on Admin API.

•To view the API methods used for performing searches, click on Search API.

For general information about the administrative REST API, see REST API reference.

Trademarks, Legal disclaimer, Third-party software in this documentation

Replacing a failed instance

You are here:

If an instance suffers an unrecoverable failure, you need to replace that instance with a new one.

Steps:

1.In the Administration App, view the Monitoring > Instances page to determine whether the failed instance was a master instance.

2.Select a new server or virtual machine to add as a new instance to the system. For information on instance requirements, see Requirements for running system instances.

3.Remove the failed instance from the system. For information, see Removing instances.

WARNING! If the failed instance was a master, after removing it, you have only two master instances remaining. If any other instance fails while you are in this state, the system becomes completely unavailable until you add a third master back to the system by completing this procedure.

4.Add the replacement instance to the system. For information, see Adding new instances.

Important: If the instance you are replacing was a master instance, when you run setup on the replacement instance, the list of masters that you specify for the -m option needs to include:

•The IP addresses of the two remaining healthy master instances.

•The IP address of the new instance that you're adding.

For example, in a system with master instance IPs ranging from 192.0.2.1 to 192.0.2.3 and you are replacing instance 192.0.2.3 with 192.0.2.5, run setup with these options:

sudo bin/setup -i 192.0.2.5 -m 192.0.2.1,192.0.2.2,192.0.2.5

This does not apply when you're replacing a worker instance. In that case, specify the IP addresses of the 3 existing masters.

Trademarks, Legal disclaimer, Third-party software in this documentation

We've Moved!

Product Documentation has moved to docs.hitachivantara.com

About master and worker instances

Single-instance systems versus multi-instance systems

One instance

Multiple instances