Skip to main content

We've Moved!

Product Documentation has moved to docs.hitachivantara.com
Hitachi Vantara Knowledge

Monitoring

Your system provides a number of mechanisms that allow you to monitor the health and performance of the system and all of its instances and services.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

Monitoring instances

You can use the System Management application, CLI, and REST API to view a list of all instances in the system.

Related topics:

Service list

Services

System Management application instructions
Related CLI command(s)

getInstance

listInstances

For information on running CLI commands, see CLI reference.

Related REST API method(s)

GET /instances

GET /instances/{uuid}

For information on specific REST API methods, in the System Management application, click on the help icon (help-icon.png). Then:

To view the administrative REST API methods, click on REST API - Admin.

For general information about the administrative REST API, see REST API reference.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

Monitoring services

You can use System Management application, CLI, and REST API to view the status of all services for the system.

System Management application instructions
Related CLI command(s)

getService

listServices

For information on running CLI commands, see CLI reference.

Related REST API method(s)

GET /services

GET /services/{id}

For information on specific REST API methods, in the System Management application, click on the help icon (help-icon.png). Then:

To view the administrative REST API methods, click on REST API - Admin.

For general information about the administrative REST API, see REST API reference.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

Monitoring jobs

You can use System Management application, CLI, and REST API to monitor jobs. You can monitor information for all jobs, for all jobs of a type, or for each job individually.

 

Note: Some functions described here are not used with HCP for cloud scale. They are not visible in the System Management application, or have no effect when used.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

Monitoring all jobs

You can use System Management application, CLI, and REST API to view the status of all jobs for the system. For information on jobs, see Jobs.

 

Note: Some functions described here are not used with HCP for cloud scale. They are not visible in the System Management application, or have no effect when used.

System Management application instructions

To view the status of all jobs, in the System Management application, click on Jobs.

On this page, the All Jobs section shows cumulative information for all jobs in the system.

Service Units — The total number of service units currently being spent for running jobs.

This value is determined by where your system's job types are allowed to run.

Each job type has its own service unit cost. If an instance is configured to run multiple job types, only the job type with the highest service unit cost counts. 

For example, suppose that your system has 4 instances and supports two job types: X, which costs 50 service units, and Y, which costs 25. Job type X is configured to run on 3 instances. Job type Y is configured to run on those same 3 instances, plus an additional instance (4 total). In this case, your total service unit cost for jobs is equal to 175:

50 + 50 + 50 + 25 = 175

For information on configuring which instances job types can run on, see Configuring where jobs run.

Total CPU — The total CPU percentage utilization for all jobs across all instances in the system.

Total Memory — The total RAM consumed by all jobs across all instances in the system.

Total Disk — The total disk space consumed by all jobs across all instances in the system.

The Job Types section shows a box for each job type that the system supports.

Related CLI command(s)

listJobs

For information on running CLI commands, see CLI reference.

Related REST API method(s)

GET /jobs

For information on specific REST API methods, in the System Management application, click on the help icon (). Then:

To view the administrative REST API methods, click on REST API - Admin.

For general information about the administrative REST API, see REST API reference.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

Monitoring job types

You can use System Management application, CLI, and REST API to view the status of all jobs of a particular type. For information on jobs, see Jobs.

 

Note: Some functions described here are not used with HCP for cloud scale. They are not visible in the System Management application, or have no effect when used.

System Management application instructions

To view the detailed status for all jobs of a type, in the System Management application, click on Jobs. Then click on the box for the job type you want.

The Job Type page shows cumulative information for all jobs of the selected type.

Total CPU — The total CPU percentage utilization for all jobs of this type across all instances in the system.

Total Memory — The total RAM consumed by all jobs of this type across all instances in the system.

Total Disk — The total disk space consumed by all jobs of this type across all instances in the system.

The Jobs section lists each individual job of this type. For more information, see Monitoring individual jobs.

The Instances section shows a list of all system instances where jobs of this type are currently running.

The Pool section shows the instances that jobs of this type are allowed to run on. For information on configuring this, see Configuring where jobs run.

The Events section shows a list of all events related to this job type.

Related CLI command(s)

getJobTypes

For information on running CLI commands, see CLI reference.

Related REST API method(s)

GET /jobs/types

For information on specific REST API methods, in the System Management application, click on the help icon (). Then:

To view the administrative REST API methods, click on REST API - Admin.

For general information about the administrative REST API, see REST API reference.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

Monitoring individual jobs

You can use System Management application, CLI, and REST API to view the status of individual jobs. For information on jobs, see Jobs.

 

Note: Some functions described here are not used with HCP for cloud scale. They are not visible in the System Management application, or have no effect when used.

System Management application instructions

To view the detailed status for all jobs of a type, in the System Management application:

1.Click on Jobs.

2.Click on the box for the job type you want.

3.Click on the box for the job you want.

The Job page shows this information about the job:

Error Count — A tally of the number of errors encountered by the system when running the job or relocating it to other instances.

Status — One of these:

oIdle — The job has not run yet or, for jobs that have a schedule, the job is not scheduled to be running at the current time.

oPending — The job has been submitted for execution but is not yet running.

oRunning — The job is being executed.

oCompleted — The job finished execution without error.

oFailed — The job finished execution with errors.

oCanceled — The job has been canceled and is no longer running.

If the task is configured to run according to a schedule, the Status panel shows this icon:

For more information, see Scheduling jobs.

Service Units — The per-instance cost for this job's type. Individual jobs do not cost any service units to run. For more information, see Monitoring job types.

Total CPU — The total CPU percentage utilization for this job across all instances in the system.

Total Memory — The total RAM consumed by this job across all instances in the system.

Total Disk — The total disk space consumed by this job across all instances in the system.

The Instances section shows a list of all system instances where the job is currently running.

To view information about the volumes that the job uses, click on the row for an instance in the Instances section. For more information, see Viewing volumes.

The Pool section shows the instances that job is allowed to run on. For information on configuring this, see Configuring where jobs run.

The Events section shows a list of all events related to this job.

Related CLI command(s)

getJobStatus

For information on running CLI commands, see CLI reference.

Related REST API method(s)

GET /jobs/status/{uuid}

For information on specific REST API methods, in the System Management application, click on the help icon (). Then:

To view the administrative REST API methods, click on REST API - Admin.

For general information about the administrative REST API, see REST API reference.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

Monitoring processes

The Processes page lets you view information about what the system is doing. This includes any service operations you started and any internal maintenance processes the system needs to run.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

Monitoring service operations

You can use the System Management application, CLI, and REST API to monitor all service operations. These operations include:

The initial deployments of services when the system was installed.

Service relocation operations that you initiate. For information, see Moving and scaling services.

For each service operation, the system shows:

The name of the service involved

The status of the operation

The number of steps completed out of the total number of steps for the operation

System Management application instructions

1.Click on the Processes panel.

The Service Operations tab shows information about in-progress and completed service operations.

Related CLI command(s)

listSystemTasks

getSystemTask

For information on running CLI commands, see CLI reference.

REST API instructions

GET /tasks/system

GET /tasks/system/{uuid}

For information on specific REST API methods, in the System Management application, click on the help icon (help-icon.png). Then:

To view the administrative REST API methods, click on REST API - Admin.

For general information about the administrative REST API, see REST API reference.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

Monitoring system processes

You can use System Management application, REST API, and CLI to view the progress of internal system processes. These include package installation tasks and regularly scheduled system maintenance activities such as log rotation.

For each process, your system shows:

The process name

The process state

The times at which each step in the process run occurred

System Management application instructions

1.In the System Management application, click on Processes.

2.To view the currently running processes, click on the System tab.

3.To view the scheduled processes, click on the Scheduled tab.

Related CLI command(s)

listSystemTasks

getSystemTask

System processes have a type of SCHEDULED or ONE-TIME.

For information on running CLI commands, see CLI reference.

REST API instructions

GET /tasks/system

GET /tasks/system/{uuid}

System processes have a type of SCHEDULED or ONE-TIME.

For information on specific REST API methods, in the System Management application, click on the help icon (help-icon.png). Then:

To view the administrative REST API methods, click on REST API - Admin.

For general information about the administrative REST API, see REST API reference.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

System events

Your system maintains a log of system events that you can view through the System Management application, CLI, and REST API.

System Management application instructions

To view all system events, in the System Management application, click on Events.

Related CLI command(s)

queryEvents

To view events through the CLI, your requests need to specify which events you want to retrieve.

For example, this JSON request body searches the event log for all events that have a severity level of warning:

{

"severities": [

"warning"

]

}

For information on running CLI commands, see CLI reference.

REST API instructions

POST /events

To view events through the REST API, your requests need to specify which events you want to retrieve.

For example, this JSON request body searches the event log for all events that have a severity level of warning:

{

"severities": [

"warning"

]

}

For information on specific REST API methods, in the System Management application, click on the help icon (help-icon.png). Then:

To view the administrative REST API methods, click on REST API - Admin.

For general information about the administrative REST API, see REST API reference.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

Alerts

Your system displays alert messages to notify you of situations that require your attention. You can view these alerts through the System Management application, CLI, and REST API.

Each alert corresponds to a system event. For information on:

Viewing system events, see System events.

Receiving emails when events occur, see Creating email notification rules.

System Management application instructions

To view alerts, click on the user icon (adminIcon_27x20.png) in the top righthand corner of each System Management application page, then click on Notifications.

Related CLI command(s)

listAlerts

For information on running CLI commands, see CLI reference.

REST API instructions

GET /alerts

For information on specific REST API methods, in the System Management application, click on the help icon (help-icon.png). Then:

To view the administrative REST API methods, click on REST API - Admin.

For general information about the administrative REST API, see REST API reference.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

Creating email notification rules

For the system to send email notifications, you need to create a rule that specifies who to email, what email server to use, what events to send emails about, and what information to include in email messages.

System Management application instructions

1.Click on the Configuration panel.

2.Click on Notifications.

3.Click on the Create button.

4.In the Type field, select Email.

5.Enter a name for the notification rule.

6.Configure the SMTP settings and message settings for the notification rule.

7.Specify a comma-separated list of emails to send notifications to.

8.Specify a severity filter for the notification role.

9.Click on the Create button.

Related CLI command(s)

createNotificationRule

For information on running CLI commands, see CLI reference.

REST API instructions

POST /notifications

For information, see the administrative REST API documentation page and click on notifications > POST /notifications.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

Creating syslog notification rules

When you create a syslog notification rule, the system sends log messages to your syslog server for each applicable system event.

System Management application instructions

1.Click on the Configuration panel.

2.Click on Notifications.

3.Click on the Create button.

4.In the Type field, select Syslog.

5.Enter a name for the notification rule.

6.Configure the settings for the notification rule.

7.Specify a severity filter for the notification rule.

8.Click on the Create button.

Related CLI command(s)

createNotificationRule

For information on running CLI commands, see CLI reference.

REST API instructions

POST /notifications

For information, see the administrative REST API documentation page and click on notifications > POST /notifications.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

Logs and diagnostic information

Each service and job type maintains its own set of logs. By default, the logs are maintained in the /<install-directory>/log directory on each instance in the system. During installation, each service and job type can be configured to store its logs in a different, non-default location.

Log management

You can manage any of the system log files yourself if you want to. That is, you can delete or archive them as necessary.

 

Note: Deleting log files may make it more difficult for system support personnel to resolve issues you may encounter.

System logs are managed automatically in these ways:

All log files are periodically added to a compressed file and moved to /<install-directory>/retired/. This occurs at least once a day, but can also occur:

oWhenever you run the log_download script.

oHourly, if the system instance's disk space is more than 60% full.

When a log file grows larger than 10MB in size, the system stops writing to that file, renames it, and begins writing to a new file. For example, if exampleService.log.0 grows too large, it is renamed to exampleService.log.1 and the system creates a new exampleService.log.0 to write to.

Retrieving logs and diagnostic information

The log_download tool lets you easily retrieve logs and diagnostic information from all instances in the system. This tool is located at this path on each instance:

/<install-directory>/bin/log_download

For information on running the tool, run:

/<install-directory>/bin/log_download -h

 

Note:  

When using the log_download tool, if you specify the --output option, do not specify an output path that contains colons, spaces, or symbolic links. If you omit the --output option, you cannot run the script from within a directory path that contains colons, spaces, or symbolic links.

When you run the log_download script, all log files are automatically compressed and moved to the /<install-directory>/retired/ directory.

If an instance is down, you need to specify the --offline option to collect the logs from that instance. If your whole system is down, you need to run the log_download script with the --offline option on each instance.

Default log locations

By default, each service stores its logs in its own directory at this path:

<install-directory>/log

This table shows the default log directory names for each service. Depending on how your system was configured when first deployed, your system's logs may not be stored in these directories.

For information on the services, see Service list.

Default log directory name Related service Contains information about

com.hds.ensemble.plugins.service.adminApp

System-Management-application

The System Management application.

com.hds.ensemble.plugins.service.cassandra

Database

System configuration data.

Document fields and values.

com.hds.ensemble.plugins.service.chronos

Scheduling

Workflow task scheduling.

com.hds.ensemble.plugins.service.elasticsearch

Metrics

The storage and indexing of:

System events

Performance and failure metrics for workflow tasks

com.hds.ensemble.plugins.service.haproxy

Network-Proxy Network requests between instances.

com.hds.ensemble.plugins.service.kafka

Message Queue

Transmission of data between instances.

com.hds.ensemble.plugins.service.logstash

Logging

The transport of system events and workflow task metrics to the Metrics service.

com.hds.ensemble.plugins.service.marathon

Service-Deployment

The deployment of high-level services across system instances. High-level services are the ones that you can move and configure (such as Index), not the services grouped under System Services.

com.hds.ensemble.plugins.service.mesosAgent

Cluster-Worker

Work ordered by the Cluster-Coordination service.

com.hds.ensemble.plugins.service.mesosMaster

Cluster-Coordination

Hardware resource allocation.

com.hds.ensemble.plugins.service.remoteAction

Watchdog

Internal system processes.

com.hds.ensemble.plugins.service.sentinel

Sentinel

Internal system processes.

com.hds.ensemble.plugins.service.watchdog

Watchdog

General diagnostic information.

com.hds.ensemble.plugins.service.zookeeper

Synchronization

Coordination of actions and database operations across instances.

com.hitachi.aspen.foundry.service.mapi.gateway MAPI-Gateway Management of API requests.
com.hitachi.aspen.foundry.service.metadata.cache Metadata-Cache API requests Transmission of system metadata between instances.
com.hitachi.aspen.foundry.service.metadata.coordination Metadata-Coordination Transmission of system metadata between instances.
com.hitachi.aspen.foundry.service.metadata.gateway Metadata-Gateway Transmission of system metadata between instances.
com.hitachi.aspen.foundry.service.metadata.async.policy.engine Metadata-Policy-Engine Update operations.
com.hitachi.aspen.foundry.service.clientaccess.data S3-Gateway Transmission of S3 requests to endpoints.
com.hitachi.aspen.foundry.service.jaeger.agent Tracing-Agent Usage of tracing operations.
com.hitachi.aspen.foundry.service.jaeger.collector Tracing-Collector Usage of tracing collections.
com.hitachi.aspen.foundry.service.jaeger.query Tracing-Query Usage of tracing queries.

Trademarks, Legal disclaimer, Third-party software in this documentation

© 2017 - 2019 Hitachi Vantara Corporation. All rights reserved.

 

  • Was this article helpful?