Each service maintains its own set of logs. By default, log files are maintained in the folder
install_path/hcpcs/log on each instance in the system. During installation, you can configure each service to store its logs in a different (that is, non-default) location.
The following table lists the available log levels.
|ALL||FATAL, ERROR, WARN, INFO, DEBUG, TRACE|
|TRACE||FATAL, ERROR, WARN, INFO, DEBUG, TRACE|
|DEBUG||FATAL, ERROR, WARN, INFO, DEBUG|
|INFO||FATAL, ERROR, WARN, INFO|
|WARN||FATAL, ERROR, WARN (default)|
You can manage any of the log files yourself. That is, you can delete or archive them as necessary.
System logs are managed automatically in these ways:
- Retirement: All log files are periodically added to a compressed file and moved to install_path/hcpcs/retired/. This occurs at least once a day, but can also occur:
- Whenever you run the
- Hourly, if the system instance's disk space is more than 60% full.
- At the optimum time for a specific service.
- Whenever you run the
- Rotation: When a log file grows larger than 10MB in size, the system stops writing to that file, renames it, and begins writing to a new file. For example, if the file
exampleService.log.0grows to 10 MB, it is renamed to
exampleService.log.1and the system creates a new file named
exampleService.log.0to write to.
- Removal: When a log file becomes older than 90 days, it is removed. If the system instance's disk space is more than 70% full, log files are deleted when they become older than one day.
- When an optimum number of log files for a specific service is reached, the system can overwrite the oldest file. For example, if a service is limited to 20 log files, when the file
exampleService.log.19is filled, the system overwrites the file named
Retrieving logs and diagnostic information
log_download lets you easily retrieve logs and diagnostic information from all instances in the system. This tool is located at this path on each instance:
For information about running the tool, use this command:
- When using the tool
log_download, if you specify the option
--output, do not specify an output path that contains colons, spaces, or symbolic links. If you omit the option
--output, you cannot run the script from within a folder path that contains colons, spaces, or symbolic links.
- When you run the script
log_download, all log files are automatically compressed and moved to the folder install_path/hcpcs/retired/.
- If an instance is down, you need to specify the option
--offlineto collect the logs from that instance. If your whole system is down, you need to run the script log_download with the option
--offlineon each instance.
Default log locations
By default, each service stores its logs on each instance on which the service instance runs, in its own folder at this path:
This table shows the default log folder names for each service. Depending on how your system was configured when first deployed, your system's logs might not be stored in these folders.
|Service||Default log folder name||Contains information about|
|Admin-App||com.hds.ensemble.plugins.service.adminApp||The System Management application.|
|Scheduling||com.hds.ensemble.plugins.service.chronos||Workflow task scheduling.|
|N/A||com.hds.ensemble.plugins.service.containerAction||Created by custom actions run by service plugins.|
|Metrics||com.hds.ensemble.plugins.service.elasticsearch||The storage and indexing of:|
|Network-Proxy||com.hds.ensemble.plugins.service.haproxy||Network requests between instances.|
|Message Queue||com.hds.ensemble.plugins.service.kafka||The transmission of data between instances.|
|Logging||com.hds.ensemble.plugins.service.logstash||The transport of system events and workflow task metrics to the Metrics service.|
|Service-Deployment||com.hds.ensemble.plugins.service.marathon||The deployment of high-level services across system instances. High-level services are the ones that you can move and configure, not the services grouped under System Services.|
|Cluster-Worker||com.hds.ensemble.plugins.service.mesosAgent||The work ordered by the Cluster-Coordination service.|
|Cluster-Coordination||com.hds.ensemble.plugins.service.mesosMaster||Hardware resource allocation.|
|Watchdog||com.hds.ensemble.plugins.service.remoteAction||Internal system processes.|
|Sentinel||com.hds.ensemble.plugins.service.sentinel||The internal system processes.|
|Watchdog||com.hds.ensemble.plugins.service.watchdog||General diagnostic information.|
|Synchronization||com.hds.ensemble.plugins.service.zookeeper||The coordination of actions and database activities across instances.|
|S3-Gateway||com.hitachi.aspen.foundry.service.clientaccess.data||The client access data service.|
|Data-Lifecycle||com.hitachi.aspen.foundry.service.data-lifecycle.service||The data lifecycle service.|
|Tracing-Agent||com.hitachi.aspen.foundry.service.jaeger.agent||The tracing agent service.|
|Tracing-Collector||com.hitachi.aspen.foundry.service.jaeger.collector||The tracing collector service.|
|Tracing-Query||com.hitachi.aspen.foundry.service.jaeger.query||The tracing query service.|
|MAPI-Gateway||com.hitachi.aspen.foundry.service.mapi.gateway||The management API gateway.|
|Metadata-Policy-Engine||com.hitachi.aspen.foundry.service.metadata.async.policy.engine||The metadata asynchronous policy engine.|
|Grafana||com.hitachi.aspen.foundry.service.metrics.grafana||The dashboard service.|
|Mirror-Out-Policy||com.hitachi.aspen.foundry.service.policy.mirror.out||The mirror out (synch-to) service.|
|Mirror-In-Policy||com.hitachi.aspen.foundry.service.policy.mirror.in||The mirror in (synch-from) service.|
|S3-Notifications||com.hitachi.aspen.foundry.service.policy.s3.notifications||The S3 notifications service.|
|Metadata-Cache||com.hitachi.aspen.foundry.service.metadata.cache||The metadata cache.|
Note: This service is deprecated.
|Metadata-Gateway||com.hitachi.aspen.foundry.service.metadata.gateway||The metadata gateway.|
|Message-Queue||com.hitachi.aspen.foundry.service.rabbitmq.server||The message broker.|
|Key-Management-Server||com.hitachi.aspen.foundry.service.vault.vault||The key management server.|