Skip to main content

We've Moved!

Product Documentation has moved to docs.hitachivantara.com
Hitachi Vantara Knowledge

Storage components

Within the Hitachi Content Platform for cloud scale (HCP for cloud scale) system, the Object Storage Management application lets you manage and monitor storage components.

The starting point for storage component management is the page Storage Component in the application Object Storage Management. The procedures in this module begin at this page.

Adding a storage component

You can use the Object Storage Management application or an API method to add a storage component to the HCP for cloud scale system.

TipTo improve performance and availability, and to avoid transfer fees, add storage components that are local to the HCP for cloud scale site.

The storage component must contain an HCP for cloud scale bucket before you can add the storage component to the HCP for cloud scale system.

To add a storage component, it must be available and you need the following information about it:

  • Storage component type
  • Endpoint information (host name or IP address)
  • If an HCP S Series storage component, the management host name and administrative user credentials
  • If used, the proxy host and port and the proxy user name and password
  • API port
  • S3 credentials (the access key and secret key to use for access to the storage component bucket)

Object Storage Management application instructions

The Add Storage Component wizard helps you add a storage component.

The storage component must contain an HCP for cloud scale bucket.

To add a storage component:

Procedure

  1. From the page Storage Component, click Add storage component.

    The wizard Add Storage Component opens. The first page describes the process and the information needed.
  2. Click Start.

    The page Connection opens.
  3. Specify the following:

    1. Storage Component Name (optional): Type the display name you choose for the storage component, up to 1,024 alphanumeric characters.

      If you leave this blank, the storage component is listed without a name.
    2. Storage Type: Select AMAZON_S3, HCP_S3, HCPS_S3 (HCP Series S), or GENERIC_S3.

    3. Region (optional): Type a region name of up to 1,024 characters.

      HCP for cloud scale doesn't validate this value except for its length.
    4. Host: Type either the IP address or the host name of the storage component. Type as many as 255 URI unreserved characters A-Z, a-z, 0-9, hyphen (-), period (.), underscore (_), and tilde (~) only.

  4. Click Next.

    The Connection Advanced page opens.
  5. Specify the following:

    1. Select the protocol used, either HTTPS (the default) or HTTP.

    2. If Use Default is selected, the applicable default port number is filled in. If you cancel the selection Use Default, type the Port number.

    3. If the storage component type is HCP S Series, additional information is required to execute MAPI requests at the management host. Type the login credentials for an administrative user in the boxes Username and Password, select the communication protocol (HTTPS or HTTP), and type the management system URL in the box MAPI URL. The account must have permissions to execute MAPI methods on the storage component.

    4. If you select Proxy, type values in the boxes Proxy Host (following the same naming rules as Host) and Proxy Port, and if the proxy needs authentication, type the Proxy User Name and Proxy Password.

  6. Click Next.

    The page Activation opens.
  7. Specify the following:

    1. Bucket Name: Type the name of the bucket on the storage component. The name can be from 3 to 63 characters long and must contain only lowercase characters (a-z), numbers (0-9), periods (.), or hyphens (-).

      NoteThe bucket must already exist on the storage component and should be empty.
    2. (Optional) To use path-style URLs to access buckets, select Use Path Style Always.

    3. Authenticate: Select the AWS Signature version, either V2 or V4.

    4. Type the Access Key.

    5. Type the Secret Key.

  8. Click Next.

    The page Review opens.
  9. Review the configuration of the storage component.

    • If the information is not correct, click Back to return to the wizard page and correct the information.
    • If the information is correct, click Create.

Results

The storage component is added, the page Storage Component is updated, and the storage component is added to the list.
NoteAfter you add the storage component, if its state is UNVERIFIED, edit the definition parameters.

If verification of the storage component fails, this error message appears:

Error activating the storage component. Please check and update your configuration before trying again.

The message gives additional details that you can use to troubleshoot the problem.

Related REST API methods

POST /storage_component/create
NoteAfter you define the storage component, if its state is UNVERIFIED, check the parameters you used when defining it.

For information about specific API methods, see the MAPI Reference or, in the Object Storage Management application, click the profile icon and select REST API.

Modifying a storage component

You can use the Object Storage Management application or an API method to modify a storage component.

Object Storage Management application instructions

You can modify the configuration of a storage component.

Procedure

  1. From the Storage Component page, click the Edit Component icon by the storage component you want to modify.

    The Edit Storage Component wizard opens.
  2. Edit connection information as needed. When you're finished click Next.

    The Connection Advanced page opens.
  3. Edit advanced connection information as needed. When you're finished click Next.

    The Activation page opens.
  4. Edit activation information as needed. When you're finished click Next.

    The Review page opens.
  5. Review the edited configuration of the storage component.

    • If the information is not correct, click Back to return to the wizard page with the information to correct.
    • If the information is correct, click Create.

Related REST API methods

POST /storage_component/update

For information about specific API methods, see the MAPI Reference or, in the Object Storage Management application, click the profile icon and select REST API.

Activating a storage component

You can use the Object Storage Management application or an API method to activate a storage component.

A storage component is displayed as UNVERIFIED if HCP for cloud scale cannot reach the storage component with the supplied parameters.

Object Storage Management application instructions

You can activate a storage container that is in the state INACTIVE.

To activate a storage component:

Procedure

  1. From the page Storage Component, navigate to the storage component you want to activate.

  2. Click the more icon for the storage component and then select Activate Now.

    The storage component state changes to ACTIVE.

Related REST API methods

POST /storage_component/update_state

For information about specific API methods, see the MAPI Reference or, in the Object Storage Management application, click the profile icon and select REST API.

Deactivating a storage component

You can use the Object Storage Management application or an API method to deactivate a storage component.

You might deactivate a storage component for maintenance purposes.

After you mark a storage component as INACTIVE, read, write, and healthcheck requests are rejected.

Object Storage Management application instructions

You can deactivate a storage container that is in the state ACTIVE.

To deactivate a storage component:

Procedure

  1. Select Storage.

    The Storage Component page opens.
  2. For the storage component you want to deactivate, click Yes, Inactivate.

    The storage component state changes to INACTIVE.

Related REST API methods

POST /storage_component/update_state

For information about specific API methods, see the MAPI Reference or, in the Object Storage Management application, click the profile icon and select REST API.

Marking a storage component as read-only

You can use the Object Storage Management application or API methods to mark a storage component as read-only.

You might mark a storage component as read-only if it is nearly full.

Once you mark a storage component as read-only, write requests are directed to different storage components.

Object Storage Management application instructions

To mark a storage component as read-only:

Procedure

  1. From the page Storage Component, navigate to the storage component you want to mark.

  2. Click the more icon for the storage component and then select Set read-Only.

    A window opens, displaying details about the storage component affected by the change, and prompts you to confirm your action.
  3. Click Mark Read-only.

    The storage component is marked as read-only.

Related REST API methods

PATCH /storage_component/update
POST /storage_component/update_state

For information about specific API methods, see the MAPI Reference or, in the Object Storage Management application, click the profile icon and select REST API.

Marking a storage component as read-write

You can use the Object Storage Management application or API methods to mark a storage component as read-write.

This makes the storage component available for writing new objects.

Object Storage Management application instructions

To mark a read-only storage component as read-write:

Procedure

  1. From the page Storage Component, navigate to the storage component you want to mark.

  2. Click the more icon for the storage component and then select Set read-write.

    A window opens, displaying details about the storage component, and prompts you to confirm your action.
  3. Click Open for Writes.

    The storage component is marked as read-write.

Related REST API methods

PATCH /storage_component/update
POST /storage_component/update_state

For information about specific API methods, see the MAPI Reference or, in the Object Storage Management application, click the profile icon and select REST API.

Viewing storage components

You can use the Object Storage Management application or an API method to list the storage components defined in the system.

For each storage component, the list gives its name, type, region, and state.

The storage component types are:

  • AMAZON_S3: An Amazon Web Services S3 compatible node
  • HCP_S3: A Hitachi Content Platform node
  • HCPS_S3: An HCP S Series node
  • GENERIC_S3: An S3 compatible node

The possible storage component states are:

  • ACTIVE: Available to serve requests
  • INACTIVE: Not available to serve requests (access is administratively paused)
  • INACESSIBLE: Available to serve requests, but HCP for cloud scale is having access issues (for example, network, authentication, or certificate issues)
  • UNVERIFIED: Not available to serve requests (unreachable by specified parameters, or awaiting administrative activation)

Object Storage Management application instructions

The storage components defined in the HCP for cloud scale system are listed in the Storage Components section of the Storage Components page.

Related REST API methods

POST /storage_component/list

For information about specific API methods, see the MAPI Reference or, in the Object Storage Management application, click the profile icon and select REST API.

Displaying storage component analytics

The Storage Component page displays counts of active, inactive, and unverified storage components, and information about system-wide available, free, and used storage capacity. The page also displays information about individual storage components.

The Storage Component page displays the following areas of information.

System-wide information

The top area of the page displays the following rolled-up information for HCP S Series storage components configured in the system:

  • Total capacity - the total number of bytes available
  • Used capacity - the total number of bytes used
  • Free capacity - the total number of bytes unused
  • Total objects - the total count of objects stored
  • Active storage - the number of storage components that can receive objects
  • Inactive storage - the number of storage components that cannot receive objects
  • Unverified storage - the number of storage components whose state can't be determined

The calculation of used capacity includes:

  • HCP S Series storage components configured for capacity monitoring
  • Storage components set to read-only status
  • Storage components that are inactive

Metrics for capacity usage are for Metadata Gateway instances only, so adding used capacity to free capacity will not equal the total capacity on the system. Also, multiple services are running on a system instance, all sharing the disk capacity. Therefore, the available capacity for the Metadata Gateway service on one node can be consumed by a different service running on the same node.

NoteIf the MAPI Gateway service restarts, capacity values are shown as 0 until fresh metrics are obtained.

The calculation of available system capacity does not include:

  • HCP S Series storage components not configured for capacity monitoring
  • Storage components other than HCP S Series storage components
  • Storage components set to read-only status
  • Storage components that are inactive
Per-storage component information

The central area of the page displays information for each HCP S Series storage component configured for capacity monitoring in the system:

  • User-defined name
  • Type (HCP S Series, displayed as HCPS_S3)
  • AWS region (us-west-2 is supported)
  • State:
    • ACTIVE: Available to serve requests
    • INACTIVE: Not available to serve requests (access is administratively paused)
    • UNVERIFIED: Not available to serve requests (unreachable by specified parameters, or awaiting administrative activation)
  • Whether or not the storage component is set to read-only status
  • Disk capacity: A graphical display of used capacity as a percentage of total capacity. You can configure a warning threshold, which is displayed as a red line. If the capacity exceeds the threshold the bar turns red. For example: Example of disk capacity information for two storage components, displayed as a bar with used capacity as a percentage of the total. The top bar represents a storage component that has exceeded the user-defined capacity threshold and is displayed in red. The bottom bar represents a storage component that is within the warning threshold and is displayed in blue.
  • Total capacity (used plus free) in bytes.
  • Available (free) capacity in bytes.

Capacity alerts are generated by the MAPI Gateway service. Use the System Management application to configure the capacity alert threshold for individual storage components or the overall system.

A more button (GUID-5579C4E8-D9C8-46DD-9EE7-51DA387A62CE-low.png), to the right of each storage component, opens a menu of actions that you can perform on that storage component.

Active object information

The bottom area of the page displays a graph over time of the count of active objects stored in the system.

Displaying counts of storage components

You can use the Object Storage Management application or an API method to display counts of storage components in the system.

Object Storage Management application instructions

To display storage counts, select Storage.

The infographic displays the count of active, inactive, and unverified storage components.

Related REST API methods

POST /storage_component/list

For information about specific API methods, see the MAPI Reference or, in the Object Storage Management application, click the profile icon and select REST API.

Metrics

HCP for cloud scale uses a third-party, open-source software tool, running over HTTPS as a service, to provide storage component metrics through a browser.

The Metrics service collects metrics for these HCP for cloud scale services:

  • S3 Gateway
  • MAPI Gateway
  • Policy Engine
  • Metadata Coordination
  • Metadata Gateway

By default the Metrics service collects all storage component metrics and you cannot disable collection. By default, the Metrics service collects data every ten seconds (the Scrape Interval) and retains data for 15 days (the Database Retention); you can configure these values in the service by using the System Management application.

NoteMetrics related to HCP for cloud scale instances and services are collected and provided by the System Management application. Collection of these metrics cannot be disabled.

Displaying the active object count

The Object Storage Management application displays a count of active objects stored in the system.

Object Storage Management application instructions

To display the Active Object Count report, select Storage. The Storage Components page opens.

The page displays a line graph showing the total number of active objects in the system over time.

Displaying metrics

You can use the metrics service to display or graph metrics, or use the service API to obtain metrics.

Object Storage Management application instructions

You can display and graph metrics using the metrics GUI.

To display metrics, select Tools and then select the Metrics panel. The metrics tool opens in a separate browser window.

The metrics tool is a third-party, open-source package. For information about using the metrics tool, see the documentation provided with the tool.

Available metrics

Metrics provide information about the operation of a service. Metrics are collected while the service is active. If a service restarts, its metrics are restarted.

NoteIf a metric is measured over an interval (for example, http_s3_servlet_requests_latency_seconds), but doesn't have at least two data points, the value is reported as NaN.
Metrics from all services

The following metrics are available from all services.

MetricDescription
http_healthcheck_requests_totalThe count of the total number of requests made to the health verification API.
http_monitoring_requests_totalThe count of the total number of requests made to the monitoring API.
scrape_duration_secondsThe duration in seconds of the scrape (collection interval).
scrape_samples_post_metric_relabelingThe count of samples remaining after metric relabeling was applied.
scrape_samples_scrapedThe count of samples the target exposed.
up1 if the instance is healthy (reachable) or 0 if collection of metrics from the instance failed.
MAPI Gateway

The following metrics are available from the MAPI Gateway service. These metrics are collected every five minutes.

MetricDescription
storage_available_capacity_bytesThe number of bytes free on an HCP S Series node.
storage_total_capacity_bytesThe number of bytes total, available and used, on an HCP S Series node.
storage_used_capacity_bytesThe number of bytes used on an HCP S Series node.

Each metric is reported with a label, store, identifying it as being either from a specific HCP S Series domain name or the aggregate total. You can also retrieve the metrics using this label. For example, to retrieve the used storage capacity of the storage component hcps10.company.com, you would specify:

storage_used_capacity_bytes{store="hcps10.company.com"}

To retrieve the used storage capacity of all available storage components, specify:

storage_used_capacity_bytes{store="aggregate"}
NoteIf storage components other than HCP S Series nodes are configured, aggregate totals aren't reported.
Message Queue

The Message Queue service supports a large number of general metrics. Information on these metrics is available at https://github.com/rabbitmq/rabbitmq-prometheus/blob/master/metrics.md.

Metadata Coordination

The following metrics are available from the Metadata Coordination service.

MetricDescription
mcs_copies_per_partitionThe count of number of copies of each metadata partition of each key space (to verify protection).
mcs_disk_usage_per_instanceThe total disk usage of each metadata instance.
mcs_disk_usage_per_partitionThe disk usage of each metadata partition of each key space.
mcs_failed_moves_per_keyspaceThe count of number of unsuccessful requests for metadata partition moves per keyspace.
mcs_failed_splits_per_keyspaceThe count of number of unsuccessful requests for metadata partition splits per keyspace.
mcs_moves_per_keyspaceThe count of number of successful requests for metadata partition moves per keyspace.
mcs_partitions_per_instanceThe count of total number of metadata partitions per metadata instance (to verify balance).
mcs_splits_per_keyspaceThe count of number of successful requests for metadata partition splits per keyspace.
Metadata Gateway

The following metrics are available from the Metadata Gateway service.

Note
  1. Client count metrics are an approximation and might not correspond to the actual count.
  2. Depending on when garbage collection tasks run, the ratio of client objects size to stored objects size might show a discrepancy.
MetricDescription
async_action_countThe count of actions performed.
async_action_latency_seconds_bucketA histogram for the duration, in seconds, of actions on buckets. For actions comprising multiple steps, this is the total of all steps.
async_action_latency_seconds_countThe count of action latency measurements taken.
async_action_latency_seconds_sumThe sum of action latency in seconds.
async_concurrenyA gauge for the number of concurrent actions.
async_duq_latency_seconds_bucketA histogram for the duration, in seconds, of operations on the durable update queue.
async_duq_latency_seconds_countThe count of durable update queue latency measurements.
async_getwork_database_countThe number of driver work checks accessing the database.
async_getwork_optimized_countThe number of driver work checks avoiding the database.
async_duq_latency_seconds_sumThe sum of actions on durable update queue in seconds.
metadata_available_capacity_bytesThe free bytes per instance (node) for the Metadata Gateway service. The label store is either the instance or aggregate.

Note: Because multiple service instances can run on a node, all consuming the same shared disk space, the value returned by this metric might be more than the actual capacity available.

metadata_clientobject_active_countThe count of client objects in metadata that are in the ACTIVE state.
metadata_clientobject_active_encrypted_​countThe count of encrypted client objects in metadata that are in the ACTIVE state.
metadata_clientobject_active_​unencrypted_​countThe count of unencrypted client objects in metadata that are in the ACTIVE state.
metadata_clientobject_and_part_active_​spacethe space occupied by client objects and parts in metadata that are in the ACTIVE state.
metadata_clientobject_part_active_countThe count of client object parts in metadata that are in the ACTIVE state.
metadata_storedObject_active_spaceThe space occupied by stored objects on the back-end storage components.
metadata_used_capacity_bytesThe used bytes per instance (node) for the Metadata Gateway service. The label store gives the domain name of the instance.

Note: Because multiple service instances can run on a node, all consuming the same shared disk space, combining this value with the value of metadata_available_capacity_​bytes won't give the total capacity of the service.

update_queue_inprogressThe count of update queue entries in progress.
update_queue_sizeThe size of the update queue.
Policy Engine

The following metrics are available from the Policy Engine service.

MetricDescription
scheduled_policy_errorsThe count of errors that occurred during execution of each scheduled policy.
scheduled_policy_items_acceptedThe count of items processed by each scheduled policy. This is the count of items examined which had operations outstanding.
scheduled_policy_items_examinedThe count of items examined by each scheduled policy.
S3 Gateway

The following metrics are available from the S3 Gateway service.

MetricDescription
http_s3_servlet_errors_totalThe count of total number of errors returned by the s3 servlet, grouped by error.
http_s3_servlet_get_object_response_​bytes_per_bucketThe count of total bytes in the body of S3 GET object responses per bucket.
http_s3_servlet_get_object_response_​bytes_totalThe count of total bytes in the body of S3 GET object responses.
http_s3_servlet_ingest_object_bytes_per_​bucketThe count of total objects ingested for the specified bucket.
http_s3_servlet_operations_totalThe count of total number of S3 operations made to the s3 servlet for each method, grouped by operation.
http_s3_servlet_post_object_bytes_totalThe count of total bytes of objects posted to S3.
http_s3_servlet_put_copied_bytes_totalThe count of total bytes of objects PUT copied (previously copied) to S3.
http_s3_servlet_put_object_bytes_totalThe count of total bytes of objects PUT (previously copied) to S3.
http_s3_servlet_put_object_part_bytes_​totalThe count of total bytes of PUT part operations (previously copied) to S3.
http_s3_servlet_requests_histogram_​latency_secondsThe latency in seconds as measured by a histogram timer, grouped by operation.
http_s3_servlet_requests_histogram_​latency_​seconds_bucketThe latency in seconds as measured by a histogram timer, grouped by bucket.
http_s3_servlet_requests_histogram_​latency_​seconds_countThe count of s3 servlet request observations; used with sum to determine average.
http_s3_servlet_requests_histogram_​latency_​seconds_sumSum of s3 servlet request latency in seconds; used with count to determine average.
http_s3_servlet_requests_latency_secondsThe latency in seconds as measured by a summary timer, grouped by operation.
http_s3_servlet_requests_latency_seconds:hour_averageThe latency in seconds over the last hour as measured by a summary timer.
http_s3_servlet_requests_latency_seconds_count
http_s3_servlet_requests_latency_seconds_sumThe sum of request latency in seconds.
http_s3_servlet_requests_per_bucketThe count of total put, get, or deletion requests made to the specified bucket.
http_s3_servlet_requests_totalThe count of total number of requests made to the s3 servlet, grouped by method.
http_s3_servlet_unimplemented_api_​request_totalThe count of total number of requests made for unimplemented S3 methods.
http_s3_servlet_unimplemented_bucket_​api_​request_totalThe count of total number of requests made for unimplemented S3 methods per bucket, grouped by API.
http_s3_servlet_unimplemented_object_​api_request_totalThe count of total number of requests made for unimplemented S3 methods per object, grouped by API.
http_s3_servlet_unimplemented_service_​api_request_totalThe count of total number of requests made for unimplemented S3 methods per service, grouped by API.
http_s3_servlet_unknown_api_requests_​totalThe count of total number of requests made for unknown S3 methods, grouped by API.
s3select_total_bytes_scannedThe count of bytes scanned in the object
s3select_total_bytes_processedThe count of bytes processed by the request
s3select_total_bytes_returnedThe count of bytes returned from the request
s3select_input_typeCount of requests by file type
s3select_output_typeCount of responses by file type
sync_from_bytesThe count of bytes synchronized from external storage (sync-from) by this instance.
sync_from_bytes_copiedThe count of bytes synchronized by full copy from external storage (sync-from) by this instance.
sync_from_bytes_putcopiedThe count of bytes synchronized by put-copy from external storage (sync-from) by this instance.
sync_from_object_count_failedThe count of objects that failed to synchronize from external storage (sync-from) by this instance.
sync_from_object_count_succeededThe count of objects synchronized from external storage (sync-from) by this instance.
sync_from_object_size_totalTotal size of object data synchronized from external storage (sync-from) by this instance.
sync_from_objectsTotal size of object data synchronized from external storage (sync-from) by this instance.
sync_to_bytesThe count of bytes synchronized to external storage (sync-to) by this instance.
sync_to_bytes_copiedThe count of bytes synchronized by full copy to external storage (sync-to) by this instance.
sync_to_bytes_putcopiedThe count of bytes synchronized by put-copy (previously copied) to external storage (sync-to) by this instance.
sync_to_object_count_failedThe count of objects that failed to synchronize to external storage (sync-to) by this instance.
sync_to_object_count_succeededThe count of objects synchronized to external storage (sync-to) by this instance.
sync_to_objectsThe count of objects synchronized to external storage (sync-to) by this instance.
sync_to_object_size_totalThe total size of object data synchronized to external storage (sync-to) by this instance.

Examples of metric expressions

By using metrics in formulas, you can generate useful information about the behavior and performance of the HCP for cloud scale system.

Available capacity

The following expression graphs the total capacity of the storage component store54.company.com over time. Information is returned for HCP S Series storage components only. The output includes the label store, which identifies the storage component by domain name. The system collects data every five minutes.

storage_total_capacity_bytes{store="store54.company.com"}

The following expression graphs the used capacity of all HCP S Series storage components in the system over time. (This is similar to the information displayed on the Storage Components page.) Information is returned only if all storage components in the system are HCP S Series nodes. The output includes the label aggregate. The system collects data every five minutes.

storage_used_capacity_bytes(store="aggregate"}
Growth of active-object count

The following expression graphs the count of active objects (metadata_clientobject_active_count) over time. (This is similar to the graph displayed on the Storage Components page.) You can use this formula to determine the growth in the number of active objects.

sum(metadata_clientobject_active_count)
Sum of update queues

The following expression graphs the size of all update queues. You can use this formula to determine whether the system is keeping up with internal events that are processed asynchronously in response to S3 activity. If this graph increases over time, you might want to increase capacity.

sum(update_queue_size)
Changes in S3 put requests over time

The following expression graphs the count of S3 put requests, summed across all nodes, at one-minute intervals. If you remove the specifier {operation="S3PutObjectOperation"} the expression graphs all S3 requests.

sum(rate(http_s3_servlet_operations_total{operation="S3PutObjectOperation"}[1m]))
Request time service levels

The following expression divides the latency of requests (async_duq_latency_seconds_bucket) in seconds by the number of requests (async_duq_latency_seconds_count), for the bucket getWork and requests less than or equal to 10 ms, and graphs it over time. You can use this formula to determine the percentage of requests completed in a given amount of time.

sum(rate(async_duq_latency_seconds_bucket{op="getWork",le="0.01"}[1m]))/
sum(rate(async_duq_latency_seconds_count{op="getWork"}[1m]))

Here is a sample graph of data from a lightly loaded system:

Prometheus graph of sample service-level data from a lightly loaded system
Request time quantile estimates

The following expression estimates the quantile for the latency of requests (async_duq_latency_seconds_bucket) in seconds for the bucket getWork. You can use this formula to estimate the percentage of requests completed in a given amount of time.

histogram_quantile(.9, sum(rate(async_duq_latency_seconds_bucket{op="getWork"}[1m])) by (le))

Here is a sample graph of data from a lightly loaded system:

Prometheus graph of sample quantile data from a lightly loaded system

Tracing requests and responses

HCP for cloud scale uses an open-source software tool, running over HTTPS as a service, for service tracing through a browser.

The Tracing service supports end-to-end, distributed tracing of S3 requests and responses by HCP for cloud scale services. Tracing helps you monitor performance and troubleshoot possible issues.

Tracing involves three service instances:

  • Tracing Query: serves traces
  • Tracing Agent: receives spans from tracers
  • Tracing Collector: receives spans from Tracing Agent service using Tchannel

Displaying traces

You can display traces using the tracing service GUI.

To begin tracing, select Tools and then click the Tracing panel. The tracing tool opens in a separate browser window.

When tracing, you can specify:

  • Service to trace
  • Operation to trace (all or specific) for each service
  • Tags
  • Lookback period (by default, over the last hour)
  • Minimum duration
  • Number of results to display (by default, 20)

The service displays all found traces with a chart giving the time duration for each trace. You can select a trace to display how the trace is served by difference services in cascade and the time spent on each service.

For information about the tracing tool, see the documentation provided with the tool.

Traceable operations

The following operations are traceable.

ComponentOperation
async-policy-engineAction Pipeline Action: BucketIdToNameMapAction
Action Pipeline Action: BucketLookupForAsyncPolicyAction
Action Pipeline Action: BucketOwnerIdToNameMapAction
Action Pipeline Action: BucketUpdateSecondaryAction
Action Pipeline Action: ClientObjectDispatchRemoveBack​ReferencesAction
Action Pipeline Action: ClientObjectLookupAction
Action Pipeline Action: ClientObjectModifyInProgressListAction
Action Pipeline Action: ClientObjectModifyListAction
Action Pipeline Action: ClientObjectUpdateSecondaryAction
Action Pipeline Action: DequeueAction
Action Pipeline Action: MetadataAction
BUCKET
CLIENT_OBJECT
STORED_OBJECT_BACK_REFERENCE
balance-engineBalanceCluster
BalanceEngineOperation
controlApi.ControlApiService
RefreshCluster
client-access-serviceAction Pipeline Action: BucketAuthorizationAction
Action Pipeline Action: BucketCountLimitAction
Action Pipeline Action: BucketCreateAction
Action Pipeline Action: BucketRegionValidationAction
Action Pipeline Action: BucketUpdateAclAction
Action Pipeline Action: ClientObjectInitiateMultipartAction
Action Pipeline Action: ClientObjectListInProgressMultipartAction
Action Pipeline Action: ClientObjectListVersionsAction
Action Pipeline Action: ClientObjectSizeLimitAction
Action Pipeline Action: ClientObjectTableLookupAction
Action Pipeline Action: ClientObjectUpdateAclAction
Action Pipeline Action: CompleteMultipartUploadAction
Action Pipeline Action: DataContentAction
Action Pipeline Action: DataDeletionAction
Action Pipeline Action: NotAnonymousAuthorizationAction
Action Pipeline Action: ObjectAuthorizationAction
Action Pipeline Action: ObjectDataPlacementAction
Action Pipeline Action: ObjectGetCurrentExpirationAction
Action Pipeline Action: ObjectGetMultipartAbortDateAction
Action Pipeline Action: ObjectGetUndeterminedExpirationAction
Action Pipeline Action: ObjectLookupAction
Action Pipeline Action: PartDataPlacementAction
Action Pipeline Action: PutAclAction
Action Pipeline Action: RequestBucketLookupAction
Action Pipeline Action: RequestVersionIdValidationAction
Action Pipeline Action: UploadIdValidationAction
Action Pipeline Action: UserLookupBucketsAction
Action Pipeline Action: VersionIdNotEmptyValidationAction
expiration-rules-engineEvaluateOperation
foundry-auth-clientFoundryAuthorizeOperation
FoundryValidateOperation
jaeger-query/api/dependencies
/api/services
/api/services/{service}/operations
/api/traces
mapi-serviceGET
POST
metadata-clientBucketService/Create
BucketService/List
BucketService/ListBucketOwnerListing
BucketService/LookupBucketNameById
BucketService/LookupByName
BucketService/UpdateACL
ClientObjectService/CloseNew
ClientObjectService/ClosePart
ClientObjectService/DeleteSpecific
ClientObjectService/List
ClientObjectService/LookupLatest
ClientObjectService/LookupSpecific
ClientObjectService/OpenNew
ClientObjectService/OpenPart
ClientObjectService/setACLOnLatest
ClientObjectService/Delete
ConfigService/List
ConfigService/LookupById
ConfigService/Set
StoredObjectService/Close
StoredObjectService/Delete
StoredObjectService/List
StoredObjectService/Lookup
StoredObjectService/MarkForCleanup
StoredObjectService/Open
UpdateQueueService/SecondaryEnqueue
UserService/LookupById
UserService/LookupOrCreate
UserService/UpdateAddAuthToken
metadata-coordination-serviceStatus.Service/GetStatus
metadata-gateway-serviceStatus.Service/GetStatus
BucketService/Create
BucketService/List
BucketService/ListBucketOwnerListing
BucketService/LookupBucketNameById
BucketService/LookupByName
BucketService/UpdateACL
ClientObjectService/CloseNew
ClientObjectService/ClosePart
ClientObjectService/DeleteSpecific
ClientObjectService/List
ClientObjectService/LookupLatest
ClientObjectService/LookupSpecific
ClientObjectService/OpenNew
ClientObjectService/OpenPart
ClientObjectService/setACLOnLatest
ConfigService/Delete
ConfigService/List
ConfigService/LookupById
ConfigService/Set
StoredObjectService/Close
StoredObjectService/Delete
StoredObjectService/List
StoredObjectService/Lookup
StoredObjectService/MarkForCleanup
StoredObjectService/Open
UpdateQueueService/SecondaryEnqueue
UserService/LookupById
UserService/LookupOrCreate
UserService/UpdateAddAuthToken
metadata-policy-clientPolicyService/ExecutePolicy
metadata-policy-serviceServiceStatus/GetStatus
PolicyService/ExecutePolicy
ScheduledDeleteBackendObjectsJob
ScheduledDeleteFailedWritesJob
ScheduledExpirationJob
ScheduledIncompleteMultipart​Expiration​Job
ScheduledStorageComponentHealth​Check​Job
storage-component-clientInMemoryStorageComponent​Verify​Operation
InMemoryStorageDeleteOperation
InMemoryStorageReadOperation
InMemoryStorageWriteOperation
storage-component-managerStorageComponentManager Operation: Create
StorageComponentManager Operation: List
StorageComponentManager Operation: Lookup
StorageComponentManager Operation: Update
tomcat-servletS3 Operation