Skip to main content

We've Moved!

Product Documentation has moved to docs.hitachivantara.com
Hitachi Vantara Knowledge

Statistics

The statistics available in the Content Software for File system and how to work with them is described.

Overview

As the Content Software for File system runs, it collects hundreds of statistics on system performance. These statistics help analyze the Content Software for File system performance and determine the source of any issue.

Five different categories of statistics are available for review:

  • CPU
  • Object Store
  • Operations
  • Operations (Driver)
  • Operations (NFS)
  • Operations (NFSw)
  • SSD

When you select each category, a list of the possible statistics related to the category is displayed, from which you can select a specific chart.

The default statistics page displays charts of the last hour of operation, presenting the system operation average value per second in one minute range.

GUID-FB9ABC04-7951-482C-B11B-5D9DA1E7F374-low.png

This Statistics view screen offers a number of options to drill-down into the statistics, according to category. Options include:

  • Mousing over the scrollable graph area to view various performance metrics of the Content Software for File cluster.
  • Troubleshooting or obtaining a correlation between events and performance (using the top line which provides links to events that occurred).
  • Adding more statistics to the view (using the Statistics menu).
  • Displaying different statistics simultaneously and toggling between them. By default, the graph area shows Ops/sec for the last hour. Using the Hour, Day, Week buttons at the bottom-right enables changing of the time interval.
  • Displaying, hiding. deleting, and zooming-in on statistics from defined timelines and dates.
  • Bookmarking specific statistics for future reference and sharing with others (using the URL).

Drill-down options

This Statistics page provides several options to drill down into the charts according to the selected category.

The options include:

  • Move the mouse over the scrollable chart area to view the performance metrics of the Weka cluster.
  • Troubleshoot or obtain a correlation between events and performance using links to events that occurred.
  • Add charts to the Statistics page, or remove charts.
  • Display different charts up to five on the statistics page. The default statistics page shows OPS (total), Throughput (total), and read/write latency for the last hour. You can change the interval by selecting the Hour, Day, or Week buttons or specify a timeframe.
  • Display and zoom in on statistics from defined timelines and dates.
  • Bookmark specific statistics for future reference and share with others (using the URL).
NoteThe page shows only the statistics of the backend and clients that are part of the cluster. The page does not show statistics in the following cases:
  • A host is removed.
  • A client is not connected to the cluster for more than the retention period.

The Weka cluster does not hold historical statistics data.

Working with statistics using the GUI

How to manage the statistics using the GUI.

Viewing statistics

From the menu, select Investigate > Statistics.

GUID-2C4B8CA7-920A-4ABC-81B6-AD251BB8A714-low.png

Adding a chart to the statistics page

You can add charts to the statistics page to display up to a maximum of five charts.

Procedure

  1. From the menu, select Investigate > Statistics.

  2. On the Statistics page, select +Add.

  3. In the Add Chart dialog, do one of the following:

    • From the Categories pane, select a category, and then from the Statistics Name pane select the required chart.
    • Search for a chart using the Filter. Type a keyword or two related to the chart, and then from the Statistics Name pane select the required chart.
    GUID-1B2FF4F9-A47A-4CCC-B8BB-02B5AFB763D2-low.png

Removing a chart from the statistics page

You can remove a chart that is no longer required to free space for adding another chart to the statistics page. For example, if the Statistics page already has the maximum number of five charts.

Procedure

  1. On the upper left corner of the chart, select X.

    GUID-7C5A4B4E-7925-4357-971E-F16E0F835672-low.png

Setting the timeframe

The Statistics page contains a time axis for all the displayed charts. To investigate charts in a specific timeframe, you can set the interval in the time axis to the last hour, last day, or last week. You can also set a timeframe for a specific period (start and end time).

Procedure

  1. To display the charts for the last period: Hour, Day, or Week, in the Last line, select the relevant button.

  2. To display the charts for a specific period, in the Range line select the calendar, and set the start time and end time for the timeframe.

    GUID-A3D04E66-0D07-4B10-A6DA-D5D6194631B6-low.png

Displaying events from a chart

If events occur during the period of the displayed charts, a purple box indicates the number of the events. To investigate the events, show and correlate them with the statistics data.

Procedure

  1. On the time axis, select the purple box (it only appears if events occur).

  2. From the popup box, select Show All.

    GUID-BD60F946-B9A5-479E-A94B-5D2FD9A08DFA-low.png

List of statistics

Attribute cache

TypeDescriptionUnits
GP_GETATTR_CACHE_MISSNumber of general purpose getAttr cache misses per secondOps/Sec
GP_GETATTRNumber of general purpose getAttr calls per secondOps/Sec

Block cache

TypeDescriptionUnits
BUCKET_CACHED_METADATA_BLOCKSBucket number of cached metadata blocksBlocks
BUCKET_CACHED_REGISTRY_L2_BLOCKSBucket number of cached registry L2 blocksBlocks
BUCKET_CACHE_METADATA_HITSBucket block cache metadata hitsQueries
BUCKET_CACHE_METADATA_MISSESBucket block cache metadata missesQueries
BUCKET_CACHE_REGISTRY_L2_HITSBucket block cache registry L2 hitsQueries
BUCKET_CACHE_REGISTRY_L2_MISSESBucket block cache registry L2 missesQueries
BUCKET_REGISTRY_L2_BLOCKS_NUMBucket number of registry L2 blocksBlocks

Block writes

Description
BLOCK_FULL_WRITESFull block writesWrites
BLOCK_PARTIAL_WRITESPartial block writesWrites

Bucket

TypeDescriptionUnits
BUDGET_UNDERFLOW_BLOCKSBlocks/Sec
CHOKING_LEVEL_ALLThrottling level applied on all types of IOs%
CHOKING_LEVEL_NON_MUTATINGThrottling level applied on non-mutating only types of IOs%
DESTAGED_BLOCKS_COUNTBlocks/Sec
DESTAGE_COUNTDestages/Sec
DIR_MOVE_TIMEOps
EXTENT_BLOCKS_COUNTBlocks
FREEABLE_LRU_BUFFERSBuffers
HASH_BLOCKS_COUNTBlocks
INODE_BLOCKS_COUNTBlocks
INODE_REFRESHER_QUEUE_LENGTHItems
JOURNAL_BLOCKS_COUNTBlocks
JOURNAL_ITERATIONSHistogram of number of batches of stripes committed in a single request
READSNumber of read operations per secondOps/Sec
READ_BYTESNumber of bytes read per secondBytes/Sec
READ_LATENCYAverage latency of READ operationsMicroseconds
REGISTRY_L1_BLOCKS_COUNTBlocks
REGISTRY_L2_BLOCKS_COUNTBlocks
REGISTRY_SEARCHES_COUNTQueries/Sec
RESIDENT_BLOCKS_COUNTBlocks/Sec
SNAPSHOT_CREATION_TIMESnaps
SPATIAL_SQUELCH_BLOCKS_COUNTBlocks
SUCCESSFUL_DATA_WEDGINGSAttempts/Sec
SUPERBLOCK_BLOCKS_COUNTBlocks
TAKEOVERS_SUCCESSFULTakeover Attempts/Sec
TAKEOVER_ATTEMPTSTakeover Attempts/Sec
TEMPORAL_SQUELCH_BLOCKS_COUNTBlocks
UNSUCCESSFUL_DATA_WEDGINGSAttempts/Sec
USER_DATA_BUFFERS_IN_USEBuffers
WRITESNumber of write operations per secondOps/Sec
WRITE_BYTESNumber of byte writes per secondBytes/Sec
WRITE_LATENCYAverage latency of WRITE operationsMicroseconds

Bucket failovers

TypeDescriptionUnits
BUCKET_FAILOVERSAmount of times swapping from a remote primary node to a secondaryFailovers
INVALID_BUCKET_TERMNumber of times a remote bucket rejected a request becaue the term was invalidExceptions
REMOTE_BUCKET_IS_SECONDARYNumber of times a remote bucket reported it is secondary and cannot serve usExceptions

Bucket rebalances

TypeDescriptionUnits
BUCKET_INITSNumber of bucket initializationsTimes
BUCKET_INIT_LATENCY_HISTMilliseconds
BUCKET_INIT_LATENCYAverage latency of bucket initializationSeconds

CPU

TypeDescriptionUnits
CPU_UTILIZATIONPercentage of the CPU time utilized for handling I/Os%

Chocking

TypeDescriptionUnits
CHOKING_LEVEL_ALLThrottling level applied on all types of IOs, both mutating and non-mutating
CHOKING_LEVEL_NON_MUTATINGThrottling level applied on non-mutating only types of IOs

Clients

TypeDescriptionUnits
CLIENTS_CONNECTEDClients connectedClients/Sec
CLIENTS_DISCONNECTEDClients left or were removedClients/Sec
CLIENTS_LEFTClients leftClients/Sec
CLIENTS_RECONNECTEDClients reconnected instead of an old instance of theirsClients/Sec
CLIENTS_REMOVEDClients removedClients/Sec

Config

TypeDescriptionUnits
AVERAGE_CHANGES_IN_CHANGESETAverage changes in changesetChanges/Sec
AVERAGE_CHANGES_IN_GENERATIONAverage changes in generationChanges/Sec
BACKEND_NODE_REJOIN_TIMEMilliseconds
CHANGESET_COMMIT_LATENCYAverage latency of committing a config changesetMicroseconds
CLIENT_NODE_REJOIN_TIMEMilliseconds
GENERATION_COMMIT_LATENCYAverage latency of committing a config generationMicroseconds
HEARTBEAT_PROCESSING_TIME_OLDSeconds
HEARTBEAT_PROCESSING_TIMESeconds
LEADER_HEARTBEAT_PROCESSING_TIME_OLDSeconds
LEADER_HEARTBEAT_PROCESSING_TIMESeconds
TOTAL_CHANGESETS_COMMITTEDTotal number of changesets committedChange Sets
TOTAL_COMMITTED_CHANGESTotal number of config changes committedChanges
TOTAL_GENERATIONS_COMMITTEDNumber of generations committed per secondGenerations

Filesystem OBS

TypeDescriptionUnits
BACKPRESSURED_BUCKETS_IN_FSSNumber of backpressured bucketsBuckets
CONCURRENT_DEMOTESHow many demotes are executed concurrently Demotes
DEMOTE_EXTENT_OBS_FETCH_BACKPRESSURENumber of extent BACKPRESSURE fetch operations per secondOps/Sec
DEMOTE_EXTENT_OBS_FETCH_IMMEDIATE_RELEASENumber of extent IMMEDIATE_RELEASE fetch operations per secondOps/Sec
DEMOTE_EXTENT_OBS_FETCH_MANHOLENumber of extent MANHOLE fetch operations per secondOps/Sec
DEMOTE_EXTENT_OBS_FETCH_MIGRATENumber of extent MIGRATE fetch operations per secondOps/Sec
DEMOTE_EXTENT_OBS_FETCH_POLICYNumber of extent POLICY fetch operations per secondOps/Sec
DEMOTE_EXTENT_OBS_FETCH_RECLAMATION_REUPLOADNumber of extent RECLAMATION_REUPLOAD fetch operations per secondOps/Sec
DEMOTE_EXTENT_OBS_FETCH_STOWNumber of extent STOW fetch operations per secondOps/Sec
DEMOTE_EXTENT_OBS_FETCHNumber of extent fetch operations per secondOps/Sec
DEMOTE_WAITING_FOR_SLOTAverage time waiting for a demotion concurrency slotMicroseconds
DESERIALIZED_EXTENTS_WITH_INVALID_BLOBSNumber of deserialized extents with invalid blob idExtents
DOWNLOADSNumber of promotes operations per secondOps/Sec
DOWNLOAD_LATENCYLatency of promote operationsMicroseconds
FAILED_DOWNLOADSNumber of failed promotes operations per secondOps/Sec
FAILED_UPLOADSNumber of failed demotes operations per secondOps/Sec
OBS_4K_IOPS_READNumber of object storage dedicated 4K read operations per secondOps/Sec
OBS_BACKPRESSURE_FREEDNumber of bytes freed from disk due to backpressureBytes/Sec
OBS_BLOB_HEADER_DOWNLOAD_LATENCYAverage latency of blob header downloadMicroseconds
OBS_BLOB_SCAVENGE_LATENCYAverage latency of blob scavengesMicroseconds
OBS_BLOB_TIERING_DURATIONMicroseconds
OBS_COMPLETELY_ALIVE_BLOBSPercentage of blobs with only live extents linked to them%
OBS_COMPLETELY_DEAD_BLOBSPercentage of blobs with no live extent linked to them%
OBS_EXTENTS_PREFETCHNumber of pre-fetched extentsOps/Sec
OBS_FREEDNumber of bytes freed from disk because they are in the OBSBytes/Sec
OBS_IMMEDIATE_RELEASE_FREEDNumber of bytes freed from disk due to immediate releaseBytes/Sec
OBS_INODES_PREFETCHNumber of pre-fetched inodesOps/Sec
OBS_INODES_RELEASENumber of pre-fetched inodesOps/Sec
OBS_ONGOING_RECLAMATIONSNumber of ongoing reclamationsOps
OBS_POLICY_FREEDNumber of bytes freed from disk due to policyBytes/Sec
OBS_PROMOTE_EXTENT_WRITE_LATENCYMicroseconds
OBS_PROMOTE_EXTENT_WRITEOps/Sec
OBS_PROMOTE_WRITEBytes/Sec
OBS_READReads that needed data from the OBSOps/Sec
OBS_RECLAMATION_PURGED_BYTESNumber of bytes purged per secondBytes/Sec
OBS_RECLAMATION_SCAVENGED_BLOBSNumber of blobs scavenged per secondOps/Sec
OBS_RECLAMATION_SCAVENGED_BYTESNumber of blobs scavenged per secondBytes/Sec
OBS_RECLAMATION_WAIT_FOR_DESTAGEAverage time waiting for destage on space reclamationMicroseconds
OBS_RELOC_DOWNLOADNumber of relocation blobs downloaded per secondOps/Sec
OBS_RELOC_UPLOADNumber of relocation blobs uploaded per secondOps/Sec
OBS_SCAVENGED_BLOB_WASTE_LEVELHistogram of waste level found in blobs
OBS_SHARED_DOWNLOADS_LATENCYMicroseconds
OBS_SHARED_DOWNLOADSOps/Sec
OBS_TRUNCATETruncates that needed data from the OBSOps/Sec
OBS_UNEXPECTED_TAG_ON_DOWNLOADUnexpected tag when downloading an extentOccurences
OBS_WRITEWrites that needed data from the OBSOps/Sec
STOW_SERIALIZED_EXTENT_DATANumber of extent descriptors uploaded that contain dataExtent descriptors
STOW_SERIALIZED_EXTENT_DESCSNumber of extent descriptors uploadedExtent descriptors
STOW_SERIALIZED_EXTENT_REDIRECTSNumber of extent descriptors uploaded that redirect to previous snapshotExtent descriptors
TIERED_FS_BREAKING_POLICYTiered Filesystem Breaking Policy CounterActivations
TIMEOUT_DOWNLOADSNumber of timeout’ed promotes operations per secondOps/Sec
TIMEOUT_OPERATIONSTotal timeouted operations per secondOps/Sec
TIMEOUT_UPLOADSNumber of timeout’ed demotes operations per secondOps/Sec
UNEXPECTED_BLOCK_VERSION_POST_UPGRADEUnexpected block version after upgrade completedOccurences
UNEXPECTED_HASHBLOCK_KV_VERSION_POST_UPGRADEUnexpected hash block KV version after upgrade completedOccurences
UPLOADSNumber of upload attempts per secondOps/Sec
UPLOAD_CHOKING_LATENCYAverage latency of waiting for demote choking budgetMicroseconds
UPLOAD_LATENCYLatency of demoteMicroseconds

Frontend

TypeDescriptionUnits
FE_IDLE_CYCLESCycles/Sec
FE_IDLE_TIMEPercentage of the CPU time not utilized for handling I/Os on frontend%

Frontend encryption

TypeDescriptionUnits
FE_BLOCKS_DECRYPTEDNumber of blocks decrypted in the frontendBlocks
FE_BLOCKS_ENCRYPTEDNumber of blocks encrypted in the frontendBlocks
FE_BLOCK_CRYPTO_LATENCYAverage latency of frontend block cryptoMicroseconds
FE_BLOCK_DECRYPT_DURATIONDuration of decryption of blocks in the frontendMicroseconds
FE_BLOCK_ENCRYPT_DURATIONDuration of encryption of blocks in the frontendMicroseconds
FE_FILENAMES_DECRYPTEDNumber of filenames decrypted in the frontendFilenames
FE_FILENAMES_ENCRYPTEDNumber of filenames encrypted in the frontendFilenames
FE_FILENAME_CRYPTO_LATENCYAverage latency of frontend filename cryptoMicroseconds
FE_FILENAME_DECRYPT_DURATIONDuration of decryption of filenames in the frontendMicroseconds
FE_FILENAME_ENCRYPT_DURATIONDuration of encryption of filenames in the frontendMicroseconds

Garbage collection

TypeDescriptionUnits
GC_FREE_SIZE_AFTER_SCANGC pool size after the scan endsBytes
GC_FREE_SIZE_BEFORE_SCANGC pool size before the scan startsBytes
GC_SCAN_TIMEGC scan timeMsec
GC_USED_SIZE_AFTER_SCANGC used size after the scan endsBytes
GC_USED_SIZE_BEFORE_SCANGC used size before the scan startsBytes

JRPC

TypeDescriptionsUnits
JRPC_SERVER_PROCESSING_AVGMicroseconds
JRPC_SERVER_PROCESSING_TIME

Journal

TypeDescriptionUnits
JOURNAL_CURRENT_OPSOperations currently in journalJournal Entries
JOURNAL_OPS_INOperations added to the journalJournal Entries/Sec
JOURNAL_OPS_OUTOperations removed from the journalJournal Entries/Sec

Memory

TypeDescriptionUnits
RSS_CURRENTMB
RSS_PEAKMB

Network

TypeDescriptionUnits
BAD_RECV_CSUMNumber of packets received with a bad checksumPackets/Sec
CORRUPT_PACKETSNumber of packets received and deemed corruptedPackets/Sec
DOUBLY_RECEIVED_PACKETSNumber of packets that were received multiple timesPackets/Sec
DROPPED_LARGE_PACKETSNumber of large packets dropped in the socket backendPackets/Sec
DROPPED_PACKETSNumber of packets received that we droppedPackets/Sec
ECN_ENCOUNTEREDNumber of ECN Encountered packetsPackets/Sec
FAULT_RECV_DELAYED_PACKETSNumber of received packets delayed due to a fault injectionPackets/Sec
FAULT_RECV_DROPPED_PACKETSNumber of received packets dropped due to a fault injectionPackets/Sec
FAULT_SENT_DELAYED_PACKETSNumber of sent packets delayed due to a fault injectionPackets/Sec
FAULT_SENT_DROPPED_PACKETSNumber of sent packets dropped due to a fault injectionPackets/Sec
GOODPUT_RX_RATIOPercentage of goodput RX packets out of total data packets received%
GOODPUT_TX_RATIOPercentage of goodput TX packets out of total data packets sent%
GW_MAC_RESOLVE_FAILURESNumber of times we failed to ARP resolve the gateway IPFailures
GW_MAC_RESOLVE_SUCCESSESNumber of times we succeeded to ARP resolve the gateway IPSuccesses
NODE_RECONNECTEDNumber of reconnectionsReconnects/Sec
PACKETS_PUMPEDNumber of packets received in each call to recvPackets
PEER_RTTRTT per peer nodeMicroseconds
PORT_RX_BYTESNumber of bytes receivedBytes/Sec
PORT_RX_PACKETSNumber of packets receivedPackets/Sec
PORT_TX_BYTESNumber of bytes transmittedBytes/Sec
PORT_TX_PACKETSNumber of packets transmittedPackets/Sec
PUMPS_TXQ_FULLNumber of times we couldn’t send any new packets to the NIC queuePumps/Sec
PUMPS_TXQ_PARTIALNumber of times we only sent some of our queued packets to the NIC queuePumps/Sec
PUMP_DURATIONDuration of each pump
PUMP_INTERVALInterval between pumps
RDMA_ADD_CHUNK_FAILURESFailures/Sec
RDMA_BINDING_FAILOVERSFail-overs/Sec
RDMA_CANCELED_COMPLETIONSCompletions/Sec
RDMA_CLIENT_BINDING_INVALIDATIONSInvalidations/Sec
RDMA_COMPLETIONSCompletions/Se
RDMA_COMP_DURATION
RDMA_COMP_FAILURESFailures/Sec
RDMA_COMP_LATENCYAverage time of RDMA requests completionMicroseconds
RDMA_NET_ERR_RETRY_EXCEEDEDOccurences/Sec
RDMA_POOL_ALLOC_FAILEDFailures/Sec
RDMA_POOL_LOW_CAPACITYFailures/Sec
RDMA_PORT_WAITING_FIBERSWaiting fibers
RDMA_REQUESTSRequests/Sec
RDMA_RX_BYTESBytes/Sec
RDMA_SERVER_BINDING_RESTARTSRestarts/Sec
RDMA_SUBMIT_FAILURESFailures/Sec
RDMA_SUBMIT_TIMEOUTSTimeouts/Sec
RDMA_TX_BYTESBytes/Sec
RECEIVED_CONTROL_PACKETSNumber of received control packetsPackets/Sec
RECEIVED_DATA_PACKETSNumber of received data packetsPackets/Sec
RECEIVED_PACKETSNumber of packets receivedPackets/Sec
RECEIVED_PACKET_GENERATIONSThe generation ("resend count") of the first incarnation of the packet seen by the receiver (indicates packet loss)

REORDERED_PACKETS

Number of reordered packetsPackets/Sec
RESEND_BATCH_SIZENumber of packets sent in a resend batch
RESENT_DATA_PACKETSNumber of data packets resentPackets/Sec
SEND_QUEUE_TIMEOUTSNumber of packets cancelled due to envelope timeout and were not in the send windowPackets/Sec
SEND_WINDOW_TIMEOUTSNumber of packets cancelled due to envelope timeout while in the send windowPackets/Sec
SENT_ACKSNumber of ACK packets sentPackets/Sec
SENT_CONTROL_PACKETSNumber of control packets sentPackets/Sec
SENT_DATA_PACKETSNumber of data packets sentPackets/Sec
SENT_PACKETSNumber of sent packetsPackets/Sec
SENT_REJECTSNumber of rejects sentPackets/Sec
SHORT_CIRCUIT_SENDSNumber of packets sent to the same nodePackets/Sec
SLOW_PATH_CSUMNumber of packets that went through checksum calculation on the CPUPackets/Sec
TIMELY_RESENDSNumber of packets resent due to timely resendPackets/Sec
TIME_TO_ACKHistogram of time to ack a data packetPackets/Sec
TIME_TO_FIRST_SENDTime from queueing to first send
UCX_SEND_CBPackets/Sec
UCX_SEND_ERRORPackets/Sec
UCX_SENT_PACKETS_ASYNCPackets/Sec
UCX_SENT_PACKETS_IMMEDIATEPackets/Sec
UCX_TXQ_FULLPackets/Sec
UDP_SENDMSG_FAILED_EAGAINNumber of packets that failed to be sent on the socket backend with EAGAINPackets/Sec
UDP_SENDMSG_FAILED_OTHERNumber of packets that failed to be sent on the socket backend with an unknown errorPackets/Sec
UDP_SENDMSG_PARTIAL_SENDNumber of packets that we failed to send but in the same pump some packets were sentPackets/Sec
UNACKED_RESENDSNumber of packets resent after receiving an ackPackets/Sec
ZERO_CSUMNumber of checksum zero receivedPackets/Sec

Object storage

TypeDescriptionUnits
FAILED_OBJECT_DELETESNumber of failed object deletes per second (any failure reason)Ops/Sec
FAILED_OBJECT_DOWNLOADSNumber of failed object download per second (any failure reason)Ops/Sec
FAILED_OBJECT_HEAD_QUERIESNumber of failed object head queries per second (any failure reason)Ops/Sec
FAILED_OBJECT_OPERATIONSTotal failed operations per secondOps/Sec
FAILED_OBJECT_UPLOADSNumber of failed object uploads per second (any failure reason)
OBJECT_DELETESNumber of object deletes per secondOps/Sec
OBJECT_DELETE_DURATIONMilliseconds
OBJECT_DELETE_LATENCYLatency of deleting an objectMicroseconds
OBJECT_DOWNLOADS_BACKGROUNDNumber of BACKGROUND objects downloaded per secondOps/Sec
OBJECT_DOWNLOADS_FOREGROUNDNumber of FOREGROUND objects downloaded per secondOps/Sec
OBJECT_DOWNLOADSNumber of objects downloaded per secondOps/Sec
OBJECT_DOWNLOAD_BYTES_BACKGROUNDNumber of BACKGROUND bytes sent to the object storageBytes/Sec
OBJECT_DOWNLOAD_BYTES_FOREGROUNDNumber of FOREGROUND bytes sent to the object storageBytes/Sec
OBJECT_DOWNLOAD_DURATIONMilliseconds
OBJECT_DOWNLOAD_LATENCYLatency of downloading an objectMicroseconds
OBJECT_DOWNLOAD_SIZEBytes
OBJECT_HEAD_DURATIONMilliseconds
OBJECT_HEAD_LATENCYLatency of deleting an objectMilliseconds
OBJECT_HEAD_QUERIESNumber of object head queries per secondOps/Sec
OBJECT_OPERATIONSTotal operations per secondOps/Sec
OBJECT_UPLOADS_BACKPRESSURENumber of BACKPRESSURE upload attempts per secondOps/Sec
OBJECT_UPLOADS_IMMEDIATE_RELEASENumber of IMMEDIATE_RELEASE upload attempts per secondOps/Sec
OBJECT_UPLOADS_MANHOLENumber of MANHOLE upload attempts per secondOps/Sec
OBJECT_UPLOADS_MIGRATENumber of MIGRATE upload attempts per secondOps/Sec
OBJECT_UPLOADS_POLICYNumber of POLICY upload attempts per secondOps/Sec
OBJECT_UPLOADS_RECLAMATION_REUPLOADNumber of RECLAMATION_REUPLOAD upload attempts per secondOps/Sec
OBJECT_UPLOADS_STOWNumber of STOW upload attempts per secondOps/Sec
OBJECT_UPLOADSNumber of object uploads per secondOps/Sec
OBJECT_UPLOAD_BYTES_BACKPRESSURENumber of BACKPRESSURE bytes sent to the object storageBytes/Sec
OBJECT_UPLOAD_BYTES_IMMEDIATE_RELEASENumber of IMMEDIATE_RELEASE bytes sent to the object storageBytes/Sec
OBJECT_UPLOAD_BYTES_MANHOLENumber of MANHOLE bytes sent to object storageBytes/Sec
OBJECT_UPLOAD_BYTES_MIGRATENumber of MIGRATE bytes sent to the object storageBytes/Sec
OBJECT_UPLOAD_BYTES_POLICYNumber of POLICY bytes sent to the object storageBytes/Sec
OBJECT_UPLOAD_BYTES_RECLAMATION_REUPLOADNumber of RECLAMATION_REUPLOAD bytes sent to the object storageBytes/Sec
OBJECT_UPLOAD_BYTES_STOWNumber of STOW bytes sent to the object storageBytes/Sec
OBJECT_UPLOAD_DURATIONMilliseconds
OBJECT_UPLOAD_LATENCYLatency of uploading an objectMicroseconds
OBJECT_UPLOAD_SIZEBytes
OBS_READ_BYTESNumber of bytes read from the object storageBytes/Sec
OBS_WRITE_BYTESNumber of bytes sent to the object storageBytes/Sec
ONGOING_DOWNLOADSNumber of ongoing downloadsOps
ONGOING_REMOVESNumber of ongoing removesOps
ONGOING_UPLOADSNumber of ongoing uploadsOps
READ_BYTESNumber of bytes read from the object storageBytes/Sec
REQUEST_COUNT_DELETENumber of HTTP DELETE requests per secondRequests/Sec
REQUEST_COUNT_GETNumber of HTTP GET requests per secondRequests/Sec
REQUEST_COUNT_HEADNumber of HTTP HEAD requests per secondRequests/Sec
REQUEST_COUNT_INVALIDNumber of HTTP INVALID requests per secondRequests/Sec
REQUEST_COUNT_POSTNumber of HTTP POST requests per secondRequests/Sec
REQUEST_COUNT_PUTNumber of HTTP PUT requests per secondRequests/Sec
RESPONSE_COUNT_ACCEPTEDNumber of HTTP ACCEPTED responses per secondResponses/Sec
RESPONSE_COUNT_BAD_GATEWAYNumber of HTTP BAD_GATEWAY responses per secondResponses/Sec
RESPONSE_COUNT_BAD_REQUESTNumber of HTTP BAD_REQUEST responses per secondResponses/Sec
RESPONSE_COUNT_CONFLICTNumber of HTTP CONFLICT responses per secondResponses/Sec
RESPONSE_COUNT_CONTINUENumber of HTTP CONTINUE responses per secondResponses/Sec
RESPONSE_COUNT_CREATEDNumber of HTTP CREATED responses per secondResponses/Sec
RESPONSE_COUNT_EXPECTATION_FAILEDNumber of HTTP EXPECTATION_FAILED responses per secondResponses/Sec
RESPONSE_COUNT_FORBIDDENNumber of HTTP FORBIDDEN responses per secondResponses/Sec
RESPONSE_COUNT_FOUNDNumber of HTTP FOUND responses per secondResponses/Sec
RESPONSE_COUNT_GATEWAY_TIMEOUTNumber of HTTP GATEWAY_TIMEOUT responses per secondResponses/Sec
RESPONSE_COUNT_GONENumber of HTTP GONE responses per secondResponses/Sec
RESPONSE_COUNT_HTTP_VERSION_NOT_SUPPORTEDNumber of HTTP HTTP_VERSION_NOT_SUPPORTED responses per secondResponses/Sec
RESPONSE_COUNT_INSUFFICIENT_STORAGENumber of HTTP INSUFFICIENT_STORAGE responses per secondResponses/Sec
RESPONSE_COUNT_INVALIDNumber of HTTP INVALID responses per secondResponses/Sec
RESPONSE_COUNT_LENGTH_REQUIREDNumber of HTTP LENGTH_REQUIRED responses per secondResponses/Sec
RESPONSE_COUNT_METHOD_NOT_ALLOWEDNumber of HTTP METHOD_NOT_ALLOWED responses per secondResponses/Sec
RESPONSE_COUNT_MOVED_PERMANENTLYNumber of HTTP MOVED_PERMANENTLY responses per secondResponses/Sec
RESPONSE_COUNT_NON_AUTH_INFONumber of HTTP NON_AUTH_INFO responses per secondResponses/Sec
RESPONSE_COUNT_NOT_ACCEPABLENumber of HTTP NOT_ACCEPABLE responses per secondResponses/Sec
RESPONSE_COUNT_NOT_FOUNDNumber of HTTP NOT_FOUND responses per secondResponses/Sec
RESPONSE_COUNT_NOT_IMPLEMENTEDNumber of HTTP NOT_IMPLEMENTED responses per secondResponses/Sec
RESPONSE_COUNT_NOT_MODIFIEDNumber of HTTP NOT_MODIFIED responses per secondResponses/Sec
RESPONSE_COUNT_NO_CONTENTNumber of HTTP NO_CONTENT responses per secondResponses/Sec
RESPONSE_COUNT_OKNumber of HTTP OK responses per secondResponses/Sec
RESPONSE_COUNT_PARTIAL_CONTENTNumber of HTTP PARTIAL_CONTENT responses per secondResponses/Sec
RESPONSE_COUNT_PAYMENT_REQUIREDNumber of HTTP PAYMENT_REQUIRED responses per secondResponses/Sec
RESPONSE_COUNT_PRECONDITION_FAILEDNumber of HTTP PRECONDITION_FAILED responses per secondResponses/Sec
RESPONSE_COUNT_PROXY_AUTH_REQUIREDNumber of HTTP PROXY_AUTH_REQUIRED responses per secondResponses/Sec
RESPONSE_COUNT_REDIRECT_MULTIPLE_CHOICESNumber of HTTP REDIRECT_MULTIPLE_CHOICES responses per secondResponses/Sec
RESPONSE_COUNT_REQUESTED_RANGE_NOT_SATISFIABLENumber of HTTP REQUESTED_RANGE_NOT_SATISFIABLE responses per secondResponses/Sec
RESPONSE_COUNT_REQUEST_HEADER_FIELDS_TOO_LARGENumber of HTTP REQUEST_HEADER_FIELDS_TOO_LARGE responses per secondResponses/Sec
RESPONSE_COUNT_REQUEST_TIMEOUTNumber of HTTP REQUEST_TIMEOUT responses per secondResponses/Sec
RESPONSE_COUNT_REQUEST_TOO_LARGENumber of HTTP REQUEST_TOO_LARGE responses per secondResponses/Sec
RESPONSE_COUNT_RESET_CONTENTNumber of HTTP RESET_CONTENT responses per secondResponses/Sec
RESPONSE_COUNT_SEE_OTHERNumber of HTTP SEE_OTHER responses per secondResponses/Sec
RESPONSE_COUNT_SERVER_ERRORNumber of HTTP SERVER_ERROR responses per secondResponses/Sec
RESPONSE_COUNT_SERVICE_UNAVAILABLENumber of HTTP SERVICE_UNAVAILABLE responses per secondResponses/Sec
RESPONSE_COUNT_SWITCHING_PROTOCOLNumber of HTTP SWITCHING_PROTOCOL responses per secondResponses/Sec
RESPONSE_COUNT_TEMP_REDIRECTNumber of HTTP TEMP_REDIRECT responses per secondResponses/Sec
RESPONSE_COUNT_UNAUTHORIZEDNumber of HTTP UNAUTHORIZED responses per secondResponses/Sec
RESPONSE_COUNT_UNPROCESSABLE_ENTITYNumber of HTTP UNPROCESSABLE_ENTITY responses per secondResponses/Sec
RESPONSE_COUNT_UNSUPPORTED_MEDIA_TYPENumber of HTTP UNSUPPORTED_MEDIA_TYPE responses per secondResponses/Sec
RESPONSE_COUNT_URI_TOO_LONGNumber of HTTP URI_TOO_LONG responses per secondResponses/Sec
RESPONSE_COUNT_USE_PROXYNumber of HTTP USE_PROXY responses per secondResponses/Sec
WAITING_FOR_BUCKET_DOWNLOAD_BANDWIDTHMilliseconds
WAITING_FOR_BUCKET_DOWNLOAD_FLOWMilliseconds
WAITING_FOR_BUCKET_REMOVE_FLOWMilliseconds
WAITING_FOR_BUCKET_UPLOAD_BANDWIDTHMilliseconds
WAITING_FOR_BUCKET_UPLOAD_FLOWMilliseconds
WAITING_FOR_GROUP_DOWNLOAD_BANDWIDTHMilliseconds
WAITING_FOR_GROUP_DOWNLOAD_FLOWMilliseconds
WAITING_FOR_GROUP_REMOVE_FLOWMilliseconds
WAITING_FOR_GROUP_UPLOAD_BANDWIDTHMilliseconds
WAITING_FOR_GROUP_UPLOAD_FLOWMilliseconds
WAITING_IN_BUCKET_DOWNLOAD_QUEUEMilliseconds
WAITING_IN_BUCKET_REMOVE_QUEUEMilliseconds
WAITING_IN_BUCKET_UPLOAD_QUEUEMilliseconds
WAITING_IN_GROUP_DOWNLOAD_QUEUEMilliseconds
WAITING_IN_GROUP_REMOVE_QUEUEMilliseconds
WAITING_IN_GROUP_UPLOAD_QUEUEMilliseconds
WRITE_BYTESNumber of bytes sent to the object storageBytes/Sec

Operations(NFS)

TypeDescriptionUnits
ACCESS_LATENCYAverage latency of ACCESS operationsMicroseconds
ACCESS_OPSNumber of ACCESS operation per secondOps/Sec
COMMIT_LATENCYAverage latency of COMMIT operationsMicroseconds
COMMIT_OPSNumber of COMMIT operation per secondOps/Sec
CREATE_LATENCYAverage latency of CREATE operationsMicroseconds
CREATE_OPSNumber of CREATE operation per secondOps/Sec
FSINFO_LATENCYAverage latency of FSINFO operationsMicroseconds
FSINFO_OPSNumber of FSINFO operation per secondOps/Sec
GETATTR_LATENCYAverage latency of GETATTR operationsMicroseconds
GETATTR_OPSNumber of GETATTR operation per secondOps/Sec
LINK_LATENCYAverage latency of LINK operationsMicroseconds
LINK_OPSNumber of LINK operation per secondOps/Sec
LOOKUP_LATENCYAverage latency of LOOKUP operationsMicroseconds
LOOKUP_OPSNumber of LOOKUP operation per secondOps/Sec
MKDIR_LATENCYAverage latency of MKDIR operationsMicroseconds
MKDIR_OPSNumber of MKDIR operation per secondOps/Sec
MKNOD_LATENCYAverage latency of MKNOD operationsMicroseconds
MKNOD_OPSNumber of MKNOD operation per secondOps/Sec
OPSTotal number of operationsOps/Sec
PATHCONF_LATENCYAverage latency of PATHCONF operationsMicroseconds
PATHCONF_OPSNumber of PATHCONF operation per secondOps/Sec
READDIR_LATENCYAverage latency of READDIR operationsMicroseconds
READDIR_OPSNumber of READDIR operation per secondOps/Sec
READLINK_LATENCYAverage latency of READLINK operationsMicroseconds
READLINK_OPSNumber of READLINK operation per secondOps/Sec
READSNumber of read operations per secondOps/Sec
READ_BYTESNumber of bytes read per secondBytes/Sec
READ_DURATIONMicroseconds
READ_LATENCYAverage latency of READ operationsMicroseconds
READ_SIZESNFS read sizes histogram
REMOVE_LATENCYAverage latency of REMOVE operationsMicroseconds
REMOVE_OPSNumber of REMOVE operation per secondOps/Sec
RENAME_LATENCYAverage latency of RENAME operationsMicroseconds
RENAME_OPSNumber of RENAME operation per secondOps/Sec
SETATTR_LATENCYAverage latency of SETATTR operationsMicroseconds
SETATTR_OPSNumber of SETATTR operation per secondOps/Sec
STATFS_LATENCYAverage latency of STATFS operationsMicroseconds
STATFS_OPSNumber of STATFS operation per secondOps/Sec
SYMLINK_LATENCYAverage latency of SYMLINK operationsMicroseconds
SYMLINK_OPSNumber of SYMLINK operation per secondOps/Sec
THROUGHPUTNumber of byte read/writes per secondBytes/Sec
WRITESNumber of write operations per secondOps/Sec
WRITE_BYTESNumber of byte writes per secondBytes/Sec
WRITE_DURATIONMicroseconds
WRITE_LATENCYAverage latency of WRITE operationsMicroseconds
WRITE_SIZESNFS write sizes histogram

Operations (NFSw)

TypeDescriptionUnits
AVG_COPY_OPSAverage copy operations per secondOps/Sec
AVG_DELETE_OPSAverage delete operations per secondOps/Sec
AVG_GET_OPSAverage get operations per secondOps/Sec
AVG_LIST_V1_OPSAverage list v1 operations per secondOps/Sec
AVG_LIST_V2_OPSAverage list v2 operations per secondOps/Sec
AVG_MULTIPART_UPLOAD_OPSAverage multipart upload operations per secondOps/Sec
AVG_PUT_OBJECTPART_OPSAverage put objectpart operations per secondOps/Sec
AVG_PUT_OPSAverage put operations per secondOps/Sec
READ_BYTESNumber of byte reads per secondBytes/Sec
TOTAL_BUCKET_CREATE_OPSTotal bucket create operations per secondOps/Sec
TOTAL_BUCKET_DELETE_OPSTotal bucket delete operation per secondsOps/Sec
TOTAL_BUCKET_LIST_OPSTotal bucket list operations per secondOps/Sec
TOTAL_COPY_LATENCYAverage latency of Copy operationsMicroseconds
TOTAL_COPY_OPSTotal Copy operationsOps
TOTAL_DELETE_OPSTotal delete operationsOps
TOTAL_GET_BUCKET_ACL_OPSTotal get bucket acl operations per secondOps/Sec
TOTAL_GET_BUCKET_NOTIFICATION_OPSTotal get bucket notifications operations per secondOps/Sec
TOTAL_GET_LATENCYAverage latency of Get operationsMicroseconds
TOTAL_GET_OPSTotal Get operationsOps
TOTAL_LIST_V1_OPSTotal list v1 operationsOps
TOTAL_LIST_V2_OPSTotal list v2 operationsOps
TOTAL_MULTIPART_UPLOAD_LATENCYAverage latency of Multipart upload operationsMicroseconds
TOTAL_MULTIPART_UPLOAD_OPSTotal multipart upload operationsOps
TOTAL_PUT_BUCKET_ACL_OPSTotal put bucket acl operations per secondOps/Sec
TOTAL_PUT_LATENCYAverage latency of Put operationsMicroseconds
TOTAL_PUT_OBJECTPART_OPSTotal put objectpart operationsOps
TOTAL_PUT_OPSTotal put operationsOps
WRITE_BYTESNumber of byte writes per secondsBytes/Sec

Operations(driver)

TypeDescriptionUnits
DIRECT_READ_SIZESBlocks/Sec
DIRECT_WRITE_SIZESBlocks
DIRECT_WRITE_SIZES_RATEBlocks/Sec
DIRECT_WRITE_SIZESBlocks
DOORBELL_RING_COUNTOps
FAILED_1HOP_READSNumber of failed single hop reads per secondOps/Sec
FILEATOMICOPEN_LATENCYAverage latency of FILEATOMICOPEN operationsMicroseconds
FILEATOMICOPEN_OPSNumber of FILEATOMICOPEN operation per secondOps/Sec
FILECLOSE_LATENCYAverage latency of FILECLOSE operationsMicroseconds
FILECLOSE_OPSNumber of FILECLOSE operation per secondOps/Sec
FILEOPEN_LATENCYAverage latency of FILEOPEN operationsMicroseconds
FILEOPEN_OPSNumber of FILEOPEN operation per secondOps/Sec
FLOCK_LATENCYAverage latency of FLOCK operationsMicroseconds
FLOCK_OPSNumber of FLOCK operation per secondOps/Sec
GETATTR_LATENCYAverage latency of GETATTR operationsMicroseconds
GETATTR_OPSNumber of GETATTR operation per secondOps/Sec
GETXATTR_LATENCYAverage latency of GETXATTR operationsMicroseconds
GETXATTR_OPSNumber of GETXATTR operation per secondOps/Sec
IOCTL_OBS_PREFETCH_LATENCYAverage latency of IOCTL_OBS_PREFETCH operationsMicroseconds
IOCTL_OBS_PREFETCH_OPSNumber of IOCTL_OBS_PREFETCH operation per secondOps/Sec
IOCTL_OBS_RELEASE_LATENCYAverage latency of IOCTL_OBS_RELEASE operationsMicroseconds
IOCTL_OBS_RELEASE_OPSNumber of IOCTL_OBS_RELEASE operation per secondOps/Sec
LINK_LATENCYAverage latency of LINK operationsMicroseconds
LINK_OPSNumber of LINK operation per secondOps/Sec
LISTXATTR_LATENCYAverage latency of LISTXATTR operationsMicroseconds
LISTXATTR_OPSNumber of LISTXATTR operation per secondOps/Sec
LOOKUP_LATENCYAverage latency of LOOKUP operationsMicroseconds
LOOKUP_OPSNumber of LOOKUP operation per secondOps/Sec
MKNOD_LATENCYAverage latency of MKNOD operationsMicroseconds
MKNOD_OPSNumber of MKNOD operation per secondOps/Sec
OPSTotal number of operationsOps/Sec
RDMA_WRITE_REQUESTSNumber of RDMA write request operations per secondOps/Sec
READDIR_LATENCYAverage latency of READDIR operationsMicroseconds
READDIR_OPSNumber of READDIR operation per secondOps/Sec
READLINK_LATENCYAverage latency of READLINK operationsMicroseconds
READLINK_OPSNumber of READLINK operation per secondOps/Sec
READSNumber of read operations per secondOps/Sec
READ_BYTESNumber of bytes read per secondBytes/Sec
READ_CHECKSUM_ERRORSOps
READ_DURATIONMicroseconds
READ_LATENCY_NO_QOSAverage latency of READ operations without QoS delayMicroseconds
READ_LATENCYAverage latency of READ operationsMicroseconds
READ_QOS_DELAYAverage QoS delay for READ operationsMicroseconds
READ_RDMA_SIZES_RATEBlocks/Sec
READ_RDMA_SIZESBlocks
READ_SIZES_RATEBlocks/Sec
READ_SIZESBlocks
RENAME_LATENCYAverage latency of RENAME operationsMicroseconds
RENAME_OPSNumber of RENAME operation per secondOps/Sec
REQUESTS_COMPLETEDOps
REQUESTS_FETCHEDOps
RMDIR_LATENCYAverage latency of RMDIR operationsMicroseconds
RMDIR_OPSNumber of RMDIR operation per secondOps/Sec
RMXATTR_LATENCYAverage latency of RMXATTR operationsMicroseconds
RMXATTR_OPSNumber of RMXATTR operation per secondOps/Sec
SETATTR_LATENCYAverage latency of SETATTR operationsMicroseconds
SETATTR_OPSNumber of SETATTR operation per secondOps/Sec
SETXATTR_LATENCYAverage latency of SETATTR operationsMicroseconds
SETXATTR_OPSNumber of SETXATTR operation per secondOps/Sec
STATFS_LATENCYAverage latency of STATFS operationsMicroseconds
STATFS_OPSNumber of STATFS operation per secondOps/Sec
SUCCEEDED_1HOP_READSNumber of succesfull single hop reads per secondOps/Sec
SYMLINK_LATENCYAverage latency of SYMLINK operationsMicroseconds
SYMLINK_OPSNumber of SYMLINK operation per secondOps/Sec
THROUGHPUTNumber of byte read/writes per secondBytes/Sec
UNLINK_LATENCYAverage latency of UNLINK operationsMicroseconds
UNLINK_OPSNumber of UNLINK operation per secondOps/Sec
WRITESNumber of write operations per secondOps/Sec
WRITE_BYTENumber of byte writes per secondBytes/Sec
WRITE_DURATIONMicroseconds
WRITE_LATENCY_NO_QOSAverage latency of WRITE operations without QoS delayMicroseconds
WRITE_LATENCYAverage latency of WRITE operationsMicroseconds
WRITE_QOS_DELAYAverage QoS delay for WRITE operationsMicroseconds
WRITE_RDMA_SIZES_RATEBlocks/Sec
WRITE_RDMA_SIZESBlocks/
WRITE_SIZES_RATEBlocks/Sec
WRITE_SIZESBlocks/Sec

Operations

TypeDescriptionUnits
ACCESS_LATENCYAverage latency of ACCESS operationsMicroseconds
ACCESS_OPSNumber of ACCESS operation per secondOps/Sec
COMMIT_LATENCYAverage latency of COMMIT operationsMicroseconds
COMMIT_OPSNumber of COMMIT operation per secondOps/Sec
CREATE_LATENCYAverage latency of CREATE operationsMicroseconds
CREATE_OPSNumber of CREATE operation per secondOps/Sec
FILEATOMICOPEN_LATENCYAverage latency of FILEATOMICOPEN operationsMicroseconds
FILEATOMICOPEN_OPSNumber of FILEATOMICOPEN operation per secondOps/Sec
FILECLOSE_LATENCYAverage latency of FILECLOSE operationsMicroseconds
FILECLOSE_OPSNumber of FILECLOSE operation per secondOps/Sec
FILEOPEN_LATENCYAverage latency of FILEOPEN operationsMicroseconds
FILEOPEN_OPSNumber of FILEOPEN operation per secondOps/Sec
FLOCK_LATENCYAverage latency of FLOCK operationsMicroseconds
FLOCK_OPSNumber of FLOCK operation per secondOps/Sec
FSINFO_LATENCYAverage latency of FSINFO operationsMicroseconds
FSINFO_OPSNumber of FSINFO operation per secondOps/Sec
GETATTR_LATENCYAverage latency of GETATTR operations Microseconds
GETATTR_OPSNumber of GETATTR operation per secondOps/Sec
LINK_LATENCYAverage latency of LINK operationsMicroseconds
LINK_OPSNumber of LINK operation per secondOps/Sec
LOOKUP_LATENCYAverage latency of LOOKUP operationsMicroseconds
LOOKUP_OPSNumber of LOOKUP operation per secondOps/Sec
MKDIR_LATENCYAverage latency of MKDIR operationsMicroseconds
MKDIR_OPSNumber of MKDIR operation per secondOps/Sec
MKNOD_LATENCYAverage latency of MKNOD operationsMicroseconds
MKNOD_OPSNumber of MKNOD operation per secondOps/Sec
OPSTotal number of operationsOps/Sec
PATHCONF_LATENCYAverage latency of PATHCONF operationsMicroseconds
PATHCONF_OPSNumber of PATHCONF operation per secondOps/Sec
READDIR_LATENCYAverage latency of READDIR operationsMicroseconds
READDIR_OPSNumber of READDIR operation per secondOps/Sec
READLINK_LATENCYAverage latency of READLINK operationsMicroseconds
READLINK_OPSNumber of READLINK operation per secondOps/Sec
READSNumber of read operations per secondOps/Sec
READ_BYTESNumber of bytes read per secondBytes/Sec
READ_DURATIONMicroseconds
READ_LATENCYAverage latency of READ operationsMicroseconds
REMOVE_LATENCYAverage latency of READ operationsMicroseconds
REMOVE_OPSNumber of REMOVE operation per secondOps/Sec
RENAME_LATENCYAverage latency of RENAME operationsMicroseconds
RENAME_OPSNumber of RENAME operation per secondOps/Sec
RMDIR_LATENCYAverage latency of RMDIR operationsMicroseconds
RMDIR_OPSNumber of RMDIR operation per secondOps/Sec
SETATTR_LATENCY Average latency of SETATTR operationsMicroseconds
SETATTR_OPSNumber of SETATTR operation per secondOps/Sec
STATFS_LATENCYAverage latency of STATFS operationsMicroseconds
STATFS_OPSNumber of STATFS operation per secondOps/Sec
SYMLINK_LATENCYAverage latency of SYMLINK operationsMicroseconds
SYMLINK_OPSNumber of SYMLINK operation per secondOps/Sec
THROUGHPUTNumber of byte read/writes per secondBytes/Sec
UNLINK_LATENCYAverage latency of UNLINK operationsMicroseconds
UNLINK_OPSNumber of UNLINK operation per secondOps/Sec
WRITESNumber of write operations per secondOps/Sec
WRITE_BYTESNumber of byte writes per secondBytes/Sec
WRITE_DURATIONMicroseconds
WRITE_LATENCYAverage latency of WRITE operationsMicroseconds

RAFT

TypeDescriptionUnits
Bucket_LEADER_CHANGESChanges of leaderChanges
Bucket_REQUESTS_COMPLETEDRequests to leader completed successfullyRequests
Configuration_LEADER_CHANGESChanges of leaderChanges
Configuration_REQUESTS_COMPLETEDRequests to leader completed successfullyRequests
Invalid_LEADER_CHANGESChanges of leaderChanges
Invalid_REQUESTS_COMPLETEDRequests to leader completed successfullyRequests
SYNCLOG_TIMEOUTSNumber of times timeouted on syncing logs to nodeTimeouts
Test_LEADER_CHANGESChanges of leaderChanges
Test_REQUESTS_COMPLETEDRequests to leader completed successfullyRequests

RAID

TypeDescriptionUnits
LONG_RPC_TIMEOUTSLong RPC timeouts encounteredOccurences
RAID_BLOCKS_IN_PREPARED_STRIPEFree blocks in prepared stripe
RAID_CHUNKS_CLEANED_BY_SHIFTDirty chunks cleaned by being shifted outOccurences
RAID_CHUNKS_SHIFTEDDirty chunks that shifted outOccurences
RAID_COMMITTED_STRIPESNum stripes writtenStripes
RAID_PLACEMENT_SWITCHESNum placement switchesSwitches
RAID_READ_BATCHES_PER_REQUEST_HISTOGRAMHistogram of number of batches of stripes read in a single request
RAID_READ_BLOCKS_STRIPE_HISTOGRAMHistogram of number of blocks read from a single stripe
RAID_READ_BLOCKSNumber of blocks read by the RAIDBlocks/Sec
RAID_READ_DEGRADEDDegraded mode readsBlocks/Sec
RAID_READ_IOSRaw read blocks performed by the RAIDBlocks/Sec
RAID_STALE_WRITES_DETECTEDStale write detected in readOccurences

RPC

TypeDescriptionUnits
CLIENT_CANCELED_REQUESTSCalls/Sec
CLIENT_DROPPED_RESPONSESCalls/Sec
CLIENT_RECEIVED_EXCEPTIONSCalls/Sec
CLIENT_RECEIVED_RESPONSESCalls/Sec
CLIENT_RECEIVED_TIMEOUTSCalls/Sec
CLIENT_ROUNDTRIP_AVG_LOWMicroseconds
CLIENT_ROUNDTRIP_AVGMicroseconds
CLIENT_RPC_CALLS_LOWRPC/Sec
CLIENT_RPC_CALLSRPC/Sec
CLIENT_SENT_REQUESTSCalls/Sec
FIRST_RESULTSNumber of first results per secondOps/Sec
SERVER_ABORTSCalls/Sec
SERVER_DROPPED_REQUESTSCalls/Sec
SERVER_PROCESSING_AVGMicroseconds
SERVER_PROCESSING_TIME
SERVER_RECEIVED_REQUESTSCalls/Sec
SERVER_REJECTSCalls/Sec
SERVER_RPC_CALLSRPC/Sec
SERVER_SENT_EXCEPTIONSCalls/Sec
SERVER_SENT_RESPONSESCalls/Sec
TIME_TO_FIRST_RESULTAverage latency to the first result of a MultiCallMicroseconds

Reactor

TypeDescription Units
BACKGROUND_CYCLESNumber of cycles spent in background fibersCycles/Sec
BACKGROUND_FIBERSNumber of background fibers that are ready to run and eager to get CPU cyclesFibers
BACKGROUND_TIMEPercentage of the CPU time utilized for background operations%
BucketInvocationState_CAPACITYNumber of data structures allocated to the BucketInvocationState poolStructs
BucketInvocationState_STRUCT_SIZENumber of bytes in each struct of the BucketInvocationState poolBytes
BucketInvocationState_USEDNumber of structs in the BucketInvocationState pool which are currently being usedStructs
Bucket_CAPACITYNumber of data structures allocated to the Bucket poolStructs
Bucket_STRUCT_SIZENumber of bytes in each struct of the Bucket poolBytes
Bucket_USEDNumber of structs in the Bucket pool which are currently being usedStructs
CLASS_BLOB!(RAID)_CAPACITYNumber of data structures allocated to the CLASS_BLOB!(RAID) poolStructs
CLASS_BLOB!(RAID)_STRUCT_SIZENumber of bytes in each struct of the CLASS_BLOB!(RAID) poolBytes
CLASS_BLOB!(RAID)_USEDNumber of structs in the CLASS_BLOB!(RAID) pool which are currently being usedStructs
CYCLES_PER_SECONDNumber of cycles the cpu runs per secondCycles/Sec
ChainedSpan_CAPACITYNumber of data structures allocated to the ChainedSpan poolStructs
ChainedSpan_STRUCT_SIZENumber of bytes in each struct of the ChainedSpan poolBytes
ChainedSpan_USEDNumber of structs in the ChainedSpan pool which are currently being usedStructs
Charter_CAPACITYNumber of data structures allocated to the Charter poolStructs
Charter_STRUCT_SIZENumber of bytes in each struct of the Charter poolBytes
Charter_USEDNumber of structs in the Charter pool which are currently being usedStructs
CrossDestageDesc_CAPACITYNumber of data structures allocated to the CrossDestageDesc poolStructs
CrossDestageDesc_STRUCT_SIZENumber of bytes in each struct of the CrossDestageDesc poolBytes
CrossDestageDesc_USEDNumber of structs in the CrossDestageDesc pool which are currently being usedStructs
DEFUNCT_FIBERSNumber of defunct buffers, which are really just memory structures allocated for future fiber needs.Fibers
DeferredTask2_CAPACITYNumber of data structures allocated to the DeferredTask2 poolStructs
DeferredTask2_STRUCT_SIZENumber of bytes in each struct of the DeferredTask2 poolBytes
DeferredTask2_USEDNumber of structs in the DeferredTask2 pool which are currently being usedStructs
EXCEPTIONSNumber of execptions caught by the reactorExceptions/Sec
GenericBaseBlock_CAPACITYNumber of data structures allocated to the GenericBaseBlock poolStructs
GenericBaseBlock_STRUCT_SIZENumber of bytes in each struct of the GenericBaseBlock poolBytes
GenericBaseBlock_USEDNumber of structs in the GenericBaseBlock pool which are currently being usedStructs
HOGGED_TIMEHistogram of time used by hogger fibers (only in debug builds)
IDLE_CALLBACK_INVOCATIONSNumber of background work invocationsInvocations/Sec
IDLE_CYCLESNumber of cycles spent in idleCycles/Sec
IDLE_TIMEPercentage of the CPU time not utilized for handling I/Os%
NODE_HANG
OUTRAGEOUS_HOGGERSNumber of hoggers taking really excessive amount of time to runInvocations
ObsGateway_CAPACITYNumber of data structures allocated to the ObsGateway poolStructs
ObsGateway_STRUCT_SIZENumber of bytes in each struct of the ObsGateway poolBytes
ObsGateway_USEDNumber of structs in the ObsGateway pool which are currently being usedStructs
PENDING_FIBERSNumber of fibers pending for external events, such as a network packet, or SSD response. Upon such external event they will change state to scheduled fibersFibers
QueuedBlock_CAPACITYNumber of data structures allocated to the QueuedBlock poolStructs
QueuedBlock_STRUCT_SIZEBytes
QueuedBlock_USEDNumber of bytes in each struct of the QueuedBlock poolStructs
ReadBlocksImpl!(RAID)_CAPACITYNumber of data structures allocated to the ReadBlocksImpl!(RAID) poolStructs
ReadBlocksImpl!(RAID)_STRUCT_SIZENumber of bytes in each struct of the ReadBlocksImpl!(RAID) poolBytes
ReadBlocksImpl!(RAID)_USEDNumber of structs in the ReadBlocksImpl!(RAID) pool which are currently being usedStructs
SCHEDULED_FIBERSNumber of current fibers that are ready to run and eager to get CPU cyclesFibers
SLEEPY_FIBERSNumber of SLEEPY fibersSleepy fiber detections
SLEEPY_RPC_SERVER_FIBERSNumber of SLEEPY RPC server fibersStructs
SSD_CAPACITYNumber of data structures allocated to the SSD poolStructs
SSD_STRUCT_SIZENumber of bytes in each struct of the SSD poolBytes
SSD_USEDNumber of structs in the SSD pool which are currently being usedStructs
STEP_CYCLESHistogram of time spent in a fiber
TIMER_CALLBACKSCurrent number of timer callbacksCallbacks
TOTAL_FIBERS_COUNTNumber of fibersFibers
TimedCallback_CAPACITYNumber of data structures allocated to the TimedCallback poolStructs
TimedCallback_STRUCT_SIZENumber of bytes in each struct of the TimedCallback poolBytes
TimedCallback_USEDNumber of structs in the TimedCallback pool which are currently being usedStructs
UploadFileInfo_CAPACITYNumber of data structures allocated to the UploadFileInfo poolStructs
UploadFileInfo_STRUCT_SIZENumber of bytes in each struct of the UploadFileInfo poolBytes
UploadFileInfo_USEDNumber of structs in the UploadFileInfo pool which are currently being usedStructs
networkBuffers_CAPACITYNumber of data structures allocated to the networkBuffers poolStructs
networkBuffers_USEDNumber of structs in the networkBuffers pool which are currently being usedStructs
rdmaNetworkBuffers_CAPACITYNumber of data structures allocated to the rdmaNetworkBuffers poolStructs
rdmaNetworkBuffers_USEDNumber of structs in the rdmaNetworkBuffers pool which are currently being usedStructs

SSD

TypeDescriptionUnits
DRIVE_ACTIVE_IOSThe number of in flight IO against the SSD at the time of samplingIOs
DRIVE_FORFEITSNumber of IOs forfeited due to lack of memory buffersOperations/Sec
DRIVE_IDLE_CYCLESNumber of cycles spent in idleCycles/Sec
DRIVE_IDLE_TIMEPercentage of the CPU time not utilized for handling I/Os%
DRIVE_IO_OVERLAPPEDNumber of overlapping IOsOperations
DRIVE_IO_TOO_LONGNumber of IOs that took longer than expectedOperations/Sec
DRIVE_LATENCYMeasure the latencies up to 5ms (higher latencies will be grouped together)
DRIVE_LOADDrive Load at sampling timeLoad
DRIVE_MEDIA_BLOCKS_READBlocks read from the SSD mediaBlocks/Sec
DRIVE_MEDIA_BLOCKS_WRITEBlocks written to the SSD mediaBlocks/Sec
DRIVE_MEDIA_ERRORSSSD Media ErrorsIO/Sec
DRIVE_NON_MEDIA_ERRORSSSD Non-Media ErrorsIO/Sec
DRIVE_PENDING_IOSThe number of IOs waiting to start executing at the time of samplingIO
DRIVE_PUMPED_IOSNumber of requests returned in a pumpOperations/Sec
DRIVE_PUMPS_DELAYEDNumber of Drive pumps that got delayedOperations/Sec
DRIVE_PUMPS_SEVERELY_DELAYEDNumber of Drive pumps that got severely delayedMicroseconds
DRIVE_PUMP_LATENCYLatency between SSD pumpsMicroseconds
DRIVE_READ_LATENCYDrive Read Execution LatencyMicroseconds
DRIVE_READ_OPSDrive Read OperationsIO/Sec
DRIVE_REMAINING_IOSNumber of requests still in the drive after a pump
DRIVE_REQUEST_BLOCKSMeasure drive request size distribution
DRIVE_SSD_PUMPSNumber of drive pumps that resulted in data flowin from/to drivePump/Sec
DRIVE_UTILIZATIONPercentage of time the drive had an active IO submitted to it%
DRIVE_WRITE_LATENCYDrive Write Execution LatencyMicroseconds
DRIVE_WRITE_OPSDrive Write OperationsIO/Sec
SSDS_IOSIOs performed on the SSD serviceIO/Sec
SSDS_IO_ERRORSIO errors on the SSD serviceBlocks/Sec
SSD_BLOCKS_READ Number of blocks read from the SSD serviceBlocks/Sec
SSD_BLOCKS_WRITTENNumber of blocks written to the SSD serviceBlocks/Sec
SSD_CHUNK_ALLOCSRate of chunk allocationsChunks/Sec
SSD_CHUNK_FREESRate of chunk freesChunks/Sec
SSD_E2E_BAD_CSUMErrors in reading blocks from the SSD serviceIO/Sec
SSD_READ_ERRORSErrors in reading blocks from the SSD serviceBlocks/Sec
SSD_READ_LATENCYAvg. latency of read requests from the SSD serviceMicroseconds
SSD_READ_REQS_LARGE_NORMALNumber of large normal read requests from the SSD serviceIO/Sec
SSD_READ_REQSNumber of read requests from the SSD serviceIO/Sec
SSD_WRITES_REQS_LARGE_NORMALNumber of large normal priority write requests to the SSD serviceIO/Sec
SSD_WRITESNumber of write requests to the SSD serviceIO/Sec
SSD_WRITE_ERRORSErrors in writing blocks to the SSD serviceBlocks/Sec
SSD_WRITE_LATENCYLatency of writes to the SSD serviceMicroseconds

Scrubber

TypeDescriptionUnits
ACTUALLY_FALSE_FREENumber of blocks that were detected as false-used and freedBlocks/Sec
CLEANED_CHUNKSNumber of chunks that were cleaned by the scrubberChunks/Sec
DEGRADED_READSNumber of degraded reads for scrubbingRequests/Sec
FALSE_FREE_CHECKED_BLOCKSNumber of blocks that were scrubbed-false-usedBlocks/Sec
FALSE_FREE_CHECK_LATENCYAverage latency of checking false free per blockMicros
FALSE_USED_CHECKED_BLOCKSNumber of blocks that were scrubbed-false-usedBlocks/Sec
FALSE_USED_CHECK_LATENCYAverage latency of checking false used per blockMicros
FALSE_USED_EXTRA_NOTIFIEDNumber of blocks that were notified as used by the mark-extra-used mechanismBlocks/Sec
INTERRUPTSNumber of scrubs that were interruptedOccurences/Sec
NETWORK_BUDGET_WAIT_LATENCYAverage latency of waiting for our network budgetMicros
NOT_ACTUALLY_FALSE_FREENumber of blocks that were detected as usedBlocks/Sec
NOT_REALLY_DIRTY_BLOCKSNumber of marked dirty blocks that ScrubMissingWrites found were actually cleanBlocks/Sec
NUM_COPY_DISCARDED_BLOCKSNumber of copied blocks that were discardedBlocks/Sec
NUM_COPY_DISCARDSNumber of times we discarded scrubber copy workOccurences/Sec
NUM_INVENTED_STRIPES_DISCARDSNumber of times we discarded all scrubber work due to invented stripesOccurences/Sec
NUM_INVENTED_STRIPES_DISCARD_BLOCKSNumber of blocks that were discarded due to invented stripesBlocks/Sec
NUM_SCRUBBER_DISCARD_INTERMEDIATESNumber of times we discarded all intermediate scrubber workOccurences/Sec
NUM_SMW_DISCARDED_BLOCKSNumber of SMW’d blocks that were discardedBlocks/Sec
NUM_SMW_DISCARDSNumber of times we discarded scrubber SMW workOccurences/Sec
PLACEMENT_SELECTION_LATENCYAverage latency of scrubbed placement selectionMicros
READS_CALLEDNumber of blocks that were readBlocks/Sec
READ_BATCH_SOURCE_BLOCKSNumber of source blocks to read in batch
READ_BLOCKS_LATENCYAverage latency of read blocksMicros
RELOCATED_BLOCKSNumber of blocks that were relocated for evictionBlocks/Sec
RELOCATE_BLOCKS_LATENCYAverage latency of relocating blocksMicros
RETRUSTED_UNPROTECTED_DIRTY_BLOCKSNumber of dirty blocks that ScrubMissingWrites retrusted because they were unprotectedBlocks/Sec
REWRITTEN_DIRTY_BLOCKSNumber of dirty blocks that ScrubMissingWrites rewrote to clean themBlocks/Sec
SCRUB_BATCHES_LATENCYAverage latency of scrub batchesMillis
SCRUB_FALSE_FREE_FAILED_READSNumber of blocks that we failed to read for scrub-false-freeBlocks/Sec
SCRUB_FALSE_FREE_FAILEDNumber of placements we failed to fully scrub-false-freeOccurences/Sec
SCRUB_FALSE_FREE_PLACEMENTSNumber of placements we finished scrub-false-usedOccurences/Sec
SCRUB_FALSE_FREE_WAS_UNPROTECTEDNumber of blocks that were false marked freed and unprotectedBlocks/Sec
SCRUB_FALSE_USED_FAILED_READSNumber of blocks that we failed to read for scrub-false-usedBlocks/Sec
SCRUB_FALSE_USED_FAILEDNumber of placements we failed to fully scrub-false-usedOccurences/Sec
SCRUB_FALSE_USED_PLACEMENTSNumber of placements we finished scrub-false-usedOccurences/Sec
SCRUB_FALSE_USED_WAS_UNPROTECTEDNumber of blocks that were false marked used and unprotectedBlocks/Sec
SCRUB_PREPARATION_FAILEDNumber of times we failed to prepare() a task and aborted scrub of placementOccurences/Sec
SFU_CHECKSNumber of blocks that were scrubbed-false-usedBlocks/Sec
SFU_CHECK_FREENumber of blocks that were detected as false-used and freedBlocks/Sec
SFU_CHECK_SECONDARYNumber of blocks that were detected as secondaryBlocks/Sec
SFU_CHECK_USEDNumber of blocks that were detected as usedBlocks/Sec
SOURCE_READSNumber of source/committed superset blocks directly read by the scrubberBlocks/Sec

TARGET_COPIED_CHUNKS

Number of chunks that were copied to target by the scrubberChunks/Sec
UPDATE_PLACEMENT_INFO_LATENCYAverage latency of updating the placement info quorumMicros
UPDATE_PLACEMENT_INFONumber of times we ran updatePlacementInfoOccurences/Sec
WONT_CLEAN_COPYINGNumber of actually dirty blocks that ScrubMissingWrites refused to clean because they will be moved to target anywayBlocks/Sec
WRITES_CALLEDNumber of blocks that were writtenBlocks/Sec
WRITE_BATCH_SOURCE_BLOCKS Number of source blocks to write in batch
WRITE_BATCH_TARGET_BLOCKSNumber of target blocks to write in batch
WRITE_BLOCKS_LATENCYAverage latency of writing blocksMicros

Squelch

TypeDescriptionUnits
BLOCKS_PER_DESQUELCHBlocks
EXTENT_DESQUELCHES_NUMTimes
EXTENT_SQUELCH_BLOCKS_READBlocks
HASH_DESQUELCHES_NUMTimes
HASH_SQUELCH_BLOCKS_READBlocks
INODE_DESQUELCHES_NUMTimes
INODE_SQUELCH_BLOCKS_READBlocks
JOURNAL_DESQUELCHES_NUMTimes
JOURNAL_SQUELCH_BLOCKS_READBlocks
MAX_BLOCKS_WITH_TEMPORAL_SQUELCH_ITEMS_IN_BUCKETNumber of block with temporal squelch items in bucketBlocks
MAX_TEMPORAL_SQUELCH_ITEMS_IN_BUCKETSquelch items
REGISTRY_L1_DESQUELCHES_NUMTimes
REGISTRY_L1_SQUELCH_BLOCKS_READBlocks
REGISTRY_L2_DESQUELCHES_NUMTimes
REGISTRY_L2_SQUELCH_BLOCKS_READBlocks
SPATIAL_SQUELCH_DESQUELCHES_NUMTimes
SPATIAL_SQUELCH_SQUELCH_BLOCKS_READBlocks
SUPERBLOCK_DESQUELCHES_NUMTimes
SUPERBLOCK_SQUELCH_BLOCKS_READBlocks
TEMPORAL_SQUELCH_DESQUELCHES_NUMTimes
TEMPORAL_SQUELCH_SQUELCH_BLOCKS_READBlocks

Statistics

TypeDescriptionUnits
GATHER_FROM_NODE_LATENCY_NETTime spent on responding to a stats gathering request(not including metadata)Seconds/Sec
GATHER_FROM_NODE_LATENCYTime spent responding to a stats gathering request(not including metadata)Seconds/Sec
GATHER_FROM_NODE_SLEEPTime spent in-between responding to a stats gathering request(not including metadata)Seconds/Sec
TIMES_QUERIED_STATSTimes
TIMES_QUERIEDNumber of times the node was queried for stats (not including metadata)Times

 

  • Was this article helpful?