Getting started
Introducing Hitachi Content Platform for cloud scale
Hitachi Content Platform for cloud scale (HCP for cloud scale) is a software-defined object storage solution that is based on a massively parallel microservice architecture, and is compatible with the Amazon S3 application programming interface (API).
HCP for cloud scale is especially well suited to service applications requiring high bandwidth and compatibility with Amazon S3 APIs.
HCP for cloud scale has the ability to federate S3-compatible storage from virtually any private or public source, and present the combined capacity in a single, centrally managed, global namespace.
You can install HCP for cloud scale on any server, in the cloud or on premise, that supports the minimum requirements.
HCP for cloud scale lets you manage and scale storage components. You can add storage components, monitor their states, and take them online or offline for purposes of maintenance and repair. The HCP for cloud scale system provides functions to send notification of alerts, track and monitor throughput and performance, and trace actions through the system.
Storage components, buckets, and objects
A storage component is an Amazon S3-compatible storage system, running independently, that is manageable by HCP for cloud scale as a back end to store object data. To an S3 client using HCP for cloud scale, the existence, type, and state of storage components are transparent.
HCP for cloud scale supports the following storage systems:
- Amazon S3
- Hitachi Content Platform (HCP)
- HCP S Series Nodes
- Any Amazon S3-compatible storage service
An HCP for cloud scale bucket is modeled on a storage service bucket. A bucket is a logical collection of secure data objects that is created and managed by a client application. HCP for cloud scale uses buckets to manage storage components, and an HCP for cloud scale site can be thought of as a logical collection of secure buckets. Buckets have associated metadata such as ownership and lifecycle status. HCP for cloud scale buckets are owned by an HCP for cloud scale user, and access is controlled on a per-bucket basis by Amazon ACL support using S3 APIs. Buckets are contained in a specific region; HCP for cloud scale supports one region.
- HCP for cloud scale buckets are not stored in storage components, so HCP for cloud scale clients can create buckets even before adding storage components.
- Storage component buckets are created by storage component administrators, and are not visible to HCP for cloud scale clients.
- If you want to empty a bucket and reuse it, don't just delete the bucket and create a new one with the same name. After a bucket is deleted, the name becomes available for anyone to use, and another account might take it first. Instead, empty the bucket and keep it.
An object consists of data and associated metadata. The metadata is a set of name-value pairs that describe the object. Every object is contained in a bucket. An object is handled as a single unit by all HCP for cloud scale transactions, services, and internal processes.
For information about Amazon S3, see Introduction to Amazon S3.
Data access
HCP for cloud scale supports the Amazon Simple Storage Service (S3) application programming interface (API), which allows client applications to store and retrieve unlimited amounts of data from configured storage services.
Data access control
HCP for cloud scale uses ownership and access control lists (ACLs) as data access control mechanisms in S3 APIs.
Ownership is implemented as follows:
- An HCP for cloud scale bucket is owned by the user who creates the bucket, and the owner cannot be changed
- A user has full control of the buckets that user owns
- A user has full control of the objects that user creates
- A user can only list the buckets that user owns
ACLs allow the assignment of privileges (read, write, or full control) to other user accounts besides the owner to access bucket and objects.
Data security
HCP for cloud scale supports encryption of data sent between systems ("in flight") and data stored persistently within the system ("at rest").
HCP for cloud scale uses Secure Sockets Layer (SSL) to provide security for both incoming and outgoing communications. To enable SSL security, two certificates are required:
- System certificate: the certificate HCP for cloud scale uses for its GUI and APIs (incoming communications)
- Client certificate: the certificates of IDPs, storage components, and SMTP servers (outgoing communications)
- Upload a PKCS12 certificate chain and password and apply it as the active system certificate.
- Download a certificate signing request (CSR), then use it to obtain, upload, and apply a certificate signed by a certificate authority (CA).
- Generate a new self-signed certificate and apply it as the active system certificate.
You can manage certificates, as well as view the installed certificates and their details, using the System Management application.
HCP for cloud scale supports data-in-flight encryption (HTTPS) for all external communications. Data-in-flight encryption is always enabled for these data paths:
- S3 API (HTTP is also enabled on a different port)
- Management API
- System Management App user interface (GUI)
- Object Storage Management App GUI
- Between HCP for cloud scale and an identity provider (IDP) server
- Between HCP for cloud scale and each application using TLS or SSL
- Between HCP for cloud scale and each managed storage component
- Between HCP for cloud scale and each SMTP server using SSL or STARTTLS
HCP for cloud scale stores three kinds of data persistently:
- HCP for cloud scale services data
- HCP for cloud scale metadata and user-defined metadata
- User data (object data)
Scalability of instances, service instances, and storage components
You can increase or decrease the capacity, performance, and availability of an HCP for cloud scale site by adding or removing the following:
- Instances: Additional physical computer nodes or virtual machines
- Service instances: Copies of services running on additional instances
- Storage components: S3-compatible systems used to store object data
In a multi-instance site, you might add additional instances if you want to improve system performance or if you are running out of disk space on one or more instances. You might remove instances if you are retiring hardware, if an instance is down and cannot be recovered, or if you decide to run a site with fewer instances.
When you add an instance, you can also scale floating services (such as the Metadata Gateway) to the new instance. When you scale a floating service, HCP for cloud scale automatically rebalances itself.
In a multi-instance site, you can manually change where a service instance runs:
- You can configure it to run on additional instances. For example, you can increase the number of instances of the S3-Gateway service to improve throughput of S3 API transactions without having to add a compute instance.
- You can configure it run on fewer instances. For example, you can free up resources on an instance to run other services.
- You can configure it to run on different instances. For example, you can move the service instances off a hardware instance to retire it.
- For a floating service, instead of specifying a specific instance on which it runs, you can specify a pool of eligible instances, any of which can run the service.
Some services have a fixed number of instances and therefore cannot be scaled. These include:
- Metadata-Coordination
- Metadata-Cache
You might add additional storage components to a site under these circumstances:
- The existing storage components are running out of available capacity
- The existing storage components do not provide the performance you require
- The existing storage components do not provide the functionality you require
Supported limits
HCP for cloud scale limits the number of instances (nodes) in a system to 160.
HCP for cloud scale does not limit the number of the following entities.
| Entity | Minimum | Maximum | Notes |
| Buckets | None | Unlimited | |
| Users (external) | None | Unlimited | The local user can do all operations including MAPI calls and S3 API calls. However, it is recommended that HCP for cloud scale be configured with an identity provider (IdP) with users to enforce role-based access control. |
| Groups (external) | Unlimited | ||
| Roles | Unlimited | ||
| Objects | None | Unlimited | |
| Storage components | 1 | Unlimited |
High availability
HCP for cloud scale provides high availability for multi-instance sites. High availability requires at least four service instances: three master instances, which run essential services, and at least one worker instance. The best practice is to run the three master instances on separate physical hardware (or, if running on virtual machines, on at least three separate physical hosts), and to run HCP for cloud scale services on more than one instance.
Site availability
An HCP for cloud scale site has three master instances, and can tolerate the failure of one master instance without interruption of service. Even if two or all three master instances fail, HCP for cloud scale services may be functional (but you cannot move or scale service instances until master instances are restored).
Service availability
HCP for cloud scale services provide high availability as follows:
- The Metadata Gateway service always has at least three service instances. When the system starts up, the nodes "elect a leader" using the raft consensus algorithm. The leader processes all GET and PUT requests. If the followers cannot identify the leader, they elect a new leader. The Metadata Gateway service can tolerate service instance failure, and service remains available without loss of data, so long as at least two service instances are healthy.
- The Metadata Coordination service always has one service instance. If that instance fails, HCP for cloud scale automatically starts another instance. Until startup is complete, the Metadata Gateway service cannot scale.
- The Metadata Cache service always has one service instance. If that instance fails, HCP for cloud scale automatically starts another instance. Until startup is complete, performance decreases.
Metadata availability
Metadata is available as long as two services are available:
- S3 Gateway
- Metadata Gateway
Object data availability
Object data is available as long as these items are available:
- S3 Gateway service (at least one instance)
- The storage component containing the requested data
- At least two functioning Metadata Gateway service instances (of the required three)
Network availability
You can install each HCP for cloud scale instance with an internal and an external network interface. If you want to avoid networking single points of failure, you can:
- Configure two external network interfaces in each HCP for cloud scale instance
- Use two switches, and connect each network interface to one of them
- Bind the two network interfaces (that is, as Active-Passive) into one virtual network interface
- Install HCP for cloud scale using the virtual network interface
Failure recovery
HCP for cloud scale actively monitors the health and performance of the system and its resources, provides real-time visual health representations, issues alert messages when needed, and can automatically take action to recover from the following types of failures:
- Instances (nodes)
- Product services (software processes)
- System services (software processes)
- Storage components
Instance failure recovery
If an instance (a compute node) fails, HCP for cloud scale automatically adds new service instances to other available instances (compute nodes) to maintain the recommended minimum number of service instances. Data on the failed instance is not lost and remains consistent. However, while the instance is down, data redundancy may degrade.
HCP for cloud scale only adds new service instances automatically for floating services. Depending on the remaining number of instances and service instances running, you may need to add new service instances or deploy a new instance.
Service failure recovery
HCP for cloud scale monitors service instances and automatically restarts them if they are not healthy.
For floating services, you can configure a pool of eligible HCP for cloud scale instances and the number of service instances that should be running at any time. You can also set the minimum and maximum number of instances running each service. If a service instance failure causes the number of service instances to go below the minimum, HCP for cloud scale brings up another one on one of the HCP for cloud scale instances in the pool that doesn't already have that service instance running.
Persistent services run on the specific instances that you specify. If one of those service instances fails, HCP for cloud scale restarts the service instance in the same HCP for cloud scale instance. HCP for cloud scale does not automatically bring up a new service instance on a different HCP for cloud scale instance.
Storage component failure recovery
HCP for cloud scale performs regular health checks to detect storage component failures.
If HCP for cloud scale detects a failure, it sets the storage component state to INACCESSIBLE, so that HCP for cloud scale will not try to write new objects to it. HCP for cloud scale can send an alert when this event happens. While a storage component is down, the data in it is not accessible.
HCP for cloud scale keeps checking a failed storage component and, when it detects that the storage component is healthy again, automatically sets its state to ACTIVE. HCP for cloud scale can send an alert when this event happens as well. Once the storage component is repaired and brought back online, the data its contains is again accessible, and HCP for cloud scale can write new objects to it.
Support for Amazon S3 APIs
HCP for cloud scale is compatible with the Amazon Simple Storage Service (Amazon S3) REST API, which allows clients to store objects in containers called buckets. A bucket is a collection of objects and has its own individual settings, such as ownership and lifecycle. Using HCP for cloud scale, you can perform common read and write operations on objects and buckets, and manage ACL settings through the client access data service.
For information about using Amazon S3, see the Amazon S3 API documentation.
The following tables list the supported Amazon S3 API features and describes any implementation differences between Amazon and HCP for cloud scale S3 APIs.
| Feature | Implementation differences |
| Authentication with AWS Signature Version 4 | Fully implemented. |
| Addressing virtual host (such as http://bucket.server/object) | Fully implemented. |
| Addressing Path style (such as http://server/bucket/object ) | Fully implemented. |
| Signed/Unsigned payload | Fully implemented. |
| Chunked request | Fully implemented. |
| Pre-signed URL | Fully implemented. |
| Feature | Implementation differences |
| GET service (list buckets) | Fully implemented. |
| Feature | Implementation differences |
| GET Bucket (list objects) V1 | Fully implemented. |
| GET Bucket (list objects) V2 | Fully implemented. |
| PUT Bucket |
To support legacy S3 buckets, HCP for cloud scale supports bucket names of less than three characters. When anonymous requests to create or delete a bucket use an invalid bucket name, Amazon S3 performs an access check first and returns 403. HCP for cloud scale returns 400 if the bucket name validation fails. |
| DELETE Bucket | |
| HEAD Bucket | |
| PUT Bucket ACL | In AWS each grantee is specified as a type-value pair, where the type is one of the following:
emailAddress. HCP for cloud scale fully supports id. HCP for cloud scale supports uri for the predefined groups Authenticated Users and All Users. HCP for cloud scale does not support the |
| GET Bucket ACL | |
| List Multipart Uploads | Fully implemented. |
| GET Bucket Lifecycle (except transition action) | HCP for cloud scale does not support Object Transition actions. If these actions are included it will throw a Malformed XML exception. |
| PUT Bucket Lifecycle (except transition action) | |
| DELETE Bucket Lifecycle (except transition action) | |
| GET Bucket Versioning | Version Listing Requests do not strictly comply to documented behavior for NextKeyMarker/NextVersionIdMarker. S3 documentation currently states that these values "specifies the first key not returned that satisfies the search criteria." However, HCP for cloud scale specifies the last key returned in the current response. S3 V1 object listings do not call out as specific a requirement and V2 object listings use a continuation token (opaque to the caller); internally, HCP for cloud scale shares the same listing logic across all three listing types. |
| GET Bucket Object Versions | Fully implemented. |
| GET Bucket Location | You must be the bucket owner. The only location supported is us-west-2. |
| Feature | Implementation differences |
| GET Object | If a lifecycle policy is configured for a bucket, HCP for cloud scale displays the expiration date of an object (in the x-amz-expiration header) fetched using the ?versionId subresource. Amazon only displays this when performing unversioned GET requests. |
| HEAD Object | If a lifecycle policy is configured for a bucket, HCP for cloud scale displays the expiration date of an object (in the x-amz-expiration header) fetched using the ?versionId subresource. Amazon only displays this when performing unversioned HEAD requests. |
| PUT Object | Content-Type Validations: Amazon is extremely liberal in what is accepted for the Content-Type of an object. HCP for cloud scale adds additional checks for what is allowed. |
| PUT Object (Copy) | Conditional headers are not supported. Server-side encryption is not supported. Multiple AWS regions are not supported; as a result, cross-region limitations are not supported. |
| PUT Object (Part Copy) | Conditional headers are not supported. Server-side encryption is not supported. |
| Object and Version Encoding | Amazon AWS S3 Object and Version listing documentation mentions the ability to pass an encoding parameter (url). This is so the object name XML in the response to the client can be escaped to avoid names containing invalid XML characters. This encoding is only documented as applied to object names and not Owner/DisplayNames. Additionally, there is no mention of escaping for Bucket Listing requests. The Owner/DisplayName is a concern as there is a possibility that user display names may not be able to contain characters that could cause XML parsing issues. Amazon may be able to restrict this, though it does not currently return a display name for all regions. HCP for cloud scale uses Foundry IDPs, thus controlling restriction is not in the realm of HCP for cloud scale. Bucket name restrictions should prevent problematic bucket names from being created. For security, HCP for cloud scale passes the user display name through a URI encoder before returning it in XML responses. |
| Object tagging | Amazon wraps eTags in double-quotes. For XML listings (v1 object, v2 object, version) it escapes these, for example:
It's not necessary for HCP for cloud scale to perform this because double-quotes do not need to be escaped within content, only attributes. |
| Expiration Date URL Encoding (x-amz-expiration header) The RuleID portion of the x-amz-expiration header is URL-encoded by HCP for cloud scale using the same encoding strategy that Amazon suggests for V4 authentication. This may result in encoded strings that do not exactly match how Amazon encodes RuleIDs in general. However, decoding them should always return the original string. | |
| GET Object ACL | In AWS each grantee is specified as a type-value pair, where the type is one of the following:
emailAddress. HCP for cloud scale fully supports id. HCP for cloud scale supports uri for the predefined groups Authenticated Users and All Users. HCP for cloud scale does not support the |
| PUT Object ACL | |
| DELETE Multiple Objects | Fully implemented. |
| POST Object | Fully implemented. |
| Initiate/Complete/Abort Multipart Upload | Fully implemented. |
| Upload Part | Fully implemented. |
| List Multipart Uploads | Fully implemented. |
The following lists are the unsupported Amazon S3 API features.
Authentication API
- Authentication v2 (deprecated by AWS)
Bucket APIs
- GET/PUT/DELETE Bucket Website
- GET/PUT/DELETE Bucket Policy
- GET/PUT/DELETE Bucket Tagging
- GET/PUT/DELETE Bucket CORS (cross-origin resource sharing)
- PUT Bucket Versioning (versioning is always On)
- GET/PUT Bucket Logging
- GET Bucket Notification
- GET/PUT Bucket requestPayment
- GET/PUT/DELETE Bucket Inventory
- List Bucket Inventory Configurations
- GET/PUT/DELETE Bucket Replication
- GET/DELETE Bucket Metrics
- List Bucket Metrics Configurations
- GET/PUT/DELETE Bucket Analytics
- List Bucket Analytics Configurations
- PUT/GET Bucket Accelerate
- Server-Side Encryption with Customer-Provided Encryption Keys (SSE-C)
- Server-Side Encryption with Storage-Managed Encryption Keys (SSE-S3)
Object APIs
- Options Object
- GET/POST Object Torrent
- SELECT Object Content (SQL)
Logging in
User accounts reside in an external identity provider (IdP). To log in you need this information:
- The IP address of the HCP for cloud scale instance that you're using
- Your user name as assigned by your system administrator
- Your password as assigned by your system administrator
- The security realm where your user account is defined
Procedure
Open a web browser and go to https://instance_ip_address:8000
instance_ip_address is the IP address of the HCP for cloud scale instance you're usingEnter your username and password.
In the Security Realm field, select the location where your user account is defined.
To log in using the local administrator account, without using an external IdP, select Local. If no IdP is configured yet, Local is the only available option,Click LOGIN.
Results
security/clearCache to allow immediate login.HCP for cloud scale applications
After you log in, HCP for cloud scale presents you with applications you can launch:
- Object Storage Management: Manage and monitor storage components, data objects, alerts, and regions
- S3 User Credentials: Generate S3 access and secret keys
- System Management (sometimes referred to in the application as the Admin App): Manage and monitor cluster instances, software services, system security, user accounts, and other cluster configuration parameters
You can return to the Applications page to switch back and forth between these applications as needed.
Switching between applications
HCP for cloud scale uses OAuth2 as a service provider to authenticate single sign-on (SSO) access. You only need one set of login credentials for all HCP for cloud scale applications, and you can switch between applications without logging in again.
To switch between applications:
Procedure
Click the Open menu (
), in the right corner of the top navigation bar, and select the application you want to use.NoteThe System Management application is also identified in the user interface as Admin-App.The application opens.
Serial number
You can use the Object Storage Management application or APIs to enter and display your HCP for cloud scale serial number.
A serial number is required to activate the HCP for cloud scale software. You must enter the serial number before proceeding further.
Entering your serial number
The Object Storage Management application displays the product serial number. An administrative account with appropriate permissions can enter or edit this number.
Object Storage Management application instructions
To enter your product serial number:
Procedure
Select Dashboard and click on the Edit icon next to the Serial Number field.
The Add Serial Number window opens.Enter your serial number and click Add.
Related API method
POST /serial_number/set
For information about specific API methods, see the MAPI Reference or, in the Object Storage Management application, click the profile icon and select REST API.
Displaying your serial number
You can use the Object Storage Management application or APIs to displays the product serial number.
Object Storage Management application instructions
The product serial number is displayed in the upper right corner of the Dashboard page.
Related API method
POST /serial_number/get
For information about specific API methods, see the MAPI Reference or, in the Object Storage Management application, click the profile icon and select REST API.
About HCP for cloud scale APIs
The Hitachi Content Platform for cloud scale (HCP for cloud scale) system includes a set of RESTful application programming interfaces (APIs) that you can use for writing applications that exercise its functions and manage the system.
Anything you can do in the Object Storage Management, S3 User Credentials, or System Management application GUIs you can also do using APIs.
The Object Storage Management application includes a RESTful API to administrative functions such as managing storage components, configuring Amazon S3 settings, and obtaining or revoking S3 user credentials. For more information on the Object Storage Management API, see the MAPI Reference.
The System Management application includes a RESTful API to system management functions such as system monitoring, service monitoring, user registration, and configuration. For more information on the System Management API, see the online help in the System Management application.
Unless otherwise noted, HCP for cloud scale is fully compatible with Amazon S3 APIs.
Object Storage Management APIs
The Object Storage Management application provides a RESTful HTTPS interface for the following functions:
- Managing storage components and Amazon Simple Storage Service (Amazon S3) settings
- Managing administrative resources such as serial numbers and system events
- Managing user resources such as S3 user credentials and OAuth tokens
You can execute all functions supported in the Object Storage Management application using RESTful APIs.
All URLs for the APIs have the following base, or root, uniform resource identifier (URI):
https://hcpcs_ip_address:9099/mapi/v1
System Management APIs
The System Management application provides a RESTful HTTPS interface for managing the following:
- Alerts
- Business objects
- Certificates
- Events
- Instances
- Jobs
- Licenses
- Notifications
- Packages
- Plugins
- Security
- Services
- Setup
- Tasks
- Updates
You can execute all functions supported in the System Management application using RESTful APIs.
For information on the System Management APIs, see the System Management online help.
Listing service ports
You can list service port information for ports available for customer use.
POST /public/discovery/get_service_port
For information about specific API methods, see the MAPI Reference or, in the Object Storage Management application, click the profile icon and select REST API.