Hitachi Content Platform for Cloud Scale v1.2.0 Release Notes
About this document
This document provides late-breaking information about HCP for cloud scale v1.2.0. It includes information that was not available at the time the technical documentation for this product was published, as well as a list of new features; resolved issues; and known issues and, where applicable, their workarounds.
Intended audience
This document is intended for Hitachi Vantara partners who license and use HCP for cloud scale.
Getting help
Hitachi Vantara Support Connect is the destination for technical support of products and solutions sold by Hitachi Vantara. To contact technical support, log on to Hitachi Vantara Support Connect for contact information: https://support.hitachivantara.com/en_us/contact-us.html.
Hitachi Vantara Community is a global online community for Hitachi Vantara customers, partners, independent software vendors, employees, and prospects. It is the destination to get answers, discover insights, and make connections. Join the conversation today! Go to community.hitachivantara.com, register, and complete your profile.
About this release
This is v1.2.0.1 of Hitachi Content Platform for cloud scale (HCP for cloud scale).
Major features
HCP for cloud scale is a software-defined object storage solution that is based on a massively parallel microservice architecture, and is compatible with the Amazon Simple Storage Service (Amazon S3) application programming interface (API). HCP for cloud scale is especially well suited to service applications requiring high bandwidth and compatibility with Amazon S3 APIs.
Major features in v1.2.0
HCP for cloud scale v1.2.0 includes the following major features.
Bucket synchronization supports copying of objects between HCP for cloud scale buckets and buckets in external systems. Buckets can be synchronized in two directions:
- Synchronization to a bucket (sync-to) allows automatic, asynchronous copying of objects in buckets in an HCP for cloud scale system to external storage systems.
- Synchronization from a bucket (sync-from) allows automatic, asynchronous copying of objects in buckets in external storage systems to an HCP for cloud scale bucket.
Bucket synchronization is configured using AWS S3 cross-region replication, so you can use the S3 SDK and S3 command-line interface (CLI) provided by AWS.
An external storage system can be another HCP for cloud scale system, AWS S3, or any S3-compatible system.
Features in v1.1.1
HCP for cloud scale v1.1.1 addresses the following:
- Improvements to stability, performance, and scalability
- Increased capacity to ingest objects
- Improvements for more accurate display of metrics for object counts
- Improvements to the user interface
- Other bug fixes
Major features in v1.1.0
HCP for cloud scale v1.1.0 includes the following new major features.
The Metadata Gateway service stores and protects metadata and serves it to other services. During initial deployment, you deploy Metadata Gateway service instances to at least three nodes in the HCP for cloud scale system. In v1.1.0 you can deploy up to a maximum of every node in the system. Using the System Management application, you can add or remove Metadata Gateway service instances.
The Metadata Coordination service manages Metadata Gateway scaling. The service does the following:
- Constantly monitors the Metadata Gateway service and balances data among Metadata Gateway instances as needed
- Moves data into new Metadata Gateway instances
- Moves data out of a Metadata Gateway instance set for removal
The Metadata Policy Engine service executes asynchronous metadata updates. In v1.1.0 the service supports operations that run across multiple partitions.
HCP for cloud scale supports these Amazon S3 APIs:
- S3 PUT Copy: This API lets you create a copy of an object already stored in an HCP for cloud scale system. You must have read access to the source object and write access to the destination bucket.
- S3 PUT Part Copy: This API lets you create a copy, part by part, of an object already stored in an HCP for cloud scale system. You specify a part using a byte range of from 5 MB to 5 GB. You must have read access to the source object and write access to the destination bucket.
In v1.1.0, conditional headers, server-side encryption, and multiple AWS regions are not supported.
Major features in v1.0.0
HCP for cloud scale v1.0.0 includes the following major features.
- S3 gateway: HCP for cloud scale supports the Amazon Simple Storage Service (Amazon S3) application programming interface (API), which allows client applications to store and retrieve unlimited amounts of data from configured storage services.
- Scalability and availability: You can increase the capacity and performance of a site by adding additional instances (physical compute nodes or virtual machines). HCP for cloud scale provides high availability by supporting multiple instances of essential services on different compute nodes.
- Diverse storage backends: HCP for cloud scale supports Amazon S3, Hitachi Content Platform (HCP), HCP S Series Nodes, and Amazon S3 compatible storage systems.
- Data and metadata management: HCP for cloud scale provides a comprehensive framework to add metadata associated with data.
- Asynchronous policy engine: The engine accepts and processes metadata updates of user requests.
- Object storage management: Management and monitoring of storage components is available through the Object Storage Management application, which provides a convenient graphical user interface (GUI).
- System management: Management and monitoring of instances, services, processes, events, and HCP for cloud scale administrative users (roles and groups) is available through the System Management application, which provides a convenient GUI.
- REST API for all management operations: All functions are available through a management application programing interface (API). Object storage management functions are available through HCP for cloud scale management APIs, and administrative functions are available through system management APIs.
- Syslog and email event notification: You can configure notifications by email or syslog of degraded health of storage components, instances, or services instances.
- External management: HCP for cloud scale supports integration with the Hitachi Remote Ops system to provide powerful and secure 24/7 monitoring.
- Distributed tracing: HCP for cloud scale supports a third-party, open-source software tool, running as a service, to provide end-to-end, distributed tracing of S3 requests and operations through a browser.
- Distributed metrics: HCP for cloud scale supports a third-party, open-source software tool, running as a service, to provide application metrics through a browser.
System requirements
This section lists the hardware, networking, and operating system requirements for running an HCP for cloud scale system with one or more instances.
Hardware requirements
To install HCP for cloud scale on on-premises hardware for production use, you must provision at least four instances (nodes) with sufficient CPU, RAM, disk space, and networking capabilities. This table shows the minimum and recommended hardware requirements for each instance in an HCP for cloud scale system.
Resource | Minimum | Recommended |
RAM | 32 GB | 128 GB |
CPU | 8-core | 24-core |
Available disk space | 500 GB 10k SAS RAID | (2) 1.92 TB SSD |
Network interface controller (NIC) | (1) 10 Gb Ethernet NICs | (4) 10 Gb Ethernet NICs |
Software requirements
This table shows the minimum and recommended software requirements for each instance in an HCP for cloud scale system.
Resource | Minimum | Recommended |
IP addresses | (1) static | (2) static |
Firewall Port Access | Port 443 for S3 API Port 8000 for System Management App GUI Port 9084 for MAPI and Object Storage Management App GUI | Same |
Network Time | IP address of time service (NTP) | Same |
Operating system and Docker minimum requirements
Each server or virtual machine you provide must meet these requirements:
- 64-bit Linux distribution
- Docker version 1.13.1 or later installed
- IP and DNS addresses configured
Operating system and Docker qualified versions
This table shows the operating systems, Docker and SELinux configurations with which the HCP for cloud scale system has been qualified.
Operating system | Docker version | Docker storage configuration | SELinux setting |
Fedora 27 | Docker 18.03.0-ce | direct-lvm | Enforcing |
Red Hat Enterprise Linux 7.4 | Docker 18.03.0-ce | direct-lvm | Enforcing |
Ubuntu 16.04-LTS | Docker 17.03.0-ce | aufs | N/A |
CentOS 7.4 | Docker 18.03.1-ce | overlay2 | Enforcing |
Docker considerations
The Docker installation directory on each instance must have at least 20 GB available for storing the HCP for cloud scale Docker images.
Make sure that the Docker storage driver is configured correctly on each instance before installing HCP for cloud scale. After installing, changing the Docker storage driver requires a reinstallation of HCP for cloud scale. To view the current Docker storage driver on an instance, run docker info.
If you want to enable SELinux on the system instances, you need to use a Docker storage driver that supports it. The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.
If you are using the Docker devicemapper storage driver:
- Make sure that there's at least 40 GB of Docker metadata storage space available on each instance. HCP for cloud scale requires 20 GB to install successfully and an additional 20 GB to successfully update to a later version. To view Docker metadata storage usage on an instance, run docker info.
- On a production system, do not run
devicemapper
inloop-lvm
mode. This can cause slow performance or, on certain Linux distributions, HCP for cloud scale may not have enough space to run.
SELinux considerations
You should decide whether you want to run SELinux on system instances and enable or disable it before installing HCP for cloud scale. Enabling or disabling SELinux on an instance requires you to reboot the instance. To view whether SELinux is enabled on an instance, run: sestatus
If you want to enable SELinux on the system instances, you need to use a Docker storage driver that supports it. The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.
Time source requirements
If you are installing a multi-instance system, each instance should run NTP (network time protocol) and use the same external time source. For information, see support.ntp.org.
Supported browsers
The following browsers are qualified for use with HCP for cloud scale software. Other browsers or versions may also work.
- Google Chrome (latest version as of the date of this publication)
- Mozilla Firefox (latest version as of the date of this publication)
Resolved issues
The following issues with the Object Storage Management application and the System Management application have been resolved in this release.
Object Storage Management
The following table lists resolved issues in HCP for cloud scale affecting object storage management.
Issue | Area affected | Description |
ASP-2664 | Amazon S3 | Amazon S3 POST redirect support Amazon S3 POST redirection is not supported. Resolution This issue is resolved. |
ASP-3082 | S3 Gateway | PUT object with Content-Length greater than actual content passed can cause corrupted data The following scenario can cause data corruption:
This scenario is not recommended. Resolution This issue is resolved. |
ASP-4228 | Amazon S3 | Bucket names with capital letters allowed You can create a bucket with a name that includes capital letters. However, the bucket does not appear in listings, and you cannot put objects into the bucket. Resolution This issue is resolved. This behavior is compliant with Amazon S3. |
ASP-4385 | Management API |
Display name different between MAPI and SW API The value of Resolution This issue is resolved. The values displayed are now identical. |
ASP-4423 | System Reports | Active Objects graph incorrect at some times The Active Objects graph is refreshed every ten seconds. During metadata balancing, metrics gauges can momentarily return inaccurate counts. Therefore, the Active Objects graph can show incorrect object counts at some times. Resolution This issue is resolved. |
ASP-4444 | Online help | Online help searches can time out Searching the Object Storage Management online help for strings containing non-alphanumeric characters such as a hyphen can cause the browser to become unresponsive and the search to time out. Resolution This issue is resolved. Searches within the online help now function as expected. |
ASP-4508 | S3 |
Text of error message differs from S3 API description The error message text for NoSuchVersion does not match Amazon S3. Resolution This issue is resolved. The message text now matches. |
ASP-4510 | Metrics | Client object active count becomes negative when deleting versions Under some circumstances the client object count displayed on the Dashboard can be lower than the current value. It can display as a negative number. Resolution This issue is resolved. |
ASP-4518 | Storage Management | Unclear error message when storage component verification fails When storage component verification fails, the error message displayed is generic. Resolution This issue is resolved. The error message now provides additional diagnostic information. |
System Management
The following table lists resolved issues in HCP for cloud scale affecting system management.
Issue | Area affected | Description |
ENS-4567 ENS-7922 | Installing packages, updates | Installing component or update packages sometimes reports failure but actually succeeds Occasionally, package installation may report failure with this error: Installation Failed Installation task was haltedAlso, uploading an update package may fail with this error: Error loading update package from data connection However, your uploads may have actually succeeded. Resolution This issue is resolved. |
ENS-6160 ENS-6276 ENS-7924 | Service deployment | Services fail to deploy with thread or process creation errors Services may fail to deploy when both of the following are true:
And:
Resolution This issue is resolved. |
S3 User Credentials
The following table lists resolved issues in HCP for cloud scale affecting S3 user credentials.
Issue | Area affected | Description |
ASP-4867 | S3 user credentials | Name of previous user displayed In the S3 User Credentials application, after you generate credentials for a user, if you log in as a different user to generate credentials, the previous user name might appear instead of the current user name. Resolution This issue is resolved. |
Known issues
The following issues with the Object Storage Management application and the System Management application have been identified in this release.
Object Storage Management
The following table lists additional known issues in HCP for cloud scale affecting object storage management.
Issue | Area affected | Description |
ASP-1138 | System Management | Cannot update cluster name After changing the cluster hostname, the change is not automatically made externally visible. Workaround Restart the S3 Gateway:
|
ASP-1387 | Metadata Gateway | New buckets and objects not immediately visible Newly created buckets and objects are not immediately visible. Some client applications (such as Cloudberry Explorer) immediately retrieve the list of buckets to display the new bucket or object, which is not visible. Workaround If the new bucket or object is not immediately visible, refresh the list manually. |
ASP-2422 | Tracing Agent | Incorrect alert message during manual deployment When manually deploying a four-node, multi-instance system, the Tracing Agent service returns an alert that the service is below the recommended instance count even when the correct number of service instances are deployed. Workaround If you have deployed the correct number of instances you can safely ignore this alert. |
ASP-2913 | Deployment | Services must be deployed with default port values The following services must be deployed with their default port values:
|
ASP-3081 | Management API | API job endpoints are not supported A number of API endpoints refer to jobs. Jobs are not supported in this release. |
ASP-3119 | MAPI Gateway | Blocked thread on authorization timeout Authentication and authorization use a system management authorization client service which has a different timeout interval. If a management API authorization or authentication request times out but the underlying client service doesn't, the thread is blocked. Workaround Stop and restart the MAPI Gateway service container. |
ASP-3170 | MAPI Gateway | Certain APIs are public The MAPI schema includes public APIs, which do not require OAuth tokens. Workaround None needed. The public APIs do not require OAuth tokens. |
ASP-3297 | Storage Management | Cannot write to storage even though storage is available The storage component to which data is written is selected at random. If a filled storage component is selected, the write may fail. Workaround Use the MAPI endpoint |
ASP-3353 | Amazon S3 | Bucket Lifecycle Version 1 not supported Bucket Lifecycle Version 1 is not supported in this release. |
ASP-3746 | Metadata Gateway | Metadata Gateway failure can cause underprotection A Metadata Gateway service instance becomes unavailable if the container or physical node hosting it fails. This can cause underprotection. Workaround If the System Management application indicates that a Metadata Gateway service is unavailable, remove the Metadata Gateway instance. This starts the service on another node instance. If there no instances available to host a Metadata Gateway, the System Management application doesn't remove the instance and instead displays an error message. In this situation, use the System Management application to do this:
|
ASP-4231 | Metadata Gateway | Leader node can go down without sending its latest snapshots, affecting synchronization A snapshot provides the current state of the state machine from a leader node (service instance) to a follower service instance that is out of synch. If a leader node runs out of space to store snapshots and can't send out its latest snapshot, the follower node will be unable to resynchronize. Workaround Bring down the leader service instance, increase its storage space, and restart the service. |
ASP-4410 | Installation | Metadata Cache can fail to deploy during installation It is possible for installation not to finish because the metadata-cache service fails to deploy during installation. Workaround Reinstall the system. |
ASP-5364 | Amazon S3 | S3 operations can fail with fresh user tokens If the cache is unavailable when a user token is generated, the cache may not contain the token, and authentication for S3 operations for that user will fail. Workaround Wait one minute and repeat the operation. |
ASP-5397 | Amazon S3 | Objects placed in bucket configured for sync-from show status of REPLICA If an object is placed directly into a bucket configured for synchronization (sync-from), the replication status of the object is incorrectly set to REPLICA. |
ASP-5408 | Amazon S3 | When bucket has multiple sync-from rules for same external queue, objects may not be synchronized The bucket sync-from function only supports one rule for the same external SQS queue and external bucket. Workaround To use multiple rules for an external bucket, use one SQS queue for each rule. |
ASP-5432 | Amazon S3 | SQS queue name not validated The SQS queue name in a sync-from request is not validated, so a name not valid to AWS is accepted. Workaround Follow AWS naming conventions for the SQS queue name: up to 80 alphanumeric characters, hyphens (-), or underscores (_). Queue names are case-sensitive (for example, |
ASP-5433 | Amazon S3 | Prefix in bucket synchronization rule not validated A prefix in a bucket synchronization rule request is not validated, so a prefix not valid to AWS is accepted. Workaround Follow AWS metadata conventions for prefixes. For more information, see https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html. |
ASP-5434 | Amazon S3 | Prefix length in bucket synchronization rule not validated The string length of a prefix in a bucket synchronization rule request is not validated, so a prefix not valid to AWS is accepted. Workaround Follow AWS metadata conventions for prefixes: up to 1024 characters. |
ASP-5438 | MAPI Gateway | Transient errors when activating or changing state of storage component Activating or changing the state of a storage component might appear to fail with a 503 (service unavailable) error. Workaround Use the API |
ASP-5441 | Storage Management | Storage component alerts not dismissed Once raised, a storage component alert might not be cleared automatically if the condition that triggered it is not resolved. For example, if a storage component is taken permanently offline, the alert is not cleared. Workaround If a storage component alert is raised, resolve the underlying issue that triggered it. If a storage component is taken permanently offine, inactivate it. |
ASP-5476 | Storage Management | Bucket synchronization fails if SQS queue is unavailable If an Amazon SQS queue is unavailable when a bucket synchronization rule is defined, or fails while synchronization is active, synchronization fails without any indication of an error. Workaround Ensure that the SQS queue is active before defining a synchronization rule. If the queue becomes unavailable:
|
System Management
The following table lists known issues in HCP for cloud scale affecting system management.
Issue | Area affected | Description |
ASP-3379 | Configuration | Cannot set refresh token timeout value The Refresh Token Timeout configuration value in the System Management application (Configuration > Security > Settings) has no effect. |
ENS-4051 | Services, system deployment, update | Service fails to start during system deployment and update Rarely, a system deployment, service management operation, or system update fails because a service fails to start. When this happens, the System Management application is inaccessible from the instance where the failure occurred. The logs in the watchdog-service log directory contain this error: Error response from daemon: Conflict. The name "service-name" is already in use by container Docker-container-id. You have to remove (or rename) that container to be able to reuse that name.Workaround Restart the Docker service on the instance where the service failed to start. For example, if you are using systemd to run Docker, run: systemctl restart dockerAfter restarting Docker, retry the system deployment, service management operation, or system update. |
ENS-7957 | System update | Network types cannot be configured for new services before system update Before starting an update, you are prompted to specify the network configuration for any new services included in the version that you're updating to. However, you can specify only the port numbers for the new service. You cannot specify the network type (that is, internal or external) for the service to use. Each new service gets the default network type, which is determined by the service itself. |
ENS-7962 | System update | Uploading an update package fails after the failure and recovery of a system instance If a system instance enters the Down state, when you attempt to upload an update package, the upload fails. However, after the system instance recovers, when you try again to upload an update package, the upload again fails, even though the system is in a healthy state. Workaround
|
ENS-7964 | Volumes | Volume configuration is not displayed correctly in System Management application During installation, you can configure volumes for system services by specifying different values in the volume.config file on each system instance. Each volume is correctly configured with the settings you specify, but the page Monitoring > Services > Service Details in the System Management application incorrectly shows each volume as having identical configurations. |
ENS-8299 | Service deployment | Database service fails to deploy The Cassandra service can fail to deploy with the error Could not contact node over JMX. The log file on the node running the service instance includes the following entry: java.lang.RuntimeException: A node required to move the data consistently is down (/nnn.nnn.nnn.nnn). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false Workaround
|
ENS-8666 | Volumes | With some Docker versions, services and jobs that use NFS-backed volumes may fail to deploy while SELinux is enabled Services and jobs can be configured to use the Docker local volume driver for storing data on NFS servers. Such services and jobs may fail to deploy when the following are true:
Workarounds Do one of the following:
|
ENS-10750 | Updates | Update volume prechecks not performed Validation of volume configuration values is not honored by the upgrade process. As a result, invalid configuration values are passed to Docker. Workaround Use caution when specifying volume values. |
FNDD-970 | MAPI | If IdP is unavailable, threads blocked HCP for cloud scale uses a System Management function to validate tokens. The function does not time out. If the identity provider is unavailable, the requesting thread is blocked. |
Documentation set
- Hitachi Content Platform for Cloud Scale Object Storage Management Application Help (MK‑HCPCS000‑02): This Help system explains how to use the HCP for cloud scale Object Storage Management application to configure and operate a common object storage interface for clients to interact with.
- Hitachi Content Platform for Cloud Scale System Management Application Help (MK‑HCPCS001‑03): This Help system contains the instructions for using the HCP for cloud scale System Management application to configure HCP for cloud scale for your users, enable and disable system features, and monitor the system and its connections.
- Installing Hitachi Content Platform for Cloud Scale (MK‑HCPCS002‑03): This document contains instructions for installing the HCP for cloud scale software.
- Hitachi Content Platform for Cloud Scale Management API Reference (MK‑HCPCS007‑01): This document describes the management REST API commands available for customer use in the HCP for cloud scale software.
- Hitachi Content Platform for Cloud Scale Copyrights and Third-party Licenses (MK‑HCPCS003‑03): This document contains copyright and license information for third-party software distributed with or embedded in the HCP for cloud scale operating system, core software, and applications.
- Hitachi Content Platform for Cloud Scale Release Notes (RN‑HCPCS004‑04, this document): This document is for customers, and describes new features, product documentation, resolved and known issues, and other useful information about this release of the product.
Documentation corrections
The following issues were identified with the documentation, including the online help, after its publication.
Object Storage Management application online help
The following items refer to the online help information available in the Object Management application profile menu under Help.
When defining a storage component, the host and proxy host names can include any alphanumeric character.
If you use the AWS command-line interface to configure bucket synchronization, use at least aws-cli
1.16.211 and aws-sdk
1.11.610.
If a sync-from action fails, it is retried, and the SQS message about the failure is retained. To avoid a possible accumulation of SQS failure messages, the best practice is to define a suitable retention policy for SQS and to delete the sync-from rule once the desired results are obtained.
System Management application online help
The following items refer to the online help information available in the System Management application profile menu under Help.
The online Help refers to the generic System Management application, not the version specific to HCP for cloud scale. For information about the HCP for cloud scale System Management application, go to https://knowledge.hitachivantara.com/Documents/Storage/HCP_for_Cloud_Scale/1.2.0/System_Management.
Job configuration is not supported in this release. Job functions are not available in the navigation pane of the System Management application.
Management API reference
The following refers to the management API reference information available in the Object Storage Management application profile menu under REST API.
The information describes endpoints related to jobs. Jobs are not supported in this release.