Skip to main content

We've Moved!

Product Documentation has moved to docs.hitachivantara.com
Hitachi Vantara Knowledge

Hitachi Content Platform for Cloud Scale v1.2.0 Release Notes

About this document

This document provides late-breaking information about HCP for cloud scale v1.2.0. It includes information that was not available at the time the technical documentation for this product was published, as well as a list of new features; resolved issues; and known issues and, where applicable, their workarounds.

Intended audience

This document is intended for Hitachi Vantara partners who license and use HCP for cloud scale.

Getting help

Hitachi Vantara Support Connect is the destination for technical support of products and solutions sold by Hitachi Vantara. To contact technical support, log on to Hitachi Vantara Support Connect for contact information: https://support.hitachivantara.com/en_us/contact-us.html.

Hitachi Vantara Community is a global online community for Hitachi Vantara customers, partners, independent software vendors, employees, and prospects. It is the destination to get answers, discover insights, and make connections. Join the conversation today! Go to community.hitachivantara.com, register, and complete your profile.

About this release

This is v1.2.0.1 of Hitachi Content Platform for cloud scale (HCP for cloud scale).

Major features

HCP for cloud scale is a software-defined object storage solution that is based on a massively parallel microservice architecture, and is compatible with the Amazon Simple Storage Service (Amazon S3) application programming interface (API). HCP for cloud scale is especially well suited to service applications requiring high bandwidth and compatibility with Amazon S3 APIs.

Major features in v1.2.0

HCP for cloud scale v1.2.0 includes the following major features.

Bucket synchronization

Bucket synchronization supports copying of objects between HCP for cloud scale buckets and buckets in external systems. Buckets can be synchronized in two directions:

  • Synchronization to a bucket (sync-to) allows automatic, asynchronous copying of objects in buckets in an HCP for cloud scale system to external storage systems.
  • Synchronization from a bucket (sync-from) allows automatic, asynchronous copying of objects in buckets in external storage systems to an HCP for cloud scale bucket.

Bucket synchronization is configured using AWS S3 cross-region replication, so you can use the S3 SDK and S3 command-line interface (CLI) provided by AWS.

An external storage system can be another HCP for cloud scale system, AWS S3, or any S3-compatible system.

Features in v1.1.1

HCP for cloud scale v1.1.1 addresses the following:

  • Improvements to stability, performance, and scalability
  • Increased capacity to ingest objects
  • Improvements for more accurate display of metrics for object counts
  • Improvements to the user interface
  • Other bug fixes

Major features in v1.1.0

HCP for cloud scale v1.1.0 includes the following new major features.

Flexible metadata scaling

The Metadata Gateway service stores and protects metadata and serves it to other services. During initial deployment, you deploy Metadata Gateway service instances to at least three nodes in the HCP for cloud scale system. In v1.1.0 you can deploy up to a maximum of every node in the system. Using the System Management application, you can add or remove Metadata Gateway service instances.

The Metadata Coordination service manages Metadata Gateway scaling. The service does the following:

  • Constantly monitors the Metadata Gateway service and balances data among Metadata Gateway instances as needed
  • Moves data into new Metadata Gateway instances
  • Moves data out of a Metadata Gateway instance set for removal
Scalable policy engine

The Metadata Policy Engine service executes asynchronous metadata updates. In v1.1.0 the service supports operations that run across multiple partitions.

Amazon S3 object store APIs

HCP for cloud scale supports these Amazon S3 APIs:

  • S3 PUT Copy: This API lets you create a copy of an object already stored in an HCP for cloud scale system. You must have read access to the source object and write access to the destination bucket.
  • S3 PUT Part Copy: This API lets you create a copy, part by part, of an object already stored in an HCP for cloud scale system. You specify a part using a byte range of from 5 MB to 5 GB. You must have read access to the source object and write access to the destination bucket.

In v1.1.0, conditional headers, server-side encryption, and multiple AWS regions are not supported.

Major features in v1.0.0

HCP for cloud scale v1.0.0 includes the following major features.

Object Store
  • S3 gateway: HCP for cloud scale supports the Amazon Simple Storage Service (Amazon S3) application programming interface (API), which allows client applications to store and retrieve unlimited amounts of data from configured storage services.
  • Scalability and availability: You can increase the capacity and performance of a site by adding additional instances (physical compute nodes or virtual machines). HCP for cloud scale provides high availability by supporting multiple instances of essential services on different compute nodes.
  • Diverse storage backends: HCP for cloud scale supports Amazon S3, Hitachi Content Platform (HCP), HCP S Series Nodes, and Amazon S3 compatible storage systems.
  • Data and metadata management: HCP for cloud scale provides a comprehensive framework to add metadata associated with data.
  • Asynchronous policy engine: The engine accepts and processes metadata updates of user requests.
Management and Monitoring
  • Object storage management: Management and monitoring of storage components is available through the Object Storage Management application, which provides a convenient graphical user interface (GUI).
  • System management: Management and monitoring of instances, services, processes, events, and HCP for cloud scale administrative users (roles and groups) is available through the System Management application, which provides a convenient GUI.
  • REST API for all management operations: All functions are available through a management application programing interface (API). Object storage management functions are available through HCP for cloud scale management APIs, and administrative functions are available through system management APIs.
  • Syslog and email event notification: You can configure notifications by email or syslog of degraded health of storage components, instances, or services instances.
Supportability
  • External management: HCP for cloud scale supports integration with the Hitachi Remote Ops system to provide powerful and secure 24/7 monitoring.
  • Distributed tracing: HCP for cloud scale supports a third-party, open-source software tool, running as a service, to provide end-to-end, distributed tracing of S3 requests and operations through a browser.
  • Distributed metrics: HCP for cloud scale supports a third-party, open-source software tool, running as a service, to provide application metrics through a browser.

System requirements

This section lists the hardware, networking, and operating system requirements for running an HCP for cloud scale system with one or more instances.

Hardware requirements

To install HCP for cloud scale on on-premises hardware for production use, you must provision at least four instances (nodes) with sufficient CPU, RAM, disk space, and networking capabilities. This table shows the minimum and recommended hardware requirements for each instance in an HCP for cloud scale system.

ResourceMinimumRecommended
RAM32 GB128 GB
CPU8-core24-core
Available disk space500 GB 10k SAS RAID(2) 1.92 TB SSD
Network interface controller (NIC)(1) 10 Gb Ethernet NICs(4) 10 Gb Ethernet NICs

Software requirements

This table shows the minimum and recommended software requirements for each instance in an HCP for cloud scale system.

ResourceMinimumRecommended
IP addresses(1) static(2) static
Firewall Port AccessPort 443 for S3 API

Port 8000 for System Management App GUI

Port 9084 for MAPI and Object Storage Management App GUI

Same
Network TimeIP address of time service (NTP)Same

Operating system and Docker minimum requirements

Each server or virtual machine you provide must meet these requirements:

  • 64-bit Linux distribution
  • Docker version 1.13.1 or later installed
  • IP and DNS addresses configured
Additionally, you should install all relevant patches on the operating system and perform appropriate security hardening tasks.
ImportantInstall the current Docker version suggested by your operating system, unless that version is earlier than 1.13.1. The system cannot run with Docker versions prior to 1.13.1.

Operating system and Docker qualified versions

This table shows the operating systems, Docker and SELinux configurations with which the HCP for cloud scale system has been qualified.

Operating systemDocker versionDocker storage configurationSELinux setting
Fedora 27Docker 18.03.0-cedirect-lvmEnforcing
Red Hat Enterprise Linux 7.4Docker 18.03.0-cedirect-lvmEnforcing
Ubuntu 16.04-LTSDocker 17.03.0-ceaufsN/A
CentOS 7.4Docker 18.03.1-ceoverlay2Enforcing

Docker considerations

The Docker installation directory on each instance must have at least 20 GB available for storing the HCP for cloud scale Docker images.

Make sure that the Docker storage driver is configured correctly on each instance before installing HCP for cloud scale. After installing, changing the Docker storage driver requires a reinstallation of HCP for cloud scale. To view the current Docker storage driver on an instance, run docker info.

If you want to enable SELinux on the system instances, you need to use a Docker storage driver that supports it. The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.

If you are using the Docker devicemapper storage driver:

  • Make sure that there's at least 40 GB of Docker metadata storage space available on each instance. HCP for cloud scale requires 20 GB to install successfully and an additional 20 GB to successfully update to a later version. To view Docker metadata storage usage on an instance, run docker info.
  • On a production system, do not run devicemapper in loop-lvm mode. This can cause slow performance or, on certain Linux distributions, HCP for cloud scale may not have enough space to run.

SELinux considerations

You should decide whether you want to run SELinux on system instances and enable or disable it before installing HCP for cloud scale. Enabling or disabling SELinux on an instance requires you to reboot the instance. To view whether SELinux is enabled on an instance, run: sestatus

If you want to enable SELinux on the system instances, you need to use a Docker storage driver that supports it. The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.

Time source requirements

If you are installing a multi-instance system, each instance should run NTP (network time protocol) and use the same external time source. For information, see support.ntp.org.

Supported browsers

The following browsers are qualified for use with HCP for cloud scale software. Other browsers or versions may also work.

  • Google Chrome (latest version as of the date of this publication)
  • Mozilla Firefox (latest version as of the date of this publication)

Resolved issues

The following issues with the Object Storage Management application and the System Management application have been resolved in this release.

Object Storage Management

The following table lists resolved issues in HCP for cloud scale affecting object storage management.

IssueArea affectedDescription
ASP-2664Amazon S3Amazon S3 POST redirect support

Amazon S3 POST redirection is not supported.

Resolution

This issue is resolved.

ASP-3082S3 GatewayPUT object with Content-Length greater than actual content passed can cause corrupted data

The following scenario can cause data corruption:

  1. A PUT operation is executed on an object or object part
  2. The call specifies a Content-Length header that is larger than the actual size of the object
  3. The call does not specify a Content-MD5 header

This scenario is not recommended.

Resolution

This issue is resolved.

ASP-4228Amazon S3Bucket names with capital letters allowed

You can create a bucket with a name that includes capital letters. However, the bucket does not appear in listings, and you cannot put objects into the bucket.

Resolution

This issue is resolved. This behavior is compliant with Amazon S3.

ASP-4385Management API

Display name different between MAPI and SW API

The value of displayName is different between the management API user/list and the S3 API get-bucket-acl.

Resolution

This issue is resolved. The values displayed are now identical.

ASP-4423System ReportsActive Objects graph incorrect at some times

The Active Objects graph is refreshed every ten seconds. During metadata balancing, metrics gauges can momentarily return inaccurate counts. Therefore, the Active Objects graph can show incorrect object counts at some times.

Resolution

This issue is resolved.

ASP-4444Online helpOnline help searches can time out

Searching the Object Storage Management online help for strings containing non-alphanumeric characters such as a hyphen can cause the browser to become unresponsive and the search to time out.

Resolution

This issue is resolved. Searches within the online help now function as expected.

ASP-4508S3

Text of error message differs from S3 API description

The error message text for NoSuchVersion does not match Amazon S3.

Resolution

This issue is resolved. The message text now matches.

ASP-4510MetricsClient object active count becomes negative when deleting versions

Under some circumstances the client object count displayed on the Dashboard can be lower than the current value. It can display as a negative number.

Resolution

This issue is resolved.

ASP-4518Storage ManagementUnclear error message when storage component verification fails

When storage component verification fails, the error message displayed is generic.

Resolution

This issue is resolved. The error message now provides additional diagnostic information.

System Management

The following table lists resolved issues in HCP for cloud scale affecting system management.

IssueArea affectedDescription
ENS-4567

ENS-7922

Installing packages, updatesInstalling component or update packages sometimes reports failure but actually succeeds

Occasionally, package installation may report failure with this error:

Installation Failed Installation task was halted

Also, uploading an update package may fail with this error: Error loading update package from data connection

However, your uploads may have actually succeeded.

Resolution

This issue is resolved.

ENS-6160

ENS-6276

ENS-7924

Service deployment Services fail to deploy with thread or process creation errors

Services may fail to deploy when both of the following are true:

  • Your instances have large numbers of CPUs
  • Your instances have systemd installed
In this situation, services may fail to deploy with errors about thread or process creation failures, such as:

{:timestamp=>"2017-09-14T17:13:43.824000+0000", :message=>"The error reported is: \n unable to create new native thread"}

And:

Failed to launch container: Failed to run 'docker -H unix:///var/run/docker.sock pull com.hds.ensemble/cassandra:25.0.0.1523': exited with status 2; stderr='runtime/cgo: pthread_create failed: Resource temporarily unavailable

Resolution

This issue is resolved.

S3 User Credentials

The following table lists resolved issues in HCP for cloud scale affecting S3 user credentials.

IssueArea affectedDescription
ASP-4867S3 user credentialsName of previous user displayed

In the S3 User Credentials application, after you generate credentials for a user, if you log in as a different user to generate credentials, the previous user name might appear instead of the current user name.

Resolution

This issue is resolved.

Known issues

The following issues with the Object Storage Management application and the System Management application have been identified in this release.

Object Storage Management

The following table lists additional known issues in HCP for cloud scale affecting object storage management.

IssueArea affectedDescription
ASP-1138System ManagementCannot update cluster name

After changing the cluster hostname, the change is not automatically made externally visible.

Workaround

Restart the S3 Gateway:

  1. In the System Management application, select Services.
  2. Select the S3-Gateway service.
  3. Click REPAIR.
ASP-1387Metadata GatewayNew buckets and objects not immediately visible

Newly created buckets and objects are not immediately visible. Some client applications (such as Cloudberry Explorer) immediately retrieve the list of buckets to display the new bucket or object, which is not visible.

Workaround

If the new bucket or object is not immediately visible, refresh the list manually.

ASP-2422Tracing AgentIncorrect alert message during manual deployment

When manually deploying a four-node, multi-instance system, the Tracing Agent service returns an alert that the service is below the recommended instance count even when the correct number of service instances are deployed.

Workaround

If you have deployed the correct number of instances you can safely ignore this alert.

ASP-2913DeploymentServices must be deployed with default port values

The following services must be deployed with their default port values:

  • Metadata Cache
  • Tracing Agent
  • Tracing Collector
  • Tracing Query
ASP-3081Management APIAPI job endpoints are not supported

A number of API endpoints refer to jobs. Jobs are not supported in this release.

ASP-3119MAPI GatewayBlocked thread on authorization timeout

Authentication and authorization use a system management authorization client service which has a different timeout interval. If a management API authorization or authentication request times out but the underlying client service doesn't, the thread is blocked.

Workaround

Stop and restart the MAPI Gateway service container.

ASP-3170MAPI GatewayCertain APIs are public

The MAPI schema includes public APIs, which do not require OAuth tokens.

Workaround

None needed. The public APIs do not require OAuth tokens.

ASP-3297Storage ManagementCannot write to storage even though storage is available

The storage component to which data is written is selected at random. If a filled storage component is selected, the write may fail.

Workaround

Use the MAPI endpoint PATCH /storage_component/update to set the filled storage component's readOnly flag to true.

ASP-3353Amazon S3Bucket Lifecycle Version 1 not supported

Bucket Lifecycle Version 1 is not supported in this release.

ASP-3746Metadata GatewayMetadata Gateway failure can cause underprotection

A Metadata Gateway service instance becomes unavailable if the container or physical node hosting it fails. This can cause underprotection.

Workaround

If the System Management application indicates that a Metadata Gateway service is unavailable, remove the Metadata Gateway instance. This starts the service on another node instance.

If there no instances available to host a Metadata Gateway, the System Management application doesn't remove the instance and instead displays an error message. In this situation, use the System Management application to do this:

  1. Add a new Metadata Gateway instance.
  2. Remove the unavailable Metadata Gateway instance.
ASP-4231Metadata GatewayLeader node can go down without sending its latest snapshots, affecting synchronization

A snapshot provides the current state of the state machine from a leader node (service instance) to a follower service instance that is out of synch. If a leader node runs out of space to store snapshots and can't send out its latest snapshot, the follower node will be unable to resynchronize.

Workaround

Bring down the leader service instance, increase its storage space, and restart the service.

ASP-4410InstallationMetadata Cache can fail to deploy during installation

It is possible for installation not to finish because the metadata-cache service fails to deploy during installation.

Workaround

Reinstall the system.

ASP-5364Amazon S3S3 operations can fail with fresh user tokens

If the cache is unavailable when a user token is generated, the cache may not contain the token, and authentication for S3 operations for that user will fail.

Workaround

Wait one minute and repeat the operation.

ASP-5397Amazon S3Objects placed in bucket configured for sync-from show status of REPLICA

If an object is placed directly into a bucket configured for synchronization (sync-from), the replication status of the object is incorrectly set to REPLICA.

ASP-5408Amazon S3When bucket has multiple sync-from rules for same external queue, objects may not be synchronized

The bucket sync-from function only supports one rule for the same external SQS queue and external bucket.

Workaround

To use multiple rules for an external bucket, use one SQS queue for each rule.

ASP-5432Amazon S3SQS queue name not validated

The SQS queue name in a sync-from request is not validated, so a name not valid to AWS is accepted.

Workaround

Follow AWS naming conventions for the SQS queue name: up to 80 alphanumeric characters, hyphens (-), or underscores (_). Queue names are case-sensitive (for example, My-queue and my-queue are different queues).

ASP-5433Amazon S3Prefix in bucket synchronization rule not validated

A prefix in a bucket synchronization rule request is not validated, so a prefix not valid to AWS is accepted.

Workaround

Follow AWS metadata conventions for prefixes. For more information, see https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html.

ASP-5434Amazon S3Prefix length in bucket synchronization rule not validated

The string length of a prefix in a bucket synchronization rule request is not validated, so a prefix not valid to AWS is accepted.

Workaround

Follow AWS metadata conventions for prefixes: up to 1024 characters.

ASP-5438MAPI GatewayTransient errors when activating or changing state of storage component

Activating or changing the state of a storage component might appear to fail with a 503 (service unavailable) error.

Workaround

Use the API storage_component/list to check the state of the storage component. If the state actually hasn't changed, try the update again.

ASP-5441Storage ManagementStorage component alerts not dismissed

Once raised, a storage component alert might not be cleared automatically if the condition that triggered it is not resolved. For example, if a storage component is taken permanently offline, the alert is not cleared.

Workaround

If a storage component alert is raised, resolve the underlying issue that triggered it. If a storage component is taken permanently offine, inactivate it.

ASP-5476Storage ManagementBucket synchronization fails if SQS queue is unavailable

If an Amazon SQS queue is unavailable when a bucket synchronization rule is defined, or fails while synchronization is active, synchronization fails without any indication of an error.

Workaround

Ensure that the SQS queue is active before defining a synchronization rule.

If the queue becomes unavailable:

  1. Issue a DELETE bucket replication S3 API request to remove the synchronization rule.
  2. Once the queue is available again, issue a PUT bucket replication S3 API request to redefine the rule.

System Management

The following table lists known issues in HCP for cloud scale affecting system management.

IssueArea affectedDescription
ASP-3379ConfigurationCannot set refresh token timeout value

The Refresh Token Timeout configuration value in the System Management application (Configuration > Security > Settings) has no effect.

ENS-4051Services, system deployment, updateService fails to start during system deployment and update

Rarely, a system deployment, service management operation, or system update fails because a service fails to start. When this happens, the System Management application is inaccessible from the instance where the failure occurred.

The logs in the watchdog-service log directory contain this error:

Error response from daemon: Conflict. The name "service-name" is already in use by container Docker-container-id. You have to remove (or rename) that container to be able to reuse that name.

Workaround

Restart the Docker service on the instance where the service failed to start. For example, if you are using systemd to run Docker, run:

systemctl restart docker

After restarting Docker, retry the system deployment, service management operation, or system update.

ENS-7957System updateNetwork types cannot be configured for new services before system update

Before starting an update, you are prompted to specify the network configuration for any new services included in the version that you're updating to. However, you can specify only the port numbers for the new service. You cannot specify the network type (that is, internal or external) for the service to use. Each new service gets the default network type, which is determined by the service itself.

ENS-7962System updateUploading an update package fails after the failure and recovery of a system instance

If a system instance enters the Down state, when you attempt to upload an update package, the upload fails. However, after the system instance recovers, when you try again to upload an update package, the upload again fails, even though the system is in a healthy state.

Workaround

  1. In the System Management application, go to the page Monitoring > Processes and for the task Upload Plugin Bundle click Retry Task.
  2. Upload the update package again.
ENS-7964VolumesVolume configuration is not displayed correctly in System Management application

During installation, you can configure volumes for system services by specifying different values in the volume.config file on each system instance. Each volume is correctly configured with the settings you specify, but the page Monitoring > Services > Service Details in the System Management application incorrectly shows each volume as having identical configurations.

ENS-8299Service deploymentDatabase service fails to deploy

The Cassandra service can fail to deploy with the error Could not contact node over JMX. The log file on the node running the service instance includes the following entry: java.lang.RuntimeException: A node required to move the data consistently is down (/nnn.nnn.nnn.nnn). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false

Workaround

  1. Restart the Cassandra container running on that node.
  2. Redeploy the service.
ENS-8666VolumesWith some Docker versions, services and jobs that use NFS-backed volumes may fail to deploy while SELinux is enabled

Services and jobs can be configured to use the Docker local volume driver for storing data on NFS servers. Such services and jobs may fail to deploy when the following are true:

  • System instances are running a version of Docker later than 1.12.6
  • SELinux is enabled

Workarounds

Do one of the following:

  • Update Docker to Community Edition (ce) version 18.03 or later
  • Disable SELinux
ENS-10750UpdatesUpdate volume prechecks not performed

Validation of volume configuration values is not honored by the upgrade process. As a result, invalid configuration values are passed to Docker.

Workaround

Use caution when specifying volume values.

FNDD-970MAPIIf IdP is unavailable, threads blocked

HCP for cloud scale uses a System Management function to validate tokens. The function does not time out. If the identity provider is unavailable, the requesting thread is blocked.

Documentation set

The documentation supporting HCP for cloud scale v1.2.0 consists of these titles:
  • Hitachi Content Platform for Cloud Scale Object Storage Management Application Help (MK‑HCPCS000‑02): This Help system explains how to use the HCP for cloud scale Object Storage Management application to configure and operate a common object storage interface for clients to interact with.
  • Hitachi Content Platform for Cloud Scale System Management Application Help (MK‑HCPCS001‑03): This Help system contains the instructions for using the HCP for cloud scale System Management application to configure HCP for cloud scale for your users, enable and disable system features, and monitor the system and its connections.
  • Installing Hitachi Content Platform for Cloud Scale (MK‑HCPCS002‑03): This document contains instructions for installing the HCP for cloud scale software.
  • Hitachi Content Platform for Cloud Scale Management API Reference (MK‑HCPCS007‑01): This document describes the management REST API commands available for customer use in the HCP for cloud scale software.
  • Hitachi Content Platform for Cloud Scale Copyrights and Third-party Licenses (MK‑HCPCS003‑03): This document contains copyright and license information for third-party software distributed with or embedded in the HCP for cloud scale operating system, core software, and applications.
  • Hitachi Content Platform for Cloud Scale Release Notes (RN‑HCPCS004‑04, this document): This document is for customers, and describes new features, product documentation, resolved and known issues, and other useful information about this release of the product.

Documentation corrections

The following issues were identified with the documentation, including the online help, after its publication.

Object Storage Management application online help

The following items refer to the online help information available in the Object Management application profile menu under Help.

Adding a storage component

When defining a storage component, the host and proxy host names can include any alphanumeric character.

About bucket synchronization

If you use the AWS command-line interface to configure bucket synchronization, use at least aws-cli 1.16.211 and aws-sdk 1.11.610.

Delete bucket synchronization rules

If a sync-from action fails, it is retried, and the SQS message about the failure is retained. To avoid a possible accumulation of SQS failure messages, the best practice is to define a suitable retention policy for SQS and to delete the sync-from rule once the desired results are obtained.

System Management application online help

The following items refer to the online help information available in the System Management application profile menu under Help.

The online Help refers to the generic System Management application, not the version specific to HCP for cloud scale. For information about the HCP for cloud scale System Management application, go to https://knowledge.hitachivantara.com/Documents/Storage/HCP_for_Cloud_Scale/1.2.0/System_Management.

Job configuration is not supported in this release. Job functions are not available in the navigation pane of the System Management application.

Management API reference

The following refers to the management API reference information available in the Object Storage Management application profile menu under REST API.

Jobs

The information describes endpoints related to jobs. Jobs are not supported in this release.