Hitachi Content Platform for Cloud Scale v2.3.0 Release Notes
About this document
This document gives late-breaking information about HCP for cloud scale v2.3.0. It includes information that was not available at the time the technical documentation for this product was published, a list of new features, a list of resolved issues, and a list of known issues and where applicable their workarounds.
Intended audience
This document is intended for customers and Hitachi Vantara partners who license and use HCP for cloud scale.
Getting help
The Hitachi Vantara Support Website is the destination for technical support of products and solutions sold by Hitachi Vantara. To contact technical support, log on to the Hitachi Vantara Support Website for contact information: https://support.hitachivantara.com/en_us/contact-us.html.
Hitachi Vantara Community is a global online community for Hitachi Vantara customers, partners, independent software vendors, employees, and prospects. It is the destination to get answers, discover insights, and make connections. Join the conversation today! Go to community.hitachivantara.com, register, and complete your profile.
About this release
This is build 2.3.0.1 of the Hitachi Content Platform for cloud scale (HCP for cloud scale) software.
Major features
HCP for cloud scale is a software-defined object storage solution that is based on a massively parallel microservice architecture and is compatible with the Amazon Simple Storage Service (Amazon S3) application programming interface (API). HCP for cloud scale is especially well suited to service applications requiring high bandwidth and compatibility with the Amazon S3 API.
Features in v2.3.0
HCP for cloud scale v2.3.0 includes the following features.
In this release, listing a bucket accurately reflects the contents of the bucket. A list is correct even immediately after writing to the bucket to add, modify, or delete objects. This feature supports analytical operations. List operations have no impact on the performance of write operations.
The user interface for the Object Storage Management application has been updated for increased security and consistency with other Hitachi Vantara products. The look and feel of the Object Storage Management application is now consistent with that of the S3 Console application.
System requirements
This section lists the hardware, networking, and operating system requirements for running an HCP for cloud scale system with one or more instances.
Hardware requirements
To install HCP for cloud scale on on-premises hardware for production use, you must provision at least four instances (nodes) with sufficient CPU, RAM, disk space, and networking capabilities. This table shows the hardware minimum requirements and best-practice resources for each instance in an HCP for cloud scale system.
| Resource | Minimum | Best |
| RAM | 32 GB | 128 GB |
| CPU | 8-core | 24-core |
| Available disk space | 500 GB 10k SAS RAID | (4) 2000 GB 15k SAS RAID SSD |
| Network interface controller (NIC) | (1) 10 Gb Ethernet NICs | (4) 10 Gb Ethernet NICs |
Software requirements
The following table shows the minimum requirements and best-practice software configurations for each instance in an HCP for cloud scale system.
| Resource | Minimum | Best |
| IP addresses | (1) static | (2) static |
| Firewall Port Access | Port 443 for S3 API Port 8000 for System Management App GUI Port 9099 for MAPI and Object Storage Management App GUI | Same |
| Network Time | IP address of time service (NTP) | Same |
Operating system and Docker minimum requirements
Each server or virtual machine you provide must have the following:
- 64-bit Linux distribution
- Docker version installed: Docker Community Edition 18.09.0 or later
- IP and DNS addresses configured
Additionally, you should install all relevant patches on the operating system and perform appropriate security hardening tasks.
To execute scripts provided with the product on RHEL, you should install Python.
Operating system and Docker qualified versions
This table shows the operating system, Docker, and SELinux configurations with which the HCP for cloud scale system has been qualified.
| Operating system | Docker version | Docker storage configuration | SELinux setting |
| Red Hat or CentOS Enterprise Linux 8.1 | Docker Community Edition 19.03.12 or later | overlay2 | Enforcing |
If you are installing on Amazon Linux, before deployment, edit the file /etc/security/limits.conf on every node to add the following two lines:
* hard nofile 65535 * soft nofile 65535
Docker considerations
The Docker installation folder on each instance must have at least 20 GB available for storing the HCP for cloud scale Docker images.
Make sure that the Docker storage driver is configured correctly on each instance before installing HCP for cloud scale. To view the current Docker storage driver on an instance, run docker info.
If you are using the Docker devicemapper storage driver:
- Make sure that there's at least 40 GB of Docker metadata storage space available on each instance. HCP for cloud scale needs 20 GB to install successfully and an additional 20 GB to successfully update to a later version. To view Docker metadata storage usage on an instance, run docker info.
- On a production system, do not run
devicemapperinloop-lvmmode. This can cause slow performance or, on certain Linux distributions, HCP for cloud scale might not have enough space to run.
SELinux considerations
You should decide whether you want to run SELinux on system instances and enable or disable it before installing HCP for cloud scale. To enable or disable SELinux on an instance, you must restart the instance. To view whether SELinux is enabled on an instance, run: sestatus
To enable SELinux on the system instances, use a Docker storage driver that supports it. The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.
Time source requirements
If you are installing a multi-instance system, each instance should run NTP (network time protocol) and use the same external time source. For information, see support.ntp.org.
Supported browsers
The following browsers are qualified for use with HCP for cloud scale software. Other browsers or versions might also work.
- Google Chrome (latest version as of the date of this publication)
- Mozilla Firefox (latest version as of the date of this publication)
Installation or upgrade considerations
This section provides information about installing or upgrading HCP for cloud scale software.
After installing the software, but before putting the system into production, you might want to assign a name to the cluster; update Secure Socket Layer (SSL) certificates for the system, storage components, or synchronized buckets; or enable encryption.
After renaming a cluster:
- Update or upload the system certificate.
- Restart (repair) the S3 Gateway service.
- Restart (repair) the MAPI Gateway service.
- If encryption is enabled, restart (repair) the Key Management Server (KMS) service and unseal the vault.
After uploading the system certificate:
- Restart (repair) the S3 Gateway service.
- Restart (repair) the MAPI Gateway service.
- If encryption is enabled, restart (repair) the Key Management Server (KMS) service and unseal the vault.
After uploading a client certificate for a storage component:
- Restart (repair) the S3 Gateway service.
- Restart (repair) the MAPI Gateway service.
After uploading a client certificate for an S3-compatible remote system used for bucket synchronization:
- Restart (repair) the S3 Gateway service.
- Restart (repair) the Policy Engine service.
After enabling encryption:
- Restart (repair) the S3 Gateway service.
- Restart (repair) the Policy Engine service.
For more information on these tasks see the Administration Guide or the online help available in the system.
As part of the upgrade to v2.3.0, table entries are migrated to new tables. This process is resource intensive and continues for some time after the update completes. Performance is affected until it completes. While migration is ongoing, an alert in the Object Storage Management application reports the migration status. The alert clears when the migration is completed.
Before beginning an upgrade, ensure that the heap size for the Sentinel service is 8 GB. If the value is too small, upgrade pre-check fails and reports a Sentinel service error.
To help the migration of larger systems finish more quickly and smoothly, you can increase resources to the following services beforehand:
- Allocate 64 GB of container memory and 32 GB of heap memory to the Metadata Gateway service.
- Allocate 16 GB of container memory and 8 GB of heap memory to the S3 Gateway service.
- Allocate 16 GB of container memory and 8 GB of heap memory to the Metadata Coordination service.
- Allocate 8 GB of container memory to the Message Queue service.
You can monitor the progress of migration using the metric deprecated_metadata_clientobject_active_count in Prometheus. This metric gives the count of objects in deprecated partitions. It decrements during the migration, reaching 0 when all objects are migrated. You can use this metric to estimate of the time remaining, though even after the value reaches 0 there will still be some cleanup.
Data Lifecycle Service operations are paused during migration and resume after migration finishes. In some cases you can force them to run.
Some information might not be accurate during migration:
- Listings might not be strongly consistent until migration has finished. In particular, deletions might not be updated.
- Chargeback metrics might not be accurate during the migration time period. However, the count of total objects and bytes in the system should remain accurate.
- Some metadata metrics will appear inaccurate during migration:
- metadata_clientobject_active_encrypted_count
- metadata_clientobject_active_unencrypted_count
The update process can trigger alarms on the Status tab of the System Management application, or emails if notification is enabled, that services have gone down, exited abnormally, or become underprotected. These notifications are expected during an update and you can safely ignore them. Update alarms should clear automatically when the update finishes.
Version 2.3 might alter I/O performance for small objects with applications that employ a specific object naming pattern. To optimize your bucket for high request rates, consider using an object key naming pattern to either:
- Distribute objects across multiple prefixes (folders)
- Introduce differentiating characters prepended to object names
An upgrade from a version before v2.2.1 to v2.3 can take several hours, and it's best to monitor the process. The upgrade can appear to stall for extended periods of time, and you should let the process finish. However, if the process stops and displays an error message (for example, The update has failed! Click retry to attempt the update again), click Retry and the process will resume.
Resolved issues
The following HCP for cloud scale issues are resolved in this release.
S3 Console
The following table lists resolved issues affecting the S3 Console application.
| Issue | Area affected | Description |
| ASP-10436 | S3 Console | Put-Object-Retention request fails with "malformed XML" error The order of headers in Put-Object-Retention requests is enforced by the S3 Console XSD, which caused requests with headers in a different order to fail with HTTP 400 "malformed XML" errors. Resolution This issue is resolved. The order is not enforced. |
Known issues
The following issues with HCP for cloud scale have been identified in this release.
Don't try to initialize the encryption key management server (the Vault service) manually outside of HCP for cloud scale. Doing so results in data loss.
The S3 Console application uses the Metrics service to fetch bucket information. So the S3 Console application can display bucket information and statistics to users, configure the port hosting the Metrics service as external.
Object storage management
The following table lists known issues affecting object storage management.
| Issue | Area affected | Description |
| ASP-2422 | Tracing Agent | Incorrect alert message during manual deployment When manually deploying a four-node, multi-instance system, the Tracing Agent service returns an alert that the service is below the needed instance count even when the correct number of service instances are deployed. Workaround If you have deployed the correct number of instances you can safely ignore this alert. |
| ASP-3081 | Management API | API job methods are not supported A number of API methods refer to jobs. Jobs are not supported in this release. |
| ASP-3119 | MAPI Gateway | Blocked thread on authorization timeout Authentication and authorization use a system management authorization client service which has a different timeout interval. If a management API authorization or authentication request times out but the underlying client service doesn't, the thread is blocked. Workaround Stop and restart the MAPI Gateway service container. |
| ASP-3170 | MAPI Gateway | Certain API methods are public The MAPI schema includes public API methods, which do not need OAuth tokens. Workaround None needed. The public API methods do not need OAuth tokens. |
| ASP-3297 | Storage Management | Cannot write to storage even though storage is available The storage component to which data is written is selected at random. If a filled storage component is selected, the write might fail. Workaround Use the Object Storage Management application or the MAPI method |
| ASP-6630 | Storage Management | Setting encryption from multiple clients simultaneously can render existing storage component inaccessible If two accounts try to set the encryption flag simultaneously, either using the GUI or the management API method Workaround
|
| ASP-7239 | Storage Management | Storage component host name final segment allowed to begin with number When configuring a storage component, the last segment of the host name can't begin with a number (for example, Workaround Ensure that the last segment of the host name does not begin with a number before proceeding. |
System management
The following table lists known issues affecting system management.
| Issue | Area affected | Description |
| ASP-3379 | Configuration | Cannot set refresh token timeout value The Refresh Token Timeout configuration value in the System Management application (Configuration > Security > Settings) has no effect. |
| ASP-9921 | System update | System update requires manual steps if DARE is enabled If data-at-rest encryption (DARE) is enabled, during an upgrade you must monitor the process, detect when the Key Management Server service restarts, and then unseal the vault. S3 traffic is blocked until the vault is unsealed. Workaround The KMS service is now the last service restarted, and you can monitor the progress in the System Management application. An event is logged ( As soon as the KMS service is updated, unseal the vault. (You can keep the unseal keys at hand to enter them immediately.) |
| ENS-7957 (FNDD-476) | System update | Network types cannot be configured for new services before system update Before starting an update, you are prompted to specify the network configuration for any new services included in the version that you're updating to. However, you can specify only the port numbers for the new service. You cannot specify the network type (that is, internal or external) for the service to use. Each new service gets the default network type, which is determined by the service itself. |
| ENS-7962 (FNDD-570) | System update | Uploading an update package fails after the failure and recovery of a system instance If a system instance enters the state Down, when you try to upload an update package the upload fails. However, after the system instance recovers, when you try again to upload an update package, the upload fails again even though the system is in a healthy state. Workaround
|
| ENS-7964 (FNDD-15) | Volumes | Volume configuration is not displayed correctly in System Management application During installation, you can configure volumes for system services by specifying different values in the volume.config file on each system instance. Each volume is correctly configured with the settings you specify, but the Monitoring > Services > Service Details page in the System Management application incorrectly shows each volume as having identical configurations. |
| ENS-8299 (FNDD-545) | Service deployment | Database service fails to deploy The Cassandra service can fail to deploy with the error Could not contact node over JMX. The log file on the node running the service instance includes the following entry: java.lang.RuntimeException: A node required to move the data consistently is down (/nnn.nnn.nnn.nnn). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false Workaround
|
| ENS-10750 (FNDD-19) | Updates | Update volume prechecks not performed Validation of volume configuration values is not honored by the upgrade process. As a result, configuration values passed to Docker are not valid. Workaround Use caution when specifying volume values. |
| FNDD-970 | MAPI | If IdP is unavailable, threads blocked HCP for cloud scale uses a System Management function to validate tokens. The function does not time out. If the identity provider is unavailable, the requesting thread is blocked. |
S3 Console
The following table lists known issues affecting the S3 Console application.
| Issue | Area affected | Description |
| ASP-9373 | S3 Console | No message confirming bucket creation When a bucket is created, a confirmation message should be displayed. No message is displayed. Workaround After creating a bucket, list the buckets to verify that the bucket was created. |
Related documents
This is the set of documents supporting v2.3.0 of HCP for cloud scale. You should have these documents available before using the product.
- Hitachi Content Platform for Cloud Scale Release Notes (RN‑HCPCS004‑18): This document is for customers and describes new features, product documentation, and resolved and known issues, and provides other useful information about this release of the product.
- Installing Hitachi Content Platform for Cloud Scale (MK‑HCPCS002‑10): This document gives you the information you need to install or update the HCP for cloud scale software.
- Hitachi Content Platform for Cloud Scale Administration Guide (MK‑HCPCS008-06): This document explains how to use the HCP for cloud scale applications to configure and operate a common object storage interface for clients to interact with; configure HCP for cloud scale for your users; enable and disable system features; and monitor the system and its connections.
- Hitachi Content Platform for Cloud Scale S3 Console Guide (MK‑HCPCS009-03): This document is for end users and explains how to use the HCP for cloud scale S3 Console application to use S3 credentials and to simplify the process of creating, monitoring, and maintaining S3 buckets and the objects they contain.
- Hitachi Content Platform for Cloud Scale Management API Reference (MK‑HCPCS007‑07): This document is for customers and describes the management application programming interface (API) methods available for customer use.
Documentation corrections
The following issues were identified with the documentation, including the online help, after its publication.
Installation Guide
The following refers to the Installation Guide.
In the module "System requirements and sizing," in the topic "Operating system and Docker qualified versions," in the table: remove the note "Technical support not available for Docker Community Edition."
Online help, Administration Guide
The following refers to the online help available in the Object Storage Management application profile menu under Help as well as to the Administration Guide.
In the module "System management," in the topic "Instances" > "Requirements for running system instances" > "Operating system and Docker qualified versions," in the table: remove the note "Technical support not available for Docker Community Edition."
Management API reference information
The following refers to the management API reference information available in the Object Storage Management application profile menu under REST API.
The information describes endpoints related to jobs. Jobs are not supported in this release.