Skip to main content

We've Moved!

Product Documentation has moved to docs.hitachivantara.com
Hitachi Vantara Knowledge

Hitachi Content Platform for Cloud Scale v2.5.2 Release Notes

 

About this document

 

This document gives late-breaking information about HCP for cloud scale v2.5.2. It includes information that was not available at the time the technical documentation for this product was published, a list of new features, a list of resolved issues, and a list of known issues and where applicable their workarounds.

Intended audience

 

This document is intended for customers and Hitachi Vantara partners who license and use HCP for cloud scale.

Getting help

 

The Hitachi Vantara Support Website is the destination for technical support of products and solutions sold by Hitachi Vantara. To contact technical support, log on to the Hitachi Vantara Support Website for contact information: https://support.hitachivantara.com/en_us/contact-us.html.

Hitachi Vantara Community is a global online community for Hitachi Vantara customers, partners, independent software vendors, employees, and prospects. It is the destination to get answers, discover insights, and make connections. Join the conversation today! Go to community.hitachivantara.com, register, and complete your profile.

About this release

 

This is build 2.5.2.3 of the Hitachi Content Platform for cloud scale (HCP for cloud scale) software.

For this release

 

Features in v2.5.2

 
Resolved issues with

The HCP for cloud scale v2.5.2 release resolves an issue identified during an upgrade to v.2.5.1, where certain system services fail to deploy.

For more information, see Resolved issues.

Best practices regarding unauthenticated ports

A notice to the Cloud Scale customers:

The goal of this notice is to highlight and clarify the existing best practice for HCP for cloud scale in regards to the two unauthenticated ports in the product.

HCP for cloud scale has two unauthenticated ports: one for the Prometheus/Metrics UI (port 9191) and one for the Jaeger/Tracing UI (port 16686). These ports are used primarily for troubleshooting and do not expose sensitive data. Nevertheless, they should not be made available to your users and must be blocked in your network.

To learn more, view the recommendations as stated in the HCP for Cloud Scale Administration Guide.

System requirements

 

This section lists the hardware, networking, and operating system requirements for running an HCP for cloud scale system with one or more instances.

Hardware requirements

 

To install HCP for cloud scale on on-premises hardware for production use, you must provision at least four instances (nodes) with sufficient CPU, RAM, disk space, and networking capabilities. This table shows the hardware resources required for each instance of an HCP for cloud scale system for a minimum qualified configuration and a standard qualified configuration.

Resource

Minimum configuration

Standard configuration

CPU

Single CPU, 10-core

Dual CPU, 20+-core

RAM

128 GB

256 GB

Available disk space

(4) 1.92 TB SSD, RAID10

(8) 1.92 TB SSD, RAID10

Network interface controller (NIC) (2) 10 GB Ethernet NICs (2) 25 GB Ethernet NICs or

(4) 10 GB Ethernet NICs

ImportantEach instance uses all available RAM and CPU resources on the server or virtual machine on which it's installed.

Software requirements

 

The following table shows the minimum requirements and best-practice software configurations for each instance in an HCP for cloud scale system.

Resource Minimum Best
IP addresses (1) static (2) static
Firewall Port Access Port 443 for SSL traffic

Port 8000 for System Management App GUI

Port 8888 for Content Search App GUI

Same
Network Time IP address of time service (NTP) Same

Operating system and Docker minimum requirements

 

Each server or virtual machine you provide must have the following:

  • 64-bit Linux distribution
  • Docker version installed: Docker Community Edition 18.09.0 or later
  • IP and DNS addresses configured

Additionally, you should install all relevant patches on the operating system and perform appropriate security hardening tasks.

ImportantThe system cannot run with Docker versions before 1.13.1.

To execute scripts provided with the product on RHEL, you should install Python.

Operating system and Docker qualified versions

 

This table shows the operating system, Docker, and SELinux configurations with which the HCP for cloud scale system has been qualified.

ImportantAn issue in Docker Enterprise Edition 19.03.15 and resolved in 20.10.5 prevented HCP for cloud scale deployment. Do not install any version of Docker Enterprise Edition above 19.03.14 and below 20.10.5.
Operating system Docker version Docker storage configuration SELinux setting
Red Hat Enterprise Linux 8.4 Docker Community Edition 19.03.12 or later overlay2 Enforcing

If you are installing on Amazon Linux, before deployment, edit the file /etc/security/limits.conf on every node to add the following two lines:

*  hard  nofile  65535
*  soft  nofile  65535

Docker considerations

 

The Docker installation folder on each instance must have at least 20 GB available for storing the HCP for cloud scale Docker images.

Make sure that the Docker storage driver is configured correctly on each instance before installing HCP for cloud scale. To view the current Docker storage driver on an instance, run docker info.

NoteAfter installation, changing the Docker storage driver requires a reinstallation of HCP for cloud scale.

If you are using the Docker devicemapper storage driver:

  • Make sure that there's at least 40 GB of Docker metadata storage space available on each instance. HCP for cloud scale needs 20 GB to install successfully and an additional 20 GB to successfully upgrade to a later version. To view Docker metadata storage usage on an instance, run docker info.
  • On a production system, do not run devicemapper in loop-lvm mode. This can cause slow performance or, on certain Linux distributions, HCP for cloud scale might not have enough space to run.

SELinux considerations

 

You should decide whether you want to run SELinux on system instances and enable or disable it before installing HCP for cloud scale. To enable or disable SELinux on an instance, you must restart the instance. To view whether SELinux is enabled on an instance, run: sestatus

To enable SELinux on the system instances, use a Docker storage driver that supports it. The storage drivers that SELinux supports differ depending on the Linux distribution you're using. For more information, see the Docker documentation.

Time source requirements

 

If you are installing a multi-instance system, each instance should run NTP (network time protocol) and use the same external time source. For information, see support.ntp.org.

Supported browsers

 

The following browsers are qualified for use with HCP for cloud scale software. Other browsers or versions might also work.

  • Google Chrome (latest version as of the date of this publication)
  • Microsoft Edge (latest version as of the date of this publication)
  • Mozilla Firefox (latest version as of the date of this publication)

Installation or upgrade considerations

 

This section provides information about installing or upgrading HCP for cloud scale software.

Upgrades from versions before v2.4.0

Upgrades from versions earlier than v2.4.0 directly to v2.5.2 are not supported. If your HCP for cloud scale software is at version 2.3.4 or earlier, you must upgrade at least to version 2.4.0 before you can upgrade to v2.5.2.

Upgrading systems with synch-to policies

In v2.5.2, only synch-to policies with single destinations are supported.

An upgrade pre-check examines all buckets with synch-to policies. If a bucket has a synch-to policy with multiple destinations the upgrade does not proceed and the owner of the bucket is notified.

Upgrading from v2.4.x to v2.5.2

Before beginning an upgrade, increase the Max Heap Size value for the Metadata-Gateway service by about one third. For example, if the Max Heap Size for Metadata-Gateway is 48000m (48GB), then change it to 64000m (64GB), a 33% increase. To increase the value:

  1. In the System Management application, select Dashboard > Services.
  2. Select Metadata-Gateway.
  3. On the Configuration tab, increase the Max Heap Size value as needed.
  4. Click Update.

Before beginning an upgrade, ensure that the heap size for the Sentinel service is 8 GiB (which you enter as 8g or 8192m). If the value is too small, upgrade pre-check fails and reports a Sentinel service error.

If you very recently upgraded to HCP for cloud scale v2.3.x, your system might still be undergoing table migration for consistent listing, which was a major feature in the v2.3 upgrade. Until this migration has completed, you cannot upgrade to v2.5.2 (or any later version). The upgrade software checks the migration state and will not start while migration is ongoing. In this case you must wait for migration to finish and then restart the upgrade to v2.5.2. Depending on the number of objects in the system, table migration can take anywhere from a day to about a week.

You can monitor the progress of migration using the metric deprecated_metadata_clientobject_active_count in Prometheus. This metric gives the count of objects in deprecated partitions. It decrements during the migration, reaching 0 when all objects are migrated. You can use this metric to estimate of the time remaining, though even after the value reaches 0 there will still be some cleanup.

Upgrade if DARE is enabled

If data-at-rest encryption (DARE) is enabled, during an upgrade you must monitor the process, detect when the Key Management Server service restarts, and then unseal the vault. S3 traffic is blocked until the vault is unsealed.

The KMS service is the last service restarted, and you can monitor the progress in the System Management application. An event is logged when the service restarts (8007, Update Completed) and you can configure email notification.

As soon as the KMS service is upgraded, unseal the vault. (You can keep the unseal keys at hand to enter them immediately.)

After upgrade is completed, users see new behaviors:

  • S3 API GET and HEAD responses include an --x-amz-server-side-encryption header with all existing encrypted objects.
  • All existing S3 buckets reflect an active encryption bucket policy, which is verifiable using the GetBucketEncryption API.
  • No S3 user can read (GET) the SSE-S3 bucket policy unless the administrator explicitly adds that permission to the S3 user role.
  • No S3 user can delete the SSE-S3 bucket policy unless the administrator explicitly adds that permission to the S3 user role.
  • No S3 user can add the SSE-S3 bucket policy to new buckets unless the administrator explicitly adds that permission to the S3 user role.
  • All S3 users can electively include --x-amz-server-side-encryption headers for any S3 creation operation (PUT, COPY, or Initiate MPU)
  • Encryption and decryption can continue until the Key-Management-Server service is restarted. After the service restarts, the key management server is consulted and must be unsealed to recommence encryption and decryption.
Dashboard modifications for upgrades from v2.4.x to v2.5.1

When upgrading from HCP for cloud scale v2.4.x to v2.5.1, you might need to modify the firewall rules on member nodes to access the features of the Dashboard service on port 3000. Use the Linux firewall tool appropriate for your installation to make the modification. The following is an example:

sudo firewall-cmd --zone=public --add-port=3000/tcp --permanent 
sudo firewall-cmd --reload
Best practice for reconfigurations during upgrades

As a best practice, major cluster-level reconfigurations should be postponed until an HCP for cloud scale upgrade completes. The following system administration activities should not be run during an upgrade:

  • Changing system certificates
  • Changing LDAP/AD groups, roles, or permissions
  • Adding storage components
  • Enabling encryption, adding/deleting KMS servers, or initiating encryption rekeys
  • Changing, scaling, or restarting microservices
  • Adding or deleting node instances
Proxy users should not upgrade to v2.5.1

If your HCP for cloud scale uses a proxy to communicate with storage components, upgrading to v2.5.1 will cause loss of communication between the proxy and storage components. Because of this, proxy users should not upgrade to v2.5.1. This is a known issue that will be addressed in a future release.

Scaling the new Mirror In, Mirror Out, and S3 Notification services (optional)

During an upgrade to HCP for cloud scale v2.5.x, Policy Engine (PE) functions are split into four individual services:

  • Mirror In (sync-from replication) - new
  • Mirror Out (sync-to replication) - new
  • S3 Notification (event notification) - new
  • Policy Engine (policy routing only) - existing

After an upgrade to v2.5.x, PE functions continue to run because this service remains scaled to the number of instances that were configured prior to the upgrade. However, the functions that have been split into the three new services are no longer executed. These services will not be scaled but will be set to "0" instances by default.

After the upgrade finishes, the new services can be scaled to the desired number of instances through the System Management application, under Services Add Services. The new services do not appear in the matrix of services until they have been scaled to 1 or more instances.

An automatic scaling script is available upon request from the Hitachi Vantara Global Support Center. Customers who are already using any of the three new functions and want to minimize interruption should start this script prior to upgrading to v2.5.x.

Restarting the Sentinel service after a failed update

In some rare cases, your software upgrade may fail. Prior to escalating to Hitachi Vantara support, you need to restart (repair) the Sentinel service twice from within the System Management application in an attempt to resolve the issue.

NoteA failed update will not impact the operation of your system.
  1. Log in to the System Management application.
  2. Click Services > Sentinel > Repair.
  3. Once completed, click Repair a second time.
  4. Switch to the Update page and click Retry to continue the upgrade process.

Best practices for managing HCP S Series Nodes

 

For a system that includes HCP S Series Node storage components, the best practice is to define a separate user account on them for exclusive use by HCP for cloud scale.

If you have already configured HCP for cloud scale to use an account from an HCP S Series Node that is typically used for its management (that is, an administrative account), and then change the account’s credentials for security purposes, then HCP for cloud scale cannot communicate with the HCP S Series Node, most importantly for data access, which leads to a disruption of service.

You set up accounts when configuring a new storage component. If you have already defined storage components with single accounts, the best practice is to add another, exclusive account on each HCP S Series Node and change the HCP for cloud scale configuration to use that account.

NoteReplacing an account for an existing storage component might cause a data path disruption of a few seconds as the account is defined.

Resolved issues

 

The following HCP for cloud scale issues are resolved in this release.

Object storage management

 

The following table lists resolved issues affecting object storage management.

Issue Area affected Description
ASP-14244 Services After upgrade, some system services fail to deploy.

After upgrading a system to v2.5.1, system services fail to deploy due to the latest S3 Select version bringing forward an incompatible version of Apache Zookeeper.

Resolution

This issue is resolved and system services now deploy correctly.

Known issues

 

The following issues with HCP for cloud scale have been identified in this release.

Object storage management

 

The following table lists known issues affecting object storage management.

Issue Area affected Description
ASP-2422 Tracing Agent Incorrect alert message during manual deployment

When manually deploying a four-node, multi-instance system, the Tracing Agent service returns an alert that the service is below the needed instance count even when the correct number of service instances are deployed.

Workaround

If you have deployed the correct number of instances you can safely ignore this alert.

ASP-3119 MAPI Gateway Blocked thread on authorization timeout

Authentication and authorization use a system management authorization client service which has a different timeout interval. If a management API authorization or authentication request times out but the underlying client service doesn't, the thread is blocked.

Workaround

Stop and restart the MAPI Gateway service container.

ASP-3170 MAPI Gateway Certain API methods are public

The MAPI schema includes public API methods, which do not need OAuth tokens.

Workaround

None needed. The public API methods do not need OAuth tokens.

ASP-11259 S3 API No eTag validation on parts of mirrored multi-part uploads

MD5/eTag validation is not performed on mirrorUploadPart operations.

Workaround

Use transport-layer security (TLS) between endpoints for mirrored operations.

System management

 

The following table lists known issues affecting system management.

Issue Area affected Description
ASP-3379 Configuration Cannot set refresh token timeout value

The Refresh Token Timeout configuration value in the System Management application (Configuration > Security > Settings) has no effect.

ASP-9433 System update Upgrade can fail and cannot be completed

Some conditions that cause an upgrade to fail the pre-check (for example, if the system was improperly prepared for upgrade) render the upgrade unable to complete even if the underlying issue is fixed. Subsequent attempts fail with the message "The upgrade has failed! Click retry to attempt the update again." The detail message is "Wait for service tracing-Query on any instance to start," which is misleading because the specified service is not the cause of the error.

Workaround

If retrying the upgrade fails even after the underlying cause is fixed, contact the Hitachi Vantara Global Support Center for assistance with preparing and completing the upgrade.

ASP-11695 System update Upgrade to v2.4 hangs if multiple buckets have synch-to policies with multiple destinations

The v2.4 upgrade pre-check detects if a user bucket has a synch-to policy with multiple destinations, which is not supported in v2.4, and the upgrade does not proceed. However, if multiple buckets have synch-to policies with multiple destinations, the upgrade pre-check can hang in a checking loop.

If this happens, in the System Management application, on the Update page, the Status tab shows the status "Running_prechecks" but the Install tab displays "Update cluster - Error" with 22 steps completed.

In this state the View Details and Retry buttons are available only momentarily during each checking cycle.

Workaround

  1. Determine which buckets violate the pre-check conditions. In the System Management application, do one of the following:
    • If a browser has a high-latency connection to the HCP for cloud scale system, the View Details button might be visible long enough to click it. This option is preferable because the details are more readable and list which buckets need correction, and because once details are displayed the Retry button becomes available, which starts another update attempt without the need to locate and stop the Sentinel service.
    • If the first option is not available, check the Sentinel service to determine the node on which the service instance is running. Examine the Sentinel service log.
  2. If examining logs, look for an update error of the form:
    'com.hds.ensemble.plugin.update.UpdateOperationFailedException: Mirror configuration for the following buckets 
    contained external configurations: [bucket1, bucket2, bucket3]' error
  3. Reconfigure the synch-from policies on the buckets.
  4. If the Retry button is available, click it now; otherwise, enter the following command to stop the Sentinel service on the node, which triggers the service to restart:
    /docker stop sentinel-service
ASP-12217 System update Loss of proxy communication after upgrading to v2.5.0

If your HCP for cloud scale uses a proxy to communicate with storage components, upgrading to v2.5.0 will cause loss of communication between the proxy and storage components. Because of this, proxy users should not upgrade to v2.5.0. This is a known issue that will be addressed in a future release.

Workaround

Proxy users should not upgrade to HCP for cloud scale v2.5.0. This issue will be addressed in a future release.

ASP-12843 System update New Mirror In, Mirror Out, and S3-Notification services have no instances after upgrading to v2.5.0

During an upgrade to HCP for cloud scale v2.5.0, the Mirror In, Mirror Out , and S3 Notification services are set to "0" instances by default.

Workaround

The three new services are only required if the following functions are used:

  • Sync-to replication – requires the Mirror Out service
  • Sync-from replication – requires the Mirror In service
  • Event notification – requires the S3 Notification service

If none of these functions are currently in use or are planned to be used, then these services do not need to be scaled.

If one or more of these functions are in use or are planned to be used, see Scaling the new Mirror In, Mirror Out, and S3 Notification services (optional) in the Installation or upgrade considerations section of this document.

FNDO-373 Volumes Volume configuration is not displayed correctly in System Management application

During installation, you can configure volumes for system services by specifying different values in the volume.config file on each system instance. Each volume is correctly configured with the settings you specify, but the Monitoring > Services > Service Details page in the System Management application incorrectly shows each volume as having identical configurations.

FNDO-512 System update Network types cannot be configured for new services before system update

Before starting an update, you are prompted to specify the network configuration for any new services included in the version that you're updating to. However, you can specify only the port numbers for the new service. You cannot specify the network type (that is, internal or external) for the service to use. Each new service gets the default network type, which is determined by the service itself.

FNDO-758 MAPI If IdP is unavailable, threads blocked

HCP for cloud scale uses a System Management function to validate tokens. The function does not time out. If the identity provider is unavailable, the requesting thread is blocked.

FNDO-931 Updates Update volume prechecks not performed

Validation of volume configuration values is not honored by the upgrade process. As a result, configuration values passed to Docker are not valid.

Workaround

Use caution when specifying volume values.

FNDO-1029 System update Uploading an update package fails after the failure and recovery of a system instance

If a system instance enters the state Down, when you try to upload an update package the upload fails. However, after the system instance recovers, when you try again to upload an update package, the upload fails again even though the system is in a healthy state.

Workaround

  1. In the System Management application, go to the Monitoring > Processes page and for the Upload Plugin Bundle task click Retry Task.
  2. Upload the update package again.
FNDO-1062 Service deployment Database service fails to deploy

The Cassandra service can fail to deploy with the error Could not contact node over JMX. The log file on the node running the service instance includes the following entry: java.lang.RuntimeException: A node required to move the data consistently is down (/nnn.nnn.nnn.nnn). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false

Workaround

  1. Restart the Cassandra container running on that node.
  2. Redeploy the service.

S3 Console

 

The following table lists known issues affecting the S3 Console application.

Issue Area affected Description
ASP-13224 S3 Console Misleading message if KEK cannot be retrieved

If a user tries to download a file on a storage component whose key encryption key cannot be retrieved, the S3 Console displays a page with the message "This page isn't working" and an HTTP 503 (service unavailable) error. (In some browsers the page is entirely blank.)

Workaround

Investigate and resolve the missing-key issue.

ASP-11600 S3 Console Enabling object lock without permission fails with briefly displayed error message

If a user without permission to set object locks creates a bucket and tries to enable object locking, the bucket is created and object locking is not enabled, and a message briefly appears indicating that object locking was not enabled.

Workaround

Users need specific permissions to set object locking on their bucket. In the System Management application, select Configuration Security Roles and set the Data Service permissions data:bucket:objectlock:get and data:bucket:objectlock:set to Yes for the user's role.

If a user needs to configure both bucket object lock and bucket expiration lifecycle rules, assign the following permissions for the user's role:

  • data:bucket:objectlock:get
  • data:bucket:objectlock:set
  • data:bucket:expirationlifecycle:get
  • data:bucket:expirationlifecycle:set

Related documents

 

The following is a list of supporting documentation for HCP for cloud scale:

  • Hitachi Content Platform for Cloud Scale Release Notes: This document is for customers and describes new features, product documentation, and resolved and known issues, and provides other useful information about this release of the product.
  • Installing Hitachi Content Platform for Cloud Scale: This document gives you the information you need to install or upgrade the HCP for cloud scale software.
  • Hitachi Content Platform for Cloud Scale Administration Guide: This document explains how to use the HCP for cloud scale applications to configure and operate a common object storage interface for clients to interact with; configure HCP for cloud scale for your users; enable and disable system features; and monitor the system and its connections.
  • Hitachi Content Platform for Cloud Scale S3 Console Guide: This document is for end users and explains how to use the HCP for cloud scale S3 Console application to use S3 credentials and to simplify the process of creating, monitoring, and maintaining S3 buckets and the objects they contain.
  • Hitachi Content Platform for Cloud Scale Management API Reference: This document is for customers and describes the management application programming interface (API) methods available for customer use.