Streaming Data Platform 3.2.0-02 Release Notes
About this document
This document (RN-93HSDP005-11, November 2018) provides late-breaking information about the Hitachi Streaming Data Platform (HSDP) version 3.2.0-02. It includes information that was not available at the time the technical documentation for this product was published, as well as a list of known problems and solutions.
Intended audience
This document is intended for customers and Hitachi Vantara partners who license and use the Hitachi Streaming Data Platform.
Accessing product downloads
Product software, drivers, and firmware downloads are available on Hitachi Vantara Support Connect: https://support.hitachivantara.com/.
Log in and select Product Downloads to access the most current downloads, including important updates that may have been made after the release of the product.
Product package contents
See Available documents for the documentation provided for this release.
| Medium | DVD-ROM | Revision | Release Type |
| Software | Hitachi Streaming Data Platform | 3.2.0-02 | Full Package |
New features and important enhancements
· HSDP now supports tuple log.
· HSDP now provides a mechanism by which the external adapter of the data input source is notified of any data clogging in stream data processing.
· HSDP now allows you to specify unlimited number of times to check output stream queue for available space. This is done by specifying the querygroup.sleepOnOverStoreRetryCount property in the system-configuration or query-group properties file.
· The TCP input adapter and the cascade adapter now retry inputting data when data clogging occurs.
System requirements
Operating systems
| Operating Systems* | Version | Architecture |
| Red Hat Enterprise Linux Server | 6.5 or later | EM64T |
| AMD64 | ||
| 7.1 or later | EM64T | |
| AMD64 | ||
| SUSE Linux Enterprise Server | 11 SP2 | EM64T |
| AMD64 | ||
| 11 SP3 | EM64T | |
| AMD64 | ||
| 12 | EM64T | |
| AMD64 |
(*) Also runs under Kernel-based Virtual Machine (KVM) on the same OS and version.
Virtualization software
| Virtualization software | Version | Update | Architecture |
| VMware vSphere® ESXi™ | 5.1 | 1 | EM64T |
| AMD64 | |||
| 5.5 | 1 | EM64T | |
| AMD64 | |||
| 6.0 | None | EM64T | |
| AMD64 | |||
| 6.5 | None | EM64T | |
| AMD64 |
Prerequisite programs
The following packages for Red Hat Enterprise Linux (64-bit x86_64) must be installed:
· gdb (x86_64)
· glibc (i686)
· glibc (x86_64)
· glibc-common (x86_64)
· glibc-devel (i686)
· glibc-devel (x86_64)
· glibc-headers (x86_64)
· glibc-utils (x86_64)
· libgcc (i686)
· libstdc++ (i686)
· ncompress (x86_64)
· nscd (x86_64)
· openmotif (x86_64)
· tar (x86_64)
· To use SNMP Traps, install the following packages in Red Hat Linux:
· net-snmp
· net-snmp-utils
· net-snmp-libs
To the RTView custom data adapter, a RTView version between 6.4.0.0 and 6.6.0.0 is required on the server on which the RTView custom data adapter is deployed.
Memory and disk space requirements
Disk Space Requirements
The disk space requirements shown here apply when HSDP runs in the following configuration. The values specified in configuration files, including the maximum log file count and the maximum size, are all assumed to be the default values.
· Installation directory
· Working directory (1 directory)
· Log files to which the following components output log data, and other files necessary to perform processing.
· - TCP data input adaptor (1 adaptor)
· - Cascading adaptor (1 adaptor)
· - SDP manager
· - SDP broker
· - SDP coordinator
· - External adaptor
Disk space requirements: 2,143 MB
Memory Requirements
The minimum memory requirements shown here apply when sample queries are registered to the SDP server and an external input adaptor and external output adaptor are used to build the analytic system. (For information about sample queries, see "Example of related external adaptor".) Values such as queue size and buffer size for the SDP server and each adaptor are assumed to be the default values.
SDP server: 475MB
SDP broker: 156MB
SDP manager: 158MB
SDP coordinator: 1039MB
External adaptor 298MB
The memory requirements for each component of HSDP might also increase, depending on the following factors.
· SDP server, external adaptor:
· Whether the specified queue size and buffer size values for the SDP server and external or internal adaptors are greater than the default values
· The number of registered query groups
· The number of streams in a query
· The data size of a tuple (This depends on the number of columns and the data type.)
· Whether the tuple log function is enabled by specifying engine.useTupleLog=true
· Whether the timestamp-adjustment function is enabled
·
· SDP coordinator:
· The number of server clusters in the stream data processing system.
·
Resolved problems
| # | Corrected Problems | Applicable products | Applicable OS |
| #1 | The SDP coordinator will not start when upgrading to Hitachi Streaming Data Platform 3.0.0 or Hitachi Streaming Data Platform 3.0.0-01 without uninstalling. | Hitachi Streaming Data Platform | Red Hat Enterprise Linux Server SUSE Linux Enterprise Server |
| #2 | A TCP data input adapter could fail to receive control tuple data. | Hitachi Streaming Data Platform | Red Hat Enterprise Linux Server SUSE Linux Enterprise Server |
| #3 | When the data source mode is specified for the timestamp mode and the time adjustment function is enabled, data sent from the data send callback to an input stream will cause an overflow in the queue input stream, which will result in discarding of the data. | Hitachi Streaming Data Platform | Red Hat Enterprise Linux Server SUSE Linux Enterprise Server |
| #4 | The cascading adaptor will halt if UTF-16 (or UTF-16BE or UTF-16LE) is specified as the character encoding and then data is transmitted by specifying the type BigDecimal as an argument of the control tuple. | Hitachi Streaming Data Platform | Red Hat Enterprise Linux Server SUSE Linux Enterprise Server |
| #5 | A query group that includes a specific analysis scenario cannot be registered. | Hitachi Streaming Data Platform | Red Hat Enterprise Linux Server SUSE Linux Enterprise Server |
| #6 | The external output adaptor cannot receive data of type CHAR if US-ASCII is specified as the character encoding. | Hitachi Streaming Data Platform | Red Hat Enterprise Linux Server SUSE Linux Enterprise Server |
| #7 | The SDP coordinator might not be able to start or stop when the command | Hitachi Streaming Data Platform | Red Hat Enterprise Linux Server SUSE Linux Enterprise Server |
| #8 | The default value of | Hitachi Streaming Data Platform | Red Hat Enterprise Linux Server SUSE Linux Enterprise Server |
| #9 | A query group cannot be registered to the sender SDP server when both of the following conditions apply: data is sent to another SDP server by using a cascading adaptor for automatic startup, and different query group names are specified for different working directories in the system. | Hitachi Streaming Data Platform | Red Hat Enterprise Linux Server SUSE Linux Enterprise Server |
Known problems
1) HSDP operation is not guaranteed if an invalid property value is specified in the query group property file, the SDP manager definition file, and the external adaptor definition file, or if any of the HSDP files are named with invalid characters.
2)
3) HSDP operation is not guaranteed if the command hsdpstop is run while the internal adapter is running. When stopping HSDP, be sure to run the necessary commands according to the following procedure.
4)
5) If automatically generated adaptors are running:
6) 1. hsdpcqlstop <query group name>
7) 2. hsdpcqldel <query group name>
8) 3. hsdpstop
9)
10) If internal standard adaptors or custom data adaptors are running, and automatically generated adaptors are not used:
11)1. hsdpstopinpro <adaptor group name>
12)2. hsdpcqlstop <query group name>
13)3. hsdpcqldel <query group name>
14) 4. hsdpstop
Installing HSDP 3.2.0-02
Installation
(1) Log on to the installation-target machine as a root user.
(2) Mount the downloaded HSDP disk image as a directory.
(3) Move to the directory where the HSDP installer is placed:
# cd <CD-ROM-mount-directory>/HSDP/
(4) Run the following command:
# install.sh
or
# install_runtime.sh
When the installation is complete, HSDP is installed in the following directory:
/opt/hitachi/hsdp/
Uninstallation
Prerequisites
(1) All SDP servers must be stopped.
In each working directory, run the hsdpstatusshow command with no option specified, and verify that each SDP server is stopped.
(2) The SDP manager must be stopped.
Run the hsdpstatusshow command with -mgr option specified, and then verify that the SDP manager has stopped.
Procedure
(1) Log on to the target machine where you want to uninstall the software as a root user.
(2) Move to a directory other than /opt/hitachi/hsdp/.
(3) Run the following command:
# /opt/hitachi/hsdp/uninstall/uninstall.sh
Port numbers
HSDP uses 8 consecutive ports, from 20425 to 20432. To use ports from p to p+7, where p is the initial port number, specify p by using the –port option of the hsdpsetup command. (hsdpsetup –port p ...).
The following table shows the port numbers used by HSDP depending on the values specified for the parameters in the manager configuration file (hsdp_setup_manager.cfg).
| Default port |
|
| ||
| Number of SDP coordinators in the coordinator group | When connecting to the coordinator group of another host | - | ||
| 1-2 | 3 or more | - | - | |
| 20425 | Y | Y | Y | N |
| 20426(*) | Y | Y | Y | Y |
| 20427 | Y | Y | Y | N |
| 20428 | N | Y | N | N |
| 20429 | N | Y | N | N |
| 20430 | N | Y | N | N |
| 20431 | Y | Y | Y | N |
| 20432 | N | y | N | N |
Y: Used N: Not used *Port 20426 is reserved for HSDP.
To use the dashboard output connector, specify the dashboardPortNo attribute of either
InprocessGroupDefinition tag or RMIGroupDefinition tag in the adaptor composition definition file, or 20421 by default.
When using a TCP data input adaptor, use the following port number.
- If the SDP broker is used:
For the receiving port, use the same port number that is used by the SDP broker (Default: 20425).
- If the SDP broker is not used:
For the receiving port, use the value specified for the port attribute of the tcpcon:input tag in the definition of the TCP data input connector in the adapter configuration definition file.
When using the cascading adaptor, sending ports are automatically assigned from the ephemeral port range*.
When using an external adaptor, use the following port number.
- For the port number for the external input adaptor:
Sending ports are automatically assigned from the ephemeral port range*
- For the port number for the TCP data input adaptor that connects with the external input adaptor:
For the receiving port, use the same port number that is used for the SDP broker (Default: 20425).
- For the port number for the external output adaptor:
Receiving ports are automatically assigned from the ephemeral port range*
- For the port number for the cascading adaptor that connects with the external output adaptor:
For the sending port, use the same port number that is used for the SDP broker (Default: 20425).
* Specify the ephemeral port range in the following OS settings file.
Redhat Enterprise Linux Server: /proc/sys/net/ipv4/ip_local_port_range
Example of related external adapters
Prerequisites
(1) The SDP broker and SDP coordinator are running (use the hsdpmanager -start command).
(2) A working directory is created (use the hsdpsetup -dir command).
Procedure
(1) To start an SDP server, run the following command in the working directory:
# working-directory/bin/hsdpstart
(2) To register a query group, run the following command in the working directory:
# working-directory/bin/hsdpcql -start QueryGroupTest
(3) To start an external output adapter, run the following command in the working directory:
# working-directory/exadaptor/outputadaptor/exadpstart_output.sh
(4) To start an external input adapter, run the following command in the working directory:
# working-directory/exadaptor/inputadaptor/exadpstart_input.sh
Results
The external input adapter reads the following csv file using UTF-8 encoding and sends each line data to the SDP server:
working-directory/exadaptor/inputadaptor/input.csv
The external output adapter gets data from the SDP server and outputs received data to the following csv file:
working-directory/exadaptor/outputadaptor/output.csv
Following the above process both external adapters stop automatically. To stop the SDP server, run the following commands in the working directory, in the following order:
# working-directory/bin/hsdpcqlstop
# working-directory/bin/hsdpcqldel
# working-directory/bin/hsdpstop
Notes
When HSDP SDK is installed, source files of both external adapters are stored in the following directories:
/opt/hitachi/hsdp/sdk/samples/exadaptor/inputadaptor/src/ExternalInputAdaptor.java
/opt/hitachi/hsdp/sdk/samples/exadaptor/outputadaptor/src/ExternalOutputAdaptor.java
Usage precautions
Notes on the virtualization platform environment
HSDP is designed to run on a standard or non-virtual machine. However, you may choose to deploy it on a virtual machine. Although the software has gone through rigorous testing, running in a virtualized environment can lead to performance issues that maybe a result of virtualization of hardware resources and shared I/O. In addition, the functions that use CPU time might not work properly.
For this reason, a full operation evaluation and performance verification of the system is necessary. Perform checks after deployment, and confirm that HSDP uses appropriate hardware, virtualization software, guest OS, and applications.
Notes on host name setting
Set a loopback address (127.0.0.1) as the localhost on a server that HSDP is installed on.
Notes on IP address setting
Set a ipv4 address on a host that HSDP is installed on.
· To specify the following parameters and command arguments as the IP address format,
· specify them as IPv4 format.
·
Adaptor-configuration definition file:
· brokerAddr attribute of the cascading client connector definition
· host attribute of the cascading properties file
·
· External adaptor configuration file:
· hsdp.broker.address
· hsdp.target.name.<n>
·
· hsdpsetup command:
· -mgr -host
· -mgr -chosts
· -mgr -cmaddr
·
· Query-group properties file:
· stream.output.<streamName>.link
Notes on cascading adapter communication method
Use of TCP socket as the communication method is recommended.
To use Java RMI, restrict connections to destination port numbers of cascading adapters from all clients other than cascading adapters.
Notes on performance tuning parameters
The following parameters affect performance of the HSDP data processing. Modify parameter values based on the HSDP system throughput. An improperly configured system could result in performance deoptimization and resource waste.
Adapter-configuration definition file:
· bufferSize attribute of the TCP data input connector definition
· receiveBufferSize attribute of the TCP data input connector definition
· sendBufferSize of the cascading client connector definition
·
· System-configuration properties file:
· engine.initialQueueSize
· engine.maxQueueSize
· engine.initialEngineQueueSize
· engine.maxEngineQueueSize
·
· External adapter-definition file:
· hsdp.tcp.sendbuffersize
· hsdp.tcp.receivebuffersize
Notes on the status of the custom adapter
The statuses of the custom adapter resulting from execution of the hsdpstatusshow command are shown below. As shown below, the status changes depending on how the execute method is implemented and on the execution results.
· Running: The custom adapter is running.
This is displayed during processing of the execute method of the StreamInprocessUP interface.
· Stopped: The custom adapter is stopped.
This is displayed after the execute method of the StreamInprocessUP interface is executed and finishes normally.
· Abnormality: The custom adapter is in an abnormal state.
This is displayed after the execute method of the StreamInprocessUP interface is executed but finishes abnormally (for example, when a RuntimeException occurs).
Notes on upgrading HSDP
When an overwrite installation is performed, the setting values are inherited from the previous version.
If you are upgrading from version 03-20-01 or an earlier version, and you want to cancel the CPU allocation, manually comment out the corresponding parameter (hsdp_cpu_no_list).
Documentation
Available documents
| Document name | Document number | Published date |
| Hitachi Streaming Data Platform Setup and Configuration Guide | MK-93HSDP000-06 | July, 2018 |
| Hitachi Streaming Data Platform Messages | MK-93HSDP002-06 | |
| Hitachi Streaming Data Platform Product Overview | MK-93HSDP003-06 |
Documentation errata
Differences with the manual are indicated by bold text.
(1) Chapter 10: SDP server-definition files, section "SDP manager-definition file", Hitachi Streaming Data Platform Setup and Configuration Guide
[Classification]
Change. Changed the default value and added a description of hsdp_cpu_no_list in "Table 32 Properties of the SDP manager-definition file".
[Before change]
Default value: 0
[After change]
Description: If this parameter is omitted, CPU core allocation is not performed for each component.
Default value: No Value
(2) Chapter 11: Adapter-definition file, section "TCP data input connector definition", Hitachi Streaming Data Platform Setup and Configuration Guide
[Classification]
Change. "Table 46 List of elements and attributes" now contains notes concerning the charCode attribute of the tcpcon:input tag.
[Before change]
| Element or attribute#1 | Description | Value | Number of definitions | Specifiable for C adapters | ||
| TCPDataInputConnectorDefinition | Defines the TCP data input connector. | -- | 1 | Yes | ||
| input | Defines the input information of the connector. | 1 | Yes | |||
| @charCode | Specifies the character encoding in which TCP data is copied into the character string. | ASCII |UTF-8 |UTF-16 | UTF-16BE|UTF-16LE | 0-1 | No | ||
Notes:
#1: If the prefix character is an at sign (@), then it is an attribute.
[After change]
| Element or attribute#1 | Description | Value | Number of definitions | Specifiable for C adapters | ||
| TCPDataInputConnectorDefinition | Defines the TCP data input connector. | -- | 1 | Yes | ||
| input | Defines the input information of the connector. | 1 | Yes | |||
| @charCode#2 | Specifies the character encoding in which TCP data is copied into the character string. | ASCII |UTF-8 |UTF-16 | UTF-16BE|UTF-16LE | 0-1 | No | ||
Notes:
#1: If the prefix character is an at sign (@), then it is an attribute.
#2: If "US-ASCII" is specified for querygroup.encoding in the system-config properties file, specify "ASCII".
(3) Chapter 11: Adapter-definition file, section "Cascading", Hitachi Streaming Data Platform Setup and Configuration Guide
[Classification]
Change. The cascading client connector definition in the adapter-configuration definition file table now contains the specifiable values for the encoding attribute of the cascading tag.
[Before change]
| Element or attribute | Description | Value | Number of definitions | Specifiable for C adapters | ||
| CascadingClientConnector | Define the cascading client connector. | -- | 1 | Yes | ||
| cascading | Define the outputstream information. | 1-16 | Yes | |||
| @encoding | If the record that is specified in @source includes STRING type data, then specify the character encoding for that record. | Specify a character set that is supported by Java. ■ Default: US-ASCII | 0-1 | No | ||
[After change]
| Element or attribute | Description | Value | Number of definitions | Specifiable for C adapters | ||
| CascadingClientConnector | Define the cascading client connector. | -- | 1 | Yes | ||
| cascading | Define the outputstream information. | 1-16 | Yes | |||
| @encoding | If the record that is specified in @source includes STRING type data, then specify the character encoding for that record. | US-ASCII |UTF-8 |UTF-16 |UTF-16BE |UTF-16LE ■ Default: US-ASCII | 0-1 | No | ||
(4) Hitachi Streaming Data Platform Messages
(Descriptions without change are omitted.)
| ID | Before change | After change |
| KFSP42005-E | System processing: For an output stream, the system blocks the query group. For an input stream, the system performs the following operations. If data source mode is specified and the timestamp adjustment function is enabled, the system blocks the query group. If the preceding condition does not apply, the system operates as follows, depending on the relationship between the defined value of the engine.maxQueueSize parameter and the defined value of the unit attribute of the output element for the TCP data-input connector: (1) If the defined value of the unit attribute of the output element for the TCP data-input connector is larger than the defined value of the engine.maxQueueSize parameter, the tuples that cannot be stored to the input stream queue are discarded. (2) If the defined value of the unit attribute of the output element for the TCP data-input connector is smaller than the defined value of the engine.maxQueueSize parameter, the system continues processing without discarding the tuples (and waits until space becomes available in the input stream queue). | System processing: For an output stream, the system blocks the query group. For an input stream, the system performs the following operations. This message is output when the timestamp mode is server mode or when the timestamp adjustment function is disabled in data source mode. The system operates as follows, depending on the relationship between the defined value of the engine.maxQueueSize parameter and the defined value of the unit attribute of the output element for the TCP data-input connector. (1) If the defined value of the unit attribute of the output element for the TCP data-input connector is larger than the defined value of the engine.maxQueueSize parameter, the tuples that cannot be stored to the input stream queue are discarded. (2) If the defined value of the unit attribute of the output element for the TCP data-input connector is smaller than the defined value of the engine.maxQueueSize parameter, the system continues processing without discarding the tuples (and waits until space becomes available in the input stream queue). |
| KFSP42033-W | Message: The system is retrying registration of a tuple in the stream queue. Query group name = aa....aa, stream name = bb....bb, sleep time until retry = cc....cc, retry iterations = dd....dd … Action to take: Get a tuple from the output stream. | Message: The system is retrying registration of a tuple in the output stream queue. Query group name = aa....aa, stream name = bb....bb, sleep time until retry = cc....cc, number of retries = dd....dd … Action to take: Check the following. - The defined value of engine.maxQueueSize - Whether a large amount of data is being sent to the output stream in a short time - Whether a tuple is being acquired from the output stream |
| KFSP42044-I | Message: The system will resume sending tuples to the stream queue. (query group = aa....aa, stream = bb....bb, sleep time until retry = cc....cc, retry iterations = dd....dd) | Message: The system will resume sending tuples into the output stream queue. (query group = aa....aa, stream = bb....bb, sleep time until retry = cc....cc, number of retries = dd....dd) |
| KFSP42045-W (New message) | - | Message: The system is retrying registration of a tuple into the input stream queue. Query group name = aa….aa, stream name = bb….bb, sleep time until retry = cc….cc, number of retries = dd….dd Explanation: aa....aa: Query group name bb....bb: Stream name cc....cc: Sleep time until retry dd….dd: Number of retries Description: There is no space in the output stream queue. System processing: After the sleep time elapses, registers the tuple in the input stream queue again. Action to take: Check the following: - The defined value of engine.maxQueueSize - Whether a large amount of data is being sent to the input stream in a short time |
| KFSP42046-I (new message) | - | Message: The system is retrying registration of a tuple into the input stream queue. (query group = aa….aa, stream = bb….bb, sleep time until retry = cc….cc, number of retries = dd….dd) Explanation: aa....aa: Query group name bb....bb: Stream name cc....cc: Sleep time until retry dd….dd: Number of retries System processing: The system will resume sending tuples to the input stream queue. |
| KFHD13006-E | Description: The stream could not be opened for one of the following reasons. Reason codes: INVALID_FORMAT The format of the specified target name is invalid. DUPLICATE The same stream is specified in method arguments two or more times. MULTIPLE_STREAMS Multiple streams are specified. The openStreamOutput() method of the HSDPadaptorManager class accepts only one stream as the argument. Action to take: Take necessary action according to the reason code. INVALID_FORMAT: Check the format of the stream. DUPLICATE: Revise duplicate streams. MULTIPLE_STREAMS: Specify only one stream. | Description: The stream could not be opened for one of the following reasons. Reason codes: INVALID_FORMAT The format of the specified target name is invalid. DUPLICATE The same stream is specified in method arguments two or more times. MULTIPLE_STREAMS Multiple streams are specified. The openStreamOutput() method of the HSDPadaptorManager class accepts only one stream as the argument. START_ADAPTOR The destination adaptor failed to start. CONNECT_TIMEOUT Communication with the connection destination stream was stopped because of a timeout. SOCKET_OPTION The socket option was invalid. Action to take: Take necessary action according to the reason code. INVALID_FORMAT: Check the format of the stream. DUPLICATE: Revise duplicate streams. MULTIPLE_STREAMS: Specify only one stream. START_ADAPTOR - Refer to the external adapter log file and the log file (see #1) of the internal adapter of the SDP server of the connection destination stream, and take appropriate action. #1: For an external input adaptor, this log file starts with "ADP-tcpinput-". For an external output adaptor, this log file starts with "ADP-cascading-out-". - Run hsdpstatusshow, and then confirm that the SDP server and the query group of the connection destination stream have started. - After starting HSDP by using the hsdpstart -init command, this reason code might be displayed. If the above measures do not recover the system, restart the SDP coordinator by using the hsdpmanager command. CONNECT_TIMEOUT: - Check the status of communications with the connection destination stream. - Increase the value of hsdp.tcp.connect.timeout. SOCKET_OPTION: In the external adaptor definition file, revise the parameter whose name starts with hsdp.tcp. |