Data Ingest v2.0 Connector
The Hitachi Visualization Suite (HVS) Data Ingest v2.0 connector retrieves geographical location data from GPS supported cradlepoint routers through User Datagram Protocol (UDP) requests or NetCloud Manager APIs. This connector integration provides Hitachi Visualization Suite with real-time updates to mapped entity coordinates, which gives your site valuable data insights, such as identifying the live location of a train on its route.
Features
GPS signal retrieval
Retrieve a geographical location from a GPS supported cradlepoint router.
Manageable
The connector can be installed and uninstalled.
Configurable
Attributes, such as the polling name and entity interval, can be changed.
Logging
Log files are listed in the Windows Event Viewer.
Ingestion monitoring
Ingestions can be added and removed without restarting services.
System diagrams
UDP
The following diagram illustrates how the Visualization Suite Data Ingest v2.0 connector retrieves a geographical location from a GPS supported cradlepoint router through a UDP port.
In this example:
- The connection uses an site-to-site VPN and cradles send the National Marine Electronics Association (NMEA) Sentences through the UDP port
- Listening occurs on the UDP port from which the Visualization Suite Data Ingest v2.0 connector retrieves NMEA sentences
- The Visualization Suite Data Ingest v2.0 connector ingests data into Visualization Suite using the Visualization Suite Web API
NetCloud Manager API
The following diagram illustrates how the Visualization Suite Data Ingest v2.0 connector retrieves a geographical location from a GPS supported cradlepoint router using the NetCloud Manager API.
In this example:
- Information about routers and their locations are sent to the NetCloud Manager (NCM) System
- Within a specific time interval, the Visualization Suite Data Ingest v2.0 connector calls the NCM API to retrieve current router data which it then parses and publishes to Visualization Suite through the Visualization Suite API.
Prerequisites and configuration considerations
- The Visualization Suite Data Ingest v2.0 connector runs on a Windows machine (64 bit) and has been qualified with Windows 2012 R2.
- The Visualization Suite Data Ingest v2.0 connector is supported in Visualization Suite v6.2.0 and above.
- The Visualization Suite Data Ingest v2.0 connector has been qualified with NetCloud Manager API v2.0.
- The Visualization Suite Data Ingest v2.0 connector is supported with the use of Cradlepoint Routers, and has been qualified with IBR600 and IBR650.
Data Ingest v2.0 connector configuration overview
You can configure the Visualization Suite Data Ingest v2.0 connector to support your production environment. Afterward, you will need to complete steps to send information to the Visualization Suite API, and manage that information.
The Visualization Suite Data Ingest v2.0 connector includes the following three files. With the exception of the application settings file, which is located in the Visualization Suite Data Ingest v2.0 connector installation directory, refer to the /Ingestions/gps/ folder for the configuration files location:
- Application Settings file: appsettings.json
- UDP configuration file: config-udp.json
- NetCloud Manager API configuration file: config-api.json
Before you begin
Before configuring the Visualization Suite Data Ingest v2.0 connector, complete the Cradle configuration and/or NetCloud Manager setup.
The NetCloud Manager API is a RESTful HTTP interface that returns data in JSON format. For information about signing up for the NCM API, managing your keys, and accessing additional documentation and code examples, see NetCloud Manager: API 2.0 Overview.
HVS appsettings.json example
{ "IngestionsFolders": [ "Ingestions/gps/" ], "MonitorIntervalMs": 60000 }
About the ingestions folders
The IngestionsFolders is a specific folder from which the Visualization Suite Data Ingest v2.0 connector monitors and reads configuration files.
"IngestionsFolders": [ "Ingestions/train", "Ingestions/cars" ]
"IngestionsFolders": [ "Ingestions" ]
IngestionsFolders
A folder from which the Visualization Suite Data Ingest v2.0 connector monitors and reads configuration files, without requiring any service restart.
Administrators can specify a directory or an array of folders, but the URL of a specific configuration file cannot be specified.
MonitorIntervalMS
The interval that the monitor will use to check for changes within the folders. If an administrator specifies 10000, then every 10 seconds the monitor will check if there are new or removed ingestions files in the folders.
By default, the value is set to 60000 (1 minute).
Config-udp.json UDP Listener example
The following code is an example configuration of the Visualization Suite Data Ingest v2.0 connector for the UDP Listener, which is used to receive data from cradles, then send that data to the Visualization Suite API.
{ "$type": "HVS.DataExchange.Publishers.NmeaApiPublisher, HVS.DataExchange", "logConfig": { "level": "Warning", "sinks": [ { "type": "File", "settings": { "rollingInterval": "Day" } }, { "type": "Console" }, { "type": "EventLog" } ] }, "name": "udp-1", "apiUrl": "https://api.hitachismartcamera.com", "domain": "HVS_DOMAIN_VALUE", "email": "HVS_SERVICE_USER", "password": "HVS_SERVICE_USER_PASSWORD", "entityNamePrefix": "", "entityNameFormat": "{EntityNamePrefix}{EntityTypeName} - {DeviceId}", "entityTypeDefinition": { "id": "HVS_ENTITY_TYPE_ID" }, ///START - DEFAULT SECTION (MANDATORY AS IS) "parser": { "$type": "HVS.DataExchange.Parsers.Customers.MetroTransitUdpDataParser, HVS.DataExchange", "keys": [ "device_id", "sentence_id", "update_at", "latitude", "ns", "longitude", "we", "fix_quality", "nr_satellites", "accurancy", "altitude", "altitude_meters", "geoid_ellipsoid", "ellopsoid_meters", "dgps_date", "dgps_station_id", "checksum" ], "checkSumChar": "*", "entityTransformation": { "nameValuePath": "device_id", "coordinatesSystem": "ddm", "deviceIdValuePath": "device_id", "latValuePath": "latitude", "latNsValuePath": "ns", "lonValuePath": "longitude", "lonWeValuePath": "we", "timeStampValuePath": "update_at", "domainTimezone": "Eastern Standard Time" } }, ///END - DO NOT CHANGE THIS PART "retriever": { "$type": "HVS.DataExchange.Retrievers.UdpDataRetriever, HVS.DataExchange", "localAddress": "127.0.0.1", "localPort": 5000, "payloadFilterCsv": "$GPGGA", "apiUpdateIntervalMillis": 10000 } }
logConfig
The name of the log configuration file.
level
Log severity (such as Warning). By default, the level is 'Information'.
sinks
Output location of the log configuration file.
sinks: type
The file type.
sinks: settings
Settings associated with the file type.
sinks:settings: rollingInterval
The rolling interval for the file type (minute, hour, day, month, year).
The rollingInterval is available only for the file type.
name
The name of the Event log file.
Any white space that is included will be removed from the file name.
apiUrl
The URL of the Hitachi Visualization Suite API.
domain
The name of the Hitachi Visualization Suite domain, where NMEA data is ingested.
email
The Hitachi Visualization Suite service account for the domain.
password
The Hitachi Visualization Suite service account password for the domain.
entityNamePrefix, entityNameFormat
Format used for the entity name.
entityTypeDefinition: id
Tracking enabled Entity Type ID.
Local IP
The local IP address.
localPort
Local UDP port.
payloadFilterCsv
The NMEA comma-separated sentences that are to be managed by Visualization Suite Data Ingest v2.0 connector.
apiUpdateIntervalMillis
The number of intervals, in milliseconds, that information must be sent to the Visualization Suite API.
Config-api.json NetCloud API example
{ "$type": "HVS.DataExchange.Publishers.NmeaApiPublisher, HVS.DataExchange", "logConfig": { "level": "Warning", "sinks": [ { "type": "File", "settings": { "rollingInterval": "Day" } }, { "type": "Console" }, { "type": "EventLog" } ] }, "name": "netcloud-api-1, "apiUrl": "https://api.hitachismartcamera.com", "domain": "HVS_DOMAIN_VALUE", "email": "HVS_SERVICE_USER", "password": "HVS_SERVICE_USER_PASSWORD", "entityNamePrefix": "", "entityNameFormat": "{EntityNamePrefix}{EntityTypeName} - {DeviceId}", "entityTypeDefinition": { "id": "HVS_ENTITY_TYPE_ID" }, "parser": { "$type": "HVS.DataExchange.Parsers.Customers.MetroTransitApiDataParser, HVS.DataExchange" }, "retriever": { "$type": "HVS.DataExchange.Retrievers.Customers.MetrotransitApiDataRetriever, HVS.DataExchange", "connectionTimeout": 30, "pollingIntervalMillis": 10000, "RequestHeaders": { "X-CP-API-ID": "0a78a1f1", "X-CP-API-KEY": "d8ac761368beca70c2089412b9432373", "X-ECM-API-ID": "932a6e8b-32e1-471d-9e00-3dc59e18eeb7", "X-ECM-API-KEY": "b5cb0b29c420f433f3ec4d7e9fd48dadcbe60926" }, "CradleCloudUri": "https://www.cradlepointecm.com/api/v2/", "CradleCloudLocationEndpoint": "locations/", "CradleCloudRouterEndpoint": "routers/" } }
connectionTimeout
The interval, in seconds, before the NCM API HTTP request will time-out.
pollingIntervalMillis
The interval, in milliseconds, that the Visualization Suite Data Ingest v2.0 connector polls data from the NCM API.
RequestHeaders
For information about signing up for the NCM API, managing your keys, and accessing additional documentation and code examples, see NetCloud Manager: API 2.0 Overview.
CradleCloudUri
The URI of the NCM API.
CradleCloudLocationEndpoint
The NCM API endpoint that provides the location (latitude, longitude) of the associated devices.
CradleCloudRouterEndpoint
The NCM API endpoint that provides the Router of the associated devices.
Notes about the entity name
The Visualization Suite Data Ingest v2.0 connector lets you construct the entity name.
{EntityNamePrefix}
The value declared in the entityNamePrefix field of the config file
{EntityTypeName}
The name of the entity type associated with the specified ID
{DeviceId}
The device ID derived from the NMEA sentences
Following is an example of the default format of the Visualization Suite Data Ingest v2.0 connector entity name. If you remove the entityNameFormat, the entity name will be formatted as EntityTypeName + white space + DeviceId.
.... "entityNamePrefix": "", "entityNameFormat": "{EntityNamePrefix}{EntityTypeName} - {DeviceId}", ....
A custom format is declared using entityNameFormat.
For a usage example, if the entity type id refers to a 'Airplanes' entity type and the incoming device id is 'IBR-234', then the newly created entity will have the following sample name: IBR-234 (Airplanes)
.... "entityNamePrefix":"Jet ", "entityNameFormat":"{EntityNamePrefix} {DeviceId} ({EntityTypeName})", ....
Log configuration file
You can configure the log file by changing the applicable sections within each ingestion setting.
"logConfig": { "level": "Warning", "sinks": [ { "type": "File", "settings": { "rollingInterval": "Day" } }, { "type": "Console" }, { "type": "EventLog" } ] },
In the following table, the 'File' sink also has a setting for defining the "rollingInterval" of the file.
Available level values | Available Sinks | File RollingInterval |
Verbose | File | Minute |
Debug | Console | Hour |
Information | EventLog | Day |
Warning | Month | |
Error | Year | |
Fatal |
Uninstalling the Data Ingest v2.0 connector
Backup the config.json file, then close all open files.
Launch the Control Panel.
Go to
and select the connector.Select the option to remove the connector.
Frequently asked questions
There may be a configuration issue with endpoints, credentials, the entity Type ID, or with the source configuration. Refer to the event log file for error information to help identify the cause of the problem.
Yes. Modifying the Entity Type will not affect the operation of the Hitachi Visualization Suite Data Ingest v2.0 connector.
Routers that support GPS functionality and reporting of the Hitachi Visualization Suite Data Ingest v2.0 connector sentences.
When devices are offline, the Hitachi Visualization Suite Data Ingest v2.0 connector will not retrieve them, so you will not find offline devices in Hitachi Visualization Suite. However, when devices are online in Hitachi Visualization Suite and they change to an offline status, then they will remain in Hitachi Visualization Suite.
The device_id is a field that is used by the Hitachi Visualization Suite Data Ingest v2.0 connector and in the update API that uniquely identifies a device. The value of this field is stored in the source_id field of the Hitachi Visualization Suite entity so that it will be used as unique identifier when updating coordinates. The value matches the user defined router name. Examples:
UDP
User defined router name (for example, IBR0600-4c3)
NCM API
User defined router name - last 6-digits of MAC address (for example, IBR0600-4c3-b1c08e)
The minimum value is 1 second (1000 milliseconds). When less than the minimum value is specified, the system defaults to 10 seconds (10000 ms) and logs a warning message:
"Because a value less than the minimum required for pollingIntervalMillis/apiUpdateIntervalMillis has been specified, the default value of 10000 ms is applied. To specify a different value, update the configuration file."
The Event Viewer will display multiple log entries. The HVSDataIngestV2Log indicates a value of the name key in the configuration file.
When you update the entity type has_tracking from true to false, any further update to the entity's location is rejected. Additionally, no new entities are created--only API errors are generated.
The status of the existing entities will remain as-is, because the status update job will not consider the entity types that do not have has_tracking set to true.
A public internet is not a requirement because this implementation requires that the cradlepoint sends UDP packets over the network to the IP address of the machine that hosts the NMEA connector.
The general procedure is as follows:
- Configure your router to send UDP packets to the IP address of the machine that hosts the Hitachi Visualization Suite Data Ingest v2.0 connector. Ensure that the machines can reach each other, and configure the cradlepoints with the option to prepend the device ID in front of each sentence.
- On the Hitachi Visualization Suite Data Ingest v2.0 connector machine, open the port specified in the config file to allow UDP traffic and to receive packets from the cradlepoints.
- Ensure that the Hitachi Visualization Suite Data Ingest v2.0 connector can reach the Hitachi Visualization Suite API.
The Hitachi Visualization Suite Data Ingest v2.0 connector makes two separate calls during the pollingIntervalMillis. If the interval is set to 10 seconds, then the Hitachi Visualization Suite Data Ingest v2.0 connector service will call NetCloud Manager API 12 times (6 calls x 2 APIs) per minute. The NetCloud Manager API returns all available data for the devices with one call.
Please send us your request by emailing the details to mailto:videosupport@hitachivantara.com.