Skip to main content

We've Moved!

Product Documentation has moved to docs.hitachivantara.com
Hitachi Vantara Knowledge

Data Ingest v2.0 Connector

The Hitachi Visualization Suite (HVS) Data Ingest v2.0 connector retrieves geographical location data from GPS supported cradlepoint routers through User Datagram Protocol (UDP) requests or NetCloud Manager APIs. This connector integration provides Hitachi Visualization Suite with real-time updates to mapped entity coordinates, which gives your site valuable data insights, such as identifying the live location of a train on its route.

Features

  • GPS signal retrieval

    Retrieve a geographical location from a GPS supported cradlepoint router.

  • Manageable

    The connector can be installed and uninstalled.

  • Configurable

    Attributes, such as the polling name and entity interval, can be changed.

  • Logging

    Log files are listed in the Windows Event Viewer.

  • Ingestion monitoring

    Ingestions can be added and removed without restarting services.

System diagrams

UDP

The following diagram illustrates how the Visualization Suite Data Ingest v2.0 connector retrieves a geographical location from a GPS supported cradlepoint router through a UDP port.

In this example:

  • The connection uses an site-to-site VPN and cradles send the National Marine Electronics Association (NMEA) Sentences through the UDP port
  • Listening occurs on the UDP port from which the Visualization Suite Data Ingest v2.0 connector retrieves NMEA sentences
  • The Visualization Suite Data Ingest v2.0 connector ingests data into Visualization Suite using the Visualization Suite Web API
GUID-81C430CB-2709-4484-9A50-105F77A7248F-low.png

NetCloud Manager API

The following diagram illustrates how the Visualization Suite Data Ingest v2.0 connector retrieves a geographical location from a GPS supported cradlepoint router using the NetCloud Manager API.

In this example:

  • Information about routers and their locations are sent to the NetCloud Manager (NCM) System
  • Within a specific time interval, the Visualization Suite Data Ingest v2.0 connector calls the NCM API to retrieve current router data which it then parses and publishes to Visualization Suite through the Visualization Suite API.
GUID-E9A675CE-F5E1-4339-BBD7-C69217520C5A-low.png

Prerequisites and configuration considerations

  • The Visualization Suite Data Ingest v2.0 connector runs on a Windows machine (64 bit) and has been qualified with Windows 2012 R2.
  • The Visualization Suite Data Ingest v2.0 connector is supported in Visualization Suite v6.2.0 and above.
  • The Visualization Suite Data Ingest v2.0 connector has been qualified with NetCloud Manager API v2.0.
  • The Visualization Suite Data Ingest v2.0 connector is supported with the use of Cradlepoint Routers, and has been qualified with IBR600 and IBR650.

Data Ingest v2.0 connector configuration overview

You can configure the Visualization Suite Data Ingest v2.0 connector to support your production environment. Afterward, you will need to complete steps to send information to the Visualization Suite API, and manage that information.

The Visualization Suite Data Ingest v2.0 connector includes the following three files. With the exception of the application settings file, which is located in the Visualization Suite Data Ingest v2.0 connector installation directory, refer to the /Ingestions/gps/ folder for the configuration files location:

  • Application Settings file: appsettings.json
  • UDP configuration file: config-udp.json
  • NetCloud Manager API configuration file: config-api.json

Before you begin

Before configuring the Visualization Suite Data Ingest v2.0 connector, complete the Cradle configuration and/or NetCloud Manager setup.

The NetCloud Manager API is a RESTful HTTP interface that returns data in JSON format. For information about signing up for the NCM API, managing your keys, and accessing additional documentation and code examples, see NetCloud Manager: API 2.0 Overview.

HVS appsettings.json example

{
  "IngestionsFolders": [
    "Ingestions/gps/"
  ],
  "MonitorIntervalMs": 60000
}

About the ingestions folders

The IngestionsFolders is a specific folder from which the Visualization Suite Data Ingest v2.0 connector monitors and reads configuration files.

"IngestionsFolders": [
    "Ingestions/train",
    "Ingestions/cars"
]
"IngestionsFolders": [
    "Ingestions"
  ]
  • IngestionsFolders

    A folder from which the Visualization Suite Data Ingest v2.0 connector monitors and reads configuration files, without requiring any service restart.

    Administrators can specify a directory or an array of folders, but the URL of a specific configuration file cannot be specified.

  • MonitorIntervalMS

    The interval that the monitor will use to check for changes within the folders. If an administrator specifies 10000, then every 10 seconds the monitor will check if there are new or removed ingestions files in the folders.

    By default, the value is set to 60000 (1 minute).

Config-udp.json UDP Listener example

The following code is an example configuration of the Visualization Suite Data Ingest v2.0 connector for the UDP Listener, which is used to receive data from cradles, then send that data to the Visualization Suite API.

{
    "$type":  "HVS.DataExchange.Publishers.NmeaApiPublisher, HVS.DataExchange",
    "logConfig": {
    "level": "Warning",
    "sinks": [
      {
        "type": "File",
        "settings":
          {
             "rollingInterval": "Day"
          }
        },
      {
        "type": "Console"
      },
      {
        "type": "EventLog"
      }
      ]
     },
    "name":  "udp-1",
    "apiUrl":  "https://api.hitachismartcamera.com",
    "domain":  "HVS_DOMAIN_VALUE",
    "email":  "HVS_SERVICE_USER",
    "password":  "HVS_SERVICE_USER_PASSWORD",
    "entityNamePrefix":  "",
    "entityNameFormat":  "{EntityNamePrefix}{EntityTypeName} - {DeviceId}",
    "entityTypeDefinition":  {
                                 "id":  "HVS_ENTITY_TYPE_ID"
                             },
    ///START - DEFAULT SECTION (MANDATORY AS IS)
    "parser":  {
                   "$type":  "HVS.DataExchange.Parsers.Customers.MetroTransitUdpDataParser, HVS.DataExchange",
                   "keys":  [
                                "device_id",
                                "sentence_id",
                                "update_at",
                                "latitude",
                                "ns",
                                "longitude",
                                "we",
                                "fix_quality",
                                "nr_satellites",
                                "accurancy",
                                "altitude",
                                "altitude_meters",
                                "geoid_ellipsoid",
                                "ellopsoid_meters",
                                "dgps_date",
                                "dgps_station_id",
                                "checksum"
                            ],
                   "checkSumChar":  "*",
                   "entityTransformation":  {
                                                "nameValuePath":  "device_id",
                                                "coordinatesSystem":  "ddm",
                                                "deviceIdValuePath":  "device_id",
                                                "latValuePath":  "latitude",
                                                "latNsValuePath":  "ns",
                                                "lonValuePath":  "longitude",
                                                "lonWeValuePath":  "we",
                                                "timeStampValuePath":  "update_at",
                                                "domainTimezone":  "Eastern Standard Time"
                                            }
               },
     ///END - DO NOT CHANGE THIS PART
    "retriever":  {
                      "$type":  "HVS.DataExchange.Retrievers.UdpDataRetriever, HVS.DataExchange",
                      "localAddress":  "127.0.0.1",
                      "localPort":  5000,
                      "payloadFilterCsv":  "$GPGGA",
                      "apiUpdateIntervalMillis":  10000
                  }
}
  • logConfig

    The name of the log configuration file.

  • level

    Log severity (such as Warning). By default, the level is 'Information'.

  • sinks

    Output location of the log configuration file.

  • sinks: type

    The file type.

  • sinks: settings

    Settings associated with the file type.

  • sinks:settings: rollingInterval

    The rolling interval for the file type (minute, hour, day, month, year).

    The rollingInterval is available only for the file type.

  • name

    The name of the Event log file.

    Any white space that is included will be removed from the file name.

  • apiUrl

    The URL of the Hitachi Visualization Suite API.

  • domain

    The name of the Hitachi Visualization Suite domain, where NMEA data is ingested.

  • email

    The Hitachi Visualization Suite service account for the domain.

  • password

    The Hitachi Visualization Suite service account password for the domain.

  • entityNamePrefix, entityNameFormat

    Format used for the entity name.

  • entityTypeDefinition: id

    Tracking enabled Entity Type ID.

  • Local IP

    The local IP address.

  • localPort

    Local UDP port.

  • payloadFilterCsv

    The NMEA comma-separated sentences that are to be managed by Visualization Suite Data Ingest v2.0 connector.

  • apiUpdateIntervalMillis

    The number of intervals, in milliseconds, that information must be sent to the Visualization Suite API.

Config-api.json NetCloud API example

{
    "$type":  "HVS.DataExchange.Publishers.NmeaApiPublisher, HVS.DataExchange",
    "logConfig": {
    "level": "Warning",
    "sinks": [
      {
        "type": "File",
        "settings":
          {
           "rollingInterval": "Day"
          }
        },
      {
        "type": "Console"
      },
      {
        "type": "EventLog"
      }
      ]
     },
    "name":  "netcloud-api-1,
    "apiUrl":  "https://api.hitachismartcamera.com",
    "domain":  "HVS_DOMAIN_VALUE",
    "email":  "HVS_SERVICE_USER",
    "password":  "HVS_SERVICE_USER_PASSWORD",
    "entityNamePrefix":  "",
    "entityNameFormat":  "{EntityNamePrefix}{EntityTypeName} - {DeviceId}",
    "entityTypeDefinition":  {
                                 "id":  "HVS_ENTITY_TYPE_ID"
                             },
    "parser":  {
                   "$type":  "HVS.DataExchange.Parsers.Customers.MetroTransitApiDataParser, HVS.DataExchange"
               },
    "retriever":  {
                      "$type":  "HVS.DataExchange.Retrievers.Customers.MetrotransitApiDataRetriever, HVS.DataExchange",
                      "connectionTimeout":  30,
                       "pollingIntervalMillis":  10000,
                       "RequestHeaders":  {
                                             "X-CP-API-ID":  "0a78a1f1",
                                             "X-CP-API-KEY":  "d8ac761368beca70c2089412b9432373",
                                             "X-ECM-API-ID":  "932a6e8b-32e1-471d-9e00-3dc59e18eeb7",
                                             "X-ECM-API-KEY":  "b5cb0b29c420f433f3ec4d7e9fd48dadcbe60926"
                                         },
                      "CradleCloudUri":  "https://www.cradlepointecm.com/api/v2/",
                      "CradleCloudLocationEndpoint":  "locations/",
                      "CradleCloudRouterEndpoint":  "routers/"
                  }
}
  • connectionTimeout

    The interval, in seconds, before the NCM API HTTP request will time-out.

  • pollingIntervalMillis

    The interval, in milliseconds, that the Visualization Suite Data Ingest v2.0 connector polls data from the NCM API.

  • RequestHeaders

    For information about signing up for the NCM API, managing your keys, and accessing additional documentation and code examples, see NetCloud Manager: API 2.0 Overview.

  • CradleCloudUri

    The URI of the NCM API.

  • CradleCloudLocationEndpoint

    The NCM API endpoint that provides the location (latitude, longitude) of the associated devices.

  • CradleCloudRouterEndpoint

    The NCM API endpoint that provides the Router of the associated devices.

Notes about the entity name

The Visualization Suite Data Ingest v2.0 connector lets you construct the entity name.

  • {EntityNamePrefix}

    The value declared in the entityNamePrefix field of the config file

  • {EntityTypeName}

    The name of the entity type associated with the specified ID

  • {DeviceId}

    The device ID derived from the NMEA sentences

Following is an example of the default format of the Visualization Suite Data Ingest v2.0 connector entity name. If you remove the entityNameFormat, the entity name will be formatted as EntityTypeName + white space + DeviceId.

....  
"entityNamePrefix":  "",
"entityNameFormat":  "{EntityNamePrefix}{EntityTypeName} - {DeviceId}",
....
Custom Format

A custom format is declared using entityNameFormat.

For a usage example, if the entity type id refers to a 'Airplanes' entity type and the incoming device id is 'IBR-234', then the newly created entity will have the following sample name: IBR-234 (Airplanes)

....
"entityNamePrefix":"Jet ",
"entityNameFormat":"{EntityNamePrefix} {DeviceId} ({EntityTypeName})",
....

Log configuration file

You can configure the log file by changing the applicable sections within each ingestion setting.

"logConfig": {
"level": "Warning",
"sinks": [
  {
    "type": "File",
    "settings":
      {
        "rollingInterval": "Day"
      }
    },
  {
    "type": "Console"
  },
  {
    "type": "EventLog"
  }
  ]
 },

In the following table, the 'File' sink also has a setting for defining the "rollingInterval" of the file.

Available level valuesAvailable SinksFile RollingInterval
VerboseFileMinute
DebugConsoleHour
InformationEventLogDay
WarningMonth
ErrorYear
Fatal

Uninstalling the Data Ingest v2.0 connector

  1. Backup the config.json file, then close all open files.

  2. Launch the Control Panel.

  3. Go to All Control Panel Items Programs and Features and select the connector.

  4. Select the option to remove the connector.

Frequently asked questions

Why wasn't an entity created in Hitachi Visualization Suite?

There may be a configuration issue with endpoints, credentials, the entity Type ID, or with the source configuration. Refer to the event log file for error information to help identify the cause of the problem.

Within Hitachi Visualization Suite, can I modify the Entity Type?

Yes. Modifying the Entity Type will not affect the operation of the Hitachi Visualization Suite Data Ingest v2.0 connector.

Which devices are supported?

Routers that support GPS functionality and reporting of the Hitachi Visualization Suite Data Ingest v2.0 connector sentences.

Our site has online and offline devices. Are offline devices indicated within Hitachi Visualization Suite?

When devices are offline, the Hitachi Visualization Suite Data Ingest v2.0 connector will not retrieve them, so you will not find offline devices in Hitachi Visualization Suite. However, when devices are online in Hitachi Visualization Suite and they change to an offline status, then they will remain in Hitachi Visualization Suite.

What is the device_id value for the UDP and NetCloud API process?

The device_id is a field that is used by the Hitachi Visualization Suite Data Ingest v2.0 connector and in the update API that uniquely identifies a device. The value of this field is stored in the source_id field of the Hitachi Visualization Suite entity so that it will be used as unique identifier when updating coordinates. The value matches the user defined router name. Examples:

  • UDP

    User defined router name (for example, IBR0600-4c3)

  • NCM API

    User defined router name - last 6-digits of MAC address (for example, IBR0600-4c3-b1c08e)

What is the minimum value that can be set for pollingIntervalMillis and apiUpdateIntervalMillis?

The minimum value is 1 second (1000 milliseconds). When less than the minimum value is specified, the system defaults to 10 seconds (10000 ms) and logs a warning message:

"Because a value less than the minimum required for pollingIntervalMillis/apiUpdateIntervalMillis has been specified, the default value of 10000 ms is applied. To specify a different value, update the configuration file."

When multiple configuration files (processes) are running, what should we see as an Event Log name within the Event Viewer?

The Event Viewer will display multiple log entries. The HVSDataIngestV2Log indicates a value of the name key in the configuration file.

What happens to tracked entities with a status when I disable the tracking option of the Entity Type?

When you update the entity type has_tracking from true to false, any further update to the entity's location is rejected. Additionally, no new entities are created--only API errors are generated.

The status of the existing entities will remain as-is, because the status update job will not consider the entity types that do not have has_tracking set to true.

What are the steps to set up an on-prem system?

A public internet is not a requirement because this implementation requires that the cradlepoint sends UDP packets over the network to the IP address of the machine that hosts the NMEA connector.

The general procedure is as follows:

  1. Configure your router to send UDP packets to the IP address of the machine that hosts the Hitachi Visualization Suite Data Ingest v2.0 connector. Ensure that the machines can reach each other, and configure the cradlepoints with the option to prepend the device ID in front of each sentence.
  2. On the Hitachi Visualization Suite Data Ingest v2.0 connector machine, open the port specified in the config file to allow UDP traffic and to receive packets from the cradlepoints.
  3. Ensure that the Hitachi Visualization Suite Data Ingest v2.0 connector can reach the Hitachi Visualization Suite API.
How many different NetCloud Manager APIs call the Hitachi Visualization Suite Data Ingest v2.0 connector in order to retrieve GPS data for devices?

The Hitachi Visualization Suite Data Ingest v2.0 connector makes two separate calls during the pollingIntervalMillis. If the interval is set to 10 seconds, then the Hitachi Visualization Suite Data Ingest v2.0 connector service will call NetCloud Manager API 12 times (6 calls x 2 APIs) per minute. The NetCloud Manager API returns all available data for the devices with one call.

What is the process to have an Hitachi Visualization Suite hosted staging environment for my POC customer?

Please send us your request by emailing the details to mailto:videosupport@hitachivantara.com.