Data Export API

To make Dynatrace NAM measurements available to external data warehouses such as Splunk and Elastic:

  • Create one or more reports based on a cached data view from the last monitoring interval.
  • Schedule the reports to render CSV-like data files every monitoring interval.
  • Make the data files available through a simple API.

There is also a way to configure the NAM Server to send data directly to Elastic.

Building a report for data export

On the NAM Server, use DMI > New report to create a report for export.

  1. Create report sections based on any of the following data views:
    • Software service, operation, and site data (with editing as described below)

    • Application, transaction, and tier data (with editing as described below)

    • Application, transaction, and tier baselines

    • Synthetic and sequence transaction data

    • Layer 2 data

      Other data views are not supported for the Data Export API.

  2. Set the time range to the last monitoring interval. For example, Last 5 minutes.
  3. Save the report with no spaces in the report name. For example, MyReportName.

If you use either (or both) of the first two data views listed above, a little manual editing is needed:

  1. Open the report and select Export report definition from the report menu to export your report definition to an XML file.
  2. With a text editor, open the exported report definition file, and then find and replace as follows:
    1. Change application="CVENT" view="ClientView" to application="CVENT" view="CVCache"
    2. Change application="CVENT" view="Tier" to application="CVENT" view="TierCache"
    3. Set the time range to the last monitoring interval:
      <timeRange timeRangeId="1period" resolutionId="PERIOD" />
  3. Save the edited XML file.
  4. Import the edited XML file back into your NAM Server:
    1. On DMI > Export/Import page, click Choose File and browse for the edited XML file for your report.
    2. Click Import from file to load the edited report definition back into your NAM Server.
  5. After you have your report in XML, you can also check whether your time range definition is correct. At the top of the report, the time range should be "Last" + the monitoring interval.

After importing the report from the edited XML, it will retain the changed, cached data views of your choice.

Registering your reports for export

You need to add your report to the list in the partner-export.yaml file in your NAM Server configuration directory.

  1. On the NAM Server, open Admin console.

  2. On the Admin console page, open Configuration file manager.

  3. In the Select file or directory list, select partner-export.yaml.

  4. Add the name of your report to the list in this format: <space>-<space><reportname>

    For example:

     - MyReportName
    

    where MyReportName is the name under which you saved the report.

  5. Click Save.

  • Remember to restart you NAM Server once you change this file.
  • Data export should be enabled only on the primary cluster in a farm configuration.
  • If you want to start exporting data right away, set the PARTNER_EXPORT system property (Admin console > General properties editor) to true. Otherwise, NAM Server will start producing data files after the first API call.

Reviewing a data file

The files produced by the Data Export API are CSV-like files with some additional metadata at the beginning. In the following example, you can see how the name of data file is created.

# Version: 1
# Producer: rtm.18.0.1.95
# AmdUUID: 66291502-820f-41d6-b3d4-c0eeaa6c0f78
# Report-Name: MyReportName
# Section-Name: Software Services
# Time-Range: 2018-11-27T16:03:00 - 2018-11-27T16:04:00
# Columns: type=Time Software%20service Operations Operation%20time Total%20bytes
# Fields: type=begT:long appl:txt trans:double transTime:double c_sByte:long
# Units: type= begT:- appl:- trans:- transTime:ms c_sByte:B

1543330980000 MySoftwareService 102 5.2 9863
1543330980000 MySoftwareService2 45 2.2 63
1543330980000 MySoftwareService3 56 23.2 12863

Managing your data files

Data files generated every monitoring interval are stored in the partner_export directory within your NAM Server installation directory.

Useful properties for managing your exported data:

  • PFE_DATA_FILE_STORAGE_PERIOD - number of hours to store report export files (default: 1 week).
  • PFE_DATA_FILE_STORAGE_SPACE - total allowed occupied space (in MB) by export files (default: 20 GB total).
  • PFE_DATA_FILE_DELETE_PERIOD - how often (in hours) to check for export files to remove (default: 1 hour)

Getting a NAM public API token

To get a NAM public API token:

  1. In the NAM Console, open Security> API tokens.
  2. On the API tokens page, go to the Public API tokens section and click Generate token.
  3. In the New API token name box, enter the name by which the new token will be identified in NAM.
  4. Click Save to generate the new token and add it to the list of public API tokens in NAM.
  5. Click Edit for your token to see the token.
  6. Click Copy to copy the token to your Windows clipboard so you can paste it into another application.

Getting a list of available data export files

To get a list of files, make a GET call to the /RtmDataApiServlet end-point with parameter cmd=get_dir to get a list of all data files currently available for export.

Send your NAM public API token as the Authorization header: Authorization: Bearer yourAPItoken

Getting a data export file

To get a single exported data file, make a GET call to the /RtmDataApiServlet end-point with parameters cmd=get_entry and entry=fileName to get a file named fileName.

Send your NAM public API token as the Authorization header: Authorization: Bearer yourAPItoken

Sending data directly to Elastic

NAM Server can be configured to send data to Elastic directly.

  • The service is an add-on to the Data Export API.
  • No API calls necessary: JSONs with data are pushed to Elastic endpoints configured with NAM Server system properties.

To configure this on the NAM side:

  • Report names in this case need to end with -elasticsearch
  • Set these system properties on the NAM Server (Admin console > General properties editor):
    • ELASTIC_SEARCH_HOST: address of the machine where Elastic is installed (default: localhost)
    • ELASTIC_SEARCH_PORT: port at which Elastic can be accessed (default: 9200)
    • ELASTIC_SEARCH_SEC: whether communication with Elastic is over SSL (default: false)
    • ELASTIC_SEARCH_INDEX: name of the index used for indexing data from NAM (default: dcrum)
    • ELASTIC_SEARCH_USER and ELASTIC_SEARCH_PASS: credentials used to connect with Elastic.
      For details, see Elastic help: Setting Up User Authentication
    • ELASTIC_SEARCH_QUERY: optional parameters sent with requests to Elastic. This can be used to attach pipelines to the process when a value is set to pipeline=myPipelineName.
      For details, see Elastic help: Pipeline Definition
    • ELASTIC_SEARCH_VERSION: in case you use old (version 5) Elastic. Set it to ensure proper indexing of text data.

You can enable extensive logging of requests (with payloads) to Elastic.

Example plugins

The following pages show NAM data export in use.

  • Elastic Search and Kibana plugin
    This plugin enables you to use Kibana to display NAM data stored in ElasticSearch.
  • Splunk plugin
    This is a demo application that shows NAM-Splunk integration. It presents how to read data from the NAM Server using the REST API and provides a built-in dashboard for data visualization similar to the Health dashboard available on the NAM Server.