Time to upgrade! NAM is scheduled for end of support. It's time to move to Dynatrace our all-in-one software intelligence platform.

NAM Data Export API

To make Dynatrace NAM measurements available to external data warehouses such as Splunk and Elastic:

  • Verify that you have sufficient permission. You need the Public API user role to access the /RtmDataApiServlet endpoint for API access to this functionality. See User roles and groups for help with assigning roles.
  • Create one or more reports based on a cached data view from the last monitoring interval.
  • Schedule the reports to render CSV-like data files every monitoring interval.
  • Make the data files available through a simple API.

There is also a way to configure the NAM Server to send data directly to Elastic instead of generating data files on the NAM Server.

Building a report for data export

On the NAM Server, use DMI > New report to create a report for export.

  1. Create report sections based on any of the following data views:
    • Software service, operation, and site data (CVENT.CVCache with editing as described below)

    • Application, transaction, and tier data (CVENT.Tier with editing as described below)

    • Application, transaction, and tier baselines

    • Synthetic and sequence transaction data

    • Layer 2 data

      Other data views are not supported for the Data Export API.

  2. Set the time range to the last monitoring interval. For example, Last 5 minutes.
  3. Save the report with no spaces in the report name. For example, MyReportName.

Additional report preparation

If you use either (or both) of the first two data views listed above, a little manual editing is needed:

  1. Open the report and select Export report definition from the report menu to export your report definition to an XML file.
  2. With a text editor, open the exported report definition file, and then find and replace as follows:
    1. Change application="CVENT" view="ClientView" to application="CVENT" view="CVCache"
    2. Change application="CVENT" view="Tier" to application="CVENT" view="TierCache"
    3. Set the time range to the last monitoring interval:
      <timeRange timeRangeId="1period" resolutionId="PERIOD" />
  3. Save the edited XML file.
  4. Import the edited XML file back into your NAM Server:
    1. On DMI > Export/Import page, select Choose File and browse for the edited XML file for your report.
    2. Select Import from file to load the edited report definition back into your NAM Server.
  5. After you have your report in XML, you can also check whether your time range definition is correct. At the top of the report, the time range should be "Last" + the monitoring interval.

After importing the report from the edited XML, it will retain the changed, cached data views of your choice.

Registering your reports for export

You need to add your report to the list in the partner-export.yaml file in your NAM Server configuration directory.

  1. On the NAM Server, open Admin console.

  2. On the Admin console page, open Configuration file manager.

  3. Search for and select partner-export.yaml.

  4. In the edit box, edit the contents of the file to add the name of your report in this format: <space>-<space><reportname>

    For example:

     - MyReportName
    

    where MyReportName is the name under which you saved the report.

  5. Select Save.

  6. Restart your NAM Server after you change this file.

  • Data export should be enabled only on the primary cluster in a farm configuration.
  • If you want to start exporting data right away, set the PARTNER_EXPORT system property (Admin console > General properties editor) to true. Otherwise, NAM Server will start producing data files after the first API call.

Reviewing a data file

The files produced by the Data Export API are CSV-like files with some additional metadata at the beginning. In the following example, you can see how the name of data file is created.

# Version: 1
# Producer: rtm.18.0.1.95
# AmdUUID: 66291502-820f-41d6-b3d4-c0eeaa6c0f78
# Report-Name: MyReportName
# Section-Name: Software Services
# Time-Range: 2018-11-27T16:03:00 - 2018-11-27T16:04:00
# Columns: type=Time Software%20service Operations Operation%20time Total%20bytes
# Fields: type=begT:long appl:txt trans:double transTime:double c_sByte:long
# Units: type= begT:- appl:- trans:- transTime:ms c_sByte:B

1543330980000 MySoftwareService 102 5.2 9863
1543330980000 MySoftwareService2 45 2.2 63
1543330980000 MySoftwareService3 56 23.2 12863

Managing your data files

Data files generated every monitoring interval are stored in the partner_export directory within your NAM Server installation directory.

Useful properties for managing your exported data:

  • PFE_DATA_FILE_STORAGE_PERIOD - number of hours to store report export files. Default: 168 hours (1 week).
  • PFE_DATA_FILE_STORAGE_SPACE - total allowed occupied space (in MB) by export files. Default: 20480 MB (20 GB total).
  • PFE_DATA_FILE_DELETE_PERIOD - how often (in hours) to check for export files to remove. Default: 1 hour.

Getting a NAM public API token

To get a NAM public API token:

  1. In the NAM Console, open Security > API tokens.
  2. On the API tokens page, go to the Public API tokens section and select Generate token.
  3. In the New API token name box, enter the name by which the new token will be identified in NAM.
  4. Select Save to generate the new token and add it to the list of public API tokens in NAM.
  5. Select Edit to see the token edit controls for that token.
  6. Select Copy to copy the token to your Windows clipboard so you can paste it into another application.

GET list of available data export files

To get a list of files, make a GET call to the /RtmDataApiServlet end-point with parameter cmd=get_dir to get a list of all data files currently available for export.

Example request

Send a request of the form

GET https://nam-server-address:port/RtmDataApiServlet?cmd=get_dir

with your NAM public API token (copied from the NAM Console) in the Authorization header: Authorization: Bearer yourAPItoken

Example response

In this example, the request returned one data export file name:

Splunk_Analysis_by_Tier_5d6e5cc8_5_t

GET data export file

To get a single exported data file, you need one of the file names from the cmd=get_dir request (see above) so you can make a GET call to the /RtmDataApiServlet end-point with parameters cmd=get_entry and entry=fileName to get a file named fileName.

Example request

In this example, we use the Splunk_Analysis_by_Tier_5d6e5cc8_5_t file name to request the contents of the file.

GET https://nam-server-address:port/RtmDataApiServlet?cmd=get_entry&entry=Splunk_Analysis_by_Tier_5d6e5cc8_5_t  

We use our NAM public API token as the Authorization header: Authorization: Bearer yourAPItoken

Sending data directly to Elastic

NAM Server can be configured to push data to Elasticsearch directly.

  • The service is an add-on to the Data Export API.
  • JSON files with data are pushed to the Elasticsearch endpoints you configure in the NAM Server system properties.

To configure this on the NAM side

  1. Save your report with names that end in -elasticsearch. This tells NAM that you want to send data to Elasticsearch rather than generate files that you can get with API calls.

  2. Add the report names (now named *-elasticsearch) to the partner-export.yaml file.

    1. On the NAM Server, open Admin console.

    2. On the Admin console page, open Configuration file manager.

    3. Search for and select partner-export.yaml.

    4. In the edit box, edit the contents of the file to add the name of your report in this format: <space>-<space><reportname>

      For example:

       - MyReportName
      

      where MyReportName is the name under which you saved the report.
      The report names needs to end in -elasticsearch.

    5. Select Save.

    6. Restart your NAM Server after you change this file.

  3. Set these system properties on the NAM Server to configure the connection to your Elasticsearch installation. Add the properties if they do not already exist on your system.
    In the NAM Server navigation menu, select Admin console and open the General properties editor.

    Property Description
    ELASTIC_SEARCH_HOST Address of the machine where Elastic is installed (default: localhost).
    ELASTIC_SEARCH_PORT Port at which Elastic can be accessed (default: 9200).
    ELASTIC_SEARCH_SEC Whether communication with Elastic is over SSL (default: false).
    ELASTIC_SEARCH_INDEX Name of the index used for indexing data from NAM (default: dcrum).
    ELASTIC_SEARCH_USER and ELASTIC_SEARCH_PASS Credentials used to connect with Elastic. For details, see Elastic help: Setting Up User Authentication.
    ELASTIC_SEARCH_QUERY Optional parameters sent with requests to Elastic. This can be used to attach pipelines to the process when a value is set to pipeline=myPipelineName. For details, see Elastic help: Pipeline Definition.
    ELASTIC_SEARCH_VERSION If you are using an old (version 5) Elastic, set this to ensure proper indexing of text data.

You can enable extensive logging of requests (with payloads) to Elastic.

Elasticsearch and Kibana

Enable Elasticsearch-Kibana support

  1. Select Admin console from the NAM Server navigation menu and open General properties editor.

  2. Search for "elastic" to list Elastic-relevant properties and set their values to match your configuration.
    If you don't see the Elastic properties, select Add new to create them.

    Property Description
    ELASTIC_SEARCH_HOST Address of the machine where Elastic is installed (default: localhost).
    ELASTIC_SEARCH_PORT Port at which Elastic can be accessed (default: 9200).
    ELASTIC_SEARCH_SEC Whether communication with Elastic is over SSL (default: false).
    ELASTIC_SEARCH_INDEX Name of the index used for indexing data from NAM (default: dcrum).
  3. Import the elk-tier-example-elasticsearch report from the kibana-*.zip package.

    To import a report:

    1. Select DMI > Export/Import in the NAM Server navigation menu.
    2. Select Choose file to browse for the file.
    3. Select Import from file.
  4. Add the report name to %PATH_TO_CAS%\config\partner-export.yaml.

    # List of reports exported through RtmDataAPI (Partner friendly export engine)  
    # ##########  BSM  ##########  
     - bsmdata-EUE_RealTime_Software_Services_Metrics_MasterSlave  
     - bsmdata-EUE_RealTime_Tier_Performance_new_dims  
     - bsmdata-EUE_RealTime_Tier_UserImpact_new_dims  
    # ##########  Splunk  ##########  
     - Splunk_Analysis_by_Tier  
    # ######### Elastic Search #########  
     - elk-tier-example-elasticsearch  
    
  5. Select Admin console from the NAM Server navigation menu and open Restart report server.

Install the NAM dashboard for Kibana

To install the NAM dashboard for Kibana

  1. Add a dcrum index in Elasticsearch through the Kibana interface.
    NAM-Kibana: add a dcrum index
  2. Import visualizations and the dashboard (see kibana-*.zip for both files).
    NAM-Kibana: Import visualizations and dashboard
  3. (Optional) Change the default formatting of the following fields: tierConn, tierApplPerf, and appHealthIdx, to Number with format 0,0.[0].
    NAM-Kibana: change default formatting
    NAM-Kibana: change default formatting
  4. (Optional) Define custom coloring for selected metrics (see visualization-colorMapping.json in kibana-*.zip for a definition).
    NAM-Kibana: define custom coloring
  5. Try it out.
    NAM-Kibana: try it out

Customize data display in Kibana

If you want to customize the way data is displayed in Kibana

  1. Create your own report in NAM and save it. The report name has to end with -elasticsearch (like the example report).
  2. Add the report name to the %PATH_TO_CAS%\config\partner-export.yaml file.
  3. Restart your NAM Server.

Data from the selected report will be sent to Elasticsearch and visible in Kibana.

Splunk plugin

Splunk plugin describes a demo application that shows NAM-Splunk integration. It presents how to read data from the NAM Server using the REST API and provides a built-in dashboard for data visualization similar to the Health dashboard available on the NAM Server.