Export user session data

This topic describes the necessary configuration and other requirements needed to export user session data via HTTP.

Dynatrace can be configured to continually send user session data to a user-provided HTTP Webhook endpoint. Once enabled, user session export automatically sends JSON data for all monitored user sessions in your environment to the configured HTTP endpoint. The HTTP server that provides the Webhook must listen for HTTP PUT or HTTP POST requests on a URL specified as part of the configuration.

The data is sent in bulk to improve performance, with a flush every few seconds to keep the data rate near real-time. The data format is one JSON document per line, so you must split the data by line to get valid JSON documents.

Session data that is streamed includes:

  • All user sessions
  • User actions
  • High-level timing and error data

Export of user session data provides a way to leverage near real-time data within external data sources for big data analysis. Streaming session data to a third party system preserves your ability to continue using your current analytics tools to combine and analyze streamed data, including data mined from other sources.

You can define up to three endpoints for your user session export.

Define endpoint

To add endpoint for your user session export:

  1. Go to Settings > Integration > User session exports.
  2. Select Add item.
  3. Type the HTTP Webhook Endpoint URL to be used for sending user session data.
  4. Make sure Enable user session export is turned on. User session export is enabled by default.
  5. optional Turn on Use POST method (instead of PUT) if you're using POST in the command to send data to the HTTP endpoint.
  6. optional Under Basic authentication, turn on Activate and type the user name and the password.
  7. optional Under Send data directly to Elasticsearch, turn on Activate and fill in the Elasticsearch information as described in the the next section.
  8. optional Define the scope of your export by selecting a management zone.
  9. optional In the same section, you can also disable UI notifications for failing exports and add special settings provided by Dynatrace support for troubleshooting.
  10. Select Save changes.

Send data directly to Elasticsearch

You can also send data in bulk to Elasticsearch:

  1. Under Send data directly to Elasticsearch, turn on Activate.
  2. In the same section, specify the name of the index where the data is to be sent and the type of documents in the Elasticsearch index.

The specified Elasticsearch index to which data is sent must be reachable from Dynatrace. For SaaS, this means the server must be publicly available. You can use an SSL certificate and basic authentication to secure access.

Note: The export-URL needs to end with _bulk. The default port for Elasticsearch is 9200. For example: http://<host>:9200/_bulk.

HTTP endpoint setup

If not using the Elasticsearch destination, the data is sent to an HTTP Endpoint via a HTTP PUT call with a bulk JSON payload.

  • Operation: PUT (or POST if configured).
  • Path: <as configured>.
  • Mime-type: application/json, however sent as multiple JSON documents in bulk line-by-line.
  • Response Body: The handler should respond with HTTP 200 "OK".
  • Description: The handler receives up to 1000 JSON documents, usually less as a periodic flush of all available sessions is performed at least once every minute.

Dynatrace doesn't read the response entirely, only the HTTP status return code.

Please note that the data should be processed quickly. Consider using some asynchronous handling to perform more processing. Dynatrace will time out requests quickly and may back off sending more data for some time to avoid overloading the number of open connections.

Sample code when using the Jersey REST framework

The following sample code uses the open source Jersey REST framework to provide the necessary HTTP endpoint for receiving the data.

@Path("/export/")
public class ExportREST {
    ...

    @PUT
    @Produces(MediaType.APPLICATION_JSON)
    @Path("events")
    public JResponse<String> jsonEvents(final String data) {
        ...
        // split the bulk into single documents
        final String[] lines = StringUtils.split(data, '\n');
        for(String line : lines) {
            ...
            // handle the JSON-data
        }

        return JResponse.ok("")
            .header(HttpHeaders.SERVER, "HTTP Endpoint for Dynatrace Session data export")
            .build();
    }
}

Manual verification

  1. Expand you user session export definition and scroll to the bottom.
  2. Select Test export to verify the data stream. A successful response reads Code: 200 - Message: OK.

Alternatively, you can execute a curl command similar to the following:

curl -v -H "Content-Type: application/json" -X PUT -d '{"visitorId":"14804637803609BCTKP776NMJBOIF3R8OD6R0E4NQALJO","visitId":"16229530","startTime":1480463779085,"endTime":1480463784889,"visitType":"SYNTHETIC"}' http://localhost:3000/export/events

Possible additional flags for the command include the following:

  • insecure: Disables the SSL check.
  • http1.1: If you receive the message REFUSED_STREAM when executing the curl command, it may indicate that curl tries to use HTTP 2.0, but the receiver side cannot handle it.

The following shows a sample response returned when using the curl command above with the specified flags:

*   Trying 192.168.1.4...
* Connected to 192.168.1.4 (192.168.1.4) port 443 (#0)
> PUT /export/events HTTP/1.1
> Host: 192.168.1.4:443
> User-Agent: curl/7.47.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 156
>
* upload completely sent off: 156 out of 156 bytes
< HTTP/1.1 200 OK
< Server: Sample REST Server by [name]
< Date: Tue, 24 April 2018 07:02:50 GMT
< Transfer-encoding: chunked
< Content-type: application/json
< Vary: Accept-Encoding
<
* Connection #0 to host 192.168.1.4 left intact

Sample dataset

You can request sample data based on your own user session data.

  1. Expand you user session export definition and scroll to the bottom.
  2. Select Download Sample Export Data to get up to 50 user sessions from the previous week. A bulk download provides a JSON document for each user session, one JSON document per line, so you can see how the data looks.

The following sample shows three separate sessions sent in bulk as three lines.

{"tenantId":"xxx","userSessionId":"1394_1008","startTime":1511441593539,"endTime":1511441716896,"duration":123357,"internalUserId":"1394","userType":"REAL_USER","applicationType":"MOBILE_APPLICATION","bounce":false,"newUser":false,"userActionCount":1,"totalErrorCount":1,"ip":"2001:1800:ffff:eac2:63f5:568e:b3c6:6c54","geolocationContinentName":"North America","geolocationCountryName":"United States","geolocationRegionName":"Florida","geolocationCityName":"Delray Beach","osFamilyName":"Windows","osVersionName":"Windows 10.0 Mobile","manufacturer":"Nokia","device":"L-930","userId":"fearghasbag","screenHeight":1920,"screenWidth":1080,"screenOrientation":"PORTRAIT","displayResolution":"FHD","hasCrash":true,"isp":"SWCP-AS - Southwest Cyberport","stringTags":{ },"numTags":{ },"dateTags":{ },"userActions":[ { "name":"Checkout","type":"Custom","startTime":1511441593539,"endTime":1511441593562,"duration":23,"application":"easyTravel Demo","speedIndex":null,"errorCount":0,"apdexCategory":"UNKNOWN","networkTime":null,"serverTime":null,"frontendTime":null,"documentInteractiveTime":null,"failedImages":null,"failedXhrRequests":null,"httpRequestsWithErrors":null,"thirdPartyResources":null,"thirdPartyBusyTime":0,"cdnResources":null,"cdnBusyTime":0,"firstPartyResources":null,"firstPartyBusyTime":0,"hasCrash":false,"domCompleteTime":null,"domContentLoadedTime":null,"loadEventStart":null,"loadEventEnd":null,"navigationStart":null,"requestStart":null,"responseStart":null,"responseEnd":null,"visuallyCompleteTime":null } ],"events":[ { "type":"UserTag","name":"fearghasbag","startTime":1511441593562,"application":"MOBILE_APPLICATION-752C288D59734C79" } ],"errors":[ { "type":"Crash","name":"ThrowAsync","startTime":1511441716896,"application":"MOBILE_APPLICATION-752C288D59734C79"}]}
{"tenantId":"xxx","userSessionId":"1394_1008","startTime":1511441593539,"endTime":1511441716896,"duration":123357,"internalUserId":"1394","userType":"REAL_USER","applicationType":"MOBILE_APPLICATION","bounce":false,"newUser":false,"userActionCount":1,"totalErrorCount":1,"ip":"2001:1800:ffff:eac2:63f5:568e:b3c6:6c54","geolocationContinentName":"North America","geolocationCountryName":"United States","geolocationRegionName":"Florida","geolocationCityName":"Delray Beach","osFamilyName":"Windows","osVersionName":"Windows 10.0 Mobile","manufacturer":"Nokia","device":"L-930","userId":"fearghasbag","screenHeight":1920,"screenWidth":1080,"screenOrientation":"PORTRAIT","displayResolution":"FHD","hasCrash":true,"isp":"SWCP-AS - Southwest Cyberport","stringTags":{ },"numTags":{ },"dateTags":{ },"userActions":[ { "name":"Checkout","type":"Custom","startTime":1511441593539,"endTime":1511441593562,"duration":23,"application":"easyTravel Demo","speedIndex":null,"errorCount":0,"apdexCategory":"UNKNOWN","networkTime":null,"serverTime":null,"frontendTime":null,"documentInteractiveTime":null,"failedImages":null,"failedXhrRequests":null,"httpRequestsWithErrors":null,"thirdPartyResources":null,"thirdPartyBusyTime":0,"cdnResources":null,"cdnBusyTime":0,"firstPartyResources":null,"firstPartyBusyTime":0,"hasCrash":false,"domCompleteTime":null,"domContentLoadedTime":null,"loadEventStart":null,"loadEventEnd":null,"navigationStart":null,"requestStart":null,"responseStart":null,"responseEnd":null,"visuallyCompleteTime":null } ],"events":[ { "type":"UserTag","name":"fearghasbag","startTime":1511441593562,"application":"MOBILE_APPLICATION-752C288D59734C79" } ],"errors":[ { "type":"Crash","name":"ThrowAsync","startTime":1511441716896,"application":"MOBILE_APPLICATION-752C288D59734C79"}]}
{"tenantId":"xxx","userSessionId":"1394_1008","startTime":1511441593539,"endTime":1511441716896,"duration":123357,"internalUserId":"1394","userType":"REAL_USER","applicationType":"MOBILE_APPLICATION","bounce":false,"newUser":false,"userActionCount":1,"totalErrorCount":1,"ip":"2001:1800:ffff:eac2:63f5:568e:b3c6:6c54","geolocationContinentName":"North America","geolocationCountryName":"United States","geolocationRegionName":"Florida","geolocationCityName":"Delray Beach","osFamilyName":"Windows","osVersionName":"Windows 10.0 Mobile","manufacturer":"Nokia","device":"L-930","userId":"fearghasbag","screenHeight":1920,"screenWidth":1080,"screenOrientation":"PORTRAIT","displayResolution":"FHD","hasCrash":true,"isp":"SWCP-AS - Southwest Cyberport","stringTags":{ },"numTags":{ },"dateTags":{ },"userActions":[ { "name":"Checkout","type":"Custom","startTime":1511441593539,"endTime":1511441593562,"duration":23,"application":"easyTravel Demo","speedIndex":null,"errorCount":0,"apdexCategory":"UNKNOWN","networkTime":null,"serverTime":null,"frontendTime":null,"documentInteractiveTime":null,"failedImages":null,"failedXhrRequests":null,"httpRequestsWithErrors":null,"thirdPartyResources":null,"thirdPartyBusyTime":0,"cdnResources":null,"cdnBusyTime":0,"firstPartyResources":null,"firstPartyBusyTime":0,"hasCrash":false,"domCompleteTime":null,"domContentLoadedTime":null,"loadEventStart":null,"loadEventEnd":null,"navigationStart":null,"requestStart":null,"responseStart":null,"responseEnd":null,"visuallyCompleteTime":null } ],"events":[ { "type":"UserTag","name":"fearghasbag","startTime":1511441593562,"application":"MOBILE_APPLICATION-752C288D59734C79" } ],"errors":[ { "type":"Crash","name":"ThrowAsync","startTime":1511441716896,"application":"MOBILE_APPLICATION-752C288D59734C79"}]}