Custom queries, segmentation, and aggregation of session data

Dynatrace captures detailed user session data each time a user interacts with your monitored application. This data includes all user actions and high level performance data. Using either the Dynatrace API or Dynatrace User Sessions Query Language (USQL), you can easily run powerful queries, segmentations, and aggregations on this captured data. To assist you, this topic provides detail about keywords and functions, syntax, working with Real User Monitoring tables, automated queries, and more.

Select your preferred approach:

Keywords and functions

The following keywords have been defined to access user session data:

AND, AS, ASC, BETWEEN, BY, DESC, DISTINCT, FALSE, FROM, GROUP, IN, IS, JSON, LIMIT, NOT, NULL, OR, ORDER, SELECT, STARTSWITH, TRUE, WHERE, LIKE, FILTER

The following functions have been defined to access user session data:

SUM, MAX, MIN, AVG, MEDIAN, COUNT, YEAR, MONTH, DAY, HOUR, MINUTE, DATETIME, TOP, PERCENTILE, KEYS

Keywords, functions, and column names are case-insensitive. String-matches in WHERE conditions are case-sensitive.

Syntax

A typical query is built from the following keywords:

SELECT <columns> FROM <table> WHERE <condition> GROUP BY <grouping> ORDER BY <ordering>

However, the only mandatory elements are SELECT <columns> and FROM <table>.

sqlcopydownload
SELECT browserType, userId, city, AVG(userActionCount) AS "Average user action count", AVG(duration) AS "Average duration", count(*) AS "Sessions", SUM(totalErrorCount) AS "Errors" FROM usersession WHERE ip between '52.179.11.1' and '52.179.11.255' GROUP BY browserType, userId, city

Frequently used keywords

SELECT <columns>

Selects one or more columns from the specified data table or performs aggregation functions from the set of supported functions.

sqlcopydownload
columns: [DISTINCT] <column>, <column>, ... | function(<parameter>) |
<column> AS <alias> | JSON

sqlcopydownload
SELECT country, city, browserfamily FROM usersession
SELECT DISTINCT country, city, useractioncount FROM usersession
SELECT country, city, avg(duration) AS average FROM usersession GROUP BY country, city

FUNNEL

Allows you to use a predefined funnel format for a query. Can be used to chart the flow of specific user actions. Can also be combined with custom session properties and other conditions.

It changes the syntax of any query to the following:

sqlcopydownload
SELECT FUNNEL (<condition> AS <alias>, <condition>, ...) FROM <table> WHERE <condition>

sqlcopydownload
SELECT COUNT(*) FROM usersession where useraction.name = "AppStart"
SELECT COUNT(*) FROM usersession where useraction.name = "AppStart" AND useraction.name = "searchJourney"
SELECT COUNT(*) FROM usersession where useraction.name = "AppStart" AND useraction.name = "searchJourney"  AND useraction.name = "bookJourney"

sqlcopydownload
SELECT FUNNEL (useraction.name = "AppStart", useraction.name = "searchJourney", useraction.name = "bookJourney")
FROM usersession

sqlcopydownload
SELECT FUNNEL (useraction.name="login", useraction.name = "searchJourney", useraction.name = "bookJourney")
FROM usersession

FROM <table>

You can only specify one table. Tables for user session data are usersession, useraction, and userevent.

sqlcopydownload
SELECT country, city, browserfamily FROM usersession
SELECT name, starttime, endtime, duration FROM useraction ORDER BY duration DESC

WHERE <condition>

You can combine multiple conditions using Boolean logic and parentheses within the WHERE clause, such as WHERE (city = 'Barcelona' AND country = 'Spain') to include only cities named Barcelona that are in Spain.

sqlcopydownload
condition: (condition AND condition) | (condition OR condition) | field IN(...) |
field IS <value> | field IS NULL | field = <value> | field > <value> | field < <value> |
field <> <value> | field IS NOT <value> | field BETWEEN <value> AND <value> | ...

However, only the right-hand side of conditions can contain a value, so you can't compare between two fields.

sqlcopydownload
SELECT country, city, browserfamily FROM usersession WHERE country = 'Albania' AND screenWidth > 1000
SELECT top(country, 20), top(city, 20), TOP(duration, 10), avg(duration) AS average
FROM usersession **WHERE** duration BETWEEN 1000 AND 2000 GROUP BY top(country, 20), top(city, 20), TOP (duration, 10)

GROUP BY <grouping>

Whenever fields are aggregated, you must specify corresponding GROUP BY keywords to indicate how the aggregation is to be performed.

sqlcopydownload
grouping: <column>, ...

sqlcopydownload
SELECT city, count(*) FROM usersession GROUP BY city
SELECT MONTH(starttime) as month, count (*) FROM usersession
GROUP BY month

LIMIT <limit>

Allows you to limit the number of returned results. For example, you can select only the top 10 results when it is combined with ordering.

The framework always applies an upper limit to prevent system overload. If LIMIT is not used, 50 results are returned by default.

sqlcopydownload
SELECT city, starttime FROM usersession ORDER BY starttime DESC LIMIT 10

LIMIT can also be used to increase the number of results in cases where the LIMIT clause is missing, because then a default limit is applied.

ORDER BY <ordering>

Allows you to order the results by columns, in either ascending or descending order. The order is ascending if not specified.

The ordering is done by frequency. For example, the top 5 returned cities are the most frequently occurring ones. By specifying a field in the ORDER BY clause, you can add a sort-by value for strings, dates, and numbers.

Ordering by enums or by function values such as AVG and SUM orders the returned results, but you may not get the top items. For example, if you request the top 5 results by AVG(duration), requesting only 10 may add results even at the top.

sqlcopydownload
ordering: <column> ASC | <column> DESC | <column>, ...

sqlcopydownload
SELECT useraction.name, starttime FROM usersession ORDER BY starttime DESC

sqlcopydownload
SELECT DISTINCT city, COUNT(*) FROM usersession ORDER BY COUNT(*) DESC

sqlcopydownload
SELECT avg(duration) AS average, count(*) as number, day(startTime) as startDay
FROM usersession where duration < 2000
GROUP BY startTime
ORDER BY average

sqlcopydownload
SELECT avg(duration) AS average, count(*) as number, day(startTime) as startDay
FROM usersession where duration < 2000
GROUP BY startTime
ORDER BY number DESC, average ASC

LIKE

Allows you to compare data with an expression by using wildcard characters to match the specified pattern. You can use the following characters:

• % or *: Matches any string of 0 or more characters
• ?: Matches any single character

String values are case-sensitive. For example, SELECT city FROM usersession WHERE userId LIKE "*dynatrace*" matches me@dynatrace.com but not me@dynaTrace.com. To avoid this, use the ? wildcard character as in this example: SELECT city FROM usersession WHERE userId LIKE "*dyna?race*"

sqlcopydownload
SELECT CONDITION(COUNT(userSessionId), WHERE useraction.name LIKE '*search.html'),
       CONDITION(COUNT(userSessionId), WHERE useraction.name LIKE '*booking-payment1.html')
FROM usersession
WHERE city LIKE "%York"
    OR city LIKE "S*Francisco"
    AND city LIKE "L*inz"

sqlcopydownload
SELECT FUNNEL (useraction.name LIKE "*start", useraction.name LIKE "Jou%rney", useraction.name LIKE "bookJourn*ey") FROM usersession

sqlcopydownload
SELECT COUNT(*) FROM usersession where useraction.name LIKE "*start"
SELECT COUNT(*) FROM usersession where useraction.name LIKE "*start" AND useraction.name LIKE "Jou%rney"
SELECT COUNT(*) FROM usersession where useraction.name LIKE "*start" AND useraction.name LIKE "Jou%rney" AND useraction.name LIKE "bookJourn*ey"

FILTER

Allows you to filter for functions that have numeric values, thereby displaying only specific results from aggregations.

sqlcopydownload
SELECT useraction.application,
AVG(usersession.doubleProperties.bookings)
FILTER > 1500
FROM usersession
WHERE usersession.doubleProperties.bookings IS NOT NULL
GROUP BY useraction.application

Frequently used functions

MIN(field)

Queries the minimum value of a numeric or date field.

sqlcopydownload
SELECT MIN(duration), MAX(duration), AVG(duration), MEDIAN(duration)
FROM usersession

MAX(field)

Queries the maximum value of a numeric or date field.

sqlcopydownload
SELECT MIN(duration), MAX(duration), AVG(duration), MEDIAN(duration)
FROM usersession

AVG(field)

Queries the average value of a numeric or date field. May be NaN if the field is always null.

sqlcopydownload
SELECT MIN(duration), MAX(duration), AVG(duration), MEDIAN(duration)
FROM usersession

MEDIAN(field)

Queries the median value of a numeric or date field.

sqlcopydownload
SELECT MIN(duration), MAX(duration), AVG(duration), MEDIAN(duration)
FROM usersession

SUM(field)

Computes the sum of a numerical field.

sqlcopydownload
SELECT TOP(name, 20), SUM(duration) FROM useraction
GROUP BY name

COUNT(field), COUNT(*), COUNT(DISTINCT field)

Counts the number of rows that match.

COUNT(*): Counts the number of matching items.

COUNT(<field>): Counts the number of matching items where \<field> is not null.

COUNT(DISTINCT <field>): Counts the number of different values for \<field> within the selected items.

sqlcopydownload
SELECT country, COUNT(*), COUNT(city), COUNT(DISTINCT city)
FROM usersession
GROUP BY country

TOP(field, n)

Returns the top \<n> results from a field. The default is 1 (the top value) if n is not specified.

sqlcopydownload
SELECT TOP(name, 20), SUM(duration)
FROM useraction
GROUP BY name

sqlcopydownload
SELECT TOP(country, 20), TOP(city, 3), COUNT(*)
FROM usersession
GROUP BY country

YEAR(datefield), MONTH(datefield), DAY(datefield), HOUR(datefield), MINUTE(datefield)

Returns the given element extracted from a date field.

YEAR: The 4-digit year. MONTH: The month number between 1 and 12 DAY: The day of the month between 1 and 31. HOUR: The hour value between 0 and 23. MINUTE: The minute value between 0 and 59.

sqlcopydownload
SELECT starttime,
DATETIME(starttime), YEAR(starttime), MONTH(starttime), DAY(starttime), HOUR(starttime), MINUTE(starttime)
FROM usersession
ORDER BY starttime
DESC

DATETIME(datefield [, format [, interval]])

Formats the selected date field with the given format string. The default format is yyyy-MM-dd HH:mm.

The allowed letters within the format string are:

• y: year
• M: month
• d: day of month
• H: hour (0-23)
• h: hour (1-12)
• m: minute
• s: second
• E: day of week (Mon-Sun)

The year|month|week intervals are for a single interval. For d (days), h (hours), m (minutes), or s (seconds), you can use a number followed by the letter for the format string, such as 5m. For example, SELECT DISTINCT DATETIME(starttime, 'HH:mm', '5m'), count(*) FROM usersession counts sessions in five-minute time blocks.

sqlcopydownload
SELECT DATETIME(starttime, 'yyyy-MM') FROM usersession
SELECT DISTINCT DATETIME(starttime, 'HH:mm', '5m'), count(*) FROM usersession

CONDITION(function, condition)

Allows you to combine multiple functions with various conditions.

The allowed functions within the format string are as follows:

• MIN()
• MAX()
• AVG()
• SUM()
• PERCENTILE()
• MEDIAN()
• COUNT()

You can combine multiple conditions using Boolean logic and parentheses with the CONDITION function, such as CONDITION(COUNT(*), WHERE city = 'Barcelona' AND country = 'Spain') to include only cities named Barcelona that are in Spain.

sqlcopydownload
CONDITION(function, condition)

condition:
(condition AND condition) | (condition OR condition) | field IN(...) |
field IS <value> | field IS NULL | field = <value> | field > <value> | field < <value> |
field <> <value> | field IS NOT <value> | field BETWEEN <value> AND <value> | ...

You can also use the FILTER clause filter for functions that have numeric values, thereby displaying only specific results from aggregations.

sqlcopydownload
SELECT CONDITION(COUNT(usersessionId), WHERE userActionCount > 2 AND useraction.name = "search.jsp") FILTER > 1000, city FROM usersession GROUP BY city

sqlcopydownload
SELECT CONDITION(COUNT(usersessionId), WHERE userActionCount > 2 AND useraction.name = "search.jsp") FROM usersession

SELECT CONDITION(SUM(usersession.duration), WHERE name = "index.jsp") AS c1, CONDITION(SUM(usersession.duration), WHERE name = "search.jsp") AS c2, CONDITION(SUM(usersession.duration), WHERE name IS NOT "index.jsp" AND name IS NOT "search.jsp") AS c3 FROM useraction WHERE (duration > 1000 OR usersession.userActionCount > 4)

SELECT CONDITION(SUM(usersession.duration), WHERE name = "index.jsp") AS c1 FROM useraction WHERE (duration > 1000 OR usersession.userActionCount > 4) ORDER BY c1

SELECT DATETIME(CONDITION(MIN(startTime ), WHERE useraction.application = "RUM Default Application" ), "yyyy-MM-dd" ) FROM usersession

PERCENTILE

Represents a value below which there's a percentage of data points lower in value. Useful in detecting the speed of your application for customers who receive the slowest response time.

sqlcopydownload
SELECT name, usersession.country, usersession.browserFamily,
AVG(duration),
MEDIAN(duration),
PERCENTILE(duration, 99)
FROM useraction
WHERE useraction.name = "easytravel/rest/login"
GROUP BY usersession.country, usersession.browserFamily, name
ORDER BY usersession.continent

KEYS(customProperty)

Returns keys of user action or user session properties according to the property data type defined in the argument.

Check the table below to understand whether keys of user action properties or keys of user session properties are returned.

The <dataType> part of the function can take the following values:

• string
• long
• double
• date

sqlcopydownload
SELECT KEYS(stringProperties) FROM useraction WHERE application = "easyTravel demo application"
SELECT KEYS(useraction.longProperties) FROM usersession WHERE applicationType="WEB_APPLICATION" ORDER BY keys(useraction.longProperties)
SELECT KEYS(usersession.stringProperties) FROM useraction WHERE usersession.city ="Berlin"

For fetching distinct keys of action or session properties, use DISTINCT KEYS(customProperty).

sqlcopydownload
SELECT DISTINCT KEYS(stringProperties) FROM useraction WHERE useraction.application = "easyTravel demo application" ORDER BY keys(stringProperties)
SELECT DISTINCT KEYS(doubleProperties) FROM usersession

Mathematical operations

The following operations are supported as part of queries:

• operations on numbers
• operations on numeric and dateTime fields
• operations on certain functions such as YEAR, MONTH, DAY, HOUR, MINUTE
• operations on numeric values and display them in different uses of measurement

Syntax

Number/NumericField/DateTimeField/Function OPERATOR Number/NumericField/DateTimeField/Function

Function: YEAR, MONTH, DAY, HOUR or MINUTE Operator: +, -, *, /, %, or MOD

sqlcopydownload
SELECT 7 + 80 * 100, duration + startTime, MONTH(startTime) - 1
FROM usersession

Conditions

All conditions must start with an identifier, such as a field name, and must be compared against a value. Two fields cannot be compared against each other. Note: Quoted text is always case-sensitive.

Basic operators

The basic operators for comparison are =, !=, <>, <, >, <=, >=, IS, and IS NOT.

To check if the value of a field is present, compare the field against NULL.

sqlcopydownload
SELECT userId FROM usersession WHERE userActionCount > 3

Ranges

Ranges are handled by keywords such as BETWEEN, <lowerLimit>, and <upperLimit>.

sqlcopydownload
SELECT DISTINCT ip FROM usersession
WHERE ip BETWEEN "192.168.0.0" AND "192.168.255.255"

Sets

The IN keyword can be used shorten "WHERE" <field> = val1 OR <field> = val2 OR <field> = val3.

sqlcopydownload
SELECT userId FROM usersession WHERE city IN ("NEW YORK", "San Francisco")

String conditions

The "STARTSWITH" string condition checks whether a string or an enum field starts with the specified text.

sqlcopydownload
SELECT city FROM usersession WHERE userId STARTSWITH "dynatrace"

Datetime values

When conditions are run on a datetime field, the following value formats are supported:

For the formats where time is optional, the following time formats are supported:

sqlcopydownload
SELECT starttime FROM usersession WHERE starttime > "8.8.2018 8:00"

Advanced syntax constructs

Query IP ranges

The IP field can be queried for ranges of addresses. Both BETWEEN ip > <lower ipaddress range> AND ip < <upper ipaddress range> or BETWEEN <lower ipaddress range> AND <lower ipaddress range> work.

sqlcopydownload
SELECT * FROM usersession WHERE ip > '211.44.94.0' AND ip < '212.113.5.0'
SELECT * FROM usersession WHERE ip BETWEEN '211.44.94.0' AND '212.113.5.0'

Query timeframe selector

You can use the following keywords to select the start time and end time as defined in the timeframe selector.

• TIME_FRAME_START
• TIME_FRAME_END

sqlcopydownload
SELECT * FROM usersession WHERE startTime >= $TIME_FRAME_START AND endTime < $TIME_FRAME_END

Query relative timeframe

You can select a timeframe relative to the time when the query was run. The current time is expressed with the $NOW variable.

$NOW [+/-] DURATION("[number]TIME_UNIT")

The following time units are supported to express the duration:

• y: year
• q: quarter
• M: month
• d: day
• w: week
• h: hour
• m: minute
• s: second

sqlcopydownload
SELECT * FROM usersession WHERE startTime >= $NOW - DURATION("1q") AND endTime <= $NOW
SELECT * FROM useraction WHERE startTime BETWEEN $NOW - DURATION("2h") AND $NOW
SELECT * FROM useraction WHERE usersession.startTime >= $TIME_FRAME_START - DURATION("2h") AND $NOW - DURATION("1h")

Secondary tables for usersession, useraction, and userevent

When using SELECT with usersession, useraction, or userevent, columns from the other table can be accessed and included in the results by prefixing column names with the table name.

sqlcopydownload
SELECT city, useraction.name FROM usersession
SELECT usersession.city, name FROM useraction

sqlcopydownload
SELECT * FROM usersession
WHERE userevent.application = "a" OR useraction.application="a"

sqlcopydownload
SELECT * FROM usersession
WHERE useraction.name = "a" and useraction.name = "b"

sqlcopydownload
SELECT usersession.*, * FROM useraction
WHERE useraction.name = "a" and useraction.duration > 1000

Filters for primary tables

Filters can be applied only to primary tables. Consider the following example:

sqlcopydownload
SELECT useraction.name FROM usersession WHERE useraction.name=”abc”

sqlcopydownload
SELECT name FROM useraction WHERE name=”abc”

SELECT * FROM table

sqlcopydownload
SELECT * FROM usersession
SELECT useraction.* FROM usersession
SELECT city, useraction.* FROM usersession
SELECT *, useraction.* FROM usersession

JSON export

sqlcopydownload
SELECT usersessionId, browserFamily, useraction.name, useraction.duration, JSON
FROM usersession LIMIT 5

Escaping strings

String literals can be put within single or double quotes. However, if you want to use the same quotation mark inside the string, simply double it.

sqlcopydownload
SELECT * FROM usersession WHERE userId = "some 'custom' name for ""my user"""
SELECT * FROM usersession WHERE userId = 'some ''custom'' name for "my user"'

Funnel charting

Funnel charting allows you to track steps through your digital service and investigate the areas of struggle faced by your users. In conjunction with Session Replay, this functionality allows you to see at which point your user is struggling in your application.

sqlcopydownload
SELECT FUNNEL(
useraction.name = "AppStart (easyTravel)" AS "Open easytravel",
useraction.name = "searchJourney" AS "Search journey",
useraction.name = "bookJourney" AS "Book journey",
useraction.matchingConversionGoals = "Payment" OR useraction.matchingConversionGoals = "booking-finished" AS "Booked")
FROM usersession

You can also filter for a specific segment. An example of this is using Session properties to extract the list of high priority customers.

sqlcopydownload
SELECT FUNNEL(
useraction.name = "AppStart (easyTravel)" AS "Open easytravel",
useraction.name = "searchJourney" AS "Search journey",
useraction.name = "bookJourney" AS "Book journey",
useraction.matchingConversionGoals = "Payment" OR useraction.matchingConversionGoals = "booking-finished" AS "Booked")
FROM usersession
WHERE stringProperties.memberstatus="GOLD"

Available user session data tables and fields

For user session data, the usersession, useraction, and userevent tables are available. Secondary tables for usersession, useraction, and userevent includes a description of how data in one of those tables is available in the other.

Run USQL queries for custom reports

A REST interface allows you to get results for your custom queries. All you need is to create a unique API token with the User session query language privilege. The ability to query user session data this way is useful in automated testing, data verification, and other automated functions. It includes the following endpoints:

/table: Returns the data as a flat table, even when grouping by various items and performing hierarchical aggregations against the user session data.

/tree: Returns the data as a full hierarchical tree based on the input.

The following query-parameters are available:

query: Needs to be encoded when put into a URL, for example, %20 instead of spaces. startTimestamp/endTimestamp: Allows you to define points in time, passed as the number of milliseconds since the Unix epoch. If not specified, this defaults to the last two hours.

sqlcopydownload
curl --location --insecure -H "Content-Type: application/json" -H "Authorization: Api-Token _token_" \
-XGET "https://{your-environment-id}.live.dynatrace.com/api/v1/userSessionQueryLanguage/table?query=select%20city,count(*)%20from%20usersession%20group%20by%20city"

sqlcopydownload
{
    "columnNames": ["city",    "count(*)"],
    "values": [
        ["Dublin",    23],
        ["N. Virginia (Amazon)",    80],
        ["Portland",    56]
    ]
}

plaintextcopydownload
curl --location --insecure -H "Content-Type: application/json" -H "Authorization: Api-Token _token_" \
-XGET "https://{your-environment-id}.live.dynatrace.com/api/v1/userSessionQueryLanguage/tree?query=select%20country,city,count(*)%20from%20usersession%20group%20by%20country,city"

sqlcopydownload
{
    "branchNames": ["country",    "city"],
    "leafNames": ["count(*)"],
    "values": {
        "United States": {
            "Portland": [56],
            "N. Virginia (Amazon)": [83]
        },
        "Ireland": {
            "Dublin": [24]
        }
    }
}

Learn more about user sessions API.

Convert queries into USQL custom metrics

You can convert some queries into USQL custom metrics for your web, mobile, and custom applications.

1. In the Dynatrace menu, go to Query user sessions.
2. Enter the query, and then select Run query.
   For a list of supported fields, see the detailed guides for your web, mobile, and custom applications.
3. Select Create custom metric.
4. Enter the metric name, and then review the proposed settings.

Limitations

• Dynatrace stores and retains Real User Monitoring (user actions and user sessions) for a limited period of time. See Data retention periods for details.

• The default result set is 50 but the number of results can be increased to a maximum of 5000 by using the LIMIT keyword.

• The number of potential maximum bucketed results is limited to a maximum of 100,000. The default is 10,000.
  This affects how TOP() is applied when DISTINCT or GROUP BY is used. If no TOP() is specified, 10,000 possible results are spread evenly across the specified columns. These default values can be overwritten by specifying a TOP() for each column. The multiplied TOP()- values cannot exceed 100,000 results.

  Examples

  • The TOP() function can be used to increase the number of different values per aggregation.
  • The maximum number of different results per aggregation is limited to 1000.
  • The following query uses at most 10,000 theoretical possible results: select browserFamily, city, count * FROM usersession group by browserFamily, city
  • The following query includes TOP() and can therefore use up to 100,000 (100 * 1000) theoretical possible results: select TOP(browserFamily, 1000), TOP(city, 100), count * FROM usersession group by browserFamily, city

• Joins aren't allowed.

• Only one table is allowed per SELECT.

• Searches for string-values with regular expressions aren't supported.

• Two different fields can't be compared. For example WHERE field1 = field2 doesn't work.

• WHERE conditions only work on fields, so neither WHERE true nor WHERE COUNT(*) > 3 is supported.

• Only closed user sessions can be queried. Live user sessions aren't taken into account.

• Ordering is partially supported.

  For example, ordering by mathematical operation is not yet supported:

  sqlcopydownload
  SELECT endTime - startTime AS duration FROM usersession ORDER BY duration

• Functions aren't allowed in the GROUP BY clause. Therefore, if you want to group by month, specify an alias.

• FUNNEL cannot be used with the SELECT * functions, keywords such as JSON, and the GROUP BY, ORDER and LIMIT statements.

• For mathematical operations, support for GROUP BY, ORDER BY, and other operations on functions is not available.

• A maximum of 10 conditions can be applied to FUNNEL.