Before you start, review your monitoring objectives for the report server.
- Do not use the NAM Server as a data warehousing, usage tracking, or statistical system.
- Do not expect to report all the details on all levels all the time.
- In particular, do not attempt single-user recognition reporting before making sure the environment is small enough to allow it (unless you enable on-demand access to user activity details).
Resolving problems related to the NAM Server database size:
To measure end-user experience, provide the number of pages that exceeded the threshold and (for business impact) the number of users who had problems.
Note that keeping details for every user is not the most important metric for measuring end-user experience. For troubleshooting purposes, it is useful to know which particular users had problems and to record all the details for all of them. However, in environments with many thousands of users, it is not practical to troubleshoot individual users.
Applies to DC RUM 2017
Use single-user recognition carefully and in a limited way. Use user aggregation where possible or consider using the ADS or enabling on-demand access to user activity details by storing them directly on a local disk.
The number of users is usually the most important factor for the overall number of sessions. If the NAM Server is going to be deployed in a large environment (more than 50,000 users), you should not store single-user information in the NAM Server database. Keep the number of unique users that the NAM Server is tracking in its database down to about 50,000. This number includes aggregated and individual users that the NAM Server is monitoring.
After aggregation, a NAM Server user may be an Autonomous System Number (ASN), a CIDR block, a class of users based on access speed inside an ASN (or CIDR), an individual user known by name or IP address, or a combination of these.
If individual user information has to be reported but the size of the NAM Server database is the limiting factor, you can try the following solutions:
Use ADS for single-user recognition
ADS is a solution in small and medium size environments, provided the number of pages does not exceed ADS limits.
Enable on-demand access to user activity details
If you require user activity details, but can accept them being available through drill-downs only, you can configure the NAM Server to store user-level details directly on the local disk, thus reducing the size of the database. This option is normally used when user details are not needed on a regular basis but should still be available for occasional inspection. However, remember that large amounts of disk storage may be required for user data. For more information, see Configuring NAM Server Scalability Options.
Maintain the lowest possible number of sites.
The number of sites plays an important role in memory usage, so it is important to keep the number of sites at a reasonable level. The NAM Server has a built-in limit of 20,000 sites and will not store more than that unless explicitly reconfigured. This should happen only after checking that there is still enough RAM during normal operations and that the system performs well.
In enterprise environments, the number of sites can be manually configured. In Internet environments, where sites are auto-discovered, the system can be configured to keep only the top N most active sites. The configured limits and current settings can be checked on the Advanced Properties Editor screen.
Also, keep the number of locations down to about 500. Locations can be defined manually or based on client ASNs or CIDR blocks. The top 500 ASNs would cover more traffic but with less granularity. The top 500 CIDR blocks would cover less traffic but with better granularity. Usually ASNs work better than CIDR blocks for websites.
Maintain the lowest possible number of applications, transactions, and tiers, and make their definitions simple.
The number of business hierarchy definitions is also important in memory allocation and usage. Also, a high number of complex rules in a single definition can affect the performance of database-run reports or DMI reports that use it as a dimension. There are no limits on how many applications, transactions, or tiers can be kept in the system, but it should not be in the hundreds.
Maintain the lowest possible number of operation names (URLs/queries) and servers.
The number of operations (URLs or corresponding entities from other analysis modules such as queries, Oracle Form names, and XML/Jolt operation names) also is important in memory usage. It also influences the overall number of sessions, especially in environments (such as HTTP/HTTPS, SQL, SAP GUI, and OF) where transaction names are auto-discovered. Keep this number at a reasonable level.
The NAM Server has a built-in limit of 10,000 operations and servers; it will not store more unless explicitly reconfigured. Reconfiguration should happen only after careful checking that there is still enough RAM during normal operations and that the system performs well. Note that the NAM Server can refuse increasing the limit if there are not enough free resources. The first thing to check after a NAM Server reaches the limit is to determine whether this is because of servers or operations.
- If it is due to the number of servers, most likely you are monitoring Internet hosts or other less important traffic, and this should be filtered out or aggregated. Configure the NAM Probe to use user-defined software services only, provided that this is consistent with your monitoring requirements.
- If it is due to the number of operations, most likely the operation population is too dynamic (for example, the URLs contain dynamic parts that change for every single load). Filter out with regex definitions. Also, the NAM Probe may have been configured to auto-discover a large number of URLs. This could have been done either by URL auto-discovery thresholds, by manual mapping of too many dynamic parameter variations, or by enabling the monitoring all URLs option in Monitoring Configuration. In any of these cases, revise the configuration against requirements to limit the number of operations, which is less than a few thousand in typical cases. You can also reduce the number of operations by using appropriate NAM Probe configuration settings and by using appropriate NAM Server limits and aging.
Consider reverting any increases in the length of the reporting interval.
If the length of the reporting interval was decreased, you might need to revise this decision and revert the change. The reporting interval is set by default to 5 minutes, which is a good compromise between the need for data granularity, the amount of data kept in the database, and the requirement for near real-time experience. All capacity estimates are based on this default setting.
Decreasing the reporting interval results in more records being written to the database, while retaining complexity (regarding the number of sessions) at the same level. In other words, the shorter the reporting interval, the more stress is put on the size of the database. The size of the database has a direct impact on system performance, especially for database-based tasks such as DMI reports. Decreasing the reporting interval also provides for more raw data from which to calculate values such as trends or averages. Therefore, if you decrease the length of the reporting interval, it will have a negative effect on the length of time taken by overnight tasks. Keep monitoring the effects of decreasing the monitoring interval on overall system performance and on disk space.
In general, keep the reporting interval at the default value unless there are good reasons to change it. Do not decrease the reporting interval if the database is already large or system performance is not very good.
Consider using an external SQL server, provided that it offers faster disk I/O speed.
For more information, see SQL Server offload - advantages and disadvantages.
Consider expanding the system to split the load.
Ultimately, increasing the number of machines in the system will solve sizing problems. However, this should be treated as the last option, only in scenarios where revising requirements and session count is impossible or insufficient. Depending on the specific architecture, you can split the monitored infrastructure between different NAM systems. However, if report server database capacity is the issue, using a server farm may be the best solution.
Applies to DC RUM 2017
If you are using DC RUM 2017, see also ADS database sizing and performance.