Log data storage v1
Log Monitoring enables you to store all logs centrally within external storage. This makes log data available independent of log files themselves. This can be beneficial in the following situations:
- Short log retention periods
- Volatile log storage
- Legal requirements for keeping logs archived centrally for long time periods
In addition you can also:
- Analyze multiple logs simultaneously
- Parse log or JSON files
- Generate metrics from log content
Log storage requirements and costs
For Dynatrace SaaS customers, log files are stored in the same AWS availability zone where your Dynatrace environment resides. You don’t have to worry about storage performance, availability, or free space. Disk storage costs are included in your Log Monitoring subscription. Costs are based on the average size of your cloud-based log storage, including the amount of streamed log data and the defined retention period. For details, see DDUs for Log Monitoring and Data retention periods.
To store log files centrally on your Dynatrace Managed cluster, you must provide a common Network File System (NFS) mount point (path) that is identical throughout the cluster and available from all cluster nodes. With this approach, it's your responsibility to ensure appropriate levels of performance, availability, and free space on the mounted NFS volume. Costs are calculated based only on the amount of ingress log data (GB/day), not total storage size, so retention time doesn't influence storage costs. For deployments on AWS, we recommend that you use the Amazon Elastic File System for your log storage.
Before configuring the path for log storage, you can check your DDU consumption. In the Dynatrace menu, go to Settings > Accounting > Davis data units overview. The Davis data units (DDUs) model counts all incoming events from your log data. Each log event (log line, message, event, etc.) deducts 0.0005 DDU from your available quota. 1 GiB ingest is an equivalent of 1 million log events. See DDUs for Log Monitoring.
To set up your central location for log storage:
From the navigation menu, select Settings > Log Monitoring.
Set the Use network attached storage switch to the On position.
Click the edit button and type the mount point (for example,
/usr/local/path/to/storage) to the network resource.
Make sure that all Dynatrace cluster nodes have write access to the mount point you indicated.
Restart the Dynatrace cluster using the Restart button for each of the nodes in the cluster.
For each monitored environment, in the Set total environment quotas section, set the Log monitoring storage to a non-zero value (default 0 MB).
The log writing queue can take up to 1% of available memory on a cluster node. The CPU is typically not affected much. For metrics, a single node can process about two million log entries per second.
Because of the volume of logs, log files are compressed using Zstandard before they're sent for analysis. This task requires an average of 10% of compute power. Worst case, this task will require about 25% of compute power.
If you configured Log Monitoring to store all logs centrally, all log content needs to be read. If the log files are on NFS, each byte written to the log file also needs to be read. As a result, Log Monitoring reads from NFS at the same rate the log files are written. Depending on your network infrastructure, this may impact your network throughput, as a higher rate of log data generation will cause higher network utilization.
For the ActiveGate component, provide one physical core per 50 Mbps of traffic (compressed content).