Syslog is a standard protocol for message logging and system logs management.
Devices like routers, firewalls, hosts, switches, and many other devices across many platforms use the Syslog standard to log users' activity, system/software life-cycle events, status, or diagnostics.
In network monitoring, the Syslog protocol is very useful because of its client-server architecture, where the remote Syslog server listens to the client's log messages.
Such consolidation of logging data in a central repository allows taking advantage of unified Dynatrace Log Management and Analytics Powered by Grail for actionable insights and automation.
OpenTelemetry collector is an open source vendor-agnostic way to receive, process and export telemetry data.
Log processor and forwarding capabilities allow you to collect data/logs from different sources, unify and send them to multiple destinations
OpenTelemetry collector and its components are supported by the community. Dynatrace supports OTLP logs ingest API. Dynatrace-supported OpenTelemetry collector for syslog use cases is under consideration.
This is intended for users who want to:
Complement observability data in Dynatrace, extend the Davis AI engine and enable App Owners, DevOps, and SRE teams to analyze all syslog data centrally.
This enables you to:
Leverage the OpenTelemetry standards to get syslogs from your devices into Dynatrace where you can immediately benefit from Dynatrace AI-driven root cause analysis.
Send syslogs to remote endpoint
Example 1: Configure Rsyslog on Linux Ubuntu to forward syslogs to a remote server.
Add the following line to the syslog daemon configuration file /etc/rsyslog.conf (UDP protocol):
*.* @<OpenTelemetry Collector host IP>:54527
*.* - instructs the daemon to forward all messages to the specified OpenTelemetry Collector instance listening on port 54527 and <OpenTelemetry Collector host IP> needs to point to the IP address of OpenTelemetry Collector.
@@ - if you are using TCP, type two @ symbols
Example 2: Configuring the F5 BIG-IP system to log to a remote syslog server (11.x - 17.x)
Refer to F5 BIG-IP documentation for instructions regarding remote Syslog configuration.
Decorate syslogs with meaningful attributes
The Dynatrace software intelligence platform and its Davis AI engine depend on context-rich, high-quality data. You can provide the context for your data ingested via
Generic log ingest API that supports a set of keys and semantic attributes. You can also provide custom attributes that don't require indexing in Dynatrace Grail data lakehouse.
Apply syslog template to your f5 BIG_IP to always include host and IP
This option can be used as an alternative to OpenTelemetry collector receiver configurations described in Example 4.
modify syslog {
include "
options {
proto-template(t_isostamp);
long_hostnames(yes); use_dns(no); use_fqdn(no);
};
template t_isostamp {
template(\"$ISODATE $HOST 1xx.xx.xx.xx9 $MSGHDR$MSG\\n\");
};
destination remote_server {
tcp(\"1xx.xx.xx.xx1\"
port (54526)
template(t_isostamp)
persist-name(r1)
);
};
filter f_alllogs {
level (debug...emerg);
};
log {
source(s_syslog_pipe);
filter(f_alllogs);
destination(remote_server);
};
"
}
Example 3: OpenTelemetry Collector installation on Linux AMD:
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.87.0/otelcol-contrib_0.87.0_linux_amd64.deb
sudo dpkg -i otelcol-contrib_0.87.0_linux_amd64.deb
sudo systemctl enable otelcol-contrib
sudo systemctl start otelcol-contrib
Example 4: OpenTelemetry Collector configuration:
- Modify OpenTelemetry collector configuration file
/etc/otelcol-contrib/config.yaml
Receiver configuration
receivers:
syslog/f5:
tcp:
listen_address: "0.0.0.0:54526"
protocol: rfc5424
operators:
- type: add
field: attributes.log.source
value: syslog
- type: add
field: attributes.dt.ip_addresses
value: "1xx.xx.xx.xx1"
- type: add
field: attributes.instance.name
value: "ip-1xx-xx-x-xx9.ec2.internal"
- type: add
field: device.type
value: "f5bigip"
syslog/host:
tcp:
listen_address: "0.0.0.0:54527"
protocol: rfc5424
operators:
- type: add
field: attributes.log.source
value: syslog
- type: add
field: device.type
value: "ubuntu-syslog"
Exporter configuration:
exporters:
logging:
verbosity: detailed
otlphttp/tenant_1:
endpoint: "https://{your-tenant}.live.dynatrace.com/api/v2/otlp"
headers:
Authorization: "Api-Token {your-api-token}"
Service configuration
service:
pipelines:
logs:
receivers: [syslog/f5, syslog/host]
processors: [batch]
exporters: [logging, otlphttp/tenant_1]
- Dynatrace log processing - parse syslogs ingested to Dynatrace to improve readability
Example 5: Dynatrace processing: Syslog attributes extraction:
In the above configuration yaml file /etc/otelcol-contrib/config.yamlwe explicitly added `attributes.instance.name' attribute that can add additional configuration effort in case of change.
If syslog message contains instance name it can be extracted automatically.
In this example we assume hostname is always available and added after timestamp. Refer to F5 BIG-IP documentation for template instructions on how to include FQDN of the system by default to all syslogs.
Log sample
{
"content":"2023-10-18T06:55:01-07:00-07:00 ip-1xx-xx-x-xx9.ec2.internal 1xx.xx.x.xx9 info systemd[1]: Removed slice user-0.slice.",
"device.type": "f5bigip"
}
Add log processing rule
- In Dynatrace UI go to Settings>Log Monitoring>Processing and select Add rule
- Name a new rule
- Add matcher: device.type == "f5bigip"
- Add processor definition: PARSE(content,"NSPACE SPACE NSPACE:instance.name")
Test result:
{
"content": "2023-10-18T06:55:01-07:00-07:00 ip-1xx-xx-x-xx9.ec2.internal 1xx.xx.x.xx9 info systemd[1]: Removed slice user-0.slice.",
"device.type": "f5bigip",
"instance.name": "ip-1xx-xx-x-xx9.ec2.internal"
}
