Extend the platform, empower your team.
Distributed general-purpose cluster-computing framework for programming entire clusters.
Apache Spark monitoring in Dynatrace provides insight into the resource usage, job status, and performance of Spark Standalone clusters.
With Spark monitoring enabled globally, Dynatrace automatically collects Spark metrics whenever a new host running Spark is detected in your environment.
The cluster charts section provides all the information you need regarding jobs, stages, messages, workers, and message processing. When jobs fail, the cause is typically a lack of cores or RAM. Note that for both cores and RAM, the maximum value is not your system’s maximum, it’s the maximum value as defined by your Spark configuration. Using the workers chart, you can immediately see when one of your nodes goes down.
Monitoring is available for the three main Spark components:
Apache Spark metrics are presented alongside other infrastructure measurements, enabling in-depth cluster performance analysis of both current and historical data.
--class de.codecentric.SparkPi \
--master spark://192.168.33.100:7077 \
--conf spark.eventLog.enabled=true \