Apache Spark
- Product information
- Release notes
Overview
This extension collects JMX metrics to provide insights into resource usage, job and application status, and performance of your spark components.
Apache Spark metrics are presented alongside other infrastructure measurements, enabling in-depth cluster performance analysis of both current and historical data.
Use cases
The extension enables insights into the overall health of Spark component instances
Get started
Dynatrace
Activate this extension in your Dynatrace environment from the in-product Hub and simply select which OneAgents to enable this on.
Spark
You must configure the required component metrics to be reported to the JMX sink.
- The metrics system is configured via a configuration file that Spark expects to be present at
$SPARK_HOME/conf/metrics.properties
For example, you can enable metric collection to the JmxSink for the master, worker, driver and executor components with a command such as:
cat << EOF > $SPARK_HOME/conf/metrics.properties
*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
EOF
Refer to the Spark documentation for more details
Details
Details: This extension can query and collect almost all component instance and namespace metrics as defined in Spark Metric Providers
- Driver
- Executor
- applicationMaster (YARN)
- mesos_cluster
- master
- ApplicationSource (Spark standalone)
- worker
- shuffleservice
Compatibility information
- Dynatrace OneAgent 1.275+
- Linux or Windows OS
- Spark version 3.x
- Spark JMX metrics are enabled
Extension content
Content typeNumber of items included
screen injections
1
generic relationship
1
metric metadata
80
dashboards
1
generic type
1
screen entities lists
1
Feature sets
Below is a complete list of the feature sets provided in this version. To ensure a good fit for your needs, individual feature sets can be activated and deactivated by your administrator during configuration.
Feature setsNumber of metrics included
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| streaming.inputRate-total | spark.streaming.inputRate-total | - | Count |
| streaming.latency | spark.streaming.latency | - | MilliSecond |
| streaming.processingRate-total | spark.streaming.processingRate-total | - | Count |
| streaming.states-rowsTotal | spark.streaming.states-rowsTotal | - | Count |
| streaming.states-usedBytes | spark.streaming.states-usedBytes | - | Byte |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| shuffleService.numActiveConnections.count | spark.shuffleService.numActiveConnections.count | - | Count |
| shuffleService.numRegisteredConnections.count | spark.shuffleService.numRegisteredConnections.count | - | Count |
| shuffleService.numCaughtExceptions.count | spark.shuffleService.numCaughtExceptions.count | - | Count |
| shuffleService.registeredExecutorsSize | spark.shuffleService.registeredExecutorsSize | - | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| appStatus.stages.failedStages.count | spark.appStatus.stages.failedStages.count | - | Count |
| appStatus.stages.skippedStages.count | spark.appStatus.stages.skippedStages.count | - | Count |
| appStatus.stages.completedStages.count | spark.appStatus.stages.completedStages.count | - | Count |
| appStatus.tasks.excludedExecutors.count | spark.appStatus.tasks.excludedExecutors.count | - | Count |
| appStatus.tasks.completedTasks.count | spark.appStatus.tasks.completedTasks.count | - | Count |
| appStatus.tasks.failedTasks.count | spark.appStatus.tasks.failedTasks.count | - | Count |
| appStatus.tasks.killedTasks.count | spark.appStatus.tasks.killedTasks.count | - | Count |
| appStatus.tasks.skippedTasks.count | spark.appStatus.tasks.skippedTasks.count | - | Count |
| spark.appStatus.tasks.unexcludedExecutors.count | spark.appStatus.tasks.unexcludedExecutors.count | - | Count |
| appStatus.jobs.succeededJobs.count | spark.appStatus.jobs.succeededJobs.count | - | Count |
| appStatus.jobs.failedJobs.count | spark.appStatus.jobs.failedJobs.count | - | Count |
| appStatus.jobs.jobDuration | spark.appStatus.jobs.jobDuration | - | MilliSecond |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| HiveExternalCatalog.fileCacheHits.count | spark.HiveExternalCatalog.fileCacheHits.count | - | Count |
| HiveExternalCatalog.filesDiscovered.count | spark.HiveExternalCatalog.filesDiscovered.count | - | Count |
| HiveExternalCatalog.hiveClientCalls.count | spark.HiveExternalCatalog.hiveClientCalls.count | - | Count |
| HiveExternalCatalog.parallelListingJobCount.count | spark.HiveExternalCatalog.parallelListingJobCount.count | - | Count |
| HiveExternalCatalog.partitionsFetched.count | spark.HiveExternalCatalog.partitionsFetched.count | - | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| ApplicationSource.status | spark.ApplicationSource.status | - | Count |
| ApplicationSource.runtime_ms | spark.ApplicationSource.runtime_ms | - | Count |
| ApplicationSource.cores | spark.ApplicationSource.cores | - | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| BlockManager.disk.diskSpaceUsed_MB | spark.BlockManager.disk.diskSpaceUsed_MB | - | MegaByte |
| BlockManager.memory.maxMem_MB | spark.BlockManager.memory.maxMem_MB | - | MegaByte |
| BlockManager.memory.maxOffHeapMem_MB | spark.BlockManager.memory.maxOffHeapMem_MB | - | MegaByte |
| BlockManager.memory.maxOnHeapMem_MB | spark.BlockManager.memory.maxOnHeapMem_MB | - | MegaByte |
| BlockManager.memory.memUsed_MB | spark.BlockManager.memory.memUsed_MB | - | MegaByte |
| BlockManager.memory.offHeapMemUsed_MB | spark.BlockManager.memory.offHeapMemUsed_MB | - | MegaByte |
| BlockManager.memory.onHeapMemUsed_MB | spark.BlockManager.memory.onHeapMemUsed_MB | - | MegaByte |
| BlockManager.memory.remainingMem_MB | spark.BlockManager.memory.remainingMem_MB | - | MegaByte |
| BlockManager.memory.remainingOffHeapMem_MB | spark.BlockManager.memory.remainingOffHeapMem_MB | - | MegaByte |
| BlockManager.memory.remainingOnHeapMem_MB | spark.BlockManager.memory.remainingOnHeapMem_MB | - | MegaByte |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| LiveListenerBus.numEventsPosted.count | spark.LiveListenerBus.numEventsPosted.count | - | Count |
| LiveListenerBus.queue.appStatus.numDroppedEvents.count | spark.LiveListenerBus.queue.appStatus.numDroppedEvents.count | - | Count |
| LiveListenerBus.queue.appStatus.size | spark.LiveListenerBus.queue.appStatus.size | - | Count |
| LiveListenerBus.queue.eventLog.numDroppedEvents.count | spark.LiveListenerBus.queue.eventLog.numDroppedEvents.count | - | Count |
| LiveListenerBus.queue.eventLog.size | spark.LiveListenerBus.queue.eventLog.size | - | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| DAGScheduler.job.activeJobs | spark.DAGScheduler.job.activeJobs | - | Count |
| DAGScheduler.job.allJobs | spark.DAGScheduler.job.allJobs | - | Count |
| DAGScheduler.messageProcessingTime.count | spark.DAGScheduler.messageProcessingTime.count | - | Count |
| DAGScheduler.messageProcessingTime.oneminuterate | spark.DAGScheduler.messageProcessingTime.oneminuterate | - | PerMinute |
| DAGScheduler.messageProcessingTime.mean | spark.DAGScheduler.messageProcessingTime.mean | - | MilliSecond |
| DAGScheduler.stage.failedStages | spark.DAGScheduler.stage.failedStages | - | Count |
| spark.DAGScheduler.stage.runningStages | spark.DAGScheduler.stage.runningStages | - | Count |
| DAGScheduler.stage.waitingStages | spark.DAGScheduler.stage.waitingStages | - | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| master.workers | spark.master.workers | - | Count |
| master.aliveWorkers | spark.master.aliveWorkers | - | Count |
| master.apps | spark.master.apps | - | Count |
| master.waitingApps | spark.master.waitingApps | - | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| worker.executors | spark.worker.executors | - | Count |
| spark.worker.coresUsed | spark.worker.coresUsed | - | Count |
| spark.worker.memUsed_MB | spark.worker.memUsed_MB | - | MegaByte |
| spark.worker.coresFree | spark.worker.coresFree | - | Count |
| spark.worker.memFree_MB | spark.worker.memFree_MB | - | MegaByte |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| mesos_cluster.waitingDrivers | spark.mesos_cluster.waitingDrivers | - | Count |
| mesos_cluster.launchedDrivers | spark.mesos_cluster.launchedDrivers | - | Count |
| mesos_cluster.retryDrivers | spark.mesos_cluster.retryDrivers | - | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| executor.bytesRead.count | spark.executor.bytesRead.count | - | Byte |
| executor.bytesWritten.count | spark.executor.bytesWritten.count | - | Byte |
| executor.cpuTime.count | spark.executor.cpuTime.count | - | Count |
| executor.filesystem.file.largeRead_ops | spark.executor.filesystem.file.largeRead_ops | - | Count |
| spark.executor.filesystem.file.read_bytes | spark.executor.filesystem.file.read_bytes | - | Byte |
| spark.executor.filesystem.file.read_ops | spark.executor.filesystem.file.read_ops | - | Count |
| spark.executor.filesystem.file.write_bytes | spark.executor.filesystem.file.write_bytes | - | Byte |
| executor.filesystem.file.write_ops | spark.executor.filesystem.file.write_ops | - | Count |
| executor.recordsRead.count | spark.executor.recordsRead.count | - | Count |
| executor.recordsWritten.count | spark.executor.recordsWritten.count | - | Count |
| executor.succeededTasks.count | spark.executor.succeededTasks.count | - | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| applicationMaster.numContainersPendingAllocate | spark.applicationMaster.numContainersPendingAllocate | - | Count |
| spark.applicationMaster.numExecutorsFailed | spark.applicationMaster.numExecutorsFailed | - | Count |
| applicationMaster.numExecutorsRunning | spark.applicationMaster.numExecutorsRunning | - | Count |
| applicationMaster.numLocalityAwareTasks | spark.applicationMaster.numLocalityAwareTasks | - | Count |
| applicationMaster.numReleasedContainers | spark.applicationMaster.numReleasedContainers | - | Count |
Full version history
To have more information on how to install the downloaded package, please follow the instructions on this page.
ReleaseDate
Full version history
- Add support for MBeans which have a UUID prefix
Full version history
- JMX metrics collection for Spark component metric providers
- Metadata - metrics, topology
