Apache Spark

Apache Spark

Enhanced insights for Spark Components

Extension

Free trial Documentation

Product information
Release notes

Overview

This extension collects JMX metrics to provide insights into resource usage, job and application status, and performance of your spark components.

Apache Spark metrics are presented alongside other infrastructure measurements, enabling in-depth cluster performance analysis of both current and historical data.

Use cases

The extension enables insights into the overall health of Spark component instances

Compatibility information

Dynatrace OneAgent 1.275+
Linux or Windows OS
Spark version 3.x
Spark JMX metrics are enabled

Subscribe to new releases

Copy to clipboard

Extension content

Content typeNumber of items included

screen injections

2

generic relationship

1

document dashboard

1

metric metadata

80

screen dql table

1

dashboards

1

screen entities lists

1

generic type

1

Feature sets

Below is a complete list of the feature sets provided in this version. To ensure a good fit for your needs, individual feature sets can be activated and deactivated by your administrator during configuration.

Feature setsNumber of metrics included

Metric name	Metric key	Description	Unit
executor.bytesRead.count	spark.executor.bytesRead.count	-	Byte
executor.bytesWritten.count	spark.executor.bytesWritten.count	-	Byte
executor.cpuTime.count	spark.executor.cpuTime.count	-	Count
executor.filesystem.file.largeRead_ops	spark.executor.filesystem.file.largeRead_ops	-	Count
spark.executor.filesystem.file.read_bytes	spark.executor.filesystem.file.read_bytes	-	Byte
spark.executor.filesystem.file.read_ops	spark.executor.filesystem.file.read_ops	-	Count
spark.executor.filesystem.file.write_bytes	spark.executor.filesystem.file.write_bytes	-	Byte
executor.filesystem.file.write_ops	spark.executor.filesystem.file.write_ops	-	Count
executor.recordsRead.count	spark.executor.recordsRead.count	-	Count
executor.recordsWritten.count	spark.executor.recordsWritten.count	-	Count
executor.succeededTasks.count	spark.executor.succeededTasks.count	-	Count

Metric name	Metric key	Description	Unit
applicationMaster.numContainersPendingAllocate	spark.applicationMaster.numContainersPendingAllocate	-	Count
spark.applicationMaster.numExecutorsFailed	spark.applicationMaster.numExecutorsFailed	-	Count
applicationMaster.numExecutorsRunning	spark.applicationMaster.numExecutorsRunning	-	Count
applicationMaster.numLocalityAwareTasks	spark.applicationMaster.numLocalityAwareTasks	-	Count
applicationMaster.numReleasedContainers	spark.applicationMaster.numReleasedContainers	-	Count

Metric name	Metric key	Description	Unit
streaming.inputRate-total	spark.streaming.inputRate-total	-	Count
streaming.latency	spark.streaming.latency	-	MilliSecond
streaming.processingRate-total	spark.streaming.processingRate-total	-	Count
streaming.states-rowsTotal	spark.streaming.states-rowsTotal	-	Count
streaming.states-usedBytes	spark.streaming.states-usedBytes	-	Byte

Metric name	Metric key	Description	Unit
shuffleService.numActiveConnections.count	spark.shuffleService.numActiveConnections.count	-	Count
shuffleService.numRegisteredConnections.count	spark.shuffleService.numRegisteredConnections.count	-	Count
shuffleService.numCaughtExceptions.count	spark.shuffleService.numCaughtExceptions.count	-	Count
shuffleService.registeredExecutorsSize	spark.shuffleService.registeredExecutorsSize	-	Count

Metric name	Metric key	Description	Unit
appStatus.stages.failedStages.count	spark.appStatus.stages.failedStages.count	-	Count
appStatus.stages.skippedStages.count	spark.appStatus.stages.skippedStages.count	-	Count
appStatus.stages.completedStages.count	spark.appStatus.stages.completedStages.count	-	Count
appStatus.tasks.excludedExecutors.count	spark.appStatus.tasks.excludedExecutors.count	-	Count
appStatus.tasks.completedTasks.count	spark.appStatus.tasks.completedTasks.count	-	Count
appStatus.tasks.failedTasks.count	spark.appStatus.tasks.failedTasks.count	-	Count
appStatus.tasks.killedTasks.count	spark.appStatus.tasks.killedTasks.count	-	Count
appStatus.tasks.skippedTasks.count	spark.appStatus.tasks.skippedTasks.count	-	Count
spark.appStatus.tasks.unexcludedExecutors.count	spark.appStatus.tasks.unexcludedExecutors.count	-	Count
appStatus.jobs.succeededJobs.count	spark.appStatus.jobs.succeededJobs.count	-	Count
appStatus.jobs.failedJobs.count	spark.appStatus.jobs.failedJobs.count	-	Count
appStatus.jobs.jobDuration	spark.appStatus.jobs.jobDuration	-	MilliSecond

Metric name	Metric key	Description	Unit
HiveExternalCatalog.fileCacheHits.count	spark.HiveExternalCatalog.fileCacheHits.count	-	Count
HiveExternalCatalog.filesDiscovered.count	spark.HiveExternalCatalog.filesDiscovered.count	-	Count
HiveExternalCatalog.hiveClientCalls.count	spark.HiveExternalCatalog.hiveClientCalls.count	-	Count
HiveExternalCatalog.parallelListingJobCount.count	spark.HiveExternalCatalog.parallelListingJobCount.count	-	Count
HiveExternalCatalog.partitionsFetched.count	spark.HiveExternalCatalog.partitionsFetched.count	-	Count

Metric name	Metric key	Description	Unit
ApplicationSource.status	spark.ApplicationSource.status	-	Count
ApplicationSource.runtime_ms	spark.ApplicationSource.runtime_ms	-	Count
ApplicationSource.cores	spark.ApplicationSource.cores	-	Count

Metric name	Metric key	Description	Unit
BlockManager.disk.diskSpaceUsed_MB	spark.BlockManager.disk.diskSpaceUsed_MB	-	MegaByte
BlockManager.memory.maxMem_MB	spark.BlockManager.memory.maxMem_MB	-	MegaByte
BlockManager.memory.maxOffHeapMem_MB	spark.BlockManager.memory.maxOffHeapMem_MB	-	MegaByte
BlockManager.memory.maxOnHeapMem_MB	spark.BlockManager.memory.maxOnHeapMem_MB	-	MegaByte
BlockManager.memory.memUsed_MB	spark.BlockManager.memory.memUsed_MB	-	MegaByte
BlockManager.memory.offHeapMemUsed_MB	spark.BlockManager.memory.offHeapMemUsed_MB	-	MegaByte
BlockManager.memory.onHeapMemUsed_MB	spark.BlockManager.memory.onHeapMemUsed_MB	-	MegaByte
BlockManager.memory.remainingMem_MB	spark.BlockManager.memory.remainingMem_MB	-	MegaByte
BlockManager.memory.remainingOffHeapMem_MB	spark.BlockManager.memory.remainingOffHeapMem_MB	-	MegaByte
BlockManager.memory.remainingOnHeapMem_MB	spark.BlockManager.memory.remainingOnHeapMem_MB	-	MegaByte

Metric name	Metric key	Description	Unit
LiveListenerBus.numEventsPosted.count	spark.LiveListenerBus.numEventsPosted.count	-	Count
LiveListenerBus.queue.appStatus.numDroppedEvents.count	spark.LiveListenerBus.queue.appStatus.numDroppedEvents.count	-	Count
LiveListenerBus.queue.appStatus.size	spark.LiveListenerBus.queue.appStatus.size	-	Count
LiveListenerBus.queue.eventLog.numDroppedEvents.count	spark.LiveListenerBus.queue.eventLog.numDroppedEvents.count	-	Count
LiveListenerBus.queue.eventLog.size	spark.LiveListenerBus.queue.eventLog.size	-	Count

Metric name	Metric key	Description	Unit
DAGScheduler.job.activeJobs	spark.DAGScheduler.job.activeJobs	-	Count
DAGScheduler.job.allJobs	spark.DAGScheduler.job.allJobs	-	Count
DAGScheduler.messageProcessingTime.count	spark.DAGScheduler.messageProcessingTime.count	-	Count
DAGScheduler.messageProcessingTime.oneminuterate	spark.DAGScheduler.messageProcessingTime.oneminuterate	-	PerMinute
DAGScheduler.messageProcessingTime.mean	spark.DAGScheduler.messageProcessingTime.mean	-	MilliSecond
DAGScheduler.stage.failedStages	spark.DAGScheduler.stage.failedStages	-	Count
spark.DAGScheduler.stage.runningStages	spark.DAGScheduler.stage.runningStages	-	Count
DAGScheduler.stage.waitingStages	spark.DAGScheduler.stage.waitingStages	-	Count

Metric name	Metric key	Description	Unit
master.workers	spark.master.workers	-	Count
master.aliveWorkers	spark.master.aliveWorkers	-	Count
master.apps	spark.master.apps	-	Count
master.waitingApps	spark.master.waitingApps	-	Count

Metric name	Metric key	Description	Unit
worker.executors	spark.worker.executors	-	Count
spark.worker.coresUsed	spark.worker.coresUsed	-	Count
spark.worker.memUsed_MB	spark.worker.memUsed_MB	-	MegaByte
spark.worker.coresFree	spark.worker.coresFree	-	Count
spark.worker.memFree_MB	spark.worker.memFree_MB	-	MegaByte

Metric name	Metric key	Description	Unit
mesos_cluster.waitingDrivers	spark.mesos_cluster.waitingDrivers	-	Count
mesos_cluster.launchedDrivers	spark.mesos_cluster.launchedDrivers	-	Count
mesos_cluster.retryDrivers	spark.mesos_cluster.retryDrivers	-	Count

Full version history

To have more information on how to install the downloaded package, please follow the instructions on this page.

ReleaseDate

Full version history

New Platform/Gen3 dashboard bundled with extension
Added DQL support for Platform/Gen3 screens
Enriched entities with dt.security_context attribute

Full version history

Add support for MBeans which have a UUID prefix

Full version history

JMX metrics collection for Spark component metric providers
Metadata - metrics, topology

Discover recent additions to Dynatrace

Problems

Analyze abnormal system behavior and performance problems detected by Davis AI.

Logs

Explore all your logs without writing a single query.

Security Investigator

Fast and precise forensics for security and logs on Grail data with DQL queries.

Business Flow

Track, analyze, and optimize your critical business processes.

Cost & Carbon Optimization

Track, analyze, and optimize your IT carbon footprint and public cloud costs.

Davis Anomaly Detection

Detect anomalies in timeseries using the Davis AI

Analyze your data

Understand your data better with deep insights and clear visualizations.

Notebooks

Create powerful, data-driven documents for custom analytics and collaboration.

Dashboards

Transform complex data into clear visualizations with custom dashboards.

Automate your processes

Turn data and answers into actions, securely, and at scale.

Workflows

Automate tasks in your IT landscape, remediate problems, and visualize processes

Jira

Create, query, comment, transition, and resolve Jira tickets within workflows.

Slack

Automate Slack messaging for security incidents, attacks, remediation, and more.

Secure your cloud application

See vulnerabilities and attacks in your environment.

Security Overview

Get a comprehensive overview of the security of your applications.

Code-Level Vulnerabilities

Detect vulnerabilities in your code in real time.

Security Posture Management

Detect, prioritize, and remediate security and compliance findings with SPM.

Threats & Exploits

Understand, triage, and investigate detection findings and alerts.

Are you looking for something different?

We have hundreds of apps, extensions, and other technologies to customize your environment

Leverage our newest innovations of Dynatrace Saas

Kick-start your app creation

Kick-start your app creation

Whether you’re a beginner or a pro, Dynatrace Developer has the tools and support you need to create incredible apps with minimal effort.

Go to Dynatrace Developer

Upgrading from Dynatrace Managed to SaaS

Upgrading from Dynatrace Managed to SaaS

Drive innovation, speed, and agility in your organization by seamlessly and securely upgrading.

Log Management and Analytics

Log Management and Analytics

Innovate faster and more efficiently with unified log management and log analytics for actionable insights and automation.