Skip to technology filters Skip to main content
Dynatrace Hub

Extend the platform,
empower your team.

Popular searches:
Home hero bg
Apache SparkApache Spark
Apache Spark

Apache Spark

Enhanced insights for Spark Components

Extension
Free trialDocumentation
Overview dashboardConfiguration feature setsMetrics Explorer - Spark tagProcess group Unified Analysis Screen updatedSpark Process Unified Analysis Screen
  • Product information
  • Release notes

Overview

This extension collects JMX metrics to provide insights into resource usage, job and application status, and performance of your spark components.

Apache Spark metrics are presented alongside other infrastructure measurements, enabling in-depth cluster performance analysis of both current and historical data.

Use cases

The extension enables insights into the overall health of Spark component instances

Compatibility information

  • Dynatrace OneAgent 1.275+
  • Linux or Windows OS
  • Spark version 3.x
  • Spark JMX metrics are enabled
Dynatrace
Documentation
By Dynatrace
Dynatrace support center
Subscribe to new releases
Copy to clipboard

Extension content

Content typeNumber of items included
screen injections
2
generic relationship
1
document dashboard
1
metric metadata
80
screen dql table
1
dashboards
1
screen entities lists
1
generic type
1

Feature sets

Below is a complete list of the feature sets provided in this version. To ensure a good fit for your needs, individual feature sets can be activated and deactivated by your administrator during configuration.

Feature setsNumber of metrics included
Metric nameMetric keyDescriptionUnit
executor.bytesRead.countspark.executor.bytesRead.count-Byte
executor.bytesWritten.countspark.executor.bytesWritten.count-Byte
executor.cpuTime.countspark.executor.cpuTime.count-Count
executor.filesystem.file.largeRead_opsspark.executor.filesystem.file.largeRead_ops-Count
spark.executor.filesystem.file.read_bytesspark.executor.filesystem.file.read_bytes-Byte
spark.executor.filesystem.file.read_opsspark.executor.filesystem.file.read_ops-Count
spark.executor.filesystem.file.write_bytesspark.executor.filesystem.file.write_bytes-Byte
executor.filesystem.file.write_opsspark.executor.filesystem.file.write_ops-Count
executor.recordsRead.countspark.executor.recordsRead.count-Count
executor.recordsWritten.countspark.executor.recordsWritten.count-Count
executor.succeededTasks.countspark.executor.succeededTasks.count-Count
Metric nameMetric keyDescriptionUnit
applicationMaster.numContainersPendingAllocatespark.applicationMaster.numContainersPendingAllocate-Count
spark.applicationMaster.numExecutorsFailedspark.applicationMaster.numExecutorsFailed-Count
applicationMaster.numExecutorsRunningspark.applicationMaster.numExecutorsRunning-Count
applicationMaster.numLocalityAwareTasksspark.applicationMaster.numLocalityAwareTasks-Count
applicationMaster.numReleasedContainersspark.applicationMaster.numReleasedContainers-Count
Metric nameMetric keyDescriptionUnit
streaming.inputRate-totalspark.streaming.inputRate-total-Count
streaming.latencyspark.streaming.latency-MilliSecond
streaming.processingRate-totalspark.streaming.processingRate-total-Count
streaming.states-rowsTotalspark.streaming.states-rowsTotal-Count
streaming.states-usedBytesspark.streaming.states-usedBytes-Byte
Metric nameMetric keyDescriptionUnit
shuffleService.numActiveConnections.countspark.shuffleService.numActiveConnections.count-Count
shuffleService.numRegisteredConnections.countspark.shuffleService.numRegisteredConnections.count-Count
shuffleService.numCaughtExceptions.countspark.shuffleService.numCaughtExceptions.count-Count
shuffleService.registeredExecutorsSizespark.shuffleService.registeredExecutorsSize-Count
Metric nameMetric keyDescriptionUnit
appStatus.stages.failedStages.countspark.appStatus.stages.failedStages.count-Count
appStatus.stages.skippedStages.countspark.appStatus.stages.skippedStages.count-Count
appStatus.stages.completedStages.countspark.appStatus.stages.completedStages.count-Count
appStatus.tasks.excludedExecutors.countspark.appStatus.tasks.excludedExecutors.count-Count
appStatus.tasks.completedTasks.countspark.appStatus.tasks.completedTasks.count-Count
appStatus.tasks.failedTasks.countspark.appStatus.tasks.failedTasks.count-Count
appStatus.tasks.killedTasks.countspark.appStatus.tasks.killedTasks.count-Count
appStatus.tasks.skippedTasks.countspark.appStatus.tasks.skippedTasks.count-Count
spark.appStatus.tasks.unexcludedExecutors.countspark.appStatus.tasks.unexcludedExecutors.count-Count
appStatus.jobs.succeededJobs.countspark.appStatus.jobs.succeededJobs.count-Count
appStatus.jobs.failedJobs.countspark.appStatus.jobs.failedJobs.count-Count
appStatus.jobs.jobDurationspark.appStatus.jobs.jobDuration-MilliSecond
Metric nameMetric keyDescriptionUnit
HiveExternalCatalog.fileCacheHits.countspark.HiveExternalCatalog.fileCacheHits.count-Count
HiveExternalCatalog.filesDiscovered.countspark.HiveExternalCatalog.filesDiscovered.count-Count
HiveExternalCatalog.hiveClientCalls.countspark.HiveExternalCatalog.hiveClientCalls.count-Count
HiveExternalCatalog.parallelListingJobCount.countspark.HiveExternalCatalog.parallelListingJobCount.count-Count
HiveExternalCatalog.partitionsFetched.countspark.HiveExternalCatalog.partitionsFetched.count-Count
Metric nameMetric keyDescriptionUnit
ApplicationSource.statusspark.ApplicationSource.status-Count
ApplicationSource.runtime_msspark.ApplicationSource.runtime_ms-Count
ApplicationSource.coresspark.ApplicationSource.cores-Count
Metric nameMetric keyDescriptionUnit
BlockManager.disk.diskSpaceUsed_MBspark.BlockManager.disk.diskSpaceUsed_MB-MegaByte
BlockManager.memory.maxMem_MBspark.BlockManager.memory.maxMem_MB-MegaByte
BlockManager.memory.maxOffHeapMem_MBspark.BlockManager.memory.maxOffHeapMem_MB-MegaByte
BlockManager.memory.maxOnHeapMem_MBspark.BlockManager.memory.maxOnHeapMem_MB-MegaByte
BlockManager.memory.memUsed_MBspark.BlockManager.memory.memUsed_MB-MegaByte
BlockManager.memory.offHeapMemUsed_MBspark.BlockManager.memory.offHeapMemUsed_MB-MegaByte
BlockManager.memory.onHeapMemUsed_MBspark.BlockManager.memory.onHeapMemUsed_MB-MegaByte
BlockManager.memory.remainingMem_MBspark.BlockManager.memory.remainingMem_MB-MegaByte
BlockManager.memory.remainingOffHeapMem_MBspark.BlockManager.memory.remainingOffHeapMem_MB-MegaByte
BlockManager.memory.remainingOnHeapMem_MBspark.BlockManager.memory.remainingOnHeapMem_MB-MegaByte
Metric nameMetric keyDescriptionUnit
LiveListenerBus.numEventsPosted.countspark.LiveListenerBus.numEventsPosted.count-Count
LiveListenerBus.queue.appStatus.numDroppedEvents.countspark.LiveListenerBus.queue.appStatus.numDroppedEvents.count-Count
LiveListenerBus.queue.appStatus.sizespark.LiveListenerBus.queue.appStatus.size-Count
LiveListenerBus.queue.eventLog.numDroppedEvents.countspark.LiveListenerBus.queue.eventLog.numDroppedEvents.count-Count
LiveListenerBus.queue.eventLog.sizespark.LiveListenerBus.queue.eventLog.size-Count
Metric nameMetric keyDescriptionUnit
DAGScheduler.job.activeJobsspark.DAGScheduler.job.activeJobs-Count
DAGScheduler.job.allJobsspark.DAGScheduler.job.allJobs-Count
DAGScheduler.messageProcessingTime.countspark.DAGScheduler.messageProcessingTime.count-Count
DAGScheduler.messageProcessingTime.oneminuteratespark.DAGScheduler.messageProcessingTime.oneminuterate-PerMinute
DAGScheduler.messageProcessingTime.meanspark.DAGScheduler.messageProcessingTime.mean-MilliSecond
DAGScheduler.stage.failedStagesspark.DAGScheduler.stage.failedStages-Count
spark.DAGScheduler.stage.runningStagesspark.DAGScheduler.stage.runningStages-Count
DAGScheduler.stage.waitingStagesspark.DAGScheduler.stage.waitingStages-Count
Metric nameMetric keyDescriptionUnit
master.workersspark.master.workers-Count
master.aliveWorkersspark.master.aliveWorkers-Count
master.appsspark.master.apps-Count
master.waitingAppsspark.master.waitingApps-Count
Metric nameMetric keyDescriptionUnit
worker.executorsspark.worker.executors-Count
spark.worker.coresUsedspark.worker.coresUsed-Count
spark.worker.memUsed_MBspark.worker.memUsed_MB-MegaByte
spark.worker.coresFreespark.worker.coresFree-Count
spark.worker.memFree_MBspark.worker.memFree_MB-MegaByte
Metric nameMetric keyDescriptionUnit
mesos_cluster.waitingDriversspark.mesos_cluster.waitingDrivers-Count
mesos_cluster.launchedDriversspark.mesos_cluster.launchedDrivers-Count
mesos_cluster.retryDriversspark.mesos_cluster.retryDrivers-Count

Full version history

To have more information on how to install the downloaded package, please follow the instructions on this page.
ReleaseDate

Full version history

  • New Platform/Gen3 dashboard bundled with extension
  • Added DQL support for Platform/Gen3 screens
  • Enriched entities with dt.security_context attribute

Full version history

  • Add support for MBeans which have a UUID prefix

Full version history

  • JMX metrics collection for Spark component metric providers
  • Metadata - metrics, topology
Dynatrace Hub
Hub HomeGet data into DynatraceBuild your own app
Log Management and AnalyticsKubernetesAI and LLM ObservabilityInfrastructure ObservabilitySoftware DeliveryApplication ObservabilityApplication SecurityDigital ExperienceBusiness Observability
Filter
Type
Built and maintained by
Deployment model
SaaS
  • SaaS
  • Managed
Partner FinderBecome a partnerDynatrace Developer

All

2 Results filtered by:

Composer logo

Composer

Simplify your Dynatrace Configuration

Temenos Core Banking T24 Fastpack logo

Temenos Core Banking T24 Fastpack

T24 Fastpack gives Dynatrace full insight into Temenos T24 performance