Runtime Support Matrix
Summary
| Capability | EMR on EC2 | EMR Serverless | EMR on EKS |
|---|---|---|---|
| Runtime metadata | Yes | Yes | Yes |
| Failed job/step state | Yes | Yes | Yes |
| S3 archived logs | Yes | Yes | Yes |
| CloudWatch Logs | No | Yes | Yes |
| CloudWatch metrics | Yes | Yes | Partial |
| YARN application correlation | Yes | Not applicable | Not applicable |
| Kubernetes pod diagnostics | Not applicable | Not applicable | Optional |
| Diagnosis report | Yes | Yes | Yes |
| Runtime-specific recommendations | Yes | Yes | Yes |
Required Identifiers
| Runtime | Required |
|---|---|
| EMR on EC2 | account_id, region, cluster_id |
| EMR Serverless | account_id, region, target.serverless_application_id, target.job_run_id |
| EMR on EKS | account_id, region, target.virtual_cluster_id, target.job_run_id |
Useful Optional Identifiers
| Runtime | Optional |
|---|---|
| EMR on EC2 | target.step_id, target.yarn_application_id, deploy_mode, time_window |
| EMR Serverless | target.attempt, time_window |
| EMR on EKS | target.eks_cluster_name, target.namespace, time_window |
Report Areas
| Area | EC2 | Serverless | EKS |
|---|---|---|---|
| Infrastructure | IAM, S3, KMS, bootstrap, capacity | execution role, application, job run | execution role/IRSA, virtual cluster |
| Data | paths, records, SQL, output | paths, records, SQL, output | paths, records, SQL, output |
| Spark Runtime | driver, YARN, executors, shuffle, memory | driver worker, executor workers, shuffle, memory | driver, executors, shuffle, memory |
| Kubernetes | no | no | namespace, driver pod, executor pods, image pull, scheduling, evictions |
| Observability | S3 logs, metrics | S3 logs, CloudWatch logs, metrics | S3 logs, CloudWatch logs, Kubernetes access |
| Configuration | Spark config | worker sizing, monitoring | pod templates, Spark config |