Vibeleaderboard
Index / tool

Run Google Dataproc Locally with Docker

local.cloud
Visit local.cloud
Category
Developer Tools
Rank
No. 2875Tools index

Previous survey · No. 2654 ·

Type
TOOL
Use case
Deployment & Operations
Date

About

The author provides Docker images that package Dataproc component versions, including Hadoop and Spark, so developers and AI agents can run a Dataproc environment locally. Google does not publish an official image for this, so dev and CI testing otherwise requires starting a real cluster. A self-test command runs from the image, and images exist for several Dataproc versions.

At a glance

This guide shows how to run a Docker image with versions of Spark, Hadoop and Hive intended to match Google Dataproc. It is worth reading if you want a local smoke test for a Spark job before sending it to a cloud cluster.

Good fit

Data engineers who can reproduce a pipeline issue with a small dataset and a pinned runtime.

Consider something else when

Skip it if you need to verify Google Cloud provisioning, permissions or other managed control-plane behavior.

Start here

Compare the image version table with your target Dataproc release. Choose one small input with a known output and use it as a smoke test in an isolated environment.

Cost and requirements

The publisher documents Docker images and a multi-node option. Local runs need your own CPU, memory and storage. The multi-node example needs more memory than a single container.

Limits to know

This is a third-party runtime image. Matching component versions can catch local compatibility problems, while cloud integration and cluster behavior still need testing on Dataproc.

Sources checked 2026-11-05. Source-based assessment.

Tags

dataprocdockersparkhadoopgcptesting

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.