Are you sure you have correct set of memory parameters 😕

Readers ... what is on-heap memory and off-heap memory ?
Sir, on heap memory is controlled by JVM and off heap memory is controlled by OS.

Good !!! You are correct but that's not it, let's dive deep into the understanding and unleashing the true potential of heap memory.

On Heap Memory 🧠

The term "on-heap memory" refers to the memory that is allocated for Spark processes within the Java Virtual Machine (JVM) heap space.

or simply the executor memory that you set in spark submit command is called on heap memory.

This is the standard memory allocation area where Java objects are stored and managed by the garbage collector.

spark-submit --class com.example.MyApp 
             --master yarn 
             --executor-memory 4G  <-- "This is your on heap memory"
             --num-executors 10 
             /path/to/myapp.jar

Sections in On Heap Memory

On heap memory has 3 sections

1. Unified Memory: Used by both Spark’s execution (shuffle, join, sort, aggregation) and storage (caching) tasks. Spark dynamically balances this space depending on the workload.

2. User Memory: For your custom variables, data structures, and code logic. Basically, where your “creativity” resides 😄

3. Reserved Memory: A safety buffer Spark keeps aside to avoid complete OOM (Out Of Memory) disasters. Think of it as Spark’s version of an emergency chocolate stash 🍫

While on-heap memory is simple and tightly integrated with the JVM, it comes with one major drawback — Garbage Collection (GC).

The JVM GC occasionally pauses the application to clean up unused objects, and in massive Spark jobs, this can lead to performance hiccups.

That’s where our cool friend off-heap memory enters. 😎

Off Heap Memory

Off-heap memory lives outside the JVM heap and is managed directly by the OS (via native memory).
It’s like saying: “Hey JVM, step aside — I’ll handle this myself.”

Why Off-Heap Memory Exists

1. Reduced GC Overhead: Since objects stored off-heap aren’t tracked by the JVM GC, you get fewer GC pauses.

2. Faster Serialization/Deserialization: Data can be stored in a binary format, improving shuffle and caching efficiency.

3. Better Control: Spark can manage off-heap storage more directly, avoiding some JVM memory fragmentation issues.

To enable off-heap memory, you can configure Spark like this:

spark-submit \
  --class com.example.MyApp \
  --master yarn \
  --executor-memory 4G \
  --conf spark.memory.offHeap.enabled=true \
  --conf spark.memory.offHeap.size=2G \
  /path/to/myapp.jar

In this example:

• The on-heap memory is 4G (controlled by the JVM).

• The off-heap memory is 2G (controlled by the OS).

So your executor now has a total of 6G effective memory (4G on + 2G off).

When to Use Off-Heap Memory 🧩

You should consider enabling off-heap memory when:

• You’re caching large datasets.

• You’re seeing long GC pauses in the Spark UI.

• You want to leverage Tungsten’s memory-efficient binary format for performance gains.

However, be careful: ⚠️⚠️⚠️

If you go too wild with off-heap memory, you might bypass the JVM’s safety checks and end up with segmentation faults or mysterious “native memory exhausted” errors. 💀

Fun Easter Egg 🥚💻

Try this small test to see on-heap vs off-heap behavior yourself!

from pyspark.sql import SparkSession
import time

spark = (
    SparkSession.builder
    .appName("HeapVsOffHeap")
    .config("spark.memory.offHeap.enabled", "true")
    .config("spark.memory.offHeap.size", "512m")
    .getOrCreate()
)

data = spark.range(0, 10_000_000).toDF("id")

# Cache dataset to use memory
data.cache().count()

print("Sleeping for 30s... check 'top' or Spark UI for memory usage 👀")
time.sleep(30)

spark.stop()

👉 Run it twice:

1. Once with offHeap.enabled = false

2. Once with it true

You’ll see different memory usage patterns in your system monitor — one handled by the JVM heap, and the other directly by native memory.

Adios

“Do or do not. There is no try.”
— Yoda, Star Wars