Wrangling Big Data at EY by day, spilling code & laughs on #dataengineering at night. Let’s make data fun! 🚀💻
While exploring Generative AI, I wanted to build something practical rather than just experiment with prompts. I developed a GenAI-powered Database Query Assistant that converts natural-language quest
Every data engineer has been here. You tune your spark-submit. You set the memory, the cores, the executors. It looks right. You hit submit. Three hours later — ExecutorLostFailure. You open the Spark
I recently built an Apache Spark standalone cluster on a single Raspberry Pi 5 (8 GB RAM) using Docker.The cluster had: 1 Spark Master 4 Spark Workers: harvey, mike, donna, louis (named after Suits characters 😄) Strict memory limits per container...
Understanding executor-local shuffle ownership in Apache Spark
BigQuery decouples storage from compute using Google’s high-speed Jupiter network. It uses a multi-tenant architecture. Client Request: You submit a GoogleSQL query via the Console, API, or CLI. Query Parsing & Optimization: The query reaches the Q...
Why the "Leaf Node" is a masterpiece of distributed engineering
Are you sure you have correct set of memory parameters 😕 Readers ... what is on-heap memory and off-heap memory ?Sir, on heap memory is controlled by JVM and off heap memory is controlled by OS. Good !!! You are correct but that's not it, let's dive...
Data Engineeringing v2.0 loading
Speed Demon or Traffic Jam?
