The Hidden DSL in Catalyst
How Spark’s internal rewriting framework, Catalyst, exposes an embedded DSL with a public extension surface, the same one Delta Lake and Iceberg use to plug into the optimizer pipeline.
Spark
Anti-patterns in Catalyst rules
Six concrete anti-patterns I encountered building a real Catalyst extension: from the wrong rule type for throws, to mutable state under AQE and Spark Connect, to JVM bootstrap traps in PySpark.
Where Data System Abstractions Break: A Semiotic Reading
Many of the most surprising performance pathologies in modern data systems are semiotic failures — structural divergences between what an interface signifies and what the underlying system does.
Delta Lake MERGE Is Not a Simple Upsert. What Actually Happens at Scale.
At 10 TB, updating 200k rows can mean rewriting thousands of files. Here’s why, and what to do about it.
Spark Is Not Just Lazy. Spark Compiles Dataflow.
Why calling Spark ’lazy’ is technically reductive, and how thinking of it as a dataflow compiler changes the way you design pipelines.
Fixing Skewed Nested Joins in Spark with Asymmetric Salting
In large-scale Spark pipelines, skew can occur when a single key carries a disproportionately large nested payload. Asymmetric salting offers a targeted solution: explode, salt, join in parallel, and optionally re-aggregate.