cdelmonte.dev
Systems begin with meaning and end in mechanism. I write about the translations in between.
Explainers
One mechanism at a time, described so that other pieces can link to it instead of opening a parenthesis.
Why Sorted Parquet Files Do Not Guarantee an Ordered Scan in DataFusion
A purchases table, one Parquet file per day, every file sorted by customer: you would expect a GROUP BY customer to use that order, yet EXPLAIN often shows it ignored. This explainer follows the ordering from the file scan to the aggregate in Apache DataFusion, shows where it gets lost, how to read it in the plan, and what you can change.
From Dictionary Indices to Integers: Parquet Bit Unpacking Explained
Parquet rarely stores a column’s values directly. Dictionary encoding keeps the distinct values once and represents the column through small bit-packed integer indices. This explainer walks through dictionary encoding, how the indices are bit-packed, and the bit-unpacking step that reconstructs them into an int[] before every dictionary lookup.
Essays
A thesis about how data systems are built, argued from principle rather than from one system.
A Small Compiler for Explaining Delta Lake Pruning
How delta-explain turns one predicate and the Delta log into a trustworthy explanation of file pruning, through shared representation, safe rewrites, and conservative measurement.
Where Spark Changes Shape: UnsafeRow, the JVM, and the Relocation of Portability
Run df.explain on a Parquet scan and you find ColumnarToRow, an operator whose only job is to change the data’s shape. It is the seam where two architectural eras meet, and a record of how portability in analytical systems has relocated from the JVM runtime to the data format and the query plan.
The Disaggregation of the Lakehouse Stack
How Delta Kernel, Arrow, and pluggable execution are disaggregating the lakehouse stack. The lakehouse stack is not converging on a new dominant engine. It is converging on a layered architecture in which protocol, data representation, and query execution are increasingly isolated behind stable interfaces.
Deep Dives
A thesis checked against the mechanism: source, measurements, a reproducible lab.
Unpacking Parquet: Explicit SIMD, Scalar Baselines, and What HotSpot Makes of Them
On the JVM, optimizing a hot kernel is not only about writing faster code: it is also about understanding how much the result depends on the machine code HotSpot derives from the scalar loop. Using Parquet bit-unpacking as a concrete case, the piece shows that a SIMD speedup depends on which scalar baseline C2 is handed, when explicit vectorization is actually justified, and why a more specialized scalar routine is not necessarily faster.
Anti-patterns in Catalyst rules
Six concrete anti-patterns I encountered building a real Catalyst extension: from the wrong rule type for throws, to mutable state under AQE and Spark Connect, to JVM bootstrap traps in PySpark.
The Hidden DSL in Catalyst
How Spark’s internal rewriting framework, Catalyst, exposes an embedded DSL with a public extension surface, the same one Delta Lake and Iceberg use to plug into the optimizer pipeline.
Projects
Tools and code that came out of the writing.
delta-explain: Making Delta Lake Pruning Visible
Partition pruning and data skipping are invisible by default. delta-explain reads the Delta log directly and shows, step by step, how a WHERE predicate narrows down candidate files, with no engine required.