Working through CMU's 10-799 on diffusion models and flow matching. Currently implementing DDPM from scratch — U-Net, noise schedules, the whole pipeline. More solutions on the way.
Distilling a 108K-parameter multi-agent policy into 1.4K parameters
A practical study of soft-target policy distillation, recurrent sequence training, sampled evaluation, and the limits of what a compression experiment can establish
BlueDot TAIS puzzle #1
A BlueDot TAIS puzzle: finding the one feature a small network stores non-linearly, working out the exact shape it uses, and then training a harder version of the same trick
The time our executor started up and then did nothing
Debugging a pre-main startup hang: jemalloc's profiler init, libgcc's unwinder, and Antithesis's coverage library conspire into a self-deadlock
CMU 10-799: Diffusion & Flow Matching Implementing diffusion models and flow matching from scratch
Graphormer: Transformers for Graph-Structured Data Understanding how Graphormer adapts the transformer architecture for graph representation learning
Transformers have revolutionized NLP and computer vision, but applying them to graph-structured data presents unique challenges. Unlike sequences or grids, graphs have irregular topology without a natural ordering of nodes.
Graphormer addresses this by introducing novel structural encodings that capture graph properties within the transformer framework:
- Centrality Encoding: Captures node importance using degree information
- Spatial Encoding: Encodes pairwise distances between nodes in the attention mechanism
- Edge Encoding: Incorporates edge features along shortest paths between node pairs
These encodings allow the standard transformer architecture to reason about graph structure while leveraging the powerful attention mechanism for learning node representations.
Published as part of the GRAM workshop's blogpost track @ ICML.
Paper Presentations
Slides from paper reading sessions on databases and ML systems
German Strings for Faster Analytics How modern analytical databases optimize string storage with inline prefix techniques
String handling is a significant performance bottleneck in analytical databases. Traditional approaches store strings as pointers to heap-allocated memory, causing cache misses and memory indirection overhead during scans and comparisons.
In this article, I explore "German strings" - an optimization technique used by modern analytical databases like DuckDB. The key insight is storing a prefix of the string inline within the pointer structure itself, enabling:
- Fast prefix comparisons without dereferencing pointers
- Better cache utilization by keeping frequently-accessed data together
- Reduced memory indirection for short strings that fit entirely inline
The technique gets its name from the Umbra database system developed in Germany, which pioneered this approach.
Originally published on the e6data engineering blog.