One format, four jobs — how Apache Arrow made my simulation queryable in SQL

A game engine usually keeps four copies of the same world. The live memory the simulation ticks on. A file to save it to. A pipeline to hand it to something else. And a debug view to stare at when a run goes wrong. Four representations of the exact same state — four bodies of code, each drifting a little out of sync with the others.

I deleted three of them. There is one layout now — a standard columnar format the rest of the data world already speaks — and it does all four jobs. This is the story of why that mattered far more than the code it saved.

The private-format habit

Most engines invent their own way to store a world. It is the natural move: you know your data, you pack it tight, you move on. But a private format is a wall. Nothing else can read it. Getting data *out* means writing an exporter; getting data *in* means writing an importer; looking at it means building a viewer. Each of those is a small bespoke project — and each is one more place the four copies of the world quietly disagree.

So I made a different call: do not invent a format. Adopt the one the entire data ecosystem already standardized on — the columnar layout that dataframes, analytics engines, and file formats across the industry all speak — and lay the simulation's live memory out that way from the first byte. The standard I picked is Apache Arrow, and the choice was less about Arrow specifically than about refusing to keep a dialect of my own.

That inverts the usual relationship. The format is no longer something I convert *to* when I save — it is the shape the world already has while it is running. A snapshot stops being a conversion and becomes a copy. Most engines treat the gap between *live state* and *exportable data* as a fact of life. I treated it as a bug and closed it.

old:  live memory ──serialize──▶ save ──convert──▶ export ──write an adapter──▶ another tool
new:  live memory ═══════════════════ the same bytes ═══════════════════ file · stream · query

One layout, four jobs

Once the running world and the standard format are the same bytes, the things that used to be four separate subsystems collapse into four *readings* of one buffer. Nothing is copied to move between them. They are the same memory, looked at with four different intentions.

Job one and two: it runs the game, and it is the save file

The memory the simulation writes every tick is already in the interchange layout, so *running the game* and *holding a savable dataset* are the same act. When I want the world on disk, I write those bytes straight out as Arrow IPC — the standard's own on-the-wire form — or as Parquet when I want it compressed and columnar for later. There is no serialize step, because there is nothing to serialize into. It was already the format.

The format the CPU wanted anyway

Adopting someone else's format usually costs you something at runtime. You pack your data the way the standard likes, not the way your code likes, and you pay the mismatch every frame. Here it went the other way. The columnar layout — every field stored as its own long, contiguous run, instead of each entity as a little bundle of mixed fields — is not just convenient for saving and querying. It is exactly the shape the CPU's vector unit wants to chew on.

A modern processor can advance several bodies in a single instruction, but only when one field's data sits back-to-back in memory, so a whole lane of it loads at once. Columnar hands that over for free: every x-position is one array, every velocity another. My per-tick pass streams straight down those columns, four lanes at a time, doing the whole simulation step while the data is hot in the cache. The standard even aligns each column to a 64-byte boundary and pads its tail — which happens to be a superset of what the vector loads want, so the loads come out aligned with nothing to fix up, and the loop can run off the end into the padding instead of dropping to a slow one-at-a-time remainder.

row layout   [x y z vx vy vz] [x y z vx vy vz] [x y z vx vy vz] ...
             one entity at a time — the vector unit can't fill a lane

columnar     x  x  x  x  x  x  x  x ...     ◀ all x, contiguous
             y  y  y  y  y  y  y  y ...
             vx vx vx vx vx vx vx vx ...    ◀ load 4 (or 8) at once, one instruction

So the layout I chose for interop and for SQL is the same layout that makes the simulation fast. The thing that lets a warehouse read my world is the thing that lets a million bodies move inside a frame budget. Usually you pick one — a format that travels, or a format that runs. Columnar is both, and I never had to choose.

Job three: other systems can read it — and I can read theirs

This is the job that matters if you are not making games. Because the world speaks a standard, a third party's data pipeline can read a snapshot with tools it already owns — and, going the other way, I can pull *their* data into the engine the same way. A fleet's telemetry, a warehouse export, a stream of positions from somewhere else: if it can produce the standard format, and most modern data infrastructure can, often for free, it flows straight in. No adapter written per source. The engine stopped being an island.

Job four: the debugger is a SQL prompt

Here is the one I did not see coming. I am simulating a million moving bodies, and a handful of them are drifting somewhere they should not be. The old way to find them is a debugger, or a custom inspector — a panel someone builds that shows one entity at a time. But the whole world is already a standard dataset. So I do not inspect it. I *query* it.

I snapshot the live world to a file and point a standard analytics engine straight at it — DuckDB, from the command line — and ask, in plain SQL, the question I actually have:

# snapshot the whole live world to one standard file
export ──▶ world.arrow

# then query it like any dataset — DuckDB, plain SQL
duckdb world.arrow "SELECT parent_id, count(*)
                    FROM bodies
                    WHERE status = 'adrift'
                    GROUP BY parent_id
                    ORDER BY 2 DESC"

That is SQL, running over a snapshot of a live simulation, with a tool I did not write and a query I typed in ten seconds. Grouping, filtering, aggregation, joins — the whole vocabulary of data analysis, aimed at a running game world. No inspector to build. No export format to design. The world was already a table; I just had to ask it something.

The reason this lands harder than *nice, we saved some code* is who is holding the keyboard. When an AI agent is the thing building and running the system, the most useful representation of its state is not a human-facing panel — it is the format the agent's own tools already speak. I do not need a rendered view to understand a million bodies. I need to run a query and read the answer. Hand the agent a standard dataset and it can interrogate the world with the entire data-tooling ecosystem, out of the box, without anyone building it a bespoke window first.

A debugger built for a human is a window you look through. A debugger built for an agent is a question it can ask. Same world — but one of those scales to a million rows, and the other asks you to scroll.

One format, and the walls came down

I set out to delete some redundant code — three copies of a world I was tired of keeping in sync. What I got was larger than tidiness. The simulation can save itself, hand its state to anything, ingest almost anything, and answer questions in SQL, all because it stopped speaking a private language. The line between *runtime state* and *a dataset you can query* turned out to be a habit, not a law. When the thing running your system is an AI, that is the first line worth erasing.

Read next

The four jobs all sit on one thing — the live memory the simulation shares with the screen. How that memory works is its own story:

• A million entities, zero copies — the shared-memory architecture underneath this format — three threads on one block of memory, the simulation writing while the renderer reads, no byte copied between them. /devlog/a-million-entities-zero-copies