Bug: use-after-free in query result string delivery corrupts large multi-column reads — regression introduced in 0.17.0 (0.16.1 clean, 0.17.0 guilty, identical instrumentation)

[BikerTech]

Ladybug version

0.18.1

What operating system are you using?

Windows 11

What happened?

Authorship disclosure: this report was written by Claude, Anthropic's AI, working as the developer on the project that found these behaviors. The test campaigns and evidence are the joint work of the AI and the project's owner, who reviewed this text and filed it. This is the third report from the same project, companion to #713 (checkpoint fold race) and #714 (WAL big-record replay).

Summary

On large multi-column string reads, the engine frees the memory holding a query result's string values while the result is still being delivered. Under valgrind, the delivery worker (lbug::common::Value::copyFromColLayout → ku_string_t / string_t::getAsString, running on a lbug::common::TaskScheduler::runWorkerThread) reads string buffers that lbug::storage::MemoryBuffer::~MemoryBuffer has already freed — 3,476 invalid-read reports in a single query execution on 0.18.1, and the freed blocks are exactly the string payloads themselves (13/14/32 bytes in our replica; 14 bytes is precisely the length of one of our stored string values).

Whether any given read appears to succeed is allocator luck. On Windows, freed memory is re-rented almost immediately, so the same read fails essentially 100% of the time in the field (random illegal byte at a random position, different every run). On Linux the freed contents often survive untouched, so the same run looks clean — but the instrument flags the trespass regardless. Worse: re-rented memory sometimes contains legal text, so the engine can deliver plausible wrong values with no error at all. This is a silent-corruption class defect, not just a crash class.

We bisected the fork's release ladder under identical instrumentation per version: the regression enters at 0.17.0. 0.15.3, 0.15.4, and 0.16.1 are completely clean (zero invalid reads); 0.17.0 and every release after it through 0.18.1 are guilty with the same stack signature. This hands you a two-release diff (0.16.1 → 0.17.0) containing the defect.

Impact observed in the field (Windows 11, ladybug 0.18.1)

• A five-column, 24,498-row sorted read fails essentially every run with a UTF-8 decode error at a random position. Small reads, aggregates, and single-column full scans on the same table are clean — the defect follows the execution plan, not the data.
• All four result-consumption paths are affected (tested directly against the same database): row iteration (rows_as_dict), Arrow export, dataframe export (get_as_df), and the engine's own COPY TO CSV file export — which wrote corrupt bytes into its own output file (1 clean run in 5; the intermittent clean run proves the stored data is intact and delivery is what loses it).
• Stored data is exonerated: every count and every small probe stayed mutually consistent throughout. The disease is delivery-side, not storage-side.
• Paging does not work around it. We certified and failed three designs: parameterized skip/limit (planner falls back to full-table materialization — instant failure at every page size), literal skip/limit (fails once skip+limit crosses the ~2,048-row internal batch boundary), and keyset-cursor paging (fails ~0.19 s into the walk — rapid small-query churn re-rents freed memory just as surely). The governing pattern in all our data: corruption risk follows engine-internal materialization and allocator churn, not delivered result size.

Are there known steps to reproduce?

Reproduction (fully synthetic — no real data involved)

Two scripts. The first builds a synthetic 24,500-row table shaped like our production workload (10 columns; a 128-element DOUBLE[] plus mixed strings, some NULL, some accented UTF-8). The second is the read that gets corrupted.

build_scale.py:

import ladybug as lb, os, shutil, random, uuid, datetime, sys
path = sys.argv[1]
rows = int(sys.argv[2]) if len(sys.argv) > 2 else 24500
for f in (path, path + ".wal"):
    if os.path.exists(f):
        (shutil.rmtree if os.path.isdir(f) else os.remove)(f)
db = lb.Database(path)
conn = lb.Connection(db)
conn.execute("""
    CREATE NODE TABLE IdentityCluster (
        cluster_id STRING, embedding_centroid DOUBLE[], cohesion_score DOUBLE,
        candidate_name STRING, haifa_confidence STRING, member_count INT64,
        centroid_shifted BOOL, merge_proposed_with STRING,
        created_at TIMESTAMP, updated_at TIMESTAMP, PRIMARY KEY (cluster_id))""")
names = ["René Tremblay", "Benoît Lefebvre", "Noël Gagné", "Aimée Côté",
         "Jérôme Bélanger", "Zoé Bergeron", "François Lévesque", "Élise Boucher"]
random.seed(42)
ts = datetime.datetime.now()
for i in range(rows):
    named = (i % 260 == 0)
    conn.execute("""
        CREATE (c:IdentityCluster { cluster_id: $cid, embedding_centroid: $cent,
            cohesion_score: $coh, candidate_name: $nm, haifa_confidence: $hc,
            member_count: $mc, centroid_shifted: false, merge_proposed_with: NULL,
            created_at: $ts, updated_at: $ts })""",
        {"cid": str(uuid.uuid4()), "cent": [random.random() for _ in range(128)],
         "coh": random.random(), "nm": random.choice(names) if named else None,
         "hc": "HAIFA_INFERRED" if named else None,
         "mc": random.randint(1, 170), "ts": ts})
conn.close(); db.close()
print("built", path, rows, "rows")

vg_read_any.py:

import ladybug as lb, sys
path = sys.argv[1]
db = lb.Database(path, read_only=True,
                 buffer_pool_size=512*1024*1024, max_db_size=1<<32)
conn = lb.Connection(db)
Q = ("MATCH (c:IdentityCluster) RETURN c.cluster_id AS cluster_id, "
     "c.candidate_name AS candidate_name, c.haifa_confidence AS haifa_confidence, "
     "c.member_count AS member_count, c.cohesion_score AS cohesion_score "
     "ORDER BY c.member_count DESC")
r = conn.execute(Q)
rows = r.rows_as_dict().get_all()
print("delivered rows:", len(rows))
print("first row:", rows[0])
r.close(); conn.close(); db.close()

Per version (Linux; PYTHONMALLOC=malloc so valgrind sees the allocations):

python3 -m venv vXXX && ./vXXX/bin/pip install -q ladybug==X.XX.X
./vXXX/bin/python build_scale.py db_vXXX.lbug
PYTHONMALLOC=malloc valgrind --tool=memcheck --error-limit=no --num-callers=8 \
    ./vXXX/bin/python vg_read_any.py db_vXXX.lbug 2> vg_vXXX.log
grep -c 'Invalid read' vg_vXXX.log

On Windows, no instrument is needed: the same read on the same table fails with a UTF-8 decode error (or, occasionally and more dangerously, silently delivers wrong string values).

The release-ladder verdict (identical table, identical read, identical instrument per version)

Release	Invalid reads	Verdict
0.15.3	0	CLEAN
0.15.4	0	CLEAN
0.16.1	0	CLEAN
0.17.0	7	GUILTY — regression enters here
0.17.1	7	GUILTY
0.18.0	10	GUILTY
0.18.1	10	GUILTY

Counts above are unique valgrind invalid-read report sites per run under --num-callers=8; a deeper run on 0.18.1 with fuller stacks recorded 3,476 individual invalid-read occurrences in one query execution. Every guilty version shows the same signature — the use is in result string delivery on a TaskScheduler worker (Value::copyFromColLayout → string_t::getAsString), and the freed-by stack is storage::MemoryBuffer::~MemoryBuffer. We have not yet re-run the instrument on 0.18.2/0.18.3; the field failures and the mechanism proof are from 0.18.1.

Given the boundary, the suspect set is whatever 0.17.0 changed in result materialization / intermediate-buffer lifetime — the string payloads live in a MemoryBuffer that is destroyed while a delivery worker still holds string_t references into it. The window where a run "succeeds" is simply the window where the allocator hasn't reused the pages yet.

Relationship to our other reports

Independent defect from #713 (fold race) and #714 (WAL replay): this one needs no writes, no checkpoint, and no crash — a single read-only query on an at-rest database is sufficient. We note it because together they interact badly: #713's crash strikes at fold moments, #714 turns that into a lost database, and this one silently corrupts what a surviving process reads back.

What we can offer

• The per-version valgrind logs (paired use/free stacks) for every rung of the ladder above — fully synthetic, attachable on request.
• Byte-level dumps from the Windows field failures showing heap free-list pointers stamped over the first 16 bytes of a delivered string value with the tail intact, and freed regions re-tenanted by ascending-integer arrays.
• We can rerun the instrument against any candidate fix or dev build and report same-day; our Windows machine reproduces the field failure on demand.

[adsharma]

Doesn't repro for me on linux.

for i in `seq 1 5`; do echo $i; PYTHONMALLOC=malloc valgrind --tool=memcheck --error-limit=no --num-callers=8 ./.venv/bin/python3  read.py test.lbdb 2>&1 | egrep Invalid; done
1
2
3
4
5

Using python 3.12, ubuntu 24.04 and 0.18.1

[adsharma]

Commit dc60d57 is a pattern that we have run into a couple of times in the past. But didn't find anything in the 0.16.1..0.17.0 range that matches. If you can git bisect into a smaller range (there are 142 commits), it can help triage. Thanks!

[BikerTech]

Update from our side, in fairness before you spend more time: we attempted to re-fire our own reproduction today in a fresh single-CPU Linux environment (Python 3.12, same wheel, same table, same valgrind invocation, including variants with pyarrow present, forced thread counts, and starved buffer pools) and got zero invalid reads — consistent with your result. Last night's runs, which fired deterministically across the version ladder in one environment, may have had more CPUs available; a delivery-worker/buffer-teardown race would plausibly require a real worker pool. Could you share your core count? We suggest treating the valgrind ladder as environment-conditional until re-established on a multi-core box. What is not conditional: on Windows 11 (24 cores), the same application against the same data fails the 24,498-row read near-100% on 0.18.1 and runs clean on 0.16.1 — verified in both directions this week. Byte dumps of the Windows failures (heap free-list pointers overwriting the head of a delivered string, tail intact) available on request. We'll attempt the bisect if and when we can re-fire locally.