bench: add repeatable solver measurements #20

2026-07-30T17:12:14+01:00

mcp commented

2026-07-30 17:12:14 +01:00

Summary

add compile-time counter-free and opt-in instrumented solver paths
record nodes, loop iterations, placements, backtracks, and stable future prune/task fields
add an opt-in C++ benchmark probe and structured JSON runner
separate solve, construction, independent validation, and rendering timings
capture compiler/build/commit/hardware/worker/policy/seed metadata
support warm-ups, repetitions, per-run timeouts, median/range/MAD, errors, and interleaved overhead measurement
keep heavyweight cases outside default CTest
document and record a clean post-#7/#14 baseline

Commits

598667b — benchmark framework and cheap tests
ddf07e7 — clean baseline recorded against exact framework commit

Testing

Release CTest: 7/7 passed
Debug CTest: 7/7 passed
ASan CTest: 7/7 passed
UBSan CTest: 7/7 passed
timeout-only and invalid-output cases emit JSON and fail appropriately

Baseline

Clean Apple arm64 Release run, one warm-up and three interleaved repetitions:

order 7: counted median 3.453 s, 110,483,315 nodes, 0.53% overhead
order 8: counted median 1.817 s, 60,485,176 nodes, 0.64% overhead
stable counts; validation/rendering medians below 0.02 ms

Order 9 is supported but retains the historical ~158.69 s result because repeating it is a multi-minute run.

Review

Independent review initially found provenance, timing-semantics, failure-signaling, and overhead-ordering issues. All four were corrected and the re-review found no remaining issues.

Closes #8
Part of #16

## Summary - add compile-time counter-free and opt-in instrumented solver paths - record nodes, loop iterations, placements, backtracks, and stable future prune/task fields - add an opt-in C++ benchmark probe and structured JSON runner - separate solve, construction, independent validation, and rendering timings - capture compiler/build/commit/hardware/worker/policy/seed metadata - support warm-ups, repetitions, per-run timeouts, median/range/MAD, errors, and interleaved overhead measurement - keep heavyweight cases outside default CTest - document and record a clean post-#7/#14 baseline ## Commits 1. `598667b` — benchmark framework and cheap tests 2. `ddf07e7` — clean baseline recorded against exact framework commit ## Testing - Release CTest: 7/7 passed - Debug CTest: 7/7 passed - ASan CTest: 7/7 passed - UBSan CTest: 7/7 passed - timeout-only and invalid-output cases emit JSON and fail appropriately ## Baseline Clean Apple arm64 Release run, one warm-up and three interleaved repetitions: - order 7: counted median 3.453 s, 110,483,315 nodes, 0.53% overhead - order 8: counted median 1.817 s, 60,485,176 nodes, 0.64% overhead - stable counts; validation/rendering medians below 0.02 ms Order 9 is supported but retains the historical ~158.69 s result because repeating it is a multi-minute run. ## Review Independent review initially found provenance, timing-semantics, failure-signaling, and overhead-ordering issues. All four were corrected and the re-review found no remaining issues. Closes #8 Part of #16

mcp added 2 commits 2026-07-30 17:12:15 +01:00

bench: add repeatable solver measurements 598667b2f3

Add an opt-in benchmark probe and JSON runner that separate solve, construction, validation, and rendering time. Record stable search counters, environment metadata, warm-up and repetition policy, timeouts, errors, median spread, and instrumentation overhead.

Compile production solving without counters and interleave counted and plain trials when measuring overhead. Keep heavyweight cases outside the default correctness path while testing counter and report behavior cheaply.

Tests: Debug and Release CTest suites (7 passed each)

Refs: #8

bench: record clean solver baseline ddf07e730a

Record reproducible order-7 and order-8 measurements from the committed benchmark framework with a clean working tree. Include stable work counts, solve-time spread, validation and rendering cost, and interleaved instrumentation overhead.

Order 9 remains supported by the runner but retains its historical timing because each current run takes several minutes.

Benchmark: Apple arm64 Release, one warm-up, three repetitions

Refs: #8

mcp merged commit ddf07e730a into main

2026-07-30 17:12:29 +01:00

mcp deleted branch codex/issue-8-benchmarks

2026-07-30 17:12:29 +01:00

mcp referenced this pull request

2026-07-30 17:12:59 +01:00

roadmap: improve time to the first square solution #16

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mgrettondann/partridge-cpp#20