From ddf07e730a43c86e6a2d89283af988dec2935c6d Mon Sep 17 00:00:00 2001 From: Codex instance Date: Thu, 30 Jul 2026 17:10:01 +0100 Subject: [PATCH] bench: record clean solver baseline Record reproducible order-7 and order-8 measurements from the committed benchmark framework with a clean working tree. Include stable work counts, solve-time spread, validation and rendering cost, and interleaved instrumentation overhead. Order 9 remains supported by the runner but retains its historical timing because each current run takes several minutes. Benchmark: Apple arm64 Release, one warm-up, three repetitions Refs: #8 --- BENCHMARKING.md | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/BENCHMARKING.md b/BENCHMARKING.md index 26186ec..3ee0bf2 100644 --- a/BENCHMARKING.md +++ b/BENCHMARKING.md @@ -66,10 +66,22 @@ in #7 and completion check fix in #14. The earlier Apple M1 Release results in elapsed for order 9; they predate the structured runner and do not contain search counters. -A clean structured baseline will be recorded here after the benchmark -framework itself is committed. Order 9 may use its documented historical -result initially because its current runtime is several minutes; the default -suite includes it with a per-run timeout. +The first clean structured baseline used commit `598667b`, Apple Clang 21.0.0 +with `-O3 -DNDEBUG`, Apple arm64, one worker, one warm-up, and three measured +repetitions. The runner reported a clean working tree: + +| Order | Result | Counted solve median (range) | Nodes | Placements | Backtracks | +| --- | --- | --- | ---: | ---: | ---: | +| 7 | infeasible | 3.453 s (3.444–3.455 s) | 110,483,315 | 110,483,314 | 110,483,314 | +| 8 | solution | 1.817 s (1.814–1.817 s) | 60,485,176 | 60,485,176 | 60,485,140 | + +Counts were stable across repetitions. Interleaved counter-free medians were +3.435 seconds for order 7 and 1.805 seconds for order 8, giving counted +overheads of 0.53% and 0.64% respectively. Construction time was zero; median +independent validation and rendering times were each below 0.02 milliseconds. + +Order 9 was not rerun for this initial baseline because its documented runtime +is several minutes. The default suite includes it with a per-run timeout. New optimization issues should quote the exact JSON environment, policy, median/spread, stable counters, and counted overhead from