# Solver benchmarks

The benchmark suite records repeatable performance data independently of the
default correctness tests.  It exercises the exhaustive infeasible order 7 and
the first-solution orders 8 and 9.  The benchmark executable is opt-in:

```sh
cmake -S . -B build-benchmark -DCMAKE_BUILD_TYPE=Release \
  -DPARTRIDGE_BUILD_BENCHMARKS=ON
cmake --build build-benchmark
python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
  > benchmark.json
```

The default policy is one unrecorded warm-up followed by five repetitions per
case, with a 600-second timeout for each process.  Solver stdout is captured;
the probe renders into an in-memory stream so grids do not perturb terminal I/O.
Override the policy with `--orders`, `--warmup`, `--repetitions`, and
`--timeout`.  Order 9 uses the constructive odd-order path, searching order 8
and then tiling the enlarged border, so it is suitable for normal local
benchmarking:

```sh
python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
  --orders 8 9 --warmup 1 --repetitions 5 --timeout 60 > benchmark.json
```

The JSON contains every run and median, range, and median absolute deviation
for solve, construction, independent validation, and rendering.  Direct-search
cases report zero construction time: their setup and allocation remain part of
solve time.  Constructed odd-order cases report predecessor search and
construction separately.  The document also records compiler,
flags, build type, commit, OS/CPU metadata, worker count, search policy, seed,
timeouts, errors, invalid outputs, and the stdout policy.  Search counts must
be stable across repeated runs.  Prune and task counters are zero for the
current unpruned, single-threaded solver and reserve stable schema fields for
later work.

The runner writes its JSON report before returning a failure status if any mode
has no completed runs or produces an error or invalid output.  Timeouts are
reported but do not fail a case when another repetition completed.

Normal `partridge_cpp` calls instantiate a compile-time counter-free solver.
Use `--measure-overhead` to run both counter-free and counted variants and
report their median difference:

```sh
python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
  --orders 8 --warmup 2 --repetitions 7 --timeout 60 --measure-overhead \
  > overhead.json
```

Counter-free and counted warm-ups and measurements are interleaved.  The first
mode alternates on each repetition, limiting systematic bias from temperature,
frequency scaling, and run order.  Reported overhead is the difference between
the two independently summarized medians.

Do not use wall-clock thresholds as correctness checks.  Keep the generated
JSON outside version control unless it is being deliberately added as a named
comparison baseline.

## Post-correctness baseline

This framework starts from commit `ce39d0a` after the rendering assertion fix
in #7 and completion check fix in #14.  The earlier Apple M1 Release results in
`results.md` are approximately 1.76 seconds for order 8 and 158.69 seconds
elapsed for order 9; they predate the structured runner and do not contain
search counters.

The first clean structured baseline used commit `598667b`, Apple Clang 21.0.0
with `-O3 -DNDEBUG`, Apple arm64, one worker, one warm-up, and three measured
repetitions.  The runner reported a clean working tree:

| Order | Result | Counted solve median (range) | Nodes | Placements | Backtracks |
| --- | --- | --- | ---: | ---: | ---: |
| 7 | infeasible | 3.453 s (3.444–3.455 s) | 110,483,315 | 110,483,314 | 110,483,314 |
| 8 | solution | 1.817 s (1.814–1.817 s) | 60,485,176 | 60,485,176 | 60,485,140 |

Counts were stable across repetitions.  Interleaved counter-free medians were
3.435 seconds for order 7 and 1.805 seconds for order 8, giving counted
overheads of 0.53% and 0.64% respectively.  Construction time was zero; median
independent validation and rendering times were each below 0.02 milliseconds.

Order 9 was not rerun for this initial baseline because the former direct
search took several minutes.  The default suite includes it with a per-run
timeout.

## Odd construction comparison

The order-9 construction was measured from the issue 6 working tree based on
commit `ddf07e7`, using Apple Clang 21.0.0 with `-O3 -DNDEBUG`, macOS arm64,
one worker, one warm-up, and three measured repetitions.  Counter-free and
counted runs were interleaved:

| Order | Mode | Search median (range) | Construction median | Nodes |
| --- | --- | --- | --- | ---: |
| 8 | counter-free | 1.773 s (1.770–1.775 s) | 0 | 0 |
| 8 | counted | 1.849 s (1.848–1.850 s) | 0 | 60,485,176 |
| 9 | counter-free | 1.778 s (1.773–1.779 s) | 0.458 us | 0 |
| 9 | counted | 1.852 s (1.845–1.912 s) | 0.416 us | 60,485,176 |

All runs completed with valid results and stable counters.  The matching
order-8 and order-9 search counts demonstrate that the new path searches only
the predecessor.  Compared with the recorded 158.69-second direct order-9
elapsed time in `results.md`, the 1.778-second counter-free median plus
construction is approximately 89 times faster.  The benchmark working tree
was necessarily dirty with the issue 6 implementation.

New optimization issues should quote the exact JSON
environment, policy, median/spread, stable counters, and counted overhead from
this runner for both before and after revisions.