Codex instance
0a7ce1e49e
Provide a separately installed OR-Tools model to compare a generic constraint solver with the native first-solution path without adding a production or default-test dependency. Use no-overlap, exact-fill, edge and equal-copy symmetry constraints, validate placements independently, and record model size, memory, worker count and timings for orders 8 and 9. Keep the tool only as a reference and defer DLX absent new evidence. Tests: Release CTest (9 passed) Tests: Python reference tests and compilation checks Tests: independently validated CP-SAT orders 8 and 9 Refs: #5
115 lines
5.5 KiB
Markdown
115 lines
5.5 KiB
Markdown
# Solver benchmarks
|
||
|
||
The benchmark suite records repeatable performance data independently of the
|
||
default correctness tests. It exercises the exhaustive infeasible order 7 and
|
||
the first-solution orders 8 and 9. The benchmark executable is opt-in:
|
||
|
||
```sh
|
||
cmake -S . -B build-benchmark -DCMAKE_BUILD_TYPE=Release \
|
||
-DPARTRIDGE_BUILD_BENCHMARKS=ON
|
||
cmake --build build-benchmark
|
||
python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
|
||
> benchmark.json
|
||
```
|
||
|
||
The default policy is one unrecorded warm-up followed by five repetitions per
|
||
case, with a 600-second timeout for each process. Solver stdout is captured;
|
||
the probe renders into an in-memory stream so grids do not perturb terminal I/O.
|
||
Override the policy with `--orders`, `--warmup`, `--repetitions`, and
|
||
`--timeout`. Order 9 uses the constructive odd-order path, searching order 8
|
||
and then tiling the enlarged border, so it is suitable for normal local
|
||
benchmarking:
|
||
|
||
```sh
|
||
python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
|
||
--orders 8 9 --warmup 1 --repetitions 5 --timeout 60 > benchmark.json
|
||
```
|
||
|
||
The JSON contains every run and median, range, and median absolute deviation
|
||
for solve, construction, independent validation, and rendering. Direct-search
|
||
cases report zero construction time: their setup and allocation remain part of
|
||
solve time. Constructed odd-order cases report predecessor search and
|
||
construction separately. The document also records compiler,
|
||
flags, build type, commit, OS/CPU metadata, worker count, search policy, seed,
|
||
timeouts, errors, invalid outputs, and the stdout policy. Search counts must
|
||
be stable across repeated runs. Prune and task counters are zero for the
|
||
current unpruned, single-threaded solver and reserve stable schema fields for
|
||
later work.
|
||
|
||
The runner writes its JSON report before returning a failure status if any mode
|
||
has no completed runs or produces an error or invalid output. Timeouts are
|
||
reported but do not fail a case when another repetition completed.
|
||
|
||
Normal `partridge_cpp` calls instantiate a compile-time counter-free solver.
|
||
Use `--measure-overhead` to run both counter-free and counted variants and
|
||
report their median difference:
|
||
|
||
```sh
|
||
python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
|
||
--orders 8 --warmup 2 --repetitions 7 --timeout 60 --measure-overhead \
|
||
> overhead.json
|
||
```
|
||
|
||
Counter-free and counted warm-ups and measurements are interleaved. The first
|
||
mode alternates on each repetition, limiting systematic bias from temperature,
|
||
frequency scaling, and run order. Reported overhead is the difference between
|
||
the two independently summarized medians.
|
||
|
||
Do not use wall-clock thresholds as correctness checks. Keep the generated
|
||
JSON outside version control unless it is being deliberately added as a named
|
||
comparison baseline.
|
||
|
||
## Post-correctness baseline
|
||
|
||
This framework starts from commit `ce39d0a` after the rendering assertion fix
|
||
in #7 and completion check fix in #14. The earlier Apple M1 Release results in
|
||
`results.md` are approximately 1.76 seconds for order 8 and 158.69 seconds
|
||
elapsed for order 9; they predate the structured runner and do not contain
|
||
search counters.
|
||
|
||
The first clean structured baseline used commit `598667b`, Apple Clang 21.0.0
|
||
with `-O3 -DNDEBUG`, Apple arm64, one worker, one warm-up, and three measured
|
||
repetitions. The runner reported a clean working tree:
|
||
|
||
| Order | Result | Counted solve median (range) | Nodes | Placements | Backtracks |
|
||
| --- | --- | --- | ---: | ---: | ---: |
|
||
| 7 | infeasible | 3.453 s (3.444–3.455 s) | 110,483,315 | 110,483,314 | 110,483,314 |
|
||
| 8 | solution | 1.817 s (1.814–1.817 s) | 60,485,176 | 60,485,176 | 60,485,140 |
|
||
|
||
Counts were stable across repetitions. Interleaved counter-free medians were
|
||
3.435 seconds for order 7 and 1.805 seconds for order 8, giving counted
|
||
overheads of 0.53% and 0.64% respectively. Construction time was zero; median
|
||
independent validation and rendering times were each below 0.02 milliseconds.
|
||
|
||
Order 9 was not rerun for this initial baseline because the former direct
|
||
search took several minutes. The default suite includes it with a per-run
|
||
timeout.
|
||
|
||
## Odd construction comparison
|
||
|
||
The order-9 construction was measured from the issue 6 working tree based on
|
||
commit `ddf07e7`, using Apple Clang 21.0.0 with `-O3 -DNDEBUG`, macOS arm64,
|
||
one worker, one warm-up, and three measured repetitions. Counter-free and
|
||
counted runs were interleaved:
|
||
|
||
| Order | Mode | Search median (range) | Construction median | Nodes |
|
||
| --- | --- | --- | --- | ---: |
|
||
| 8 | counter-free | 1.773 s (1.770–1.775 s) | 0 | 0 |
|
||
| 8 | counted | 1.849 s (1.848–1.850 s) | 0 | 60,485,176 |
|
||
| 9 | counter-free | 1.778 s (1.773–1.779 s) | 0.458 us | 0 |
|
||
| 9 | counted | 1.852 s (1.845–1.912 s) | 0.416 us | 60,485,176 |
|
||
|
||
All runs completed with valid results and stable counters. The matching
|
||
order-8 and order-9 search counts demonstrate that the new path searches only
|
||
the predecessor. Compared with the recorded 158.69-second direct order-9
|
||
elapsed time in `results.md`, the 1.778-second counter-free median plus
|
||
construction is approximately 89 times faster. The benchmark working tree
|
||
was necessarily dirty with the issue 6 implementation.
|
||
|
||
New optimization issues should quote the exact JSON
|
||
environment, policy, median/spread, stable counters, and counted overhead from
|
||
this runner for both before and after revisions.
|
||
|
||
The separate optional CP-SAT reference benchmark and its model, memory, worker,
|
||
and timing report are documented in [CP_SAT_REFERENCE.md](./CP_SAT_REFERENCE.md).
|