Codex instance
d751d1b13e
Represent partial placements as column heights and branch on the narrowest local valley. This removes the board-area cell state and makes first-solution search substantially smaller for feasible orders. Expose direct-search and candidate-order benchmark controls so the skyline core can be measured independently of odd-order construction. Document the completeness argument and the 10/11 test-tier decision. Tests: Release, Debug, ASan, and UBSan CTest (10 passed each) Refs: #4
170 lines
8.4 KiB
Markdown
170 lines
8.4 KiB
Markdown
# Solver benchmarks
|
||
|
||
The benchmark suite records repeatable performance data independently of the
|
||
default correctness tests. It exercises the exhaustive infeasible order 7 and
|
||
the first-solution orders 8 and 9. The benchmark executable is opt-in:
|
||
|
||
```sh
|
||
cmake -S . -B build-benchmark -DCMAKE_BUILD_TYPE=Release \
|
||
-DPARTRIDGE_BUILD_BENCHMARKS=ON
|
||
cmake --build build-benchmark
|
||
python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
|
||
> benchmark.json
|
||
```
|
||
|
||
The default policy is one unrecorded warm-up followed by five repetitions per
|
||
case, with a 600-second timeout for each process. Solver stdout is captured;
|
||
the probe renders into an in-memory stream so grids do not perturb terminal I/O.
|
||
Override the policy with `--orders`, `--warmup`, `--repetitions`, `--timeout`,
|
||
and `--candidate-order`. Ascending candidate sizes are the production default.
|
||
Order 9 uses the constructive odd-order path, searching order 8 and then tiling
|
||
the enlarged border, so it is suitable for normal local benchmarking:
|
||
|
||
```sh
|
||
python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
|
||
--orders 8 9 --warmup 1 --repetitions 5 --timeout 60 > benchmark.json
|
||
```
|
||
|
||
Use `--direct-search` when benchmarking the skyline core rather than the public
|
||
even-predecessor construction used for odd orders from 9 onwards:
|
||
|
||
```sh
|
||
python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
|
||
--orders 6 7 8 9 --warmup 1 --repetitions 5 --timeout 60 \
|
||
--candidate-order ascending --direct-search > direct.json
|
||
```
|
||
|
||
The JSON contains every run and median, range, and median absolute deviation
|
||
for solve, construction, independent validation, and rendering. Direct-search
|
||
cases report zero construction time: their setup and allocation remain part of
|
||
solve time. Constructed odd-order cases report predecessor search and
|
||
construction separately. The document also records compiler,
|
||
flags, build type, commit, OS/CPU metadata, worker count, search policy, seed,
|
||
timeouts, errors, invalid outputs, and the stdout policy. Search counts must
|
||
be stable across repeated runs. Prune and task counters are zero for the
|
||
current unpruned, single-threaded solver and reserve stable schema fields for
|
||
later work.
|
||
|
||
The runner writes its JSON report before returning a failure status if any mode
|
||
has no completed runs or produces an error or invalid output. Timeouts are
|
||
reported but do not fail a case when another repetition completed.
|
||
|
||
Normal `partridge_cpp` calls instantiate a compile-time counter-free solver.
|
||
Use `--measure-overhead` to run both counter-free and counted variants and
|
||
report their median difference:
|
||
|
||
```sh
|
||
python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
|
||
--orders 8 --warmup 2 --repetitions 7 --timeout 60 --measure-overhead \
|
||
> overhead.json
|
||
```
|
||
|
||
Counter-free and counted warm-ups and measurements are interleaved. The first
|
||
mode alternates on each repetition, limiting systematic bias from temperature,
|
||
frequency scaling, and run order. Reported overhead is the difference between
|
||
the two independently summarized medians.
|
||
|
||
Do not use wall-clock thresholds as correctness checks. Keep the generated
|
||
JSON outside version control unless it is being deliberately added as a named
|
||
comparison baseline.
|
||
|
||
## Smallest-valley skyline
|
||
|
||
The solver stores one filled height per board column instead of one value per
|
||
cell. Equal adjacent heights form conceptual vertical bars. A valley is a
|
||
maximal bar lower than both neighbours, with board edges treated as bars of
|
||
full board height. Each node scans for the smallest-width valley, breaking
|
||
ties by lower height and then leftmost position, and tries every available
|
||
square which fits at that valley's far-left edge.
|
||
|
||
This branching remains complete: the bottom-left cell of the selected valley
|
||
must be covered, a square covering it cannot begin to the left across the
|
||
taller neighbour, and it cannot extend beyond the equal-height run without
|
||
overlap or leaving an unreachable hole. Trying every fitting available size
|
||
therefore includes the placement used by every possible completion.
|
||
|
||
For board width `W` and order `n`, the skyline scan is `O(W)`. A node tries at
|
||
most `n` candidates and each placement or exact undo changes at most `n`
|
||
heights, giving `O(W + n^2)` local work. The skyline, multiplicities,
|
||
placements, and recursion stack use `O(W + n^2)` state, compared with the
|
||
former `O(W^2)` cell grid.
|
||
|
||
One-run exploratory measurements used the issue #4 dirty worktree at base
|
||
commit `0a7ce1e`, Apple Clang 21.0.0, `-O3 -DNDEBUG`, macOS arm64, one worker,
|
||
no warm-up, and a 15-second timeout. Every completed result passed the
|
||
independent benchmark validator:
|
||
|
||
| Order | Result | Ascending time | Ascending nodes | Descending time | Descending nodes |
|
||
| --- | --- | ---: | ---: | ---: | ---: |
|
||
| 6 | infeasible | 0.040 s | 659,598 | 0.039 s | 659,598 |
|
||
| 7 | infeasible | 3.103 s | 43,604,507 | 3.071 s | 43,604,507 |
|
||
| 8 | solution | 0.585 s | 7,735,369 | 0.941 s | 12,186,125 |
|
||
| 9 direct | solution | 3.831 s | 45,840,266 | timeout | unavailable |
|
||
|
||
The infeasible orders exhaust the same tree in either direction. Ascending
|
||
was selected as the default because it reaches the first order-8 solution with
|
||
36% fewer nodes and also completed direct order 9 within the timeout;
|
||
descending direct order 9 did not.
|
||
|
||
A direct ascending order-10 probe exceeded 20 seconds. Public order 10 is
|
||
also a direct search, and public order 11 first searches order 10 before using
|
||
odd-predecessor construction. Consequently neither 10 nor 11 is in the
|
||
routine correctness suite: doing so would test the same unresolved order-10
|
||
search bottleneck, while the existing route-boundary test still verifies that
|
||
11 selects construction. Revisit both sizes when order 10 completes within a
|
||
practical test budget.
|
||
|
||
## Post-correctness baseline
|
||
|
||
This framework starts from commit `ce39d0a` after the rendering assertion fix
|
||
in #7 and completion check fix in #14. The earlier Apple M1 Release results in
|
||
`results.md` are approximately 1.76 seconds for order 8 and 158.69 seconds
|
||
elapsed for order 9; they predate the structured runner and do not contain
|
||
search counters.
|
||
|
||
The first clean structured baseline used commit `598667b`, Apple Clang 21.0.0
|
||
with `-O3 -DNDEBUG`, Apple arm64, one worker, one warm-up, and three measured
|
||
repetitions. The runner reported a clean working tree:
|
||
|
||
| Order | Result | Counted solve median (range) | Nodes | Placements | Backtracks |
|
||
| --- | --- | --- | ---: | ---: | ---: |
|
||
| 7 | infeasible | 3.453 s (3.444–3.455 s) | 110,483,315 | 110,483,314 | 110,483,314 |
|
||
| 8 | solution | 1.817 s (1.814–1.817 s) | 60,485,176 | 60,485,176 | 60,485,140 |
|
||
|
||
Counts were stable across repetitions. Interleaved counter-free medians were
|
||
3.435 seconds for order 7 and 1.805 seconds for order 8, giving counted
|
||
overheads of 0.53% and 0.64% respectively. Construction time was zero; median
|
||
independent validation and rendering times were each below 0.02 milliseconds.
|
||
|
||
Order 9 was not rerun for this initial baseline because the former direct
|
||
search took several minutes. The default suite includes it with a per-run
|
||
timeout.
|
||
|
||
## Odd construction comparison
|
||
|
||
The order-9 construction was measured from the issue 6 working tree based on
|
||
commit `ddf07e7`, using Apple Clang 21.0.0 with `-O3 -DNDEBUG`, macOS arm64,
|
||
one worker, one warm-up, and three measured repetitions. Counter-free and
|
||
counted runs were interleaved:
|
||
|
||
| Order | Mode | Search median (range) | Construction median | Nodes |
|
||
| --- | --- | --- | --- | ---: |
|
||
| 8 | counter-free | 1.773 s (1.770–1.775 s) | 0 | 0 |
|
||
| 8 | counted | 1.849 s (1.848–1.850 s) | 0 | 60,485,176 |
|
||
| 9 | counter-free | 1.778 s (1.773–1.779 s) | 0.458 us | 0 |
|
||
| 9 | counted | 1.852 s (1.845–1.912 s) | 0.416 us | 60,485,176 |
|
||
|
||
All runs completed with valid results and stable counters. The matching
|
||
order-8 and order-9 search counts demonstrate that the new path searches only
|
||
the predecessor. Compared with the recorded 158.69-second direct order-9
|
||
elapsed time in `results.md`, the 1.778-second counter-free median plus
|
||
construction is approximately 89 times faster. The benchmark working tree
|
||
was necessarily dirty with the issue 6 implementation.
|
||
|
||
New optimization issues should quote the exact JSON
|
||
environment, policy, median/spread, stable counters, and counted overhead from
|
||
this runner for both before and after revisions.
|
||
|
||
The separate optional CP-SAT reference benchmark and its model, memory, worker,
|
||
and timing report are documented in [CP_SAT_REFERENCE.md](./CP_SAT_REFERENCE.md).
|