partridge-cpp/BENCHMARKING.md

Solver benchmarks

The benchmark suite records repeatable performance data independently of the default correctness tests. It exercises the exhaustive infeasible order 7 and the first-solution orders 8 and 9. The benchmark executable is opt-in:

cmake -S . -B build-benchmark -DCMAKE_BUILD_TYPE=Release \
  -DPARTRIDGE_BUILD_BENCHMARKS=ON
cmake --build build-benchmark
python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
  > benchmark.json

The default policy is one unrecorded warm-up followed by five repetitions per case, with a 600-second timeout for each process. Solver stdout is captured; the probe renders into an in-memory stream so grids do not perturb terminal I/O. Override the policy with --orders, --warmup, --repetitions, --timeout, and --candidate-order. Ascending candidate sizes are the production default. Order 9 uses the constructive odd-order path, searching order 8 and then tiling the enlarged border, so it is suitable for normal local benchmarking:

python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
  --orders 8 9 --warmup 1 --repetitions 5 --timeout 60 > benchmark.json

Use --direct-search when benchmarking the skyline core rather than the public even-predecessor construction used for odd orders from 9 onwards:

python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
  --orders 6 7 8 9 --warmup 1 --repetitions 5 --timeout 60 \
  --candidate-order ascending --direct-search > direct.json

The JSON contains every run and median, range, and median absolute deviation for solve, construction, independent validation, and rendering. Direct-search cases report zero construction time: their setup and allocation remain part of solve time. Constructed odd-order cases report predecessor search and construction separately. The document also records compiler, flags, build type, commit, OS/CPU metadata, worker count, search policy, seed, timeouts, errors, invalid outputs, and the stdout policy. Search counts must be stable across repeated runs. Prune and task counters are zero for the current unpruned, single-threaded solver and reserve stable schema fields for later work.

The runner writes its JSON report before returning a failure status if any mode has no completed runs or produces an error or invalid output. Timeouts are reported but do not fail a case when another repetition completed.

Normal partridge_cpp calls instantiate a compile-time counter-free solver. Use --measure-overhead to run both counter-free and counted variants and report their median difference:

python3 benchmarks/run.py --binary build-benchmark/partridge_benchmark \
  --orders 8 --warmup 2 --repetitions 7 --timeout 60 --measure-overhead \
  > overhead.json

Counter-free and counted warm-ups and measurements are interleaved. The first mode alternates on each repetition, limiting systematic bias from temperature, frequency scaling, and run order. Reported overhead is the difference between the two independently summarized medians.

Do not use wall-clock thresholds as correctness checks. Keep the generated JSON outside version control unless it is being deliberately added as a named comparison baseline.

Smallest-valley skyline

The solver stores one filled height per board column instead of one value per cell. Equal adjacent heights form conceptual vertical bars. A valley is a maximal bar lower than both neighbours, with board edges treated as bars of full board height. Each node scans for the smallest-width valley, breaking ties by lower height and then leftmost position, and tries every available square which fits at that valley's far-left edge.

This branching remains complete: the bottom-left cell of the selected valley must be covered, a square covering it cannot begin to the left across the taller neighbour, and it cannot extend beyond the equal-height run without overlap or leaving an unreachable hole. Trying every fitting available size therefore includes the placement used by every possible completion.

For board width W and order n, the skyline scan is O(W). A node tries at most n candidates and each placement or exact undo changes at most n heights, giving O(W + n^2) local work. The skyline, multiplicities, placements, and recursion stack use O(W + n^2) state, compared with the former O(W^2) cell grid.

One-run exploratory measurements used the issue #4 dirty worktree at base commit 0a7ce1e, Apple Clang 21.0.0, -O3 -DNDEBUG, macOS arm64, one worker, no warm-up, and a 15-second timeout. Every completed result passed the independent benchmark validator:

Order	Result	Ascending time	Ascending nodes	Descending time	Descending nodes
6	infeasible	0.040 s	659,598	0.039 s	659,598
7	infeasible	3.103 s	43,604,507	3.071 s	43,604,507
8	solution	0.585 s	7,735,369	0.941 s	12,186,125
9 direct	solution	3.831 s	45,840,266	timeout	unavailable

The infeasible orders exhaust the same tree in either direction. Ascending was selected as the default because it reaches the first order-8 solution with 36% fewer nodes and also completed direct order 9 within the timeout; descending direct order 9 did not.

A direct ascending order-10 probe exceeded 20 seconds. Public order 10 is also a direct search, and public order 11 first searches order 10 before using odd-predecessor construction. Consequently neither 10 nor 11 is in the routine correctness suite: doing so would test the same unresolved order-10 search bottleneck, while the existing route-boundary test still verifies that 11 selects construction. Revisit both sizes when order 10 completes within a practical test budget.

Post-correctness baseline

This framework starts from commit ce39d0a after the rendering assertion fix in #7 and completion check fix in #14. The earlier Apple M1 Release results in results.md are approximately 1.76 seconds for order 8 and 158.69 seconds elapsed for order 9; they predate the structured runner and do not contain search counters.

The first clean structured baseline used commit 598667b, Apple Clang 21.0.0 with -O3 -DNDEBUG, Apple arm64, one worker, one warm-up, and three measured repetitions. The runner reported a clean working tree:

Order	Result	Counted solve median (range)	Nodes	Placements	Backtracks
7	infeasible	3.453 s (3.444–3.455 s)	110,483,315	110,483,314	110,483,314
8	solution	1.817 s (1.814–1.817 s)	60,485,176	60,485,176	60,485,140

Counts were stable across repetitions. Interleaved counter-free medians were 3.435 seconds for order 7 and 1.805 seconds for order 8, giving counted overheads of 0.53% and 0.64% respectively. Construction time was zero; median independent validation and rendering times were each below 0.02 milliseconds.

Order 9 was not rerun for this initial baseline because the former direct search took several minutes. The default suite includes it with a per-run timeout.

Odd construction comparison

The order-9 construction was measured from the issue 6 working tree based on commit ddf07e7, using Apple Clang 21.0.0 with -O3 -DNDEBUG, macOS arm64, one worker, one warm-up, and three measured repetitions. Counter-free and counted runs were interleaved:

Order	Mode	Search median (range)	Construction median	Nodes
8	counter-free	1.773 s (1.770–1.775 s)	0	0
8	counted	1.849 s (1.848–1.850 s)	0	60,485,176
9	counter-free	1.778 s (1.773–1.779 s)	0.458 us	0
9	counted	1.852 s (1.845–1.912 s)	0.416 us	60,485,176

All runs completed with valid results and stable counters. The matching order-8 and order-9 search counts demonstrate that the new path searches only the predecessor. Compared with the recorded 158.69-second direct order-9 elapsed time in results.md, the 1.778-second counter-free median plus construction is approximately 89 times faster. The benchmark working tree was necessarily dirty with the issue 6 implementation.

New optimization issues should quote the exact JSON environment, policy, median/spread, stable counters, and counted overhead from this runner for both before and after revisions.

The separate optional CP-SAT reference benchmark and its model, memory, worker, and timing report are documented in CP_SAT_REFERENCE.md.