scorio.aggregate

Test-time-scaling aggregation. Import the public API with from scorio import aggregate or its short alias from scorio import agg. The two names refer to the same module, so scorio.agg.best_of_n(...) and scorio.aggregate.best_of_n(...) are equivalent. The methods fall into four categories:

1. Confidence signals turn a trace’s token log-probabilities / top-\(k\) log-probabilities into a scalar confidence – the scores the selection rules consume.

2. Reward aggregation reduces a process reward model’s per-step scores to one per-trace reward.

3. Offline selection collapses a fixed pool of answers (+ optional scores) into one predicted answer.

4. Online early stopping decides when to stop sampling traces or generating a trace.

Signals and selection compose: many literature methods are a (signal, selection rule) pair. For example, DeepConf offline voting is weighted_majority_vote fed deepconf_confidence(); self-certainty Best-of-N is best_of_n fed self_certainty(); length-normalized likelihood-weighted self-consistency is weighted_majority_vote fed exp(mean_logprob).

Shared conventions

Each selection rule collapses a pool of sampled candidates for one question into a single predicted answer. The inputs are two aligned arrays:

• answers: (N,) for one question or (M, N) for a batch of M questions with N candidates each. Labels are compared by equality, so any hashable value works (string, int, canonicalized expression). Entries that are None, "" or NaN are treated as unparsable and excluded from every rule.

• scores: per-candidate reward-model / verifier / confidence scores of the same shape (higher is better). Required by every rule except majority_vote(), which uses only answers.

By default, every offline selection rule returns the selected answer per question: a scalar for (N,) input, or an (M,) object array for (M, N) input. Optional flags can append information about a representative candidate:

• Every selection rule supports return_index=True, which also returns a representative candidate index (-1 when a question has no valid answer).

• Score-aware selection rules also support return_score=True, which returns the score of that representative candidate (NaN when a question has no valid answer). For best_of_n(), this is the maximizing (argmax) score. majority_vote() does not support return_score because it does not take scores.

When both flags are supported and requested, values are returned in the order (selected, index, score). Requesting only one flag returns (selected, index) or (selected, score).

These rules select answers but do not calculate accuracy. To measure the accuracy of a rule across a benchmark, gather the correctness of the selected candidates and pass it to scorio.eval (e.g. eval.avg / eval.bayes) for an accuracy with a credible interval.

Confidence signals

Per-trace scalars computed from a trace’s own token statistics – the chosen-token log-probabilities (T,) and the per-position top-\(k\) log-probabilities (T, k) – with no external verifier. Most are confidences (higher = more confident). The uncertainties perplexity() and token_entropy() are lower = more confident and must be reoriented for selectors that expect higher scores. varentropy() is instead a dispersion diagnostic: neither direction is generally more confident, so it should not be used alone as a selection score without a task-specific interpretation. Choose the transformation to match the selection rule:

• Order-based selectors, including best_of_n(), majority_of_the_bests(), rank_weighted_vote(), and filtered_vote() with weighted=False, use only score ordering. Pass higher-oriented confidences directly; negate perplexity() or token_entropy().

• Additive raw-weight voting, including weighted_majority_vote() with aggregate="sum" and filtered_vote() with weighted=True, uses score magnitudes and signs. Supply non-negative weights such as exp(mean_logprob), exp(sequence_logprob), or 1 / perplexity. Do not use raw negative log-probabilities or -perplexity as vote weights.

• softmax_weighted_vote() applies its own softmax and accepts raw higher-is-better scores. logit_weighted_vote() with its default transform expects verifier probabilities strictly between zero and one.

mean_logprob(logprobs)

Mean chosen-token log-probability (length-normalized sequence log-likelihood).

Averages the log-probability the model assigned to each token it actually generated. Higher (closer to zero) means the trace was, on average, more predictable to the model and is used as a length-fair confidence for reward-free selection. As a Best-of-N score this is the sample-and-rank rule of Meena. Exponentiating it gives the length-normalized likelihood weight used by normalized weighted self-consistency, \(\exp(\text{mean\_logprob}) = 1 / \mathrm{PPL}\). Do not use the raw negative value as an additive vote weight.

References

Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., & Le, Q. V. (2020). Towards a Human-like Open-Domain Chatbot (Meena; sample-and-rank). arXiv:2001.09977. https://arxiv.org/abs/2001.09977

Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models (Sec. 2, normalized weighted sum). ICLR 2023, arXiv:2203.11171. https://arxiv.org/abs/2203.11171

Parameters:

logprobs (Any) – (T,) array-like of chosen-token log-probabilities for one trace.

Return type:

float

Returns:

The mean log-probability (<= 0); higher is more confident.

Formula:

\[s(y) = \frac{1}{T} \sum_{t=1}^{T} \log p_\theta(y_t \mid y_{<t}, x).\]

Examples

>>> round(mean_logprob([-0.1, -0.2, -0.3]), 4)
-0.2

sequence_logprob(logprobs)

Total (un-normalized) sequence log-likelihood, \(\sum_t \log p_\theta\).

The sum of the chosen-token log-probabilities – the log-probability of the whole generation. Unlike mean_logprob() it is not length-normalized, so it penalizes longer traces. Exponentiating it gives the path probability used by the paper’s unnormalized weighted sum, where paths vote with weight \(\exp(\text{sequence\_logprob})\).

References

Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models (Sec. 2, unnormalized weighted sum). ICLR 2023, arXiv:2203.11171. https://arxiv.org/abs/2203.11171

Parameters:

logprobs (Any) – (T,) array-like of chosen-token log-probabilities.

Return type:

float

Returns:

The summed log-probability (<= 0); higher is more confident.

Formula:

\[\log p_\theta(y \mid x) = \sum_{t=1}^{T} \log p_\theta(y_t \mid y_{<t}, x).\]

Examples

>>> round(sequence_logprob([-0.1, -0.2, -0.3]), 4)
-0.6

perplexity(logprobs)

Sequence perplexity, \(\exp(-\text{mean\_logprob})\).

The exponentiated mean negative log-probability. Lower is more confident (perplexity 1 is a perfectly predicted trace), so this is an uncertainty. For an order-based selector, negate it or use mean_logprob(); for magnitude-weighted voting, use reciprocal perplexity so the vote weight remains non-negative.

Parameters:

logprobs (Any) – (T,) array-like of chosen-token log-probabilities.

Return type:

float

Returns:

The perplexity (>= 1); lower is more confident.

Formula:

\[\mathrm{PPL}(y) = \exp\!\Big(-\tfrac{1}{T} \sum_{t=1}^{T} \log p_\theta(y_t \mid y_{<t}, x)\Big).\]

Examples

>>> round(perplexity([0.0, 0.0, 0.0]), 4)
1.0

self_certainty(topk_logprobs, *, aggregate='mean')

Self-certainty: average KL divergence from uniform to the next-token law.

Measures how far each step’s next-token distribution sits from maximum uncertainty (the uniform distribution): a peaked distribution is far from uniform and scores high. Averaged over the trace it is a reward-free confidence that ranks Best-of-N candidates as well as a trained verifier, and its rank-weighted vote is rank_weighted_vote().

Only the top-\(k\) log-probabilities are exposed here, so the per-token KL is computed over the renormalized top-\(k\) support (uniform reference over the \(k\) observed candidates). This is the top-\(k\) approximation of the paper’s full-vocabulary Eq. (9); because the near-zero tail tokens the paper sums over are unavailable, the value is a biased but empirically monotone proxy, not the exact quantity.

References

Kang, Z., Zhao, X., & Song, D. (2025). Scalable Best-of-N Selection for Large Language Models via Self-Certainty. NeurIPS 2025, arXiv:2502.18581. https://arxiv.org/abs/2502.18581

Parameters:

• topk_logprobs (Any) – (T, k) array-like (or a ragged length-T list of rows) of per-position top-\(k\) log-probabilities.

• aggregate (str) – How to reduce the per-token KL over the trace: "mean" (default), "min" (least-certain token), or "max".

Return type:

float

Returns:

The (top-\(k\)) self-certainty; higher is more confident.

Formula:

\[C_i^{\mathrm{KL}} = \mathrm{KL}\big(U \,\|\, p_i\big) = -\log k - \tfrac{1}{k}\sum_{j} \log \tilde p_{ij}, \qquad \mathrm{SC}(y) = \operatorname*{agg}_i C_i^{\mathrm{KL}},\]

with \(\tilde p_i\) the renormalized top-\(k\) distribution.

Examples

>>> # near one-hot position -> high self-certainty; flat -> ~0
>>> peaked = [[0.0, -20.0, -20.0]]
>>> self_certainty(peaked) > 5
True
>>> flat = [[-1.0986, -1.0986, -1.0986]]     # ~uniform over 3
>>> abs(self_certainty(flat)) < 1e-3
True

token_confidence(topk_logprobs)

DeepConf per-token confidence \(C_i = -\tfrac1k \sum_j \log P_i(j)\).

The negative mean of the top-\(k\) log-probabilities at each position – high when the model’s probability mass concentrates on a few tokens. This is the building block for every DeepConf trace-level reduction (deepconf_confidence()) and for the online early-termination rule scorio.aggregate.deepconf_online_stop(); unlike token_entropy() it uses the raw top-\(k\) log-probabilities without renormalizing.

References

Fu, Y., Wang, X., Tian, Y., & Zhao, J. (2025). Deep Think with Confidence (Eq. 2). arXiv:2508.15260. https://arxiv.org/abs/2508.15260

Parameters:

topk_logprobs (Any) – (T, k) array-like (or ragged list of rows) of per-position top-\(k\) log-probabilities.

Return type:

ndarray

Returns:

A (T,) float array of per-token confidences (>= 0); higher is more confident.

Formula:

\[C_i = -\frac{1}{k} \sum_{j=1}^{k} \log P_i(j).\]

Examples

>>> token_confidence([[0.0, -2.0], [-1.0, -3.0]]).tolist()
[1.0, 2.0]

deepconf_confidence(topk_logprobs, *, mode='mean', window=2048, tail_tokens=2048, bottom_quantile=0.10)

DeepConf trace confidence: average, tail, bottom-percentile, or lowest group.

Reduces the DeepConf per-token confidences (token_confidence()) to one trace-level score, using the reduction that best separates correct from incorrect traces. All four reductions are confidences (higher = more confident) and drop into weighted_majority_vote() (confidence-weighted vote) or filtered_vote() (confidence filtering); the group-based reductions are the statistic behind DeepConf’s online early stopping.

References

Fu, Y., Wang, X., Tian, Y., & Zhao, J. (2025). Deep Think with Confidence. arXiv:2508.15260. https://arxiv.org/abs/2508.15260

Parameters:

• topk_logprobs (Any) – (T, k) array-like (or ragged list of rows) of per-position top-\(k\) log-probabilities.

• mode (str) –

  Which reduction of the per-token confidences to return:

  • "mean" – average trace confidence \(\tfrac1T\sum_i C_i\).

  • "tail" – mean over the last tail_tokens tokens.

  • "bottom_group" – mean of the lowest bottom_quantile fraction of sliding-window (group) confidences.

  • "lowest_group" – the minimum sliding-window confidence.

• window (int) – Sliding-window (group) size in tokens for the group reductions (paper default 2048). Clamped to the trace length.

• tail_tokens (int) – Tail length for mode="tail" (paper default 2048).

• bottom_quantile (float) – Fraction of lowest groups averaged for mode="bottom_group" (paper default 0.10).

Return type:

float

Returns:

The trace confidence (>= 0); higher is more confident.

Formula:

With \(C_i\) the per-token confidence and \(C_{G_s}\) the mean of \(C_i\) over the sliding window \(G_s\) of window tokens,

\[C_{\mathrm{mean}} = \tfrac1T\sum_i C_i, \quad C_{\mathrm{tail}} = \operatorname{mean}_{i > T - \tau} C_i, \quad C_{\mathrm{lowest}} = \min_s C_{G_s}, \quad C_{\mathrm{bottom}} = \operatorname{mean}_{s:\,C_{G_s} \le Q_{\eta}(C_{G_\cdot})} C_{G_s}.\]

Examples

>>> tk = [[0.0, -2.0]] * 3 + [[-1.0, -3.0]] * 3   # conf 1,1,1,2,2,2
>>> round(deepconf_confidence(tk, mode="mean"), 4)
1.5
>>> round(deepconf_confidence(tk, mode="tail", tail_tokens=3), 4)
2.0
>>> round(deepconf_confidence(tk, mode="lowest_group", window=3), 4)
1.0

token_entropy(topk_logprobs, *, aggregate='mean')

Mean (or max) Shannon entropy of the top-k next-token distribution.

The per-token entropy of the renormalized top-\(k\) distribution, reduced over the trace. Higher is less confident (an uncertainty): its negation is Kang et al.’s negative-entropy confidence and it is also the token-level signal underlying DeepConf; negate it before using it as a selection score. With aggregate="max" it reports the single most-uncertain token.

References

Malinin, A., & Gales, M. (2021). Uncertainty Estimation in Autoregressive Structured Prediction. ICLR 2021, arXiv:2002.07650. https://arxiv.org/abs/2002.07650

Kang, Z., Zhao, X., & Song, D. (2025). Scalable Best-of-N Selection for Large Language Models via Self-Certainty (negative entropy, Eq. 7). arXiv:2502.18581. https://arxiv.org/abs/2502.18581

Parameters:

• topk_logprobs (Any) – (T, k) array-like (or ragged list of rows) of per-position top-\(k\) log-probabilities.

• aggregate (str) – "mean" (default), "max" (most-uncertain token), or "min".

Return type:

float

Returns:

The entropy in nats over the top-\(k\) support; lower is more confident.

Formula:

\[H_i = -\sum_{j} \tilde p_{ij} \log \tilde p_{ij}, \qquad H(y) = \operatorname*{agg}_i H_i,\]

with \(\tilde p_i\) the renormalized top-\(k\) distribution. The exact entropy sums over the full vocabulary; the top-\(k\) value is a truncated lower bound.

Examples

>>> round(token_entropy([[-1.0986, -1.0986, -1.0986]]), 4)  # uniform over 3
1.0986
>>> token_entropy([[0.0, -20.0, -20.0]]) < 1e-6             # near one-hot
True

varentropy(topk_logprobs, *, aggregate='mean')

Varentropy: variance of the surprisal under the top-k next-token distribution.

The probability-weighted variance of the token surprisals \(-\log \tilde p_{ij}\) around the entropy – a second-order dispersion signal that distinguishes a flatly-uncertain distribution (low varentropy) from one that is confident-but-multimodal (high varentropy). Popularized as a decoding confidence signal by the entropix project and used alongside entropy to flag branching / “forking” steps.

References

Kontoyiannis, I., & Verdu, S. (2014). Optimal Lossless Data Compression: Non-Asymptotics and Asymptotics (source varentropy). IEEE Trans. Inf. Theory 60(2).

entropix (xjdr et al., 2024). Entropy-based sampling. https://github.com/xjdr-alt/entropix

Parameters:

• topk_logprobs (Any) – (T, k) array-like (or ragged list of rows) of per-position top-\(k\) log-probabilities.

• aggregate (str) – "mean" (default), "max", or "min".

Return type:

float

Returns:

The varentropy (>= 0) over the top-\(k\) support.

Formula:

\[V\!H_i = \sum_{j} \tilde p_{ij} \big(-\log \tilde p_{ij} - H_i\big)^2 = \Big(\sum_{j} \tilde p_{ij} (\log \tilde p_{ij})^2\Big) - H_i^2, \qquad V\!H(y) = \operatorname*{agg}_i V\!H_i.\]

Examples

>>> round(varentropy([[-1.0986, -1.0986, -1.0986]]), 6)   # uniform -> 0
0.0

max_softmax_probability(topk_logprobs, *, aggregate='mean')

Maximum softmax probability (MSP), averaged over the trace.

The per-token probability of the most likely next token, reduced over the trace – the classic maximum-softmax-probability confidence / out-of- distribution baseline. The top-1 probability is exact (it is present in the top-\(k\)), so no renormalization is needed. With aggregate="min" it reports the least-confident token, a worst-step signal.

References

Hendrycks, D., & Gimpel, K. (2017). A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. ICLR 2017, arXiv:1610.02136. https://arxiv.org/abs/1610.02136

Parameters:

• topk_logprobs (Any) – (T, k) array-like (or ragged list of rows) of per-position top-\(k\) log-probabilities.

• aggregate (str) – "mean" (default), "min" (least-confident token), or "max".

Return type:

float

Returns:

The mean (or min/max) top-1 probability in (0, 1]; higher is more confident.

Formula:

\[\mathrm{MSP}(y) = \operatorname*{agg}_i \max_j p_{ij}.\]

Examples

>>> round(max_softmax_probability([[0.0, -20.0], [-0.6931, -0.6931]]), 4)
0.75

logprob_margin(topk_logprobs, *, use_prob=False, aggregate='mean')

Top1-top2 margin of the next-token distribution, averaged over the trace.

The gap between the most and second-most likely next token at each position, reduced over the trace – the margin-of-confidence / best-versus-second-best uncertainty measure. A large margin means the model had a clear preference at that step. Both ranks are within the top-\(k\), so the margin is exact.

References

Scheffer, T., Decomain, C., & Wrobel, S. (2001). Active Hidden Markov Models for Information Extraction (margin sampling). IDA 2001, LNCS 2189. https://doi.org/10.1007/3-540-44816-0_31

Parameters:

• topk_logprobs (Any) – (T, k) array-like (or ragged list of rows) of per-position top-\(k\) log-probabilities.

• use_prob (bool) – If True, use the probability margin \(p_{(1)} - p_{(2)}\); if False (default), the log-prob margin \(\log p_{(1)} - \log p_{(2)}\).

• aggregate (str) – "mean" (default), "min" (closest-call token), or "max".

Return type:

float

Returns:

The mean (or min/max) top1-top2 margin (>= 0); higher is more confident. Positions with a single candidate contribute 0.

Formula:

\[m_i = \ell_{(1)}(i) - \ell_{(2)}(i), \qquad \mathrm{margin}(y) = \operatorname*{agg}_i m_i,\]

with \(\ell_{(1)}, \ell_{(2)}\) the two largest (log-)probabilities.

Examples

>>> round(logprob_margin([[0.0, -1.0, -2.0], [-0.5, -0.7, -3.0]]), 4)
0.6

picsar(logprobs, *, answer_start=None, normalize_reasoning=False)

PiCSAR: reasoning + answer log-likelihood Best-of-N selector.

Scores a trace by the sum of its reasoning log-likelihood and its answer log-likelihood – the joint confidence in how it reasoned and what it concluded. With answer_start splitting the token stream into a reasoning span y_{<answer_start} and an answer span y_{>=answer_start}, the score is the reasoning log-likelihood (optionally length-normalized, the PiCSAR-N variant) plus the answer log-likelihood. Without a split it reduces to sequence_logprob(), the whole-trace log-likelihood.

References

Leang, J. O. J., Zhao, Z., Gema, A. P., Yang, S., Kwan, W.-C., He, X., Li, W., Minervini, P., Giunchiglia, E., & Cohen, S. B. (2026). Probabilistic Confidence via Sum of Answer and Reasoning Log-Likelihoods (PiCSAR). Findings of ACL 2026, arXiv:2508.21787. https://arxiv.org/abs/2508.21787

Parameters:

• logprobs (Any) – (T,) array-like of chosen-token log-probabilities.

• answer_start (int | None) – Index of the first answer-span token. Tokens before it are the reasoning span; from it onward is the answer span. None (default) treats the whole trace as one span (equals sequence_logprob()).

• normalize_reasoning (bool) – If True (PiCSAR-N), divide the reasoning log-likelihood by its token count before adding the (un-normalized) answer log-likelihood. Ignored when answer_start is None.

Return type:

float

Returns:

The PiCSAR score; higher is more confident.

Formula:

\[\mathrm{PiCSAR}(r, y) = \log p(r \mid x) + \log p(y \mid \langle a\rangle, r, x),\]

with PiCSAR-N replacing \(\log p(r\mid x)\) by \(\tfrac{1}{|r|}\log p(r\mid x)\).

Examples

>>> lp = [-0.1, -0.2, -0.3, -0.4]
>>> round(picsar(lp), 4)                       # whole-trace log-likelihood
-1.0
>>> round(picsar(lp, answer_start=3), 4)       # reasoning(-0.6) + answer(-0.4)
-1.0

Reward-model aggregation

prm_aggregate(step_scores, *, method='last')

Aggregate a trace’s per-step process-reward scores into one trace reward.

A process reward model emits a score \(r_t \in (0, 1)\) for each of the trace’s \(L\) reasoning steps; this reduces those scores to a single reward for reranking or weighted voting. The reduction choice reflects a different notion of trajectory quality:

• "last" – the final step’s score (the solution-level reward; the common default when scaling PRM verification).

• "min" – the worst step (a trace is only as correct as its weakest step; Math-Shepherd, and one of the two aggregations of Lightman et al.).

• "prod" – the product of step scores (joint step correctness; the other Lightman et al. aggregation).

• "mean" – the average step score (overall trajectory quality).

• "max" – the best step (included for completeness).

References

Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., & Cobbe, K. (2023). Let’s Verify Step by Step (PRM; product / min aggregation). ICLR 2024, arXiv:2305.20050. https://arxiv.org/abs/2305.20050

Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., & Sui, Z. (2024). Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (min / product). ACL 2024, arXiv:2312.08935. https://arxiv.org/abs/2312.08935

Zhang, L., Gao, J., Ren, X., Cao, Z., et al. (2025). (Benchmarks the {min, prod, mean, last, max} PRM aggregations.) arXiv:2508.01682. https://arxiv.org/abs/2508.01682

Parameters:

• step_scores (Any) – (L,) array-like of per-step reward-model scores for one trace (typically in (0, 1); higher is better). Must be non-empty and finite.

• method (str) – Reduction to apply – "last" (default), "min", "mean", "prod", or "max".

Return type:

float

Returns:

The aggregated per-trace reward (a scalar). Higher is better; feed it as a scores entry to a selection rule.

Formula:

\[R_{\min} = \min_t r_t, \quad R_{\mathrm{prod}} = \prod_{t=1}^{L} r_t, \quad R_{\mathrm{mean}} = \tfrac{1}{L}\sum_{t=1}^{L} r_t, \quad R_{\mathrm{last}} = r_L, \quad R_{\max} = \max_t r_t.\]

Examples

>>> prm_aggregate([0.9, 0.8, 0.95], method="last")
0.95
>>> prm_aggregate([0.9, 0.4, 0.95], method="min")
0.4
>>> round(prm_aggregate([0.9, 0.8, 0.95], method="mean"), 4)
0.8833
>>> import numpy as np
>>> # Build the (M, N) scores matrix for a batch of PRM step-score traces:
>>> traces = [[[0.9, 0.8], [0.7, 0.6]], [[0.5, 0.9], [0.95, 0.99]]]
>>> scores = np.array([[prm_aggregate(t, method="min") for t in q]
...                    for q in traces])
>>> scores.tolist()
[[0.8, 0.6], [0.5, 0.95]]

Reward-based selection

best_of_n(answers, scores, *, return_index=False, return_score=False)

Best-of-N: return the answer of the highest-scoring candidate.

Samples N candidates per question, scores each with a reward model or verifier, and keeps the answer of the single highest-scoring one. This is the classic verifier reranking / best-of-\(n\) selection used to trade inference compute for accuracy.

References

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168. https://arxiv.org/abs/2110.14168

Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020). Learning to Summarize from Human Feedback. NeurIPS 2020, arXiv:2009.01325.

Parameters:

• answers (Any) – (N,) or (M, N) array-like of answer labels. Candidates whose answer is unparsable (None, "", NaN) are not eligible to be selected.

• scores (Any) – (N,) or (M, N) array-like of per-candidate scores (higher is better), same shape as answers.

• return_index (bool) – If True, also return the selected candidate index per question (-1 if no candidate has a valid answer).

• return_score (bool) – If True, also return the score of the selected candidate – for Best-of-N, the maximizing (argmax) score (NaN if no candidate has a valid answer).

Return type:

Any

Returns:

The selected answer per question (scalar or (M,) object array). When return_index and/or return_score are set, a tuple (selected[, index][, score]) is returned in that order.

Formula:

\[i^\star_\alpha = \operatorname*{arg\,max}_i S_{\alpha i}, \qquad \hat z_\alpha = Z_{\alpha, i^\star_\alpha},\]

with ties in the score broken by lowest index.

Examples

>>> answers = ["A", "B", "A", "C", "A"]
>>> scores = [0.1, 0.9, 0.2, 0.3, 0.15]
>>> best_of_n(answers, scores)
'B'
>>> best_of_n(answers, scores, return_index=True)
('B', 1)
>>> best_of_n(answers, scores, return_score=True)
('B', 0.9)
>>> best_of_n(answers, scores, return_index=True, return_score=True)
('B', 1, 0.9)

majority_of_the_bests(answers, scores, *, m=None, return_index=False, return_score=False)

Majority-of-the-Bests (MoB): the mode of the bootstrapped Best-of-N answer.

Best-of-N returns a single draw from a noisy answer distribution induced by an imperfect reward model. MoB instead estimates that distribution by bootstrapping – repeatedly resample \(m\) candidates with replacement, apply Best-of-N to each resample, and record the resulting answer – and returns its mode. Because the correct answer, while rarely near certainty under a flawed reward, is often the single most likely Best-of-N outcome, taking the mode improves over vanilla Best-of-N.

This implementation uses the closed-form (\(B\to\infty\)) bootstrap distribution from the paper rather than Monte-Carlo resampling, so the result is exact and deterministic. Sorting candidates by descending score, the candidate at reward-rank \(t\) (0-indexed) is the Best-of-N winner of a size-\(m\) with-replacement resample with probability \(\left(\tfrac{n-t}{n}\right)^m - \left(\tfrac{n-t-1}{n}\right)^m\); these weights are summed within each answer group and the largest group is returned.

References

Rakhsha, A., Madan, K., Zhang, T., Farahmand, A.-m., & Khasahmadi, A. (2025). Majority of the Bests: Improving Best-of-N via Bootstrapping. arXiv:2511.18630. https://arxiv.org/abs/2511.18630

Parameters:

• answers (Any) – (N,) or (M, N) array-like of answer labels. Unparsable entries (None, "", NaN) are excluded from resampling.

• scores (Any) – (N,) or (M, N) array-like of per-candidate scores (higher is better), same shape as answers.

• m (int | None) – Bootstrap resample size. Defaults to \(\lfloor\sqrt{n}\rfloor\) over the n valid candidates, as recommended by the paper; clamped to [1, n].

• return_index (bool) – If True, also return the highest-scoring member of the winning answer group per question (-1 if none are valid).

• return_score (bool) – If True, also return the score of that representative candidate – the best reward within the winning answer group (NaN if none are valid).

Return type:

Any

Returns:

The selected answer per question (scalar or (M,) object array). When return_index and/or return_score are set, a tuple (selected[, index][, score]) is returned in that order.

Formula:

Let candidates be ranked by score, \(z_{(t)}\) the answer at rank \(t\), and

\[w_t = \Big(\tfrac{n-t}{n}\Big)^{m} - \Big(\tfrac{n-t-1}{n}\Big)^{m}, \qquad t = 0,\ldots,n-1,\]

the probability that rank \(t\) wins Best-of-N on a size-\(m\) with-replacement resample (\(\sum_t w_t = 1\)). MoB returns

\[\hat z_\alpha = \operatorname*{arg\,max}_z \sum_{t:\, z_{(t)} = z} w_t .\]

Examples

>>> answers = ["A", "A", "A", "B", "B", "C"]
>>> scores = [0.5, 0.55, 0.6, 0.9, 0.4, 0.3]
>>> best_of_n(answers, scores)             # fooled by the lone high-reward B
'B'
>>> majority_of_the_bests(answers, scores)  # robust: bootstrap mode is A
'A'

mob is an alias for majority_of_the_bests.

best_of_majority(answers, scores, *, alpha=0.0, aggregate='mean', return_index=False, return_score=False)

Best-of-Majority (BoM): highest-reward answer among the frequent ones.

Best-of-N is fooled by a single high-reward outlier; majority vote ignores reward quality. Best-of-Majority interpolates: it first keeps only answers whose empirical frequency is at least alpha (a frequency gate that discards rarely-produced answers a reward model is most likely to over-score), then, among those survivors, returns the one with the highest aggregated reward. This frequency-gated reward selection is minimax-optimal for Pass@k inference scaling; here it is used at k = 1 to return a single answer.

References

Di, Q., Ji, K., Li, X., Zhao, H., & Gu, Q. (2025). Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scaling. arXiv:2510.03199 (Algorithm 3). https://arxiv.org/abs/2510.03199

Parameters:

• answers (Any) – (N,) or (M, N) array-like of answer labels. Unparsable entries (None, "", NaN) are ignored.

• scores (Any) – (N,) or (M, N) array-like of per-candidate scores (higher is better), same shape as answers.

• alpha (float) – Minimum empirical frequency \(\alpha \in [0, 1]\) an answer must reach to be eligible. 0 (default) keeps every answer, matching weighted_majority_vote() when aggregate is "sum" or "mean". With aggregate="max", it selects the answer group whose best member has the highest score. A small positive value (e.g. enough to drop single-occurrence answers) enables the intended frequency gate. If the gate would empty the pool it is relaxed to keep all valid answers.

• aggregate (str) – Per-answer reward pooling: "mean" (default, the paper’s instantiation), "sum", or "max".

• return_index (bool) – If True, also return the highest-scoring member of the winning answer group per question (-1 if none are valid).

• return_score (bool) – If True, also return the score of that representative candidate (NaN if none are valid).

Return type:

Any

Returns:

The selected answer per question (scalar or (M,) object array). When return_index and/or return_score are set, a tuple (selected[, index][, score]) is returned in that order.

Formula:

For question \(q\), let the empirical frequency be \(\pi_q(z) = \tfrac{1}{n}\sum_i \mathbb 1[Z_{q i} = z]\), the gated set be \(\mathcal E_q(\alpha) = \{z : \pi_q(z) \ge \alpha\}\), and \(\hat r_q(z)\) be the mean, sum, or maximum of the group’s scores.

\[\hat z_q = \operatorname*{arg\,max}_{z \in \mathcal E_q(\alpha)} \hat r_q(z),\]

with ties broken by earliest appearance.

Examples

>>> answers = ["A", "A", "A", "B"]
>>> scores = [0.5, 0.55, 0.6, 0.99]
>>> best_of_n(answers, scores)                       # lone high-reward B
'B'
>>> best_of_majority(answers, scores, alpha=0.5)     # B is too rare -> A
'A'

Vote-based aggregation

Every voting rule groups candidates by answer equality and returns the winning group’s answer; they differ only in how each candidate’s vote is weighted and which candidates are allowed to vote. Several are strict generalizations of the basic rules: softmax_weighted_vote and rank_weighted_vote each expose a single knob that continuously bridges majority_vote() and best_of_n(), and filtered_vote recovers majority_vote(), weighted_majority_vote(), and best_of_n() at the limits of its keep argument.

majority_vote(answers, *, return_index=False)

Majority vote (self-consistency) over a candidate pool.

Selects the answer that appears most frequently among the sampled candidates for each question. This is the self-consistency decoding rule: sample several chain-of-thought completions, marginalize over the reasoning paths, and keep the most common final answer.

References

Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023, arXiv:2203.11171. https://arxiv.org/abs/2203.11171

Parameters:

• answers (Any) – (N,) or (M, N) array-like of answer labels. Entries that are None, "" or NaN are treated as unparsable and ignored. Answers are compared by equality, so any hashable label (string, int, canonicalized expression) works.

• return_index (bool) – If True, also return a representative candidate index per question (the first occurrence of the winning answer).

Return type:

Any

Returns:

The selected answer for each question. This is a scalar for (N,) input or an (M,) object array for (M, N) input. With return_index, a pair (selected, index) is returned, where index is -1 for a question whose candidates are all unparsable.

Formula:

Let \(Z_{\alpha\cdot}\) be the answers for question \(\alpha\) and \(c_\alpha(z) = \sum_i \mathbb{1}[Z_{\alpha i} = z]\). Then

\[\hat z_\alpha = \operatorname*{arg\,max}_{z} c_\alpha(z),\]

with ties broken by earliest appearance in the sample order.

Examples

>>> answers = ["A", "B", "A", "C", "A"]
>>> majority_vote(answers)
'A'
>>> majority_vote(answers, return_index=True)
('A', 0)
>>> batch = [["A", "B", "A"], ["X", "X", "Y"]]
>>> majority_vote(batch).tolist()
['A', 'X']

weighted_majority_vote(answers, scores, *, aggregate='sum', return_index=False, return_score=False)

Verifier-weighted majority vote over a candidate pool.

Groups candidates by answer and selects the group with the largest aggregated score. With aggregate="sum" each answer is weighted by the total score of its candidates (so agreement and quality both help); with aggregate="mean" answers are ranked by average candidate quality. When the scores are a verifier’s correctness probabilities this is the weighted self-consistency of Li et al. (2023). With aggregate="sum", unit weights recover plain majority_vote(). With aggregate="mean", unit weights give every nonempty answer group the same score, so the tie-break rule decides the result.

References

Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W. (2023). Making Language Models Better Reasoners with Step-Aware Verifier. ACL 2023, arXiv:2206.02336. https://arxiv.org/abs/2206.02336

Parameters:

• answers (Any) – (N,) or (M, N) array-like of answer labels. Unparsable entries (None, "", NaN) are ignored.

• scores (Any) – (N,) or (M, N) array-like of per-candidate scores (reward-model / verifier scores; higher is better), same shape as answers.

• aggregate (str) – "sum" (default) or "mean" group aggregation.

• return_index (bool) – If True, also return the index of the highest-scoring member of the winning group per question.

• return_score (bool) – If True, also return the score of that representative candidate – the best reward within the winning group (NaN if no candidate has a valid answer). Note this is the representative’s own score, not the aggregated group weight that decided the vote.

Return type:

Any

Returns:

The selected answer per question (scalar or (M,) object array). When return_index and/or return_score are set, a tuple (selected[, index][, score]) is returned in that order.

Formula:

With group score \(W_\alpha(z)\) the answer is

\[W_\alpha(z) = \sum_{i:\, Z_{\alpha i} = z} S_{\alpha i} \quad(\text{sum}), \qquad W_\alpha(z) = \frac{\sum_{i:\, Z_{\alpha i}=z} S_{\alpha i}} {\sum_i \mathbb{1}[Z_{\alpha i}=z]} \quad(\text{mean}),\]

and \(\hat z_\alpha = \arg\max_z W_\alpha(z)\), ties broken by earliest appearance.

Examples

>>> answers = ["A", "A", "A", "B"]
>>> scores = [0.3, 0.3, 0.3, 0.85]
>>> weighted_majority_vote(answers, scores)          # sum: A=0.9 > B=0.85
'A'
>>> weighted_majority_vote(answers, scores, aggregate="mean")  # mean: A=0.3 < B=0.85
'B'
>>> weighted_majority_vote(answers, scores, return_score=True)  # best A-member reward
('A', 0.3)

softmax_weighted_vote(answers, scores, *, temperature=1.0, return_index=False, return_score=False)

Temperature-softmax-weighted majority vote (CISC).

A weighted vote in which each candidate’s weight is a softmax of its score, \(\exp(S_i / T)\), rather than the raw score. The single temperature \(T > 0\) continuously bridges the three basic rules: \(T \to \infty\) flattens the weights to majority vote, while \(T \to 0\) concentrates all weight on the highest-scoring candidate, recovering Best-of-N (exactly so when the top score is unique). This is the Confidence-Informed Self-Consistency (CISC) rule, where the score is the model’s own confidence in each response.

References

Taubenfeld, A., Sheffer, T., Ofek, E., Feder, A., Goldstein, A., Gekhman, Z., & Yona, G. (2025). Confidence Improves Self-Consistency in LLMs. Findings of ACL 2025, arXiv:2502.06233 (Definition 3.1). https://arxiv.org/abs/2502.06233

Parameters:

• answers (Any) – (N,) or (M, N) array-like of answer labels. Unparsable entries (None, "", NaN) are ignored.

• scores (Any) – (N,) or (M, N) array-like of per-candidate scores (higher is better), same shape as answers.

• temperature (float) – Softmax temperature \(T > 0\). Smaller sharpens toward Best-of-N; larger flattens toward majority vote (float('inf') gives exactly majority vote).

• return_index (bool) – If True, also return the highest-scoring member of the winning answer group per question (-1 if none are valid).

• return_score (bool) – If True, also return the score of that representative candidate (NaN if none are valid).

Return type:

Any

Returns:

The selected answer per question (scalar or (M,) object array). When return_index and/or return_score are set, a tuple (selected[, index][, score]) is returned in that order.

Formula:

The paper’s softmax weights \(\tilde c_i = \exp(S_{\alpha i}/T) / \sum_j \exp(S_{\alpha j}/T)\) share a per-question denominator that cancels in the argmax, so the selection is equivalently

\[\hat z_\alpha = \operatorname*{arg\,max}_z \sum_{i:\, Z_{\alpha i} = z} \exp\!\big(S_{\alpha i} / T\big),\]

with ties broken by earliest appearance. (Implemented with the standard exp((S - max S) / T) shift for numerical stability, which leaves the argmax unchanged.)

Examples

>>> answers = ["A", "A", "B"]
>>> scores = [0.1, 0.2, 0.9]
>>> softmax_weighted_vote(answers, scores, temperature=0.1)  # sharp -> Best-of-N
'B'
>>> softmax_weighted_vote(answers, scores, temperature=float("inf"))  # -> majority
'A'

rank_weighted_vote(answers, scores, *, p=1.0, return_index=False, return_score=False)

Rank-weighted (Borda) majority vote.

A weighted vote that uses only the ordinal rank of the scores, not their magnitudes. Candidates are ranked by score (best first) and the rank-t candidate (t = 0 is the best) casts a vote of weight \((n - t)^p\). Because it depends only on the score ordering, the rule is invariant to any monotone rescaling of the scores – a robustness the magnitude-based weighted_majority_vote() does not have. The exponent \(p \ge 0\) interpolates between majority vote (p = 0, all weights equal) and Best-of-N (p \to \infty, the top candidate dominates); p = 1 is the standard Borda count. Introduced as self-certainty Borda voting.

References

Kang, Z., Zhao, X., & Song, D. (2025). Scalable Best-of-N Selection for Large Language Models via Self-Certainty. NeurIPS 2025, arXiv:2502.18581. https://arxiv.org/abs/2502.18581

Parameters:

• answers (Any) – (N,) or (M, N) array-like of answer labels. Unparsable entries (None, "", NaN) are ignored.

• scores (Any) – (N,) or (M, N) array-like of per-candidate scores (higher is better), same shape as answers.

• p (float) – Non-negative Borda exponent. 0 recovers majority vote; 1 is standard Borda; larger values sharpen toward Best-of-N.

• return_index (bool) – If True, also return the highest-scoring member of the winning answer group per question (-1 if none are valid).

• return_score (bool) – If True, also return the score of that representative candidate (NaN if none are valid).

Return type:

Any

Returns:

The selected answer per question (scalar or (M,) object array). When return_index and/or return_score are set, a tuple (selected[, index][, score]) is returned in that order.

Formula:

Ranking the \(n\) valid candidates for question \(\alpha\) by descending score (ties by lowest index) and letting \(r_{\alpha i} \in \{1,\ldots,n\}\) be candidate \(i\)’s rank,

\[\hat z_\alpha = \operatorname*{arg\,max}_z \sum_{i:\, Z_{\alpha i} = z} (n - r_{\alpha i} + 1)^{p},\]

with ties broken by earliest appearance.

Examples

>>> answers = ["A", "A", "B"]
>>> scores = [0.30, 0.31, 0.99]
>>> rank_weighted_vote(answers, scores, p=1.0)   # Borda: A gets 2+1, B gets 3
'A'
>>> rank_weighted_vote(answers, scores, p=0.0)   # -> majority vote
'A'

logit_weighted_vote(answers, scores, *, threshold=0.5, transform='logit', return_index=False, return_score=False)

Threshold-shifted log-odds weighted majority vote.

The Bayes-optimal aggregation of a generator’s samples with a verifier’s scores casts each candidate’s vote with a weight that is a log-likelihood ratio of its score, approximated in closed form by a threshold-shifted transform. With transform="logit" the weight is \(\operatorname{logit}(S_i) - \operatorname{logit}(b)\) (scores must be probabilities in \((0, 1)\)); with transform="linear" it is the simpler \(S_i - b\). Unlike weighted_majority_vote(), weights can be negative: candidates scoring below the threshold \(b\) vote against their own answer, so an answer supported only by low-quality candidates can be beaten by a rarer but higher-quality one.

References

Kuang, P., Wang, Y., Han, X., Liu, Y., Xu, K., & Wang, H. (2025). Optimal Aggregation of LLM and PRM Signals for Efficient Test-Time Scaling. ICLR 2026, arXiv:2510.13918 (Theorem 3.2). https://arxiv.org/abs/2510.13918

Parameters:

• answers (Any) – (N,) or (M, N) array-like of answer labels. Unparsable entries (None, "", NaN) are ignored.

• scores (Any) – (N,) or (M, N) array-like of per-candidate scores. For transform="logit" every valid score must lie strictly in (0, 1) (a verifier probability); for transform="linear" any real score is allowed.

• threshold (float) – The calibrated decision threshold \(b\). For transform="logit" it must lie in (0, 1) (default 0.5, i.e. a zero log-odds shift); for transform="linear" any real value (0.0 recovers weighted_majority_vote() with aggregate="sum").

• transform (str) – "logit" (log-odds; default) or "linear".

• return_index (bool) – If True, also return the highest-scoring member of the winning answer group per question (-1 if none are valid). Note the winning group may have a negative total weight.

• return_score (bool) – If True, also return the score of that representative candidate (NaN if none are valid).

Return type:

Any

Returns:

The selected answer per question (scalar or (M,) object array). When return_index and/or return_score are set, a tuple (selected[, index][, score]) is returned in that order.

Formula:

With per-candidate weight \(w_{\alpha i} = \operatorname{logit}(S_{\alpha i}) - \operatorname{logit}(b)\) (or \(S_{\alpha i} - b\) for the linear transform),

\[\hat z_\alpha = \operatorname*{arg\,max}_z \sum_{i:\, Z_{\alpha i} = z} w_{\alpha i},\]

with ties broken by earliest appearance. This is the paper’s practical parametric instantiation: the per-response LLM-reliability offset of the full Theorem 3.2 weight is folded into the calibrated threshold b (the zero-crossing of the log-odds weight), so only b is exposed here. The nonparametric log-density-ratio weight is left to a caller who has a calibration set.

Examples

>>> answers = ["A", "A", "B"]
>>> scores = [0.4, 0.4, 0.9]     # both A-candidates below b=0.5
>>> logit_weighted_vote(answers, scores)          # A votes negative -> B wins
'B'
>>> from scorio.aggregate import weighted_majority_vote
>>> a, s = ["A", "A", "B"], [0.4, 0.4, 0.9]
>>> logit_weighted_vote(a, s, transform="linear", threshold=0.0) == \
...     weighted_majority_vote(a, s)              # linear, b=0 -> weighted sum
True

filtered_vote(answers, scores, *, keep=0.5, weighted=True, return_index=False, return_score=False)

Confidence-filtered majority vote (DeepConf / top-K verifier-ranked vote).

Drops the lowest-scoring candidates before voting, keeping only the top-keep by score, then takes a (weighted) majority vote over the survivors. Filtering out low-quality candidates concentrates the vote on the model’s most confident responses; the retained set then votes with weight equal to the score (weighted=True, DeepConf’s confidence-weighted vote) or with unit weights (weighted=False, the classic top-K verifier-ranked majority vote of Cobbe et al.).

References

Fu, Y., Wang, X., Tian, Y., & Zhao, J. (2025). Deep Think with Confidence. arXiv:2508.15260. https://arxiv.org/abs/2508.15260

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J. (2021). Training Verifiers to Solve Math Word Problems (Sec. 5.1, top-K verifier voting). arXiv:2110.14168.

Parameters:

• answers (Any) – (N,) or (M, N) array-like of answer labels. Unparsable entries (None, "", NaN) are ignored.

• scores (Any) – (N,) or (M, N) array-like of per-candidate scores (higher is better), same shape as answers.

• keep (float) – How many top-scoring candidates to retain per question. A float in (0, 1] is a fraction of the valid candidates (default 0.5, at least one is always kept); an integer >= 1 is an explicit count. keep=1.0 keeps everyone (no filtering); keep=1 (integer) keeps only the single best, recovering Best-of-N.

• weighted (bool) – If True (default), survivors vote with weight equal to their score (DeepConf); if False, with unit weights (top-K majority vote).

• return_index (bool) – If True, also return the highest-scoring member of the winning answer group per question (-1 if none are valid).

• return_score (bool) – If True, also return the score of that representative candidate (NaN if none are valid).

Return type:

Any

Returns:

The selected answer per question (scalar or (M,) object array). When return_index and/or return_score are set, a tuple (selected[, index][, score]) is returned in that order.

Formula:

Let \(\mathcal T_\alpha\) be the k highest-scoring valid candidates for question \(\alpha\) (k from keep; score ties broken by lowest index). Then

\[\begin{split}\hat z_\alpha = \operatorname*{arg\,max}_z \sum_{i \in \mathcal T_\alpha:\, Z_{\alpha i} = z} w_{\alpha i}, \qquad w_{\alpha i} = \begin{cases} S_{\alpha i} & \text{weighted} \\ 1 & \text{unweighted,} \end{cases}\end{split}\]

with ties broken by earliest appearance.

Examples

>>> answers = ["A", "A", "A", "B", "B", "B"]
>>> scores = [0.1, 0.2, 0.3, 0.8, 0.85, 0.9]
>>> filtered_vote(answers, scores, keep=0.5)   # top 3 are all B
'B'
>>> filtered_vote(answers, scores, keep=1.0, weighted=False)  # keep all -> majority tie, first
'A'

Online early stopping

These rules turn a fixed-budget aggregator into an adaptive one. The sampling-level rules watch the stream of extracted answers and report when to stop drawing traces; the trace-level DeepConf rules report the token at which a trace’s running confidence falls below a warmup-calibrated threshold.

adaptive_consistency_stop(answers, *, threshold=0.95, return_prob=False)

Adaptive-Consistency stopping: stop sampling once the majority is decided.

Self-consistency draws a fixed number of samples; Adaptive-Consistency stops early, after each new sample, as soon as the currently leading answer is statistically unlikely to be overtaken. Placing a Dirichlet prior on the answer probabilities and reducing to the top two answers, it stops when the posterior probability that the leader’s true probability exceeds the runner-up’s clears threshold. Easy questions (an early, dominant majority) stop in a few samples; hard ones keep sampling to the budget.

References

Aggarwal, P., Madaan, A., Yang, Y., & Mausam. (2023). Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs. EMNLP 2023, arXiv:2305.11860. https://arxiv.org/abs/2305.11860

Parameters:

• answers (Any) – The sequence of extracted answers sampled so far (call after each new sample). Unparsable entries (None, "", NaN) are ignored. Answers are compared by equality.

• threshold (float) – Posterior-probability threshold \(C \in (0, 1)\) to stop (default 0.95); higher samples more before stopping.

• return_prob (bool) – If True, return (stop, prob) with the current posterior probability that the leader stays on top.

Return type:

Any

Returns:

True if sampling should stop now, else False (a (stop, prob) tuple when return_prob). Always False until at least one valid answer has been seen.

Formula:

With the two largest answer counts \(v_1 \ge v_2\) and a uniform Dirichlet prior, the top-two posterior is \(p \sim \mathrm{Beta}(v_1 + 1, v_2 + 1)\) and the rule stops when

\[\Pr(p > \tfrac12 \mid v_1, v_2) = 1 - I_{1/2}(v_1 + 1,\, v_2 + 1) \ge C,\]

with \(I\) the regularized incomplete beta function.

Examples

>>> adaptive_consistency_stop(["A"] * 8 + ["B"] * 2)   # 8 vs 2 -> decided
True
>>> adaptive_consistency_stop(["A", "A", "B", "B"])    # tie -> keep going
False
>>> stop, p = adaptive_consistency_stop(["A"] * 5, return_prob=True)
>>> stop, round(p, 4)
(True, 0.9844)

esc_stop(window_answers)

Early-Stopping Self-Consistency: stop when a whole sampling window agrees.

The simplest, hyperparameter-light early-stopping rule: draw samples in fixed windows and stop the moment one window is unanimous (zero answer entropy). A fully-agreeing window is strong evidence the question is easy and further sampling is wasted; if no window agrees, sampling continues to the budget.

References

Li, Y., Yuan, P., Feng, S., Pan, B., Wang, X., Sun, B., Wang, H., & Li, K. (2024). Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning. ICLR 2024, arXiv:2401.10480. https://arxiv.org/abs/2401.10480

Parameters:

window_answers (Any) – The extracted answers of the most recent sampling window. Answers are compared by equality; a window containing an unparsable entry (None, "", NaN) is not unanimous.

Return type:

bool

Returns:

True if every answer in the window is valid and identical (so sampling should stop), else False. An empty window returns False.

Formula:

For window answers \(W\), stop iff the answer entropy \(H(W) = 0\), i.e. all \(|W|\) answers are equal.

Examples

>>> esc_stop(["A", "A", "A"])
True
>>> esc_stop(["A", "B", "A"])
False
>>> esc_stop(["A", None, "A"])
False

deepconf_stop_threshold(warmup_confidences, *, keep=0.1)

DeepConf online stopping threshold from warmup trace confidences.

DeepConf-online first generates a small batch of warmup traces, scores each with a group-confidence measure (e.g. lowest-group confidence from deepconf_confidence()), and sets a stopping threshold s so that only the most-confident fraction keep of traces would survive. During the run, a trace whose running confidence drops below s is terminated (deepconf_online_stop()).

References

Fu, Y., Wang, X., Tian, Y., & Zhao, J. (2025). Deep Think with Confidence. arXiv:2508.15260. https://arxiv.org/abs/2508.15260

Parameters:

• warmup_confidences (Any) – Array-like of per-trace confidence scores from the warmup traces (higher = more confident).

• keep (float) – Fraction of most-confident traces to keep (default 0.10, “DeepConf-low”; 0.90 is the more permissive “DeepConf-high”). The threshold is the 1 - keep quantile of the warmup confidences.

Return type:

float

Returns:

The stopping threshold s (a scalar).

Formula:

\[s = Q_{1 - \mathrm{keep}}\big(\{C(y) : y \in \text{warmup}\}\big).\]

Examples

>>> deepconf_stop_threshold([1.0, 2.0, 3.0, 4.0, 5.0], keep=0.2)
4.2

deepconf_online_stop(topk_logprobs, threshold, *, window=2048)

DeepConf online early termination: the token where a trace should be cut.

Scans a trace’s DeepConf group confidence (sliding-window mean of the per-token confidences) and reports the first token at which a completed window’s confidence falls below threshold – the point at which DeepConf-online would stop generating this trace and discard it. Applied to an already-generated trace it emulates the online decision offline (true mid-generation termination needs generation-time streaming control, which lives outside this library).

References

Fu, Y., Wang, X., Tian, Y., & Zhao, J. (2025). Deep Think with Confidence. arXiv:2508.15260. https://arxiv.org/abs/2508.15260

Parameters:

• topk_logprobs (Any) – (T, k) array-like (or ragged list of rows) of per-position top-\(k\) log-probabilities for one trace.

• threshold (float) – Stopping threshold s (from deepconf_stop_threshold() over warmup traces). A window confidence < s triggers termination.

• window (int) – Sliding-window (group) size in tokens (paper default 2048); clamped to the trace length.

Return type:

int | None

Returns:

The 0-based token index of the last token generated before termination (the end of the first below-threshold window), or None if no window drops below threshold (the trace runs to completion and is kept).

Formula:

With group confidences \(C_{G_s}\) over windows ending at token \(s\), the termination token is

\[\min\{ s : C_{G_s} < \text{threshold} \},\]

or \(\varnothing\) (None) if no such window exists.

Examples

>>> tk = [[0.0, -2.0]] * 3 + [[-4.0, -6.0]] * 3    # conf 1,1,1,5,5,5
>>> deepconf_online_stop(tk, threshold=2.0, window=3) is None  # all >= 2? no
False
>>> # first 3-window mean is 1.0 (< 2.0) ending at token index 2
>>> deepconf_online_stop(tk, threshold=2.0, window=3)
2
>>> deepconf_online_stop(tk, threshold=0.5, window=3) is None  # never < 0.5
True