Skip to content

graphnetz.benchmark

The benchmark layer: a curated task catalogue, a model registry, the runner that executes every compatible (task, model, seed) triple, and the report those runs return.

The report is the return type

run_benchmark does not return an accuracy table. It returns a BenchmarkReport whose methods are the statistical layer: per-cell intervals, corrected pairwise tests, the quantities that say when those tests are uninformative, and rank aggregation across tasks. See Reading the report.

Running a benchmark

benchmark

Statistically robust benchmarks across a category for one or many models.

The dispatcher trains every compatible (model, task) pair across multiple seeds and returns a BenchmarkReport that exposes mean ± 95 % t-CI, paired t-tests with Holm-Bonferroni correction, publication-ready LaTeX tables, and plots.

Custom models are plugged in via the same three paths as before:

  1. Decorator / registry::

    from graphnetz import register_model

    @register_model(task_type="node_cls") class MyGNN(torch.nn.Module): def init(self, in_channels, hidden_channels, out_channels): ...

  2. Class attribute::

    class MyGNN(torch.nn.Module): task_types = {"node_cls"}

  3. Inline tuple (cls, tasks) or (cls, tasks, factory) in the models mapping::

    run_benchmark("social", {"MyGNN": (MyGNN, "node_cls")})

The default factory calls cls(in_channels, hidden_channels, out_channels); DGI-task models receive (in_channels, hidden_channels) (the third argument is dropped).

Functions:

Name Description
run_benchmark

Run a benchmark across one or more (model, task, seed) combinations.

SearchSpace dataclass 

SearchSpace(
    lr: Sequence[float] = (),
    weight_decay: Sequence[float] = (),
    hidden_channels: Sequence[int] = (),
    n_random: int | None = None,
    seed: int = 0,
    _axes: tuple[str, ...] = ("lr", "weight_decay", "hidden_channels"),
)

Hyperparameters to search over, and how to enumerate them.

Only knobs the training routines already expose are searchable: lr and weight_decay (the optimiser) and hidden_channels (the encoder width). An empty space means "no search", so SearchSpace() reproduces the unsearched protocol exactly.

n_random switches from an exhaustive grid to that many random draws (without replacement where the grid is smaller), which is the better use of a fixed budget once the grid has more than two or three axes (Bergstra & Bengio, 2012).

Methods:

Name Description
candidates

Enumerate the candidate configurations, always at least one.

candidates 

candidates() -> list[dict[str, Any]]

Enumerate the candidate configurations, always at least one.

Source code in src/graphnetz/benchmark/_search.py 
59
60
61
62
63
64
65
66
67
68
69
70
71
def candidates(self) -> list[dict[str, Any]]:
    """Enumerate the candidate configurations, always at least one."""
    axes = {name: tuple(getattr(self, name)) for name in self._axes}
    active = {k: v for k, v in axes.items() if v}
    if not active:
        return [{}]
    keys = list(active)
    grid = [dict(zip(keys, combo, strict=False)) for combo in itertools.product(*(active[k] for k in keys))]
    if self.n_random is None or self.n_random >= len(grid):
        return grid
    rng = np.random.default_rng(self.seed)
    idx = rng.choice(len(grid), size=self.n_random, replace=False)
    return [grid[int(i)] for i in sorted(idx)]

run_benchmark 

run_benchmark(
    category: str | None = None,
    models: type
    | tuple[Any, ...]
    | ModelSpec
    | dict[str, type | tuple[Any, ...] | ModelSpec]
    | None = None,
    root: str = "data/benchmark",
    hidden_channels: int = 64,
    epochs: int | None = None,
    only: list[str] | None = None,
    verbose: bool = True,
    seeds: int | Iterable[int] | None = None,
    seed: int | None = None,
    task_type: str | None = None,
    tasks: Iterable[Task] | None = None,
    device: device | str | None = "auto",
    search: SearchSpace | None = None,
) -> BenchmarkReport

Run a benchmark across one or more (model, task, seed) combinations.

Two ways to choose tasks:

  1. By category (default) -- tasks come from BENCHMARK_TASKS indexed as [category][task_type] -> list[Task]. Pass category="social" (etc.) and optionally restrict with task_type and only=.
  2. Ad-hoc -- pass tasks=[Task(...), ...] to bypass the registry entirely. Useful for benchmarking custom datasets without mutating global state. category then defaults to "custom" and is used only to namespace root/ cache directories.

The runner trains every compatible (model, task) pair across each value in seeds (default (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)) and aggregates the per-seed histories into a BenchmarkReport.

search optionally adds an inner hyperparameter loop (Algorithm 1, line 4a): for every (task, model, seed) the runner trains each candidate in the SearchSpace, scores them on the task's validation series, and keeps the winner's history. Candidates are never scored on a held-out metric. The selected configuration and the full search trace land in report.config["search_selected"], and the cost multiplier is announced before training starts, because a grid of G candidates multiplies compute by G.

Source code in src/graphnetz/benchmark/_runner.py 
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
def run_benchmark(
    category: str | None = None,
    models: type | tuple[Any, ...] | ModelSpec | dict[str, type | tuple[Any, ...] | ModelSpec] | None = None,
    root: str = "data/benchmark",
    hidden_channels: int = 64,
    epochs: int | None = None,
    only: list[str] | None = None,
    verbose: bool = True,
    seeds: int | Iterable[int] | None = None,
    seed: int | None = None,
    task_type: str | None = None,
    tasks: Iterable[Task] | None = None,
    device: torch.device | str | None = "auto",
    search: SearchSpace | None = None,
) -> BenchmarkReport:
    """Run a benchmark across one or more (model, task, seed) combinations.

    Two ways to choose tasks:

    1. **By category** (default) -- tasks come from
       [`BENCHMARK_TASKS`][graphnetz.benchmark.BENCHMARK_TASKS] indexed as
       ``[category][task_type] -> list[Task]``. Pass ``category="social"``
       (etc.) and optionally restrict with ``task_type`` and ``only=``.
    2. **Ad-hoc** -- pass ``tasks=[Task(...), ...]`` to bypass the registry
       entirely. Useful for benchmarking custom datasets without mutating
       global state. ``category`` then defaults to ``"custom"`` and is used
       only to namespace ``root/`` cache directories.

    The runner trains every compatible (model, task) pair across each
    value in ``seeds`` (default ``(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)``) and aggregates the per-seed
    histories into a [`BenchmarkReport`][graphnetz.benchmark.BenchmarkReport].

    ``search`` optionally adds an inner hyperparameter loop (Algorithm 1,
    line 4a): for every (task, model, seed) the runner trains each candidate in
    the [`SearchSpace`][graphnetz.benchmark.SearchSpace], scores them on the task's
    **validation** series, and keeps the winner's history. Candidates are never
    scored on a held-out metric. The selected configuration and the full search
    trace land in ``report.config["search_selected"]``, and the cost multiplier
    is announced before training starts, because a grid of ``G`` candidates
    multiplies compute by ``G``.
    """
    if models is None:
        msg = "run_benchmark requires `models` (a class, dict, or ModelSpec)"
        raise ValueError(msg)
    if task_type is not None and task_type not in TASK_TYPES:
        msg = f"Unknown task type {task_type!r}. Choices: {sorted(TASK_TYPES)}"
        raise ValueError(msg)
    if not isinstance(models, dict):
        spec = _spec_from(models)
        models = {spec.cls.__name__: spec}

    resolved = {name: _spec_from(value) for name, value in models.items()}
    seed_list = _normalize_seeds(seeds, seed)

    if tasks is not None:
        task_list = list(tasks)
        for t in task_list:
            if not isinstance(t, Task):
                msg = f"`tasks` must contain Task instances, got {type(t).__name__}"
                raise TypeError(msg)
            if t.task_type not in TASK_TYPES:
                msg = f"Task {t.name!r} has unknown task type {t.task_type!r}; choices: {sorted(TASK_TYPES)}"
                raise ValueError(msg)
        if task_type is not None:
            task_list = [t for t in task_list if t.task_type == task_type]
        if category is None:
            category = "custom"
    else:
        if category is None:
            msg = "run_benchmark requires either `category` or `tasks=`"
            raise ValueError(msg)
        if category not in BENCHMARK_TASKS:
            msg = f"Unknown category {category!r}. Choices: {sorted(BENCHMARK_TASKS)}"
            raise KeyError(msg)
        task_list = iter_benchmark_tasks(category=category, task_type=task_type)
    if only is not None:
        task_list = [t for t in task_list if t.name in only]
    tasks = task_list  # the loop below treats this as the working list

    histories: dict[str, dict[str, list[dict[str, list[float]]]]] = {}
    search_selected: dict[str, Any] = {}
    candidates = search.candidates() if search is not None else [{}]
    if search is not None and not search.is_empty:
        print(f"hyperparameter search: {search.describe()} -- multiplies training cost by {len(candidates)}x")
    total_combinations = (
        sum(1 for spec in resolved.values() for task in tasks if task.task_type in spec.task_type)
        * len(seed_list)
        * len(candidates)
    )
    overall_pbar = tqdm(
        total=total_combinations,
        desc="Benchmark",
        unit="run",
        disable=not verbose,
        bar_format="{l_bar}{bar}| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, {rate_fmt}]",
    )
    import inspect

    for task in tasks:
        try:
            seed_aware = "seed" in inspect.signature(task.loader).parameters
        except (TypeError, ValueError):
            seed_aware = False
        ds_cache: Any = None  # for seed-agnostic loaders, load once
        histories[task.name] = {}
        for model_name, spec in resolved.items():
            if task.task_type not in spec.task_type:
                continue
            histories[task.name][model_name] = []
            for s in seed_list:
                _seed_all(s)
                if seed_aware:
                    # Seed-aware loaders (e.g. synthetic combinatorial graphs)
                    # produce a fresh dataset per seed, so cross-seed variance
                    # captures data resampling rather than only model init.
                    ds = task.loader(f"{root}/{category}/{task.name}/seed{s}", seed=s)
                else:
                    if ds_cache is None:
                        ds_cache = task.loader(f"{root}/{category}/{task.name}")
                    ds = ds_cache
                if len(candidates) == 1:
                    history = _run_task(
                        task,
                        ds,
                        spec,
                        hidden_channels,
                        epochs or task.epochs,
                        verbose=verbose,
                        device=device,
                        hyper=candidates[0],
                    )
                else:
                    # Inner search: reseed before every candidate so the
                    # comparison between them is paired too, then keep the one
                    # that wins on validation.
                    def _trials(t=task, sp=spec, d=ds, s_=s):
                        for cand in candidates:
                            _seed_all(s_)
                            yield (
                                cand,
                                _run_task(
                                    t,
                                    d,
                                    sp,
                                    hidden_channels,
                                    epochs or t.epochs,
                                    verbose=False,
                                    device=device,
                                    hyper=cand,
                                ),
                            )
                            overall_pbar.update(1)

                    best_config, history, trace = select(_trials())
                    search_selected[f"{task.name}/{model_name}/seed{s}"] = {
                        "selected": best_config,
                        "trace": trace,
                    }
                histories[task.name][model_name].append(history)
                # Update overall progress with latest metric
                last_metrics = {k: v[-1] for k, v in history.items() if v}
                metric_str = " ".join(f"{k[:3]}={v:.3f}" for k, v in last_metrics.items())
                overall_pbar.set_postfix_str(f"{task.name}/{model_name}/s{s} | {metric_str}", refresh=False)
                if len(candidates) == 1:
                    overall_pbar.update(1)
    overall_pbar.close()

    from graphnetz.training import _resolve_device

    config = {
        "category": category,
        "task": task,
        "hidden_channels": hidden_channels,
        "epochs": epochs,
        "only": only,
        "device": str(_resolve_device(device)),
        **_provenance(),
    }
    if search is not None and not search.is_empty:
        config["search"] = search.describe()
        config["search_selected"] = search_selected  # type: ignore[assignment]
    return BenchmarkReport(seeds=seed_list, histories=histories, config=config)

The report

The plotting methods (plot, plot_forest, plot_pairwise, plot_critical_difference, plot_learning_curves) are mixed in from a separate module to keep the statistics and the figure code apart; they are part of the public surface and are documented here alongside the rest.

benchmark

Statistically robust benchmarks across a category for one or many models.

The dispatcher trains every compatible (model, task) pair across multiple seeds and returns a BenchmarkReport that exposes mean ± 95 % t-CI, paired t-tests with Holm-Bonferroni correction, publication-ready LaTeX tables, and plots.

Custom models are plugged in via the same three paths as before:

  1. Decorator / registry::

    from graphnetz import register_model

    @register_model(task_type="node_cls") class MyGNN(torch.nn.Module): def init(self, in_channels, hidden_channels, out_channels): ...

  2. Class attribute::

    class MyGNN(torch.nn.Module): task_types = {"node_cls"}

  3. Inline tuple (cls, tasks) or (cls, tasks, factory) in the models mapping::

    run_benchmark("social", {"MyGNN": (MyGNN, "node_cls")})

The default factory calls cls(in_channels, hidden_channels, out_channels); DGI-task models receive (in_channels, hidden_channels) (the third argument is dropped).

BenchmarkReport dataclass 

BenchmarkReport(
    seeds: tuple[int, ...],
    histories: dict[str, dict[str, list[dict[str, list[float]]]]],
    config: dict[str, Any] = dict(),
    ci_method: str = "t",
    bootstrap_n: int = 10000,
    bootstrap_seed: int = 0,
    pairwise_method: str = "t",
    epoch_selection: str = "final",
    SCHEMA: int = 1,
)

Bases: _ReportPlotsMixin

Structured outcome of a multi-seed benchmark run.

histories[task][model] is a list with one history dict per seed (in seed order). The report is also a read-only mapping task -> {model: history_seed_0} for backward compatibility with single-seed callers.

Methods:

Name Description
plot

Grouped bar chart of mean ± CI half-width across seeds.
plot_forest

Forest plot, one row per task with models jittered within the row.
plot_pairwise

Pairwise comparison plot, with two layouts that scale differently.
plot_critical_difference

Demšar critical-difference (CD) diagram.
plot_learning_curves

Mean ± t-CI learning curves, one panel per task, sharing y-axis.
final_metrics

Reduced metric value per (task, model, seed).
epoch_selection_support

Per task: whether "best_val" is available, and why not if it is not.
selected_epochs

The epoch each (task, model, seed) metric was taken from.
summary

Per-(task, model) mean, std, sem, CI half-width and bounds.
pairwise

Paired pairwise tests between models per task with Holm adjustment.
power

Per-comparison minimum detectable effect and observed power.
equivalence

Two one-sided tests for equivalence within margin, plus a verdict.
rank_table

Per-task ranks of the seed means; rows tasks, columns models.
mean_ranks

Mean rank per model across tasks, under a choice of task weighting.
friedman

Friedman omnibus test on per-task ranks of seed-mean metrics.
rank_stability

How much the across-task ranking depends on which tasks were run.
to_json

Serialise the report to JSON: seeds, config, and the raw histories.
from_json

Rebuild a report from to_json output (a path or the text itself).
to_latex

Booktabs LaTeX table of mean ± CI half-width with bold-best per task.
pairwise_to_latex

LaTeX booktabs table of pairwise Holm-adjusted p-values.

Attributes:

Name Type Description
epoch_selection str

How each per-epoch history is reduced to one number.
SCHEMA int

Version of the JSON payload written by to_json.

epoch_selection class-attribute instance-attribute 

epoch_selection: str = 'final'

How each per-epoch history is reduced to one number.

"final" (default) is the fixed-epoch protocol. "best_val" selects the epoch optimising the paired validation series and reports the held-out metric there -- proper checkpoint selection. Setting it once switches the summary, the pairwise tests, the rank aggregation, every plot and the LaTeX exporters together.

SCHEMA class-attribute instance-attribute 

SCHEMA: int = 1

Version of the JSON payload written by to_json.

plot 

plot(
    ax: Axes | None = None,
    *,
    ci: float = 0.95,
    ylabel: str | None = None,
    title: str | None = None,
    annotate: bool = True,
    pretty_tasks: Mapping[str, str] | None = None,
) -> tuple[Figure, Axes]

Grouped bar chart of mean ± CI half-width across seeds.

Source code in src/graphnetz/benchmark/_report_plots.py 
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
def plot(
    self,
    ax: plt.Axes | None = None,
    *,
    ci: float = 0.95,
    ylabel: str | None = None,
    title: str | None = None,
    annotate: bool = True,
    pretty_tasks: Mapping[str, str] | None = None,
) -> tuple[plt.Figure, plt.Axes]:
    """Grouped bar chart of mean ± CI half-width across seeds."""
    finals = self.final_metrics()
    pretty = dict(pretty_tasks or {})
    values: dict[str, dict[str, float]] = {}
    errors: dict[str, dict[str, float]] = {}
    for task, per_task in finals.items():
        label = pretty.get(task, task)
        values[label] = {}
        errors[label] = {}
        for model, vals in per_task.items():
            arr = np.asarray(vals, dtype=float)
            values[label][model] = float(arr.mean())
            errors[label][model] = self._ci_half(arr, ci)
    from graphnetz.plotting import pretty_metric

    return plot_grouped_bars(
        values,
        errors=errors,
        ax=ax,
        title=title,
        ylabel=ylabel or pretty_metric(self.metric_name()),
        annotate=annotate,
    )

plot_forest 

plot_forest(
    ax: Axes | None = None,
    *,
    ci: float = 0.95,
    pretty_tasks: Mapping[str, str] | None = None,
    xlabel: str | None = None,
    height_per_task: float = 0.42,
    sort_within: bool = False,
    band: bool = True,
) -> tuple[Figure, Axes]

Forest plot, one row per task with models jittered within the row.

Height scales with the number of tasks only -- adding more models widens the within-row jitter rather than adding new rows -- so the figure stays compact for many models.

sort_within=True orders the jittered positions per-task so the best mean lands at the top of the row (helps spot leaders when there are many models). Each model keeps a stable colour across tasks.

band=True shades alternating task rows (banded reading aid).

Source code in src/graphnetz/benchmark/_report_plots.py 
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
def plot_forest(
    self,
    ax: plt.Axes | None = None,
    *,
    ci: float = 0.95,
    pretty_tasks: Mapping[str, str] | None = None,
    xlabel: str | None = None,
    height_per_task: float = 0.42,
    sort_within: bool = False,
    band: bool = True,
) -> tuple[plt.Figure, plt.Axes]:
    """Forest plot, one row per task with models jittered within the row.

    Height scales with the number of *tasks* only -- adding more models
    widens the within-row jitter rather than adding new rows -- so the
    figure stays compact for many models.

    ``sort_within=True`` orders the jittered positions per-task so the
    best mean lands at the top of the row (helps spot leaders when there
    are many models).  Each model keeps a stable colour across tasks.

    ``band=True`` shades alternating task rows (banded reading aid).
    """
    from graphnetz.plotting import COLUMN_INCHES, pretty_metric

    set_plot_style()
    finals = self.final_metrics()
    tasks = sorted(finals)
    models = sorted({m for per in finals.values() for m in per})
    pretty = dict(pretty_tasks or {})
    n_tasks = len(tasks)
    n_models = len(models)
    metric = self.metric_name()
    lower_is_better = metric in _LOWER_IS_BETTER

    if ax is None:
        height = max(1.6, height_per_task * n_tasks + 1.0)
        fig, ax = plt.subplots(figsize=(COLUMN_INCHES["single"] * 1.05, height))
    else:
        fig = ax.figure  # type: ignore[assignment]

    jitter_span = 0.7
    slot_positions = (
        np.linspace(-jitter_span / 2, jitter_span / 2, n_models) if n_models > 1 else np.zeros(n_models)
    )

    # Precompute per-task offsets (mapping model_index -> within-row offset).
    per_task_offset: dict[str, dict[str, float]] = {}
    for task in tasks:
        present = [m for m in models if m in finals[task]]
        if sort_within and len(present) > 1:
            means = np.array([float(np.mean(finals[task][m])) for m in present])
            order = np.argsort(means if lower_is_better else -means)
            ordered = [present[i] for i in order]
        else:
            ordered = present
        row_offsets = (
            np.linspace(-jitter_span / 2, jitter_span / 2, len(ordered))
            if len(ordered) > 1
            else np.zeros(len(ordered))
        )
        per_task_offset[task] = dict(zip(ordered, row_offsets, strict=False))

    if band:
        for i in range(n_tasks):
            if i % 2 == 0:
                ax.axhspan(
                    i - 0.5,
                    i + 0.5,
                    facecolor="0.96",
                    edgecolor="none",
                    zorder=0,
                )

    for j, model in enumerate(models):
        xs: list[float] = []
        ys: list[float] = []
        errs: list[float] = []
        for i, task in enumerate(tasks):
            if model not in finals[task]:
                continue
            arr = np.asarray(finals[task][model], dtype=float)
            xs.append(float(arr.mean()))
            offset = per_task_offset[task].get(model, slot_positions[j])
            ys.append(i + offset)
            errs.append(self._ci_half(arr, ci))
        if xs:
            color = NATURE_COLORS[j % len(NATURE_COLORS)]
            ax.errorbar(
                xs,
                ys,
                xerr=[errs, errs],
                fmt="o",
                color=color,
                ecolor=color,
                elinewidth=1.0,
                capsize=2.0,
                markersize=3.5,
                label=model,
                zorder=3,
            )

    for i in range(n_tasks - 1):
        ax.axhline(i + 0.5, color="0.85", linewidth=0.3, zorder=1)

    ax.set_yticks(range(n_tasks))
    ax.set_yticklabels([pretty.get(t, t) for t in tasks])
    ax.set_ylim(n_tasks - 0.5, -0.5)
    ax.tick_params(axis="y", which="both", length=0)
    ax.tick_params(axis="y", which="minor", left=False)
    ax.set_xlabel(xlabel or pretty_metric(metric))
    ax.set_axisbelow(True)
    ax.xaxis.grid(True, zorder=1)
    ax.legend(
        loc="lower center",
        bbox_to_anchor=(0.5, 1.02),
        ncol=min(n_models, 4),
        frameon=False,
        handlelength=1.2,
        handletextpad=0.4,
        columnspacing=1.0,
    )
    fig.tight_layout()
    return fig, ax

plot_pairwise 

plot_pairwise(
    ax: Axes | None = None,
    *,
    ci: float = 0.95,
    alpha: float = 0.05,
    pretty_tasks: Mapping[str, str] | None = None,
    layout: str = "matrix",
    max_cols: int = 3,
    method: str | None = None,
) -> tuple[Figure, Any]

Pairwise comparison plot, with two layouts that scale differently.

layout="matrix" (default) -- one significance heatmap per task, with the lower triangle holding \(-\log_{10}(p_{\text{Holm}})\) and the upper triangle holding the signed mean difference. Scales to many models and many tasks (panels arranged in a grid with at most max_cols columns).

layout="list" -- one row per pairwise comparison with CI whiskers and a significance marker. Best for small numbers of comparisons.

method overrides self.pairwise_method ("t" or "wilcoxon") for this call only.

Source code in src/graphnetz/benchmark/_report_plots.py 
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
def plot_pairwise(
    self,
    ax: plt.Axes | None = None,
    *,
    ci: float = 0.95,
    alpha: float = 0.05,
    pretty_tasks: Mapping[str, str] | None = None,
    layout: str = "matrix",
    max_cols: int = 3,
    method: str | None = None,
) -> tuple[plt.Figure, Any]:
    """Pairwise comparison plot, with two layouts that scale differently.

    ``layout="matrix"`` (default) -- one significance heatmap per task,
    with the lower triangle holding $-\\log_{10}(p_{\\text{Holm}})$ and the
    upper triangle holding the signed mean difference.  Scales to many
    models and many tasks (panels arranged in a grid with at most
    ``max_cols`` columns).

    ``layout="list"`` -- one row per pairwise comparison with CI whiskers
    and a significance marker.  Best for small numbers of comparisons.

    ``method`` overrides ``self.pairwise_method`` (``"t"`` or
    ``"wilcoxon"``) for this call only.
    """
    if layout == "list":
        return self._plot_pairwise_list(ax=ax, ci=ci, alpha=alpha, pretty_tasks=pretty_tasks, method=method)
    if layout == "matrix":
        return self._plot_pairwise_matrix(
            ci=ci, alpha=alpha, pretty_tasks=pretty_tasks, max_cols=max_cols, method=method
        )
    msg = f"Unknown pairwise layout: {layout!r}; choices: 'matrix', 'list'"
    raise ValueError(msg)

plot_critical_difference 

plot_critical_difference(
    *,
    alpha: float = 0.05,
    title: str | None = None,
    aggregation: str = "uniform",
    weights: Mapping[str, float] | None = None,
    groups: Mapping[str, str] | None = None,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> tuple[Figure, Axes]

Demšar critical-difference (CD) diagram.

Computes mean ranks of every model across tasks and overlays the Nemenyi critical difference at level alpha. Models within CD of each other are joined by a thick horizontal "clique" bar (i.e., not significantly different). This is the canonical scalable visualization for multi-method, multi-dataset benchmarks (Demšar, 2006).

Only models present in every task are included. Requires at least two tasks and at least two such models.

aggregation selects the task weighting (see mean_ranks). Only the default "uniform" weighting has a Friedman/Nemenyi null, so any other choice draws the ranks without the CD reference and clique bars and labels itself as a diagnostic -- a weighted mean rank must not be compared against \(CD_\alpha\).

epoch_selection overrides the report-level setting, so the diagram can be redrawn under checkpoint selection without retraining.

Source code in src/graphnetz/benchmark/_report_plots.py 
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
def plot_critical_difference(
    self,
    *,
    alpha: float = 0.05,
    title: str | None = None,
    aggregation: str = "uniform",
    weights: Mapping[str, float] | None = None,
    groups: Mapping[str, str] | None = None,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> tuple[plt.Figure, plt.Axes]:
    r"""Demšar critical-difference (CD) diagram.

    Computes mean ranks of every model across tasks and overlays the
    Nemenyi critical difference at level ``alpha``.  Models within
    ``CD`` of each other are joined by a thick horizontal "clique" bar
    (i.e., not significantly different).  This is the canonical
    scalable visualization for multi-method, multi-dataset benchmarks
    (Demšar, 2006).

    Only models present in *every* task are included.  Requires at
    least two tasks and at least two such models.

    ``aggregation`` selects the task weighting (see
    [`mean_ranks`][graphnetz.benchmark.BenchmarkReport.mean_ranks]).  **Only the default
    ``"uniform"`` weighting has a Friedman/Nemenyi null**, so any other
    choice draws the ranks without the ``CD`` reference and clique bars and
    labels itself as a diagnostic -- a weighted mean rank must not be
    compared against $CD_\alpha$.

    ``epoch_selection`` overrides the report-level setting, so the diagram
    can be redrawn under checkpoint selection without retraining.
    """
    from graphnetz.plotting import COLUMN_INCHES

    set_plot_style()
    table = self.rank_table(epoch_selection=epoch_selection, strict=strict)
    if table.empty or table.shape[0] < 2 or table.shape[1] < 2:
        fig, ax = plt.subplots(figsize=(COLUMN_INCHES["single"], 1.6))
        ax.text(
            0.5,
            0.5,
            "CD diagram needs >= 2 tasks and >= 2 models common to all tasks",
            ha="center",
            va="center",
            transform=ax.transAxes,
            fontsize=8,
        )
        ax.axis("off")
        return fig, ax

    weighted = aggregation != "uniform"
    series = self.mean_ranks(
        aggregation=aggregation,
        weights=weights,
        groups=groups,
        epoch_selection=epoch_selection,
        strict=strict,
    )
    models = list(table.columns)
    avg_ranks = np.array([float(series[m]) for m in models])
    ranks = table.to_numpy(dtype=float)
    n, k = ranks.shape
    # Friedman omnibus: only interpret Nemenyi after the global null is
    # rejected (Demšar, 2006). We compute it from the same rank table.
    chi2 = _friedman_chi2(ranks)
    friedman_p = float(stats.chi2.sf(chi2, df=k - 1))
    friedman_rejected = friedman_p < alpha
    cd = _nemenyi_cd(k, n, alpha)

    order = np.argsort(avg_ranks)
    sorted_models = [models[i] for i in order]
    sorted_ranks = avg_ranks[order]

    # Maximal cliques: contiguous runs in rank order whose span < CD.
    # A weighted aggregation has no Friedman/Nemenyi null, so no clique bar
    # and no CD reference are drawn -- claiming either would be exactly the
    # unearned significance this framework exists to prevent.
    cliques_raw: list[tuple[int, int]] = []
    i = 0
    while i < k:
        j = i
        while j + 1 < k and sorted_ranks[j + 1] - sorted_ranks[i] < cd:
            j += 1
        if j > i:
            cliques_raw.append((i, j))
        i += 1
    cliques: list[tuple[int, int]] = []
    for a, b in sorted(set(cliques_raw)):
        if any(c <= a and b <= d for c, d in cliques):
            continue
        cliques = [(c, d) for c, d in cliques if not (a <= c and d <= b)]
        cliques.append((a, b))
    if weighted:
        cliques = []

    # Layout coordinates.
    fig_w = COLUMN_INCHES["double"]
    fig_h = max(2.2, 1.6 + 0.22 * k)
    fig, ax = plt.subplots(figsize=(fig_w, fig_h))

    rank_y = 0.0
    x_min, x_max = 1.0, float(k)
    ax.plot([x_min, x_max], [rank_y, rank_y], color="black", linewidth=0.8)
    for r in range(int(x_min), int(x_max) + 1):
        ax.plot([r, r], [rank_y, rank_y - 0.04], color="black", linewidth=0.6)
        ax.text(r, rank_y - 0.08, f"{r}", ha="center", va="top", fontsize=8)

    # Method leaders + side labels (left for top half, right for bottom half).
    half = (k + 1) // 2
    label_y_step = 0.16
    label_y_top = 0.32
    label_x_left = x_min - 0.5
    label_x_right = x_max + 0.5
    for idx, (model, r) in enumerate(zip(sorted_models, sorted_ranks, strict=False)):
        color = NATURE_COLORS[idx % len(NATURE_COLORS)]
        if idx < half:
            label_x = label_x_left
            ha = "right"
            ly = label_y_top + (half - idx - 1) * label_y_step
        else:
            label_x = label_x_right
            ha = "left"
            ly = label_y_top + (idx - half) * label_y_step
        ax.plot([r, r], [rank_y, ly], color="0.55", linewidth=0.5, zorder=1)
        ax.plot([r, label_x], [ly, ly], color="0.55", linewidth=0.5, zorder=1)
        ax.plot([r], [rank_y], marker="o", markersize=5.0, color=color, zorder=2)
        # Model name in ink for readability; the rank in muted grey.
        ax.text(
            label_x + (-0.05 if ha == "right" else 0.05),
            ly,
            f"{model} ({r:.2f})",
            va="center",
            ha=ha,
            fontsize=9,
            color="0.15",
        )

    # Clique bars below the rank axis (start below the tick labels).
    bar_y = rank_y - 0.16
    for a, b in cliques:
        ax.plot(
            [sorted_ranks[a] - 0.06, sorted_ranks[b] + 0.06],
            [bar_y, bar_y],
            color="black",
            linewidth=3.5,
            solid_capstyle="round",
            zorder=3,
        )
        bar_y -= 0.06

    # CD scale at the top -- uniform weighting only.
    cd_y = label_y_top + max(half - 1, 0) * label_y_step + 0.22
    if weighted:
        ax.text(
            (x_min + x_max) / 2,
            cd_y + 0.04,
            f"{aggregation}-weighted mean rank -- diagnostic only, "
            r"no Nemenyi $CD_\alpha$ applies",
            ha="center",
            va="bottom",
            fontsize=8,
            color="0.3",
        )
    else:
        ax.plot([x_min, x_min + cd], [cd_y, cd_y], color="black", linewidth=1.0)
        ax.plot([x_min, x_min], [cd_y - 0.025, cd_y + 0.025], color="black", linewidth=1.0)
        ax.plot(
            [x_min + cd, x_min + cd],
            [cd_y - 0.025, cd_y + 0.025],
            color="black",
            linewidth=1.0,
        )
        ax.text(
            x_min + cd / 2,
            cd_y + 0.04,
            rf"CD = {cd:.3f} (Nemenyi, $\alpha={alpha}$, $k={k}$, $N={n}$)",
            ha="center",
            va="bottom",
            fontsize=8,
        )
        friedman_color = "0.15" if friedman_rejected else "0.4"
        ax.text(
            x_min + cd / 2,
            cd_y + 0.18,
            rf"Friedman $\chi^2_{{{k - 1}}} = {chi2:.2f}$, $p = {friedman_p:.3g}$"
            + (" (reject)" if friedman_rejected else " (do not reject)"),
            ha="center",
            va="bottom",
            fontsize=7,
            color=friedman_color,
        )

    # Direction caption below all clique bars.
    caption_y = bar_y - 0.04
    ax.text(
        (x_min + x_max) / 2,
        caption_y,
        "Mean rank (lower rank = better)",
        ha="center",
        va="top",
        fontsize=8,
        color="0.3",
    )

    ax.set_xlim(label_x_left - 1.2, label_x_right + 1.2)
    ax.set_ylim(caption_y - 0.12, cd_y + 0.2)
    ax.axis("off")
    if title is not None:
        ax.set_title(title)
    fig.tight_layout()
    return fig, ax

plot_learning_curves 

plot_learning_curves(
    *,
    ci: float = 0.95,
    metric_key: str | None = None,
    pretty_tasks: Mapping[str, str] | None = None,
    ylabel: str | None = None,
    max_cols: int = 4,
) -> tuple[Figure, ndarray]

Mean ± t-CI learning curves, one panel per task, sharing y-axis.

Panels wrap into rows of at most max_cols so individual panels stay readable on benchmarks with many tasks; a single legend is shared by all panels below the figure.

Source code in src/graphnetz/benchmark/_report_plots.py 
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
def plot_learning_curves(
    self,
    *,
    ci: float = 0.95,
    metric_key: str | None = None,
    pretty_tasks: Mapping[str, str] | None = None,
    ylabel: str | None = None,
    max_cols: int = 4,
) -> tuple[plt.Figure, np.ndarray]:
    """Mean ± t-CI learning curves, one panel per task, sharing y-axis.

    Panels wrap into rows of at most ``max_cols`` so individual panels
    stay readable on benchmarks with many tasks; a single legend is
    shared by all panels below the figure.
    """
    set_plot_style()
    from string import ascii_lowercase

    from graphnetz.plotting import COLUMN_INCHES, panel_label, pretty_metric

    tasks = list(self.histories)
    n_tasks = max(len(tasks), 1)
    ncols = max(1, min(max_cols, n_tasks))
    nrows = (n_tasks + ncols - 1) // ncols
    width = COLUMN_INCHES["double"]
    panel_w = width / ncols
    height = panel_w / 1.3 * nrows + 0.35  # + room for the shared legend
    fig, axes_obj = plt.subplots(nrows, ncols, figsize=(width, height), sharey=True, squeeze=False)
    axes = axes_obj.ravel()
    pretty = dict(pretty_tasks or {})
    resolved_key = metric_key
    for idx, task in enumerate(tasks):
        ax = axes[idx]
        per_task = self.histories[task]
        for j, model in enumerate(per_task):
            seed_histories = per_task[model]
            if not seed_histories:
                continue
            key = metric_key or _auto_metric_key(seed_histories[0])
            resolved_key = resolved_key or key
            arr = np.array([h[key] for h in seed_histories], dtype=float)
            mean = arr.mean(axis=0)
            n = arr.shape[0]
            if n > 1:
                sem = arr.std(axis=0, ddof=1) / np.sqrt(n)
                half = sem * stats.t.ppf((1 + ci) / 2, n - 1)
            else:
                half = np.zeros_like(mean)
            epochs_axis = np.arange(1, mean.size + 1)
            color = NATURE_COLORS[j % len(NATURE_COLORS)]
            ax.plot(epochs_axis, mean, color=color, label=model, linewidth=1.4)
            ax.fill_between(epochs_axis, mean - half, mean + half, color=color, alpha=0.18, linewidth=0)
        if idx // ncols == nrows - 1 or idx + ncols >= n_tasks:
            ax.set_xlabel("Epoch")
        ax.set_title(pretty.get(task, task))
        ax.set_axisbelow(True)
        ax.yaxis.grid(True)
        ax.margins(x=0.02)
        if idx % ncols == 0:
            ax.set_ylabel(ylabel or pretty_metric(resolved_key or "metric"))
        else:
            ax.tick_params(labelleft=False)
        if n_tasks > 1:
            panel_label(ax, ascii_lowercase[idx % 26], x=-0.08 if idx % ncols else -0.18)
    for idx in range(n_tasks, nrows * ncols):
        axes[idx].axis("off")

    # One shared legend below all panels instead of crowding panel one.
    handles, labels = axes[0].get_legend_handles_labels()
    if handles:
        fig.legend(
            handles,
            labels,
            loc="lower center",
            ncol=min(len(labels), 5),
            frameon=False,
            handlelength=1.8,
            columnspacing=1.4,
            bbox_to_anchor=(0.5, 0.0),
        )
    fig.tight_layout(rect=(0, 0.35 / height, 1, 1), h_pad=2.2, w_pad=1.2)
    return fig, axes_obj if nrows > 1 else np.atleast_1d(axes_obj[0])

final_metrics 

final_metrics(
    key: str | None = None, *, epoch_selection: str | None = None, strict: bool = True
) -> dict[str, dict[str, list[float]]]

Reduced metric value per (task, model, seed).

epoch_selection overrides epoch_selection for this call. Under "best_val" a task whose trainer records no held-out series cannot be selected honestly; strict=True raises, strict=False falls back to the final epoch for that task alone (use epoch_selection_support to see which tasks fell back).

Source code in src/graphnetz/benchmark/_report.py 
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
def final_metrics(
    self,
    key: str | None = None,
    *,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> dict[str, dict[str, list[float]]]:
    """Reduced metric value per (task, model, seed).

    ``epoch_selection`` overrides [`epoch_selection`][graphnetz.benchmark.BenchmarkReport.epoch_selection] for this call.
    Under ``"best_val"`` a task whose trainer records no held-out series
    cannot be selected honestly; ``strict=True`` raises, ``strict=False``
    falls back to the final epoch for that task alone (use
    [`epoch_selection_support`][graphnetz.benchmark.BenchmarkReport.epoch_selection_support] to see which tasks fell back).
    """
    selection = epoch_selection or self.epoch_selection
    out: dict[str, dict[str, list[float]]] = {}
    for task, per_task in self.histories.items():
        out[task] = {}
        task_selection = selection
        if selection != "final" and not strict:
            sample = next(iter(per_task.values()), None)
            if sample and _selection_reason(sample[0], key or _auto_metric_key(sample[0])) is not None:
                task_selection = "final"
        for model, seed_histories in per_task.items():
            vals: list[float] = []
            for h in seed_histories:
                k = key or _auto_metric_key(h)
                vals.append(_selected_value(h, k, task_selection)[0])
            out[task][model] = vals
    return out

epoch_selection_support 

epoch_selection_support(key: str | None = None) -> DataFrame

Per task: whether "best_val" is available, and why not if it is not.

Checkpoint selection needs a validation series and a held-out series in the same history. Node classification and both link-prediction trainers record both; the graph-level trainers record only a validation metric, so selecting on it and reporting it would be optimistically biased. This surfaces that asymmetry instead of hiding it.

Source code in src/graphnetz/benchmark/_report.py 
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
def epoch_selection_support(self, key: str | None = None) -> pd.DataFrame:
    """Per task: whether ``"best_val"`` is available, and why not if it is not.

    Checkpoint selection needs a validation series *and* a held-out series
    in the same history.  Node classification and both link-prediction
    trainers record both; the graph-level trainers record only a validation
    metric, so selecting on it and reporting it would be optimistically
    biased.  This surfaces that asymmetry instead of hiding it.
    """
    rows = []
    for task, per_task in self.histories.items():
        sample = next(iter(per_task.values()), None)
        if not sample:
            continue
        metric = key or _auto_metric_key(sample[0])
        reason = _selection_reason(sample[0], metric)
        rows.append(
            {
                "task": task,
                "metric": metric,
                "best_val_supported": reason is None,
                "reason": "" if reason is None else reason,
            }
        )
    return pd.DataFrame(rows).set_index("task")

selected_epochs 

selected_epochs(
    key: str | None = None, *, epoch_selection: str | None = None, strict: bool = True
) -> DataFrame

The epoch each (task, model, seed) metric was taken from.

Auditing surface for "best_val": a selected epoch pinned at the last epoch for every seed means selection did nothing, and a wildly varying one means the run had not converged.

Source code in src/graphnetz/benchmark/_report.py 
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
def selected_epochs(
    self,
    key: str | None = None,
    *,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> pd.DataFrame:
    """The epoch each (task, model, seed) metric was taken from.

    Auditing surface for ``"best_val"``: a selected epoch pinned at the
    last epoch for every seed means selection did nothing, and a wildly
    varying one means the run had not converged.
    """
    selection = epoch_selection or self.epoch_selection
    rows = []
    for task, per_task in self.histories.items():
        task_selection = selection
        if selection != "final" and not strict:
            sample = next(iter(per_task.values()), None)
            if sample and _selection_reason(sample[0], key or _auto_metric_key(sample[0])) is not None:
                task_selection = "final"
        for model, seed_histories in per_task.items():
            for seed, h in zip(self.seeds, seed_histories, strict=False):
                k = key or _auto_metric_key(h)
                value, epoch = _selected_value(h, k, task_selection)
                rows.append(
                    {
                        "task": task,
                        "model": model,
                        "seed": seed,
                        "selection": task_selection,
                        "epoch": epoch,
                        "n_epochs": len(h[k]),
                        "value": value,
                    }
                )
    return pd.DataFrame(rows)

summary 

summary(
    ci: float = 0.95,
    method: str | None = None,
    *,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> DataFrame

Per-(task, model) mean, std, sem, CI half-width and bounds.

method overrides self.ci_method for this call only; choose "t" for Student's-t intervals (default) or "bootstrap" for percentile-bootstrap intervals (better for non-Gaussian metrics such as Hits@K, MRR, or AUC).

Source code in src/graphnetz/benchmark/_report.py 
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
def summary(
    self,
    ci: float = 0.95,
    method: str | None = None,
    *,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> pd.DataFrame:
    """Per-(task, model) mean, std, sem, CI half-width and bounds.

    ``method`` overrides ``self.ci_method`` for this call only; choose
    ``"t"`` for Student's-t intervals (default) or ``"bootstrap"`` for
    percentile-bootstrap intervals (better for non-Gaussian metrics
    such as Hits@K, MRR, or AUC).
    """
    rows = []
    finals = self.final_metrics(epoch_selection=epoch_selection, strict=strict)
    for task, per_task in finals.items():
        for model, values in per_task.items():
            arr = np.asarray(values, dtype=float)
            mean = float(arr.mean())
            std = float(arr.std(ddof=1)) if arr.size > 1 else 0.0
            sem = float(stats.sem(arr)) if arr.size > 1 else 0.0
            half = self._ci_half(arr, ci, method=method)
            rows.append(
                {
                    "task": task,
                    "model": model,
                    "n_seeds": arr.size,
                    "mean": mean,
                    "std": std,
                    "sem": sem,
                    "ci_low": mean - half,
                    "ci_high": mean + half,
                }
            )
    return pd.DataFrame(rows).set_index(["task", "model"]).sort_index()

pairwise 

pairwise(
    alpha: float = 0.05,
    method: str | None = None,
    *,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> DataFrame

Paired pairwise tests between models per task with Holm adjustment.

method overrides self.pairwise_method for this call only:

  • "t" (default) -- paired Student's t-test on per-seed final metrics.
  • "wilcoxon" -- non-parametric Wilcoxon signed-rank test on the paired differences. Recommended at small seed counts where the paired t-test's normality assumption is most fragile; see Benavoli et al., JMLR 17(5):1-36, 2016.

epoch_selection overrides epoch_selection, so the same comparison can be run under a fixed-epoch and a checkpoint-selected protocol without retraining.

Source code in src/graphnetz/benchmark/_report.py 
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
def pairwise(
    self,
    alpha: float = 0.05,
    method: str | None = None,
    *,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> pd.DataFrame:
    """Paired pairwise tests between models per task with Holm adjustment.

    ``method`` overrides ``self.pairwise_method`` for this call only:

    - ``"t"`` (default) -- paired Student's t-test on per-seed final metrics.
    - ``"wilcoxon"`` -- non-parametric Wilcoxon signed-rank test on the
      paired differences. Recommended at small seed counts where the
      paired t-test's normality assumption is most fragile; see
      Benavoli et al., *JMLR* 17(5):1-36, 2016.

    ``epoch_selection`` overrides [`epoch_selection`][graphnetz.benchmark.BenchmarkReport.epoch_selection], so the same
    comparison can be run under a fixed-epoch and a checkpoint-selected
    protocol without retraining.
    """
    finals = self.final_metrics(epoch_selection=epoch_selection, strict=strict)
    test = method or self.pairwise_method
    rows = []
    for task, per_task in finals.items():
        models = sorted(per_task)
        pairs: list[tuple[str, str, float, float, float]] = []
        ps: list[float] = []
        for i, model_a in enumerate(models):
            for model_b in models[i + 1 :]:
                a = np.asarray(per_task[model_a], dtype=float)
                b = np.asarray(per_task[model_b], dtype=float)
                p = _paired_pvalue(a, b, test)
                d = a - b
                sigma_d = float(d.std(ddof=1)) if d.size > 1 else float("nan")
                pairs.append((model_a, model_b, float(d.mean()), sigma_d, p))
                ps.append(p)
        adj = _holm_correction(np.asarray(ps, dtype=float))
        for (model_a, model_b, diff, sigma_d, p_raw), p_holm in zip(pairs, adj, strict=False):
            n = len(per_task[model_a])
            rows.append(
                {
                    "task": task,
                    "model_a": model_a,
                    "model_b": model_b,
                    "mean_diff": diff,
                    # Paired effect size (Appendix A): invariant to the seed
                    # count, so it complements the p-value with a magnitude.
                    "sigma_d": sigma_d,
                    "cohens_dz": diff / sigma_d
                    if sigma_d and np.isfinite(sigma_d) and sigma_d > 0
                    else float("nan"),
                    "p_raw": p_raw,
                    "p_holm": p_holm,
                    "significant": (not np.isnan(p_holm)) and p_holm < alpha,
                    "n_seeds": n,
                }
            )
    return pd.DataFrame(rows)

power 

power(
    alpha: float = 0.05,
    target_power: float = 0.8,
    *,
    targets: tuple[float, ...] = (0.02, 0.01, 0.005),
    epoch_selection: str | None = None,
    strict: bool = True,
) -> DataFrame

Per-comparison minimum detectable effect and observed power.

A non-significant pairwise row means one of two very different things: the models are close, or the design could not tell. This separates them. mde is the smallest true paired difference detectable at target_power; detectable flags comparisons whose observed effect clears their own MDE; seeds_for_* says how many seeds a given effect size would need.

See graphnetz.benchmark._power for the formulae.

Source code in src/graphnetz/benchmark/_report.py 
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
def power(
    self,
    alpha: float = 0.05,
    target_power: float = 0.80,
    *,
    targets: tuple[float, ...] = (0.02, 0.01, 0.005),
    epoch_selection: str | None = None,
    strict: bool = True,
) -> pd.DataFrame:
    """Per-comparison minimum detectable effect and observed power.

    A non-significant pairwise row means one of two very different things:
    the models are close, or the design could not tell.  This separates
    them.  ``mde`` is the smallest true paired difference detectable at
    ``target_power``; ``detectable`` flags comparisons whose observed
    effect clears their own MDE; ``seeds_for_*`` says how many seeds a
    given effect size would need.

    See `graphnetz.benchmark._power` for the formulae.
    """
    pw = self.pairwise(alpha=alpha, epoch_selection=epoch_selection, strict=strict)
    if pw.empty:
        return pw
    rows = []
    for _, r in pw.iterrows():
        n = int(r["n_seeds"])
        sigma_d = float(r["sigma_d"])
        effect = float(r["mean_diff"])
        mde = minimum_detectable_effect(sigma_d, n, alpha, target_power)
        row = {
            "task": r["task"],
            "model_a": r["model_a"],
            "model_b": r["model_b"],
            "mean_diff": effect,
            "sigma_d": sigma_d,
            "n_seeds": n,
            "mde": mde,
            "observed_power": observed_power(effect, sigma_d, n, alpha),
            "detectable": bool(np.isfinite(mde) and abs(effect) >= mde),
        }
        for t in targets:
            row[f"seeds_for_{t:g}"] = seeds_for_effect(t, sigma_d, alpha, target_power)
        rows.append(row)
    return pd.DataFrame(rows)

equivalence 

equivalence(
    margin: float,
    alpha: float = 0.05,
    *,
    method: str | None = None,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> DataFrame

Two one-sided tests for equivalence within margin, plus a verdict.

margin is the smallest effect size of interest, in the metric's own units (e.g. 0.01 accuracy points). Combined with the difference test, every comparison lands in exactly one of different, equivalent, trivial or undetermined -- so "we found no difference" can be stated as a positive claim where the data support it and withheld where they do not.

Both families are Holm-corrected over the comparisons within a task, matching pairwise.

Source code in src/graphnetz/benchmark/_report.py 
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
def equivalence(
    self,
    margin: float,
    alpha: float = 0.05,
    *,
    method: str | None = None,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> pd.DataFrame:
    """Two one-sided tests for equivalence within ``margin``, plus a verdict.

    ``margin`` is the smallest effect size of interest, in the metric's own
    units (e.g. ``0.01`` accuracy points).  Combined with the difference
    test, every comparison lands in exactly one of ``different``,
    ``equivalent``, ``trivial`` or ``undetermined`` -- so "we found no
    difference" can be stated as a positive claim where the data support it
    and withheld where they do not.

    Both families are Holm-corrected over the comparisons within a task,
    matching [`pairwise`][graphnetz.benchmark.BenchmarkReport.pairwise].
    """
    finals = self.final_metrics(epoch_selection=epoch_selection, strict=strict)
    diff_df = self.pairwise(alpha=alpha, method=method, epoch_selection=epoch_selection, strict=strict)
    rows = []
    for task, per_task in finals.items():
        models = sorted(per_task)
        keys: list[tuple[str, str]] = []
        ps: list[float] = []
        for i, model_a in enumerate(models):
            for model_b in models[i + 1 :]:
                a = np.asarray(per_task[model_a], dtype=float)
                b = np.asarray(per_task[model_b], dtype=float)
                ps.append(paired_tost(a, b, margin, alpha))
                keys.append((model_a, model_b))
        adj = _holm_correction(np.asarray(ps, dtype=float))
        for (model_a, model_b), p_raw, p_holm in zip(keys, ps, adj, strict=False):
            match = diff_df[
                (diff_df["task"] == task) & (diff_df["model_a"] == model_a) & (diff_df["model_b"] == model_b)
            ]
            p_diff = float(match["p_holm"].iloc[0]) if not match.empty else float("nan")
            mean_diff = float(match["mean_diff"].iloc[0]) if not match.empty else float("nan")
            rows.append(
                {
                    "task": task,
                    "model_a": model_a,
                    "model_b": model_b,
                    "mean_diff": mean_diff,
                    "margin": margin,
                    "p_tost_raw": p_raw,
                    "p_tost_holm": p_holm,
                    "p_difference_holm": p_diff,
                    "verdict": equivalence_verdict(p_diff, p_holm, alpha),
                }
            )
    return pd.DataFrame(rows)

rank_table 

rank_table(*, epoch_selection: str | None = None, strict: bool = True) -> DataFrame

Per-task ranks of the seed means; rows tasks, columns models.

Rank 1 is best, ties averaged, and the metric direction is applied per task, so a benchmark mixing accuracy with MAE ranks correctly. Only models present in every task appear -- the same restriction the CD diagram applies, surfaced as data.

Source code in src/graphnetz/benchmark/_report.py 
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
def rank_table(
    self,
    *,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> pd.DataFrame:
    """Per-task ranks of the seed means; rows tasks, columns models.

    Rank 1 is best, ties averaged, and the metric direction is applied per
    task, so a benchmark mixing accuracy with MAE ranks correctly.  Only
    models present in *every* task appear -- the same restriction the CD
    diagram applies, surfaced as data.
    """
    finals = self.final_metrics(epoch_selection=epoch_selection, strict=strict)
    if not finals:
        return pd.DataFrame()
    common: set[str] = set.intersection(*[set(per.keys()) for per in finals.values()])
    if len(common) < 2:
        return pd.DataFrame()
    models = sorted(common)
    tasks = sorted(finals)
    ranks = _rank_rows(finals, self._task_directions(), models, tasks)
    return pd.DataFrame(ranks, index=pd.Index(tasks, name="task"), columns=models)

mean_ranks 

mean_ranks(
    *,
    aggregation: str = "uniform",
    weights: Mapping[str, float] | None = None,
    groups: Mapping[str, str] | None = None,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> Series

Mean rank per model across tasks, under a choice of task weighting.

The Demšar procedure weights every task equally, which means a small synthetic task counts as much as a large public benchmark. That is a modelling choice, not a law, so it is exposed:

"uniform" The Demšar default, \(\bar r_i = \frac1N \sum_n r_{n,i}\). This is the only variant whose null distribution the Friedman and Nemenyi procedures describe. "reliability" Weight each task by the inverse mean 95 % CI half-width of its cells, so tasks whose seed variance makes their ranking unreliable contribute less. "hierarchical" Average within each group in groups (e.g. research category), then across groups, so a category with many tasks does not dominate one with few. "custom" Use weights directly, keyed by task name.

Non-uniform variants are diagnostics: they break the exchangeability the Friedman null assumes, so a weighted mean rank must not be compared against \(CD_\alpha\). Use rank_stability for uncertainty on them.

Source code in src/graphnetz/benchmark/_report.py 
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
def mean_ranks(
    self,
    *,
    aggregation: str = "uniform",
    weights: Mapping[str, float] | None = None,
    groups: Mapping[str, str] | None = None,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> pd.Series:
    r"""Mean rank per model across tasks, under a choice of task weighting.

    The Demšar procedure weights every task equally, which means a small
    synthetic task counts as much as a large public benchmark.  That is a
    modelling choice, not a law, so it is exposed:

    ``"uniform"``
        The Demšar default, $\bar r_i = \frac1N \sum_n r_{n,i}$.
        **This is the only variant whose null distribution the Friedman and
        Nemenyi procedures describe.**
    ``"reliability"``
        Weight each task by the inverse mean 95 % CI half-width of its
        cells, so tasks whose seed variance makes their ranking unreliable
        contribute less.
    ``"hierarchical"``
        Average within each group in ``groups`` (e.g. research category),
        then across groups, so a category with many tasks does not
        dominate one with few.
    ``"custom"``
        Use ``weights`` directly, keyed by task name.

    Non-uniform variants are **diagnostics**: they break the exchangeability
    the Friedman null assumes, so a weighted mean rank must not be compared
    against $CD_\alpha$.  Use [`rank_stability`][graphnetz.benchmark.BenchmarkReport.rank_stability] for uncertainty
    on them.
    """
    table = self.rank_table(epoch_selection=epoch_selection, strict=strict)
    if table.empty:
        return pd.Series(dtype=float)
    tasks = list(table.index)

    if aggregation == "uniform":
        w = _normalized_weights(None, tasks)
    elif aggregation == "custom":
        if weights is None:
            msg = "aggregation='custom' requires weights={task: weight}"
            raise ValueError(msg)
        w = _normalized_weights(weights, tasks)
    elif aggregation == "reliability":
        summary = self.summary(epoch_selection=epoch_selection, strict=strict)
        half = (summary["ci_high"] - summary["ci_low"]) / 2.0
        per_task = half.groupby(level="task").mean()
        inv = {t: 1.0 / max(float(per_task.get(t, np.nan)), 1e-12) for t in tasks}
        w = _normalized_weights(inv, tasks)
    elif aggregation == "hierarchical":
        if groups is None:
            msg = "aggregation='hierarchical' requires groups={task: group}"
            raise ValueError(msg)
        sizes: dict[str, int] = {}
        for t in tasks:
            sizes[groups.get(t, t)] = sizes.get(groups.get(t, t), 0) + 1
        # Each group contributes equally; tasks share their group's budget.
        w = _normalized_weights({t: 1.0 / sizes[groups.get(t, t)] for t in tasks}, tasks)
    else:
        msg = f"Unknown aggregation {aggregation!r}; choices: 'uniform', 'reliability', 'hierarchical', 'custom'"
        raise ValueError(msg)

    values = np.asarray(table.to_numpy(dtype=float))
    weighted = (values * w[:, None]).sum(axis=0) / w.sum()
    return pd.Series(weighted, index=table.columns).sort_values()

friedman 

friedman(
    alpha: float = 0.05, *, epoch_selection: str | None = None, strict: bool = True
) -> dict[str, float | int | bool]

Friedman omnibus test on per-task ranks of seed-mean metrics.

Returns the statistic chi2, the asymptotic \(\chi^2_{k-1}\) p-value, the rejection flag at alpha, the \((k, N)\) shape, and the Nemenyi critical_difference implied by that shape. Also returns n_for_observed_gap: the number of tasks at which \(CD_\alpha\) would shrink below the largest observed mean-rank gap -- i.e. how much benchmark breadth the current ordering would need before it could be called significant. Since \(CD_\alpha \propto 1/\sqrt N\) this is a design quantity, computed from the observed gap and independent of any assumption about how new tasks would rank.

The Nemenyi post-hoc surfaced in plot_critical_difference should only be interpreted when rejected is true (Demšar, 2006). Only models present in every task are included.

Source code in src/graphnetz/benchmark/_report.py 
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
def friedman(
    self,
    alpha: float = 0.05,
    *,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> dict[str, float | int | bool]:
    r"""Friedman omnibus test on per-task ranks of seed-mean metrics.

    Returns the statistic ``chi2``, the asymptotic $\chi^2_{k-1}$ p-value,
    the rejection flag at ``alpha``, the $(k, N)$ shape, and the Nemenyi
    ``critical_difference`` implied by that shape.  Also returns
    ``n_for_observed_gap``: the number of tasks at which $CD_\alpha$ would
    shrink below the largest observed mean-rank gap -- i.e. how much
    benchmark breadth the current ordering would need before it could be
    called significant.  Since $CD_\alpha \propto 1/\sqrt N$ this is a
    design quantity, computed from the observed gap and independent of any
    assumption about how new tasks would rank.

    The Nemenyi post-hoc surfaced in [`plot_critical_difference`][graphnetz.benchmark.BenchmarkReport.plot_critical_difference]
    should only be interpreted when ``rejected`` is true (Demšar, 2006).
    Only models present in every task are included.
    """
    empty = {
        "chi2": float("nan"),
        "p_value": float("nan"),
        "k": 0,
        "n": 0,
        "rejected": False,
        "critical_difference": float("nan"),
        "max_rank_gap": float("nan"),
        "n_for_observed_gap": 0,
    }
    table = self.rank_table(epoch_selection=epoch_selection, strict=strict)
    if table.empty or table.shape[0] < 2 or table.shape[1] < 2:
        finals = self.final_metrics(epoch_selection=epoch_selection, strict=strict)
        common = set.intersection(*[set(p) for p in finals.values()]) if finals else set()
        return {**empty, "k": len(common), "n": len(finals)}

    ranks = table.to_numpy(dtype=float)
    n, k = ranks.shape
    chi2 = _friedman_chi2(ranks)
    p = float(stats.chi2.sf(chi2, df=k - 1))
    cd = _nemenyi_cd(k, n, alpha)
    avg = ranks.mean(axis=0)
    gap = float(avg.max() - avg.min())
    # CD(N) = q sqrt(k(k+1)/6N) < gap  <=>  N > q^2 k(k+1)/(6 gap^2),
    # so scaling the current CD gives the N at which the gap would resolve.
    n_needed = int(np.ceil((cd**2) * n / (gap**2))) if gap > 0 and np.isfinite(cd) else 0
    return {
        "chi2": float(chi2),
        "p_value": p,
        "k": k,
        "n": n,
        "rejected": bool(p < alpha),
        "critical_difference": float(cd),
        "max_rank_gap": gap,
        "n_for_observed_gap": n_needed,
    }

rank_stability 

rank_stability(
    *,
    n_boot: int = 10000,
    alpha: float = 0.05,
    seed: int = 0,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> dict[str, Any]

How much the across-task ranking depends on which tasks were run.

A CD diagram at small N says nothing about whether the ordering would survive a different draw of benchmark tasks. Two answers:

Bootstrap over tasks. Resample the N tasks with replacement n_boot times, recompute mean ranks, and report the distribution of the largest mean-rank gap, the fraction of resamples reproducing the observed rank order, and for each pair the fraction of resamples in which the pair separates by more than that resample's own \(CD_\alpha\).

Leave-one-task-out jackknife. Drop each task in turn and record how far every mean rank moves -- exposing single tasks that carry the ordering.

Returns mean_ranks, order, gap_observed, gap_quantiles, order_stability, separation and jackknife.

Source code in src/graphnetz/benchmark/_report.py 
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
def rank_stability(
    self,
    *,
    n_boot: int = 10_000,
    alpha: float = 0.05,
    seed: int = 0,
    epoch_selection: str | None = None,
    strict: bool = True,
) -> dict[str, Any]:
    """How much the across-task ranking depends on *which* tasks were run.

    A CD diagram at small ``N`` says nothing about whether the ordering
    would survive a different draw of benchmark tasks.  Two answers:

    *Bootstrap over tasks.*  Resample the ``N`` tasks with replacement
    ``n_boot`` times, recompute mean ranks, and report the distribution of
    the largest mean-rank gap, the fraction of resamples reproducing the
    observed rank *order*, and for each pair the fraction of resamples in
    which the pair separates by more than that resample's own $CD_\\alpha$.

    *Leave-one-task-out jackknife.*  Drop each task in turn and record how
    far every mean rank moves -- exposing single tasks that carry the
    ordering.

    Returns ``mean_ranks``, ``order``, ``gap_observed``, ``gap_quantiles``,
    ``order_stability``, ``separation`` and ``jackknife``.
    """
    table = self.rank_table(epoch_selection=epoch_selection, strict=strict)
    if table.empty or table.shape[0] < 2:
        return {"mean_ranks": pd.Series(dtype=float), "order_stability": float("nan")}
    ranks = table.to_numpy(dtype=float)
    models = list(table.columns)
    tasks = list(table.index)
    n, k = ranks.shape

    observed = ranks.mean(axis=0)
    order = [models[i] for i in np.argsort(observed)]
    gap_observed = float(observed.max() - observed.min())

    rng = np.random.default_rng(seed)
    idx = rng.integers(0, n, size=(n_boot, n))
    boot = ranks[idx].mean(axis=1)  # [n_boot, k]
    gaps = boot.max(axis=1) - boot.min(axis=1)
    boot_order = np.argsort(boot, axis=1)
    target = np.argsort(observed)
    order_stability = float(np.mean(np.all(boot_order == target[None, :], axis=1)))

    cd_boot = _nemenyi_cd(k, n, alpha)  # same (k, N) in every resample
    separation = []
    for i in range(k):
        for j in range(i + 1, k):
            diff = np.abs(boot[:, i] - boot[:, j])
            separation.append(
                {
                    "model_a": models[i],
                    "model_b": models[j],
                    "observed_gap": float(abs(observed[i] - observed[j])),
                    "separates_frac": float(np.mean(diff > cd_boot)),
                }
            )

    jack = []
    for t in range(n):
        kept = np.delete(ranks, t, axis=0).mean(axis=0)
        for m, before, after in zip(models, observed, kept, strict=False):
            jack.append(
                {
                    "dropped_task": tasks[t],
                    "model": m,
                    "mean_rank": float(after),
                    "shift": float(after - before),
                }
            )
    return {
        "mean_ranks": pd.Series(observed, index=models).sort_values(),
        "order": order,
        "gap_observed": gap_observed,
        "critical_difference": float(cd_boot),
        "gap_quantiles": {q: float(np.quantile(gaps, q)) for q in (0.025, 0.25, 0.5, 0.75, 0.975)},
        "gap_exceeds_cd_frac": float(np.mean(gaps > cd_boot)),
        "order_stability": order_stability,
        "separation": pd.DataFrame(separation),
        "jackknife": pd.DataFrame(jack),
        "n_boot": n_boot,
    }

to_json 

to_json(path: str | Path | None = None) -> str

Serialise the report to JSON: seeds, config, and the raw histories.

The payload is the metric tensor X[task, model, seed] before any statistic is applied, so a published bundle can be re-analysed under a different CI method, pairwise test, epoch selection or aggregation without retraining. JSON rather than pickle so the artefact stays readable, diffable, and independent of the library version that wrote it. Returns the JSON text and writes it to path when given.

Source code in src/graphnetz/benchmark/_report.py 
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
def to_json(self, path: str | Path | None = None) -> str:
    """Serialise the report to JSON: seeds, config, and the raw histories.

    The payload is the metric tensor ``X[task, model, seed]`` *before* any
    statistic is applied, so a published bundle can be re-analysed under a
    different CI method, pairwise test, epoch selection or aggregation
    without retraining.  JSON rather than pickle so the artefact stays
    readable, diffable, and independent of the library version that wrote
    it.  Returns the JSON text and writes it to ``path`` when given.
    """
    import json

    payload = {
        "schema": self.SCHEMA,
        "seeds": list(self.seeds),
        "config": {k: v for k, v in self.config.items() if isinstance(v, (str, int, float, bool, type(None)))},
        "options": {
            "ci_method": self.ci_method,
            "bootstrap_n": self.bootstrap_n,
            "bootstrap_seed": self.bootstrap_seed,
            "pairwise_method": self.pairwise_method,
            "epoch_selection": self.epoch_selection,
        },
        "histories": self.histories,
    }
    text = json.dumps(payload, indent=1) + "\n"
    if path is not None:
        out = Path(path)
        out.parent.mkdir(parents=True, exist_ok=True)
        out.write_text(text)
    return text

from_json classmethod 

from_json(source: str | Path) -> BenchmarkReport

Rebuild a report from to_json output (a path or the text itself).

Source code in src/graphnetz/benchmark/_report.py 
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
@classmethod
def from_json(cls, source: str | Path) -> BenchmarkReport:
    """Rebuild a report from [`to_json`][graphnetz.benchmark.BenchmarkReport.to_json] output (a path or the text itself)."""
    import json

    text = Path(source).read_text() if Path(str(source)).exists() else str(source)
    payload = json.loads(text)
    schema = payload.get("schema")
    if schema != cls.SCHEMA:
        msg = f"unsupported report schema {schema!r}; this version reads {cls.SCHEMA}"
        raise ValueError(msg)
    options = payload.get("options", {})
    return cls(
        seeds=tuple(payload["seeds"]),
        histories=payload["histories"],
        config=payload.get("config", {}),
        ci_method=options.get("ci_method", "t"),
        bootstrap_n=options.get("bootstrap_n", 10000),
        bootstrap_seed=options.get("bootstrap_seed", 0),
        pairwise_method=options.get("pairwise_method", "t"),
        epoch_selection=options.get("epoch_selection", "final"),
    )

to_latex 

to_latex(
    path: str | Path,
    *,
    ci: float = 0.95,
    bold_best: bool = True,
    pretty_tasks: Mapping[str, str] | None = None,
    caption: str | None = None,
    label: str | None = None,
    method: str | None = None,
) -> Path

Booktabs LaTeX table of mean ± CI half-width with bold-best per task.

method overrides self.ci_method ("t" or "bootstrap").

Source code in src/graphnetz/benchmark/_report.py 
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
def to_latex(
    self,
    path: str | Path,
    *,
    ci: float = 0.95,
    bold_best: bool = True,
    pretty_tasks: Mapping[str, str] | None = None,
    caption: str | None = None,
    label: str | None = None,
    method: str | None = None,
) -> Path:
    """Booktabs LaTeX table of mean ± CI half-width with bold-best per task.

    ``method`` overrides ``self.ci_method`` (``"t"`` or ``"bootstrap"``).
    """
    finals = self.final_metrics()
    tasks = sorted(finals)
    models = sorted({m for per in finals.values() for m in per})
    best = self._best_per_task() if bold_best else {}
    pretty = dict(pretty_tasks or {})

    lines: list[str] = []
    if caption is not None or label is not None:
        lines.extend([r"\begin{table}[t]", r"  \centering"])
        if caption is not None:
            lines.append(rf"  \caption{{{caption}}}")
        if label is not None:
            lines.append(rf"  \label{{{label}}}")
    lines.append(r"\begin{tabular}{l" + "c" * len(tasks) + "}")
    lines.append(r"\toprule")
    header = "Model & " + " & ".join(pretty.get(t, t) for t in tasks) + r" \\"
    lines.append(header)
    lines.append(r"\midrule")
    for model in models:
        cells = []
        for task in tasks:
            values = np.asarray(finals[task].get(model, []), dtype=float)
            if values.size == 0:
                cells.append("--")
                continue
            mean = float(values.mean())
            half = self._ci_half(values, ci, method=method)
            if bold_best and best.get(task) == model:
                cell = rf"$\mathbf{{{mean:.3f} \pm {half:.3f}}}$"
            else:
                cell = rf"${mean:.3f} \pm {half:.3f}$"
            cells.append(cell)
        lines.append(f"{model} & " + " & ".join(cells) + r" \\")
    lines.append(r"\bottomrule")
    lines.append(r"\end{tabular}")
    if caption is not None or label is not None:
        lines.append(r"\end{table}")
    out = Path(path)
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text("\n".join(lines) + "\n")
    return out

pairwise_to_latex 

pairwise_to_latex(
    path: str | Path,
    *,
    alpha: float = 0.05,
    caption: str | None = None,
    label: str | None = None,
    method: str | None = None,
) -> Path

LaTeX booktabs table of pairwise Holm-adjusted p-values.

method overrides self.pairwise_method ("t" or "wilcoxon") for this call only.

Source code in src/graphnetz/benchmark/_report.py 
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
def pairwise_to_latex(
    self,
    path: str | Path,
    *,
    alpha: float = 0.05,
    caption: str | None = None,
    label: str | None = None,
    method: str | None = None,
) -> Path:
    """LaTeX booktabs table of pairwise Holm-adjusted p-values.

    ``method`` overrides ``self.pairwise_method`` (``"t"`` or
    ``"wilcoxon"``) for this call only.
    """
    df = self.pairwise(alpha=alpha, method=method)
    lines: list[str] = []
    if caption is not None or label is not None:
        lines.extend([r"\begin{table}[t]", r"  \centering"])
        if caption is not None:
            lines.append(rf"  \caption{{{caption}}}")
        if label is not None:
            lines.append(rf"  \label{{{label}}}")
    lines.append(r"\begin{tabular}{llcccl}")
    lines.append(r"\toprule")
    lines.append(r"Task & Comparison & $\Delta\mu$ & $p_{\text{raw}}$ & $p_{\text{Holm}}$ & Sig. \\")
    lines.append(r"\midrule")
    for _, row in df.iterrows():
        sig = r"\textbf{*}" if row["significant"] else ""
        p_raw = "n/a" if pd.isna(row["p_raw"]) else f"{row['p_raw']:.3g}"
        p_holm = "n/a" if pd.isna(row["p_holm"]) else f"{row['p_holm']:.3g}"
        lines.append(
            f"{row['task']} & {row['model_a']} vs.\\ {row['model_b']} & "
            f"${row['mean_diff']:+.3f}$ & {p_raw} & {p_holm} & {sig} \\\\"
        )
    lines.append(r"\bottomrule")
    lines.append(r"\end{tabular}")
    if caption is not None or label is not None:
        lines.append(r"\end{table}")
    out = Path(path)
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text("\n".join(lines) + "\n")
    return out

Tasks and models

benchmark

Statistically robust benchmarks across a category for one or many models.

The dispatcher trains every compatible (model, task) pair across multiple seeds and returns a BenchmarkReport that exposes mean ± 95 % t-CI, paired t-tests with Holm-Bonferroni correction, publication-ready LaTeX tables, and plots.

Custom models are plugged in via the same three paths as before:

  1. Decorator / registry::

    from graphnetz import register_model

    @register_model(task_type="node_cls") class MyGNN(torch.nn.Module): def init(self, in_channels, hidden_channels, out_channels): ...

  2. Class attribute::

    class MyGNN(torch.nn.Module): task_types = {"node_cls"}

  3. Inline tuple (cls, tasks) or (cls, tasks, factory) in the models mapping::

    run_benchmark("social", {"MyGNN": (MyGNN, "node_cls")})

The default factory calls cls(in_channels, hidden_channels, out_channels); DGI-task models receive (in_channels, hidden_channels) (the third argument is dropped).

Functions:

Name Description
iter_benchmark_tasks

Flatten BENCHMARK_TASKS to a list, optionally filtered by category/task.
task_from_dataset

Wrap an already-loaded dataset as a Task.
register_task

Register task under category in BENCHMARK_TASKS.
unregister_task

Remove a previously registered task; returns it, or None if absent.
register_model

Register a model with the benchmark dispatcher.

Attributes:

Name Type Description
BENCHMARK_TASKS dict[str, dict[str, list[Task]]]

Curated benchmark taxonomy: category -> task_type -> [Task, ...].
TASK_TYPES frozenset[str]

BENCHMARK_TASKS module-attribute 

BENCHMARK_TASKS: dict[str, dict[str, list[Task]]] = {
    "combinatorial": {
        "link_pred": [
            Task(
                "random_tsp",
                "link_pred",
                lambda root, seed=0: combinatorial.random_tsp(
                    root, num_graphs=1, num_nodes=200, k=4, seed=seed
                ),
                epochs=80,
            ),
            Task(
                "random_coloring",
                "link_pred",
                lambda root, seed=0: combinatorial.random_coloring(
                    root, num_graphs=1, num_nodes=200, edge_prob=0.1, seed=seed
                ),
                epochs=80,
            ),
        ]
    },
    "biology": {
        "graph_cls": [
            Task("mutag", "graph_cls", biology.mutag, epochs=40),
            Task("proteins", "graph_cls", biology.proteins, epochs=20),
        ],
        "link_pred": [Task("celegans", "link_pred", biology.celegans, epochs=80)],
    },
    "social": {
        "node_cls": [
            Task("cora", "node_cls", social.cora, epochs=100),
            Task("citeseer", "node_cls", social.citeseer, epochs=100),
            Task("pubmed", "node_cls", social.pubmed, epochs=100),
            Task("roman_empire", "node_cls", social.roman_empire, epochs=80),
            Task("minesweeper", "node_cls", social.minesweeper, epochs=80),
        ],
        "link_pred": [
            Task("cora_link_pred", "link_pred", social.cora, epochs=80),
            Task("citeseer_link_pred", "link_pred", social.citeseer, epochs=80),
        ],
    },
    "knowledge": {
        "link_pred": [
            Task("fb15k_237", "link_pred", knowledge.fb15k_237, epochs=20),
            Task("wordnet18rr", "link_pred", knowledge.wordnet18rr, epochs=20),
        ]
    },
    "infrastructure": {
        "link_pred": [
            Task("power_grid", "link_pred", infrastructure.power_grid, epochs=80),
            Task("euroroad", "link_pred", infrastructure.euroroad, epochs=80),
        ]
    },
    "finance": {
        "link_pred": [
            Task("product_space", "link_pred", finance.product_space, epochs=80),
            Task("board_directors", "link_pred", finance.board_directors, epochs=40),
        ]
    },
    "computing": {
        "link_pred": [
            Task(
                "internet_as",
                "link_pred",
                lambda root: computing.internet_as(root),
                epochs=40,
            ),
            Task("topology", "link_pred", computing.topology, epochs=10),
        ]
    },
    "vision": {
        "graph_cls": [
            Task(
                "mnist_superpixels",
                "graph_cls",
                lambda root: vision.mnist_superpixels(root)[:1500],
                epochs=4,
            )
        ]
    },
    "physics": {
        "graph_reg": [
            Task(
                "zinc",
                "graph_reg",
                lambda root: (
                    physics.zinc(root, subset=True, split="train"),
                    physics.zinc(root, subset=True, split="val"),
                ),
                epochs=10,
            )
        ],
        "link_pred": [
            Task(
                "ising_lattice",
                "link_pred",
                lambda root, seed=0: physics.ising_lattice(
                    root, num_graphs=1, side=20, seed=seed
                ),
                epochs=60,
            )
        ],
    },
    "security": {
        "link_pred": [
            Task("terrorists_911", "link_pred", security.terrorists_911, epochs=120)
        ]
    },
}

Curated benchmark taxonomy: category -> task_type -> [Task, ...].

TASK_TYPES module-attribute 

TASK_TYPES: frozenset[str] = frozenset(
    {"node_cls", "graph_cls", "graph_reg", "link_pred"}
)

Task dataclass 

Task(name: str, task_type: str, loader: Callable[..., Any], epochs: int = 30)

A single benchmark task_type: a dataset loader plus its training task.

ModelSpec dataclass 

ModelSpec(
    cls: type,
    task_type: frozenset[str] = frozenset(),
    factory: Callable[..., Module] | None = None,
)

How to instantiate a model and which task tasks it supports.

iter_benchmark_tasks 

iter_benchmark_tasks(
    category: str | None = None, task_type: str | None = None
) -> list[Task]

Flatten BENCHMARK_TASKS to a list, optionally filtered by category/task.

Examples:

>>> [
...     t.name
...     for t in iter_benchmark_tasks(category="biology", task_type="graph_cls")
... ]
['mutag', 'proteins']
Source code in src/graphnetz/benchmark/_tasks.py 
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
def iter_benchmark_tasks(
    category: str | None = None,
    task_type: str | None = None,
) -> list[Task]:
    """Flatten ``BENCHMARK_TASKS`` to a list, optionally filtered by category/task.

    Examples
    --------
    >>> [
    ...     t.name
    ...     for t in iter_benchmark_tasks(category="biology", task_type="graph_cls")
    ... ]
    ['mutag', 'proteins']
    """
    cats = [category] if category is not None else list(BENCHMARK_TASKS)
    out: list[Task] = []
    for c in cats:
        per_cat = BENCHMARK_TASKS.get(c, {})
        tasks = [task_type] if task_type is not None else list(per_cat)
        for k in tasks:
            out.extend(per_cat.get(k, []))
    return out

task_from_dataset 

task_from_dataset(name: str, task_type: str, dataset: Any, *, epochs: int = 30) -> Task

Wrap an already-loaded dataset as a Task.

The dataset must satisfy the conventions for task: a PyG dataset or any object exposing ds[0] plus the relevant attributes (num_features / num_classes / num_relations). The benchmark dispatcher caches the dataset, so the same instance is reused across seeds without reloading.

Source code in src/graphnetz/benchmark/_tasks.py 
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
def task_from_dataset(
    name: str,
    task_type: str,
    dataset: Any,
    *,
    epochs: int = 30,
) -> Task:
    """Wrap an already-loaded dataset as a [`Task`][graphnetz.benchmark.Task].

    The dataset must satisfy the conventions for ``task``: a PyG dataset or
    any object exposing ``ds[0]`` plus the relevant attributes (``num_features``
    / ``num_classes`` / ``num_relations``). The benchmark dispatcher caches
    the dataset, so the same instance is reused across seeds without
    reloading.
    """
    if task_type not in TASK_TYPES:
        msg = f"Unknown task {task_type!r}; choices: {sorted(TASK_TYPES)}"
        raise ValueError(msg)
    return Task(name=name, task_type=task_type, loader=lambda _root: dataset, epochs=epochs)

register_task 

register_task(category: str, task_type: Task) -> None

Register task under category in BENCHMARK_TASKS.

The task becomes visible to run_benchmark(category) and to iter_benchmark_tasks. Use unregister_task to remove it (e.g. in tearDown of a test).

Source code in src/graphnetz/benchmark/_tasks.py 
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
def register_task(category: str, task_type: Task) -> None:
    """Register ``task`` under ``category`` in [`BENCHMARK_TASKS`][graphnetz.benchmark.BENCHMARK_TASKS].

    The task becomes visible to ``run_benchmark(category)`` and to
    [`iter_benchmark_tasks`][graphnetz.benchmark.iter_benchmark_tasks]. Use [`unregister_task`][graphnetz.benchmark.unregister_task] to remove it
    (e.g. in ``tearDown`` of a test).
    """
    if not isinstance(task_type, Task):
        msg = f"task must be a Task, got {type(task_type).__name__}"
        raise TypeError(msg)
    if task_type.task_type not in TASK_TYPES:
        msg = f"Task {task_type.name!r} has unknown task {task_type.task_type!r}; choices: {sorted(TASK_TYPES)}"
        raise ValueError(msg)
    per_cat = BENCHMARK_TASKS.setdefault(category, {})
    per = per_cat.setdefault(task_type.task_type, [])
    if any(t.name == task_type.name for t in per):
        msg = f"Task {task_type.name!r} already registered in category {category!r}/{task_type.task_type!r}"
        raise ValueError(msg)
    per.append(task_type)

unregister_task 

unregister_task(category: str, name: str) -> Task | None

Remove a previously registered task; returns it, or None if absent.

Source code in src/graphnetz/benchmark/_tasks.py 
225
226
227
228
229
230
231
232
def unregister_task(category: str, name: str) -> Task | None:
    """Remove a previously registered task; returns it, or ``None`` if absent."""
    per_cat = BENCHMARK_TASKS.get(category, {})
    for task_tasks in per_cat.values():
        for i, t in enumerate(task_tasks):
            if t.name == name:
                return task_tasks.pop(i)
    return None

register_model 

register_model(
    cls: type | None = None,
    *,
    task_type: str | Iterable[str],
    factory: Callable[..., Module] | None = None,
) -> Callable[[type], type] | type

Register a model with the benchmark dispatcher.

Usable as a decorator (@register_model(task_type="node_cls")) or as a plain function (register_model(MyGNN, task_type={"graph_cls", "graph_reg"})).

Source code in src/graphnetz/benchmark/_specs.py 
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
def register_model(
    cls: type | None = None,
    *,
    task_type: str | Iterable[str],
    factory: Callable[..., torch.nn.Module] | None = None,
) -> Callable[[type], type] | type:
    """Register a model with the benchmark dispatcher.

    Usable as a decorator (``@register_model(task_type="node_cls")``) or as a
    plain function (``register_model(MyGNN, task_type={"graph_cls", "graph_reg"})``).
    """
    tasks = frozenset({task_type} if isinstance(task_type, str) else task_type)
    unknown = tasks - TASK_TYPES
    if unknown:
        msg = f"Unknown task {sorted(unknown)}; allowed: {sorted(TASK_TYPES)}"
        raise ValueError(msg)

    def _register(target: type) -> type:
        _REGISTRY[target] = ModelSpec(cls=target, task_type=tasks, factory=factory)
        return target

    return _register(cls) if cls is not None else _register

Convenience plotting

benchmark

Statistically robust benchmarks across a category for one or many models.

The dispatcher trains every compatible (model, task) pair across multiple seeds and returns a BenchmarkReport that exposes mean ± 95 % t-CI, paired t-tests with Holm-Bonferroni correction, publication-ready LaTeX tables, and plots.

Custom models are plugged in via the same three paths as before:

  1. Decorator / registry::

    from graphnetz import register_model

    @register_model(task_type="node_cls") class MyGNN(torch.nn.Module): def init(self, in_channels, hidden_channels, out_channels): ...

  2. Class attribute::

    class MyGNN(torch.nn.Module): task_types = {"node_cls"}

  3. Inline tuple (cls, tasks) or (cls, tasks, factory) in the models mapping::

    run_benchmark("social", {"MyGNN": (MyGNN, "node_cls")})

The default factory calls cls(in_channels, hidden_channels, out_channels); DGI-task models receive (in_channels, hidden_channels) (the third argument is dropped).

Functions:

Name Description
plot_benchmark

Grouped bar chart with mean ± CI error bars.

plot_benchmark 

plot_benchmark(
    results: BenchmarkReport | Mapping[str, Mapping[str, Mapping[str, list[float]]]],
    errors: Mapping[str, Mapping[str, float]] | None = None,
    ax: Axes | None = None,
    title: str | None = None,
    annotate: bool = True,
    ci: float = 0.95,
) -> tuple[Figure, Axes]

Grouped bar chart with mean ± CI error bars.

Accepts a BenchmarkReport (preferred) or the legacy dict form for a single seed. errors overrides the default t-CI half-width.

Source code in src/graphnetz/benchmark/_runner.py 
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
def plot_benchmark(
    results: BenchmarkReport | Mapping[str, Mapping[str, Mapping[str, list[float]]]],
    errors: Mapping[str, Mapping[str, float]] | None = None,
    ax: plt.Axes | None = None,
    title: str | None = None,
    annotate: bool = True,
    ci: float = 0.95,
) -> tuple[plt.Figure, plt.Axes]:
    """Grouped bar chart with mean ± CI error bars.

    Accepts a [`BenchmarkReport`][graphnetz.benchmark.BenchmarkReport] (preferred) or the legacy dict form for
    a single seed. ``errors`` overrides the default t-CI half-width.
    """
    if isinstance(results, BenchmarkReport):
        return results.plot(ax=ax, title=title, annotate=annotate, ci=ci)

    set_plot_style()
    values: dict[str, dict[str, float]] = {}
    metric_label: str | None = None
    for task_name, per_task in results.items():
        per_value: dict[str, float] = {}
        for model_name, history in per_task.items():
            metric, value = _final_metric(history)
            metric_label = metric_label or metric
            per_value[model_name] = value
        values[task_name] = per_value

    return plot_grouped_bars(
        values,
        errors=errors,
        ax=ax,
        title=title,
        ylabel=metric_label or "metric",
        annotate=annotate,
    )