API¶

Launch¶

The launch topology: launcher, application, ranks.

nunatak run – mpirun -n 8 ./solver starts one launcher process where nunatak runs, and eight ranks wherever the scheduler placed them. This module is the only place that knows launchers. It splits the command into launcher prefix and application, so per-rank collection can be interposed between the two; it reads the rank identity that the MPI runtime writes into each rank’s environment; and the same split names the real target - solver, never mpirun - for the Run naming cascade and the binary inspection of doctor.

class nunatak.launch.LaunchPlan(prefix: tuple[str, ...], application: tuple[str, ...], mpi: bool, ranks: int | None)¶

A command split at the boundary the launcher cannot cross.

prefix runs once, here; application runs in every rank, there. An empty application states that no target could be resolved behind the launcher - there is nothing to wrap, and callers say so instead of guessing.

property target: str | None¶: The argv token of the profiled binary, None when unresolved.

wrap(shim: Sequence[str]) → list[str]¶

The same launch with shim interposed inside each rank.

mpirun -n 8 ./solver becomes mpirun -n 8 <shim> ./solver: the launcher still fans out, but each rank now starts in the shim, which is where per-rank collection happens.

nunatak.launch.split(command: Sequence[str]) → LaunchPlan¶: The LaunchPlan of command: prefix, application, MPI topology.

nunatak.launch.real_target(command: list[str]) → str | None¶: Return the argv token of the profiled binary, seen through launchers. Returns None when nothing resolves - callers fall back to the first token of the command.

class nunatak.launch.RankIdentity(rank: int, world_size: int | None = None, local_rank: int | None = None)¶

What one process knows about its place in MPI_COMM_WORLD.

local_rank is the rank’s index on its own node - what designates one sampled rank per node without any communication. None states the runtime did not publish the value, never that it is zero.

nunatak.launch.rank_identity(environment: Mapping[str, str]) → RankIdentity | None¶

The rank identity the MPI runtime wrote into environment.

None outside any rank - which is exactly how a process decides it is the orchestrator and not a rank.

The rank shim: what nunatak runs inside each MPI rank.

LaunchPlan.wrap interposes python -m nunatak.rank between the launcher and the application, so this module executes once per rank, on the rank’s node. It is the counting layer of the two-layer collection: one perf stat around the whole application - a few counters, constant cost, every rank - whose per-rank aggregates reveal the load imbalance that sampling a subset of ranks never could.

Artifacts land under <run>/collect/rank-<rank>/, and that is the multi-node retrieval mechanism: the Run directory lives on the shared filesystem, so each rank writing there before it exits brings the artifacts home before the job epilogue.

The shim depends on nothing but the standard library, stays silent on the application’s stdout and stderr, and propagates the application’s exit code: measurements observe, they never interfere.

nunatak.rank.measure(directory: Path, command: Sequence[str], environment: Mapping[str, str]) → int¶

Run command in this rank, counting around it, and return its exit code.

Outside any rank identity the command runs untouched and nothing is written: a shim that cannot say where it is must not invent a Locus. When perf is missing, or fails before launching the application - it opens its events first, so a missing or header-only CSV proves the application never ran - the application still runs bare and the rank meta says the rank went uncounted: a missing capability never prevents the run.

nunatak.rank.main(argv: Sequence[str] | None = None) → int¶: Entry point of python -m nunatak.rank.

Measured pivot¶

Domain model of the measured pivot.

The vocabulary is bound to the reference glossary (CONTEXT.md): the same words are used in the code, the interface and the documentation, and the terms it proscribes appear nowhere.

The pivot holds measured data only; the Diagnostic, the roofline placement and any aggregate across Loci are recomputed on demand and never persisted: storing a conclusion would freeze a question that has not been asked yet.

class nunatak.pivot.model.Quality(*values)¶

Confidence label of a Measurement or a Ceiling.

“measured” comes from a raw counter or a successful Calibration; “estimated” results from a motivated downgrade or a theoretical model; “unavailable” states that a missing quantity is not zero.

static worst(*qualities: Quality) → Quality¶

Propagate Quality along a lineage: the worst of the inputs.

The Quality of a derived metric is never set by hand: it is always computed from the Qualities of its inputs.

class nunatak.pivot.model.ResolutionLevel(*values)¶

How far the attribution of a Hotspot could go.

Distinct from Quality: when attribution fails, the Measurement stays exact - that time really was spent at that address - and it is the identity that degrades, not the value.

class nunatak.pivot.model.PhysicalIdentity(module_id: str, offset: int)¶

Physical identity of a native Hotspot: (build-id | LC_UUID, offset).

Aggregates inside a Run and validates cross-Pass merges: raw counters from two Passes may only combine when the module is identical in both. Only native code has one. No absolute address is ever stored: the offset is relative to the module, so ASLR and function reordering cannot split a Hotspot.

class nunatak.pivot.model.LogicalIdentity(module: str, name: str | None = None, source_file: str | None = None)¶

Logical identity of a Hotspot: (module, demangled name, source file).

Used for display, for the language model, and to compare Runs. The declaration line is an attribute of the Hotspot, never part of this key.

class nunatak.pivot.model.Hotspot(logical_identity: LogicalIdentity, resolution_level: ResolutionLevel, physical_identity: PhysicalIdentity | None = None, offset: int | None = None)¶

The atomic unit of analysis: CPU function, GPU kernel or Python frame.

offset is a display and disambiguation detail for unresolved Hotspots; lines and the inlining chain, internal details of the Hotspot, arrive with the attribution chain.

property display_name: str¶

The name shown to the user, module+0x3a1c when unresolved.

An address in a gap between symbols is never attached to the neighbouring symbol: naming the gap after its neighbour would be a confident lie.

class nunatak.pivot.model.Locus(node: str, rank: int | None = None, thread: int | None = None, device: int | None = None, stream: int | None = None)¶: A point of the execution topology: node > MPI rank > thread on CPU, node > device > stream on GPU.

class nunatak.pivot.model.Measurement(hotspot: Hotspot | None, locus: Locus, counter: str, value: float | None, unit: str, quality: Quality, reason: str | None = None, sample_count: int | None = None, coverage: float | None = None, pass_index: int = 0)¶

A raw-counter value for one (Hotspot, Locus) couple.

It carries what is needed to judge its own solidity: its number of samples, its coverage ratio when counters were multiplexed, and its Pass of origin. An estimated value always carries the reason of its downgrade; “unavailable” has no value, because unavailable is not zero.

A Measurement without a Hotspot is the counting layer’s: a whole-Locus aggregate - one rank’s time, cycles, instructions, MPI volumes - which has nothing to attribute. It is what reveals load imbalance on every rank at constant cost, while sampling covers a subset. Hotspot-grained computations read hotspot_level, never measurements raw.

property relative_error: float | None¶: Sampling error, decreasing in 1/sqrt(n).

nunatak.pivot.model.hotspot_level(measurements: Iterable[Measurement]) → list[Measurement]¶

The sampling layer’s Measurements: those attached to a Hotspot.

Shares, coverage and the statistical floor are Hotspot-grained; a Locus-level aggregate mixed into their totals would drown the sampled sums under whole-process counts.

nunatak.pivot.model.locus_level(measurements: Iterable[Measurement]) → list[Measurement]¶: The counting layer’s Measurements: whole-Locus aggregates.

class nunatak.pivot.model.InlineFrame(function: str, file: str | None = None, line: int | None = None, declaration_line: int | None = None)¶

One step of a persisted inlining chain: a name, a source position.

An inline frame is nothing but a line come from another file: it is an internal detail of the Hotspot, never a unit of analysis. line is where the sampled address falls, declaration_line where the function starts; both are None, and the file with them, when the module carried no debug information.

class nunatak.pivot.model.AddressDetail(hotspot: Hotspot, offset: int, counter: str, value: float, frames: tuple[InlineFrame, ...], sample_count: int | None = None, pass_index: int = 0)¶

The internal detail of a named Hotspot at one sampled address: the inlining chain seen there and the weight it carries.

Frames are innermost first, the physical function last. Weights are aggregated over Loci: the per-line view says where time goes inside a function, imbalance stays at the Measurement grain. This is what lets a report ventilate a Hotspot by line and by inline frame - and build the transverse per-inline-frame view, the only view stable across a recompilation - on a machine where the binary and the symbolizer no longer exist.

An embedded source extract for one named Hotspot - never a whole file: the body of the physical function and its hot inline frames, a few context lines around.

file is the path DWARF recorded (the identity), resolved_path where the text was actually read. When text is None, reason says why the source is absent - not found, ambiguous - so the report can say it instead of silently showing nothing. Embedding the text keeps the Run self-sufficient: it is readable in six months, on a machine where the source tree no longer exists, and its size stays bounded.

class nunatak.pivot.model.Event(locus: Locus, kind: str, name: str, start_ns: int, duration_ns: int, pass_index: int = 0, attributes: tuple[tuple[str, str], ...] = ())¶

A timestamped fact with a duration: GPU kernel launch, MPI call.

The Event stream feeds the report timeline and the network analysis; it is distinct from the aggregated Measurements.

class nunatak.pivot.model.Collector(tool: str, version: str)¶: An external tool orchestrated during a Pass, with its detected version.

class nunatak.pivot.model.Pass(index: int, exit_code: int, collectors: tuple[Collector, ...] = (), start: str | None = None, end: str | None = None)¶: One execution of the application within a single Run.

class nunatak.pivot.model.Ceiling(name: str, value: float, unit: str, quality: Quality, reason: str | None = None)¶

An upper performance bound of the Machine, reachable in practice.

Carries a Quality like a Measurement does: “measured” from a successful Calibration, “estimated” when theoretical or measured under suspicious conditions.

class nunatak.pivot.model.Allocation(visible_cores: int | None = None, affinity_mask: tuple[int, ...] | None = None, cpu_quota: float | None = None, memory_limit_bytes: int | None = None)¶

The share of the node this process actually received.

A Ceiling only holds for a given scope - the allocation’s - and only compares to Measurements aggregated over that same scope: a job given 8 cores of a 128-core node must not recycle the whole node’s Ceilings. visible_cores is what the affinity mask allows, affinity_mask the exact cores (their placement decides which caches and NUMA nodes the job sees), cpu_quota the cgroup limit in cores, memory_limit_bytes the cgroup memory cap; None states that a bound does not exist or cannot be observed on this platform.

class nunatak.pivot.model.Machine(system: str, kernel: str, architecture: str, cpu_model: str | None = None, logical_cores: int | None = None, allocation: Allocation = <factory>, ceilings: tuple[~nunatak.pivot.model.Ceiling, ...]=())¶

The hardware a Run executes on, carrier of the roofline Ceilings.

Its identity is not a node - the node is a Locus level - but a couple hardware + allocation shape: two jobs receiving different shares of the same node are two Machines, a thousand identical nodes of a cluster are one. Every Run embeds a complete snapshot of its Machine in its manifest, so any cache remains an optimization.

class nunatak.pivot.model.Provenance(commit: str | None = None, dirty_tree: bool | None = None, dependencies: dict[str, str]=<factory>, effective_configuration: dict[str, object]=<factory>)¶

What allows a Run to be explained without replaying it.

Best-effort and descriptive, never certifying: it records what it observes and never blocks a Run. The effective configuration includes the thresholds that drive Quality: a threshold can be tuned, it cannot be tuned silently.

class nunatak.pivot.model.Degradation(name: str, message: str, remedy: str | None = None)¶: A missing capability, named, announced before the run, with the way forward. Never a refusal: a missing capability removes measurements, it does not prevent the run.

class nunatak.pivot.model.Run(name: str, created: str, command: list[str], exit_code: int, machine: ~nunatak.pivot.model.Machine, provenance: ~nunatak.pivot.model.Provenance, passes: list[~nunatak.pivot.model.Pass] = <factory>, degradations: list[~nunatak.pivot.model.Degradation] = <factory>, measurements: list[~nunatak.pivot.model.Measurement] = <factory>, events: list[~nunatak.pivot.model.Event] = <factory>, address_details: list[~nunatak.pivot.model.AddressDetail] = <factory>, source_extracts: list[~nunatak.pivot.model.SourceExtract] = <factory>)¶

A profiling session: the persisted container of the measured pivot.

A Run is a single directory, whatever the number of ranks: a directory survives scp, archiving and being attached to a ticket. It contains no analysis output and no Explanation.

Persistence of a Run: Parquet for the measured pivot, JSON for the manifest.

The manifest is readable without nunatak - plain JSON carrying the complete Machine snapshot, the Provenance, the Passes, the effective configuration and the degradations. That is what makes a Run archivable at ten years.

nunatak.pivot.persistence.write_run(directory: Path, run: Run) → Path¶

Write run as a self-sufficient directory and return its path.

Parameters: directory is the Run directory (created if needed), run the in-memory Run. Layout: manifest.json at the root, columnar data under pivot/.

nunatak.pivot.persistence.machine_to_dict(machine: Machine) → dict¶: The plain-JSON form of a Machine, shared by the Run manifest and the profile cache so the two never drift apart.

nunatak.pivot.persistence.machine_from_dict(data: dict) → Machine¶

Rebuild a Machine from its plain-JSON form.

A snapshot written before the allocation shape existed reads back with an empty Allocation: absent is not zero.

nunatak.pivot.persistence.manifest(run: Run) → dict¶

The manifest content: plain JSON, readable without nunatak.

Also the trunk of the report payload, so the two never drift apart.

nunatak.pivot.persistence.read_run(directory: Path) → Run¶

Read a Run directory written by write_run and return the Run.

Raises ValueError when the directory is not a Run or was written by a newer schema than this version understands.

Collection¶

Collection orchestration: adapters around external collectors.

Every collector is executed as a subprocess and its output parsed, never linked: the product’s license and ABI stay decoupled from every tool’s. The single execution boundary lives in execution.py; it is what the corpus records and replays.

nunatak.collect.cpu_collector(executor: Executor, config: Config) → tuple[PerfAdapter | None, str | None]¶

The CPU collector usable in this environment: (adapter, version).

(None, None) when the tool is absent; (None, version) when the tool is present but the environment forbids sampling - a capability lost to permissions is a degradation, never a failed launch. The platform is the executor’s: a replayed Linux entry stays a Linux collection wherever it is replayed.

The execution boundary for external processes: exec + parse, never link.

Everything nunatak consumes from collectors crosses this boundary, which is why adapters are substitutable by a source of recordings: the corpus wraps or replaces the executor, never the adapters.

class nunatak.collect.execution.Invocation(argv: tuple[str, ...], exit_code: int, stdout: str | None = None, stderr: str | None = None)¶: The observable outcome of one external process.

class nunatak.collect.execution.Executor¶

Interface: run an external process and report what happened.

capture=False leaves stdout/stderr connected to the caller’s - the profiled application’s output is never swallowed.

property system: str¶: The platform tools run on; a replay reports the recorded one.

sampling_blocked() → str | None¶

Why event sampling cannot work in this environment, or None.

Every executor answers for its own environment: the real one reads the kernel setting, a recording keeps its inner verdict, and a replay reports the verdict the recording preserved.

run(argv: list[str], capture: bool = True, env: dict[str, str] | None = None, cwd: str | None = None) → Invocation¶

Run argv and report what happened.

capture=True collects stdout/stderr; False leaves them attached to the caller’s streams.

class nunatak.collect.execution.SubprocessExecutor¶

The real thing. A missing or non-executable program is reported with the reserved exit codes rather than an exception: an absent tool is an expected situation, not an error of nunatak.

sampling_blocked()¶

Report the kernel setting that denies sampling, if any.

Ubuntu ships kernel.perf_event_paranoid=4: perf_event_open is denied to unprivileged users even on their own processes.

run(argv, capture=True, env=None, cwd=None)¶: Execute argv as a subprocess.

perf adapter: Linux CPU collection by event-triggered sampling.

An adapter knows how to detect the presence and version of its tool, build its command line, execute it, and declare what it produces. It knows nothing about the pivot: parsing its outputs is the ingestion’s job, versioned by detected tool version.

class nunatak.collect.perf.PerfAdapter(path: str = 'perf')¶

Produces Measurements (no Events): sampled raw counters per Hotspot.

detect(executor: Executor) → str | None¶: Version of the tool, or None when it cannot run.

collect(command: list[str], directory: Path, executor: Executor, frequency: int, events: tuple = ()) → tuple[int, list[Degradation]]¶

Run command under perf record, then extract what nunatak consumes: the perf script text and the build-id list. Returns (application exit code, degradations); raw artifacts land under directory.

With a counter group, task-clock becomes the explicit time base (a software event, no hardware counter spent) and the group’s events ride along. perf validates events before launching the application, so a rejected group fails fast - no data file is written - and the recording is retried time-only: the application never runs twice.

Counter groups per microarchitecture: what sampling attributes to Hotspots beyond time.

Every auxiliary event carries a fixed period rather than a frequency: each sample is then worth exactly its period, so the sum of periods counts the events - validated at 99.9% of perf stat on the corpus machine - and the interrupt rate stays bounded by construction, which is the only overhead lever sampling has. The time base is task-clock, a software event that consumes no hardware counter and gives every Hotspot its seconds.

Hardware-prefetch fill events are deliberately absent: sampling them inflates what they measure - the interrupt handler’s own memory traffic triggers prefetches, an observer effect measured at 20x on Zen 2 - and doubles the run time. DRAM traffic is therefore demand fills only, and every Measurement built from it says so.

An unknown microarchitecture gets no counter group: the run samples time alone, exactly as before, and the roofline placement stays unavailable with its reason.

class nunatak.collect.events.SampledEvent(event: str, canonical: str, unit: str, scale: float = 1.0, quality: Quality = Quality.MEASURED, reason: str | None = None)¶

One vendor event sampled alongside time, and what it becomes in the pivot: a canonical counter name, a unit, a scale, and the Quality its Measurements deserve - estimated when the event is an honest proxy rather than the quantity itself.

property selector: str¶: The perf record -e argument, fixed period included.

nunatak.collect.events.sampling_events(machine: Machine, cpuinfo=None) → tuple[SampledEvent, ...]¶: The counter group for this Machine, empty when its microarchitecture has none - time-only sampling, never a guess.

nunatak.collect.events.canonical(counter: str) → SampledEvent | None¶: The pivot-side meaning of a sampled event name as perf script prints it (ls_refills_from_sys.ls_mabresp_lcl_dram/period=100003/u), None for anything that is not a mapped vendor event.

Recording corpus¶

Recording corpus: capture once on real hardware, replay forever without.

The corpus is captured, never hand-written: a hand-written corpus only tests the idea we have of tool outputs, never the real ones. nunatak produces it itself: RecordingExecutor wraps the real executor during a run on real hardware and saves every invocation crossing the execution boundary; ReplayExecutor substitutes those recordings for the real tools, which is what makes adapters substitutable by a source of recordings.

Entry layout:

entry/
  meta.json               command, platform, collectors, versions
  invocations/000.json    argv, exit code, whether output was captured
  invocations/000.stdout  raw captured output (absent when not captured)
  invocations/000.stderr

nunatak.corpus.write_meta(entry: Path, command: list[str], collectors: list[dict], sampling_blocked: str | None = None) → None¶

Describe a corpus entry: what ran, where, with which collectors.

sampling_blocked preserves the recording machine’s verdict, so a replay takes the same path the recording took: an entry captured where sampling was denied must not replay as if it were allowed - it would ask for collector invocations the entry never recorded.

nunatak.corpus.read_meta(entry: Path) → dict¶: Load the description of a corpus entry.

class nunatak.corpus.RecordingExecutor(inner: Executor, entry: Path)¶

Wraps the real executor and records every invocation, in order.

sampling_blocked()¶: The recording machine’s verdict: sampling happens on real hardware.

run(argv, capture=True, env=None, cwd=None)¶: Run through the wrapped executor and persist the invocation.

class nunatak.corpus.ReplayExecutor(entry: Path)¶

Substitutes recordings for the real tools.

Invocations are matched by program base name, in recorded order (one FIFO queue per program): absolute paths legitimately differ between the recording machine and the replaying one. A program the entry never recorded is reported absent - a corpus entry declares what existed.

property system: str¶: The recorded platform, not the replaying machine’s.

sampling_blocked()¶

The recording machine’s verdict, never the replaying one’s.

An entry captured where sampling was denied must replay down the same degraded path: deciding “allowed” here would ask for collector invocations the entry never recorded. Entries written before the verdict was kept read back as unblocked - the real corpus was captured with sampling working.

run(argv, capture=True, env=None, cwd=None)¶: Serve the next recording for this program instead of running it.

Ingestion¶

Ingestion: versioned parsers turning collector outputs into Measurements.

One parser per (tool, detected version) couple. This is where the normalization into module offsets happens: no absolute address crosses this boundary, so ASLR and library load order cannot split a Hotspot. A tool version the ingestion does not recognize is declared as a degradation rather than parsed blindly - the raw outputs stay in the Run for a later nunatak to ingest.

class nunatak.ingestion.Sample(pid: int, tid: int, time_s: float, period: int, counter: str, module: str, offset: int | None)¶: One sampling hit, already normalized to a module-relative offset.

nunatak.ingestion.ingest(tool: str, version: str, directory: Path, node: str) → tuple[list[Measurement], list[Degradation]]¶

Turn the raw artifacts of one collection into Measurements.

Returns the Measurements and the named degradations met on the way; an empty Measurement list with a degradation is a valid outcome.

nunatak.ingestion.measurements_from_samples(samples: list[Sample], module_ids: dict[str, str], node: str) → list[Measurement]¶

Aggregate samples into per-(Hotspot, Locus) Measurements.

Hotspots are unresolved at this stage - (module, offset), displayed module+0x… - and the attribution chain later gives them names. module_ids maps module paths to their build-id; a module without one gets no physical identity. Output is deterministic: sorted by decreasing value.

Parser for perf script sample lines, perf 6.4 and newer.

The adapter requests the explicit field list comm,pid,tid,time,period,event,ip,sym,symoff,dso,dsoff, which yields one line per sample:

workload  4013/4013  136.565152: 1003009 task-clock:ppp:  aaaadcb806d0 main+0x50 (/tmp/workload+0x6d0)

dsoff (the +0x6d0 after the module path, added in perf 6.4) is the module-relative offset the normalization requires; the absolute ip and perf’s own symbolization are deliberately discarded - attribution works from (module, offset) and nothing else.

nunatak.ingestion.perf_script.supports(version: str) → bool¶

Whether this parser understands the detected perf version.

The upper bound is open: new releases are vetted by replaying the corpus, not by refusing to run.

nunatak.ingestion.perf_script.parse_samples(text: str)¶

Parse sample lines into Sample objects.

Returns (samples, unparsed lines). A dso without a +0x offset (pseudo modules such as [vdso]) yields a sample with offset None: the module is known, the position inside it is not.

nunatak.ingestion.perf_script.parse_buildid_list(text: str) → dict[str, str]¶: Parse perf buildid-list output: <build-id> <module path> lines.

Parser for perf stat -x, CSV output - the counting layer’s format.

Fixtures in the tests are the verbatim output of perf 6.14.11 on an AMD EPYC 7702: value, unit, event, counter runtime in nanoseconds, coverage percentage, then derived columns this parser ignores. A -o file opens with a # started on … comment. <not supported> in the value column states the counter does not exist on this machine - which is a fact to keep, never a zero.

class nunatak.ingestion.perf_stat.Count(counter: str, value: float | None, unit: str, coverage: float | None)¶

One counted event: a value over the whole process, or its absence.

coverage is time_running / time_enabled as a fraction; counting three events on hardware with more counters than that does not multiplex, but the column is authoritative, not our assumption.

nunatak.ingestion.perf_stat.parse(text: str) → tuple[list[Count], list[str]]¶

Parse perf stat -x, CSV into Counts.

Returns the Counts and the lines that did not parse - the caller turns those into a named degradation instead of guessing.

Ingestion of the counting layer: per-rank aggregates into the pivot.

Each rank directory under collect/ was written by the rank shim on the rank’s own node: a rank.json identity and, when perf was there, a perf stat CSV. The Measurements produced here are Locus-level - one value per (node, rank), no Hotspot: the counting layer has nothing to attribute, and that is what lets it cover every rank at constant cost.

nunatak.ingestion.rank_counting.ingest_counting(directory: Path) → tuple[list[Measurement], list[Degradation]]¶

Turn the rank directories under directory into Measurements.

Returns an empty list for a Run without ranks - a single-process Run has no counting layer, which is not a degradation. Uncounted ranks and ranks the world size announces but that left no artifacts are each declared once, with the rank numbers: silence about a missing rank would read as “nothing ran there”.

Attribution¶

Attribution: from module-relative addresses to named Hotspots.

Where the measured pivot records that time was spent at (module, offset), attribution says what that address is: the physical function, its source position, and the frames inlined into it - kept as internal detail of the Hotspot, never as units of analysis. It works on the distinct addresses left by the aggregation, never on the sample stream.

When attribution fails, the Measurement stays exact - that time really was spent at that address - and it is the identity that degrades, not the value: an unresolved Hotspot keeps its module+0x… display and its measured numbers.

class nunatak.attribution.AttributionChain(frames: tuple[Frame, ...] = ())¶

What symbolization established for one module-relative address.

Frames are ordered innermost first: the last one is the physical function - the thing with a symbol, an extent and an address - and the frames before it were inlined into it. An empty chain states that no symbol covers the address.

property physical: Frame | None¶: The physical function the address belongs to, None when no symbol covers it.

property resolution_level: ResolutionLevel¶

How far this chain goes: LINE with a source position, FUNCTION with a bare name, UNRESOLVED when no symbol covers the address.

The symbolizer output cannot tell .symtab from a .dynsym-only module, so the SYMBOL level never originates here: refining FUNCTION into SYMBOL requires inspecting the module’s sections.

class nunatak.attribution.Frame(function: str, file: str | None = None, line: int | None = None, declaration_line: int | None = None, start_address: int | None = None)¶

One step of an attribution chain: a demangled name, a source position.

line is where the address falls, declaration_line where the function starts; both are None when the module has no debug information, and the file with them. start_address is the symbol’s st_value - the module-relative start of the physical function, carried only by the outermost frame - which keys the function-grain physical identity.

class nunatak.attribution.ModuleSymbolization(chains: dict[int, ~nunatak.attribution.symbolizer.AttributionChain]=<factory>, error: str | None = None)¶: The outcome for one module: chains keyed by offset, and the reason when the module could not be read at all.

class nunatak.attribution.Symbolizer(path: str, major: int)¶

One usable llvm-symbolizer: an invoked path and its major version.

symbolize(executor: Executor, module: str, offsets: list[int]) → ModuleSymbolization¶

Resolve offsets (module-relative virtual addresses) in module.

One batched invocation: llvm-symbolizer takes every address on the command line and answers with one JSON entry per address, so the chains come back keyed by the offset each entry names.

nunatak.attribution.attribute(measurements: list[Measurement], symbolizer: Symbolizer | None, executor: Executor) → tuple[list[Measurement], list[AddressDetail], list[Degradation]]¶

Name the unresolved Hotspots of measurements.

Returns the new Measurements, the internal detail of the named Hotspots - the inlining chain and the weight of each sampled address, what a report needs to ventilate a Hotspot by line on a machine where the binary no longer exists - and the degradations met on the way.

Symbolization runs once per module, on its distinct sampled offsets; modules that only yield bare names are then inspected once to tell function (a .symtab name) from symbol (a .dynsym-only module). Without a symbolizer the Measurements come back untouched: doctor has already announced the missing capability. A module the symbolizer cannot read leaves its Hotspots unresolved and is declared, once, in a degradation.

nunatak.attribution.locate(executor: Executor, config: Config) → Symbolizer | None¶

The first candidate that actually runs, with its major version.

An old LLVM is still returned: the caller decides what its age implies, doctor turning it into a warning or a degradation.

llvm-symbolizer driver: module-relative addresses to attribution chains.

The nominal symbolization path is LLVM, declared as an external dependency and never bundled. Probing must invoke the candidates, because finding a file proves nothing: Homebrew’s llvm formula is keg-only, hence never on PATH, and Linux distributions install versioned, unlinked binaries.

Symbolization only ever runs on the set of distinct addresses left by the aggregation - a few thousand in practice - so its cost is one short invocation per module. The extent rule comes with the tool: llvm-symbolizer names an address only when it falls inside [st_value, st_value + st_size) of a symbol, and an address in a gap between symbols comes back empty rather than attached to its neighbour.

class nunatak.attribution.symbolizer.Frame(function: str, file: str | None = None, line: int | None = None, declaration_line: int | None = None, start_address: int | None = None)¶

One step of an attribution chain: a demangled name, a source position.

class nunatak.attribution.symbolizer.AttributionChain(frames: tuple[Frame, ...] = ())¶

What symbolization established for one module-relative address.

property physical: Frame | None¶: The physical function the address belongs to, None when no symbol covers it.

property resolution_level: ResolutionLevel¶

How far this chain goes: LINE with a source position, FUNCTION with a bare name, UNRESOLVED when no symbol covers the address.

The symbolizer output cannot tell .symtab from a .dynsym-only module, so the SYMBOL level never originates here: refining FUNCTION into SYMBOL requires inspecting the module’s sections.

class nunatak.attribution.symbolizer.ModuleSymbolization(chains: dict[int, ~nunatak.attribution.symbolizer.AttributionChain]=<factory>, error: str | None = None)¶: The outcome for one module: chains keyed by offset, and the reason when the module could not be read at all.

class nunatak.attribution.symbolizer.Symbolizer(path: str, major: int)¶

One usable llvm-symbolizer: an invoked path and its major version.

symbolize(executor: Executor, module: str, offsets: list[int]) → ModuleSymbolization¶

Resolve offsets (module-relative virtual addresses) in module.

One batched invocation: llvm-symbolizer takes every address on the command line and answers with one JSON entry per address, so the chains come back keyed by the offset each entry names.

nunatak.attribution.symbolizer.candidate_paths(config: Config) → list[str]¶

Paths worth probing for llvm-symbolizer.

PATH alone cannot be trusted: Homebrew’s llvm formula is keg-only, and Linux distributions install versioned, unlinked binaries.

nunatak.attribution.symbolizer.locate(executor: Executor, config: Config) → Symbolizer | None¶

The first candidate that actually runs, with its major version.

An old LLVM is still returned: the caller decides what its age implies, doctor turning it into a warning or a degradation.

Module inspection: which symbol and debug sections an object offers.

The symbolizer output alone cannot tell a .symtab name from a .dynsym-only one, yet the difference is what separates the function resolution level from symbol - and what tells the user whether the way forward is a debuginfo package or a -g recompile. The inventory comes from the llvm-readelf sitting next to the located llvm-symbolizer: same install, same version.

class nunatak.attribution.inspection.ModuleSections(symtab: bool, dynsym: bool, debug_info: bool)¶: Presence of the sections that grade attribution.

nunatak.attribution.inspection.readelf_path(symbolizer_path: str) → str¶: The llvm-readelf sibling of a located llvm-symbolizer, honoring versioned basenames like llvm-symbolizer-19.

nunatak.attribution.inspection.inspect(executor: Executor, readelf: str, module: str) → ModuleSections | None¶: Section inventory of module, or None when it cannot be established - llvm-readelf missing, module unreadable. The caller then keeps the level the symbolizer output already justified rather than guessing.

Source resolution and extraction: from persisted chains to code extracts.

A file named by DWARF is searched in three steps: the recorded path as it is, then the user-supplied source map (/build/x=/home/me/x), then a basename search under the repository root or the working directory. On multiple ambiguous matches nunatak does not choose: the Hotspot stays without source, with the reason - a wrong file shown confidently would be worse than none.

Only the necessary extracts are embedded in the Run, never whole files: the body of the physical function and its hot inline frames, a few context lines around. The true end of a function is not in the line table, so an extract runs from the earliest declaration to the last sampled line, plus context, and its size is capped with the truncation marked.

nunatak.attribution.source.extract(details: list[AddressDetail], source_map: dict[str, str], root: Path, checksums: dict[str, str] | None = None) → list[SourceExtract]¶

The source extracts of the named Hotspots in details.

One extract per (Hotspot, file), from a few lines before the earliest declaration to a few lines after the last sampled line. A file that cannot be resolved yields an extract without text, carrying the reason. checksums maps DWARF paths to their line-table MD5: a resolved file that disagrees with its fingerprint is neither shown nor embedded - the developer edited it since the build, and its line numbers no longer mean anything.

Staleness guard: DWARF 5 line-table checksums against the files on disk.

A developer who edited a file since the profiled binary was built would otherwise read a report pointing at lines that moved. The line table can carry an MD5 fingerprint of each source file as the compiler read it (clang emits one by default, gcc emits none): when the fingerprint is present and disagrees with the resolved file, the source is neither shown nor sent, and the extract says why. Absent fingerprint, no verdict - the refusal requires a present and discordant fingerprint, never a guess.

The fingerprints come from the llvm-dwarfdump sitting next to the located llvm-symbolizer: same install, same version.

nunatak.attribution.staleness.dwarfdump_path(symbolizer_path: str) → str¶: The llvm-dwarfdump sibling of a located llvm-symbolizer, honoring versioned basenames like llvm-symbolizer-19.

nunatak.attribution.staleness.line_table_checksums(executor: Executor, dwarfdump: str, module: str) → dict[str, str]¶

The {source path: MD5} fingerprints of module’s line tables.

Empty when the tool cannot run, the module has no line table, or the compiler emitted no checksums: an empty verdict verifies nothing and forbids nothing.

nunatak.attribution.staleness.checksums_for(executor: Executor, dwarfdump: str, details: list[AddressDetail]) → dict[str, str]¶: The line-table fingerprints of every module whose Hotspots carry source positions - the only ones an extract will be read for.

Machine¶

Machine snapshot, identity and profile cache.

A Machine is not a node: its identity is the couple hardware plus allocation shape, which is what makes Ceilings comparable to Measurements aggregated over the same scope. The profile is cached under $XDG_CACHE_HOME/nunatak/machines/ and reused across Runs; every Run still embeds a complete snapshot in its manifest, so the cache can disappear without any Run losing anything.

The identification fields and the allocation shape are captured today; the Ceilings will be produced by the Calibration, and the cache is what will keep it from replaying on every Run.

nunatak.machine.allocation(proc_self: Path = PosixPath('/proc/self/cgroup'), cgroup_root: Path = PosixPath('/sys/fs/cgroup')) → Allocation¶

The share of the node this process actually received.

The affinity mask is the ground truth of what a batch scheduler granted; the cgroup limits catch what affinity does not (CPU bandwidth quotas, memory caps). Platforms without either report None: an unobservable bound is not an absent one.

nunatak.machine.allocated_cores(machine: Machine) → float | None¶: The cores this job may actually burn: the affinity count, further capped by the cgroup quota when one exists.

nunatak.machine.snapshot(executor: Executor) → Machine¶: Best-effort description of the hardware this process runs on and of the share of it this process received.

nunatak.machine.identity(machine: Machine) → str¶

Canonical fingerprint of a Machine: hardware plus allocation shape.

A thousand identical nodes share one identity; two jobs receiving different shares of one node get two. The kernel release is left out on purpose - a kernel update does not change what the silicon can reach - and the Ceilings are the profile’s content, never its key.

nunatak.machine.cache_directory() → Path¶: Where Machine profiles live: $XDG_CACHE_HOME/nunatak/machines.

nunatak.machine.store(machine: Machine, directory: Path | None = None) → Path¶

Cache machine’s profile under its identity and return the path.

The kernel version rides along: a profile measured by other calibration kernels is stale by definition and will not be loaded.

nunatak.machine.load(machine: Machine, directory: Path | None = None) → Machine | None¶: The cached profile of machine’s identity, None when there is none, when it is unreadable, or when it was measured by another kernel version.

Calibration¶

Calibration: the operation that produces the Ceilings of a Machine.

The nominal path runs microbenchmarks on the target - a Ceiling is the maximum of its repetitions, never their mean, because a roofline only makes sense against a bound that is reachable in practice. The theoretical table is the last rung of the fallback ladder: it never produces anything better than an estimated Ceiling, and it never extrapolates - an unknown microarchitecture yields no Ceiling at all.

nunatak.calibration.merged_ceilings(measured: tuple[Ceiling, ...], theoretical: tuple[Ceiling, ...]) → tuple[Ceiling, ...]¶: One set of Ceilings for the Machine: what was measured wins, the theoretical table fills whatever stayed unmeasured - a partial calibration still deserves a complete roofline.

Theoretical peaks per microarchitecture: the last rung of the ladder.

The table holds what a core can retire per cycle - a few dozen entries that evolve slowly - and is crossed with what the system exposes at run time: the microarchitecture identifiers, the frequency, and the allocation shape. A theoretical peak is systematically unreachable in practice (turbo, throttling, cgroup limits), so it only ever yields Ceilings of Quality estimated, with the reason.

An unknown microarchitecture yields no Ceiling, never an extrapolation: a wrong ceiling labeled estimated still bends every classification built on it. Memory bandwidth has no theoretical entry at all - it depends on the DIMM population, which nothing exposes reliably - and only exists once the calibration kernel has measured it.

class nunatak.calibration.theory.MicroArchitecture(name: str, dp_flops_per_cycle: float, sp_flops_per_cycle: float)¶

What one core can retire per cycle, an FMA counting as two FLOPs.

Values are the microarchitecture’s maximum (the widest SKU): a theoretical Ceiling is an upper bound, and a lower-tier SKU under it is exactly what the estimated Quality announces.

nunatak.calibration.theory.x86_microarchitecture(text: str) → MicroArchitecture | None¶: The table entry for an x86 /proc/cpuinfo, None when not listed.

nunatak.calibration.theory.arm_microarchitecture(text: str) → MicroArchitecture | None¶: The table entry for an aarch64 /proc/cpuinfo, None when not listed.

nunatak.calibration.theory.detect(machine: Machine, cpuinfo: Path = PosixPath('/proc/cpuinfo')) → MicroArchitecture | None¶: The microarchitecture this Machine runs on, None when the table does not list it - in which case no Ceiling is produced, ever.

nunatak.calibration.theory.frequency(maximum: Path = PosixPath('/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq'), cpuinfo: Path = PosixPath('/proc/cpuinfo')) → tuple[float, str | None] | None¶

The core frequency in Hz and an honesty caveat, None when nothing is exposed.

In order: the rated maximum from cpufreq, the frequency printed in the brand string, and finally the highest currently observed frequency - the latter is not a rated maximum, and the caveat says so in the Ceiling’s reason.

nunatak.calibration.theory.ceilings(machine: Machine, microarchitecture: MicroArchitecture | None, clock: tuple[float, str | None] | None) → tuple[Ceiling, ...]¶: The estimated FLOP/s Ceilings of machine, scaled to its allocation; empty when the microarchitecture, the frequency or the core count is unknown - absence, never extrapolation.

nunatak.calibration.theory.theoretical_ceilings(machine: Machine) → tuple[Ceiling, ...]¶: The estimated Ceilings of the Machine this process runs on.

Calibration kernel driver: build locally, run autonomously, keep the max.

The embedded C sources are compiled with whatever compiler the machine offers - there is always a compiler on a cluster - cached next to the Machine profiles, and run as a separate process through the execution boundary, so a calibration records and replays like any collector. A Ceiling is the maximum of its repetitions, never their mean: the goal is an upper bound. Pollution signals (dispersion between repetitions, concurrent load, a kernel built without SIMD, a value above the theoretical peak) downgrade the Ceiling to estimated with the reason, they never discard it.

class nunatak.calibration.kernel.KernelRun(kernel: str, isa: str | None, threads: int | None, load: float | None, rates: tuple[float, ...])¶: Parsed output of one kernel invocation: the self-reported build ISA and load average travel with the rates, so a replayed calibration sees the same pollution signals as the recorded one.

nunatak.calibration.kernel.compiler(executor: Executor, config: Config) → str | None¶

The first C compiler that answers, None when the machine has none.

tools.cc in the configuration wins, then the conventional names.

nunatak.calibration.kernel.build(executor: Executor, cc: str, directory: Path) → Path | None¶

Compile the embedded kernel into directory, reusing a previous build of the same kernel version.

-march=native first; Apple’s clang on arm64 only accepts -mcpu=native, hence the second attempt.

nunatak.calibration.kernel.parse(stdout: str) → KernelRun | None¶: Parse the kernel’s output lines, None when they are not its.

nunatak.calibration.kernel.measure(executor: Executor, binary: Path, kernel: str, threads: int, repetitions: int = 5, milliseconds: int = 300) → KernelRun | None¶: One autonomous kernel process, its output parsed; None on failure.

nunatak.calibration.kernel.calibrate(executor: Executor, machine: Machine, config: Config, directory: Path | None = None, budget_seconds: float = 60.0, theoretical: dict[str, float] | None=None, clock=<built-in function monotonic>) → tuple[Ceiling, ...]¶

Measure the Ceilings of machine, in priority order, within the budget.

Returns the measured Ceilings only - possibly none, when no compiler exists or the kernel cannot be built: the theoretical table remains the caller’s fallback for whatever is missing. theoretical maps Ceiling names to the table’s peaks, so a measurement above its theoretical bound is caught and downgraded.

Analysis¶

Deterministic analysis engine: a pure function of (pivot, Machine).

Nothing here is persisted - the Diagnostic is recomputed on demand from the measured pivot and the Machine snapshot, which is what keeps a Run analyzable years later and the engine testable without hardware. Its reproducibility is the counterpart of the non-reproducible Explanation: these are the facts.

Every derived quantity carries its lineage and its Quality, propagated as the worst of its inputs and never short-circuited: a number displayed measured is measured end to end. Where a source counter does not exist, the fact is unavailable with the reason, rather than approximate: unavailable is not zero.

class nunatak.analysis.Derived(name: str, value: float | None, unit: str, quality: Quality, lineage: tuple[str, ...] = (), formula: str | None = None, reason: str | None = None)¶

A quantity computed from raw counters and Ceilings by a formula.

It remembers where it came from: lineage names its inputs, formula states the computation, and its Quality is the worst of its inputs - never set by hand. unavailable carries no value and always says why.

class nunatak.analysis.Diagnostic(hotspot: Hotspot, share: Derived, dram_intensity: Derived, achieved: Derived, attainable: Derived, envelope_fraction: Derived, imbalance: Derived, classification: str | None, classification_reason: str | None = None)¶

The deterministic verdict for one Hotspot: its share of the run, its roofline placement, and the regime it states.

A classification states a regime, never a cause; when the placement cannot be computed, classification is None and classification_reason says why.

nunatak.analysis.envelope(intensity: float, ceilings: dict[str, Ceiling]) → tuple[float, tuple[str, ...], Quality, str | None] | None¶

The roofline: min(compute peak, bandwidth x intensity).

The memory diagonal stops at the break point, it never crosses it - that formula is a testable invariant. Returns (value, lineage, quality, downgrade reason), or None when either Ceiling is missing.

nunatak.analysis.time_base(run: Run) → str | None¶: The counter Hotspot shares of time are stated against: the first clock this Run measured, cycles as last resort.

nunatak.analysis.diagnose(run: Run, floor_samples: int = 30, latency_fraction: float = 0.5, imbalance_ratio: float = 2.0) → list[Diagnostic]¶

The Diagnostics of every Hotspot above the statistical floor, ordered by decreasing share.

Placement aggregates the Measurements of all Loci before comparing: same scope on both sides, or one rank’s performance would face one node’s Ceiling. The wall time of a Hotspot is approximated by its most-loaded Locus - Loci run concurrently.

Summary¶

Terminal summary: the first reading level of the report.

At the end of run, the log restates what the report’s synthesis will say, in the same vocabulary: findings ordered by decreasing share of the sampled time, each with its quantified evidence, then “what this report does not say”. Whoever reads only the job log learns fewer details than the report shows, never less about how solid the numbers are: a downgraded value states its reason, an absent quantity is written unavailable - never zero, never a blank - and what is missing is gathered in one named section instead of scattered across footnotes.

nunatak.summary.summarize(run: Run, diagnostics: list[Diagnostic], floor_samples: int = 30) → list[str]¶

The lines of the terminal summary, findings first, admissions last.

diagnostics is the output of analysis.diagnose(run), already ordered by decreasing share; floor_samples must be the floor that produced it, so the admissions name the threshold actually applied. Returns plain lines: the Console decides how they reach the log.

Report¶

The HTML report: what the user reads, built from what was measured.

The report is a self-contained page rendered by a compiled TypeScript mini-app; everything the app shows crosses one boundary, the payload - plain JSON recomputed on demand from the Run and its Diagnostics.

nunatak.report.build(run: Run, diagnostics: list[Diagnostic], floor_samples: int = 30) → dict¶

The complete report payload for one Run.

diagnostics is the output of analysis.diagnose(run), already ordered by decreasing share; floor_samples must be the floor that produced it, so the report names the threshold actually applied. Returns a JSON-serializable dict; its trunk is the Run manifest, so the report and the Run directory never drift apart.

Report payload: everything the report shows, as plain JSON data.

The payload is the contract between the Python core and the report’s TypeScript mini-app: the app renders it, it never computes it. Like the Diagnostic it embeds, the payload is recomputed on demand and never persisted - the Run directory stays measured data only.

It carries data, not prose: values keep their unit, their Quality and their downgrade reason, absent quantities stay null next to the reason, and the app owns the wording - in the same vocabulary as the terminal summary, which derives from the same Diagnostics.

nunatak.report.payload.build(run: Run, diagnostics: list[Diagnostic], floor_samples: int = 30) → dict¶

The complete report payload for one Run.

nunatak.report.payload.without_source(payload: dict) → dict¶

The –no-source variant of a payload: the code text is withheld, the line numbers and the sample distribution stay.

A page-side toggle would be a trap: the text would remain embedded in the file it claims to hide. The variant is produced here, before the page exists, so what leaves the machine never contained a line of code. Returns a new payload; the input is not modified.

Self-contained HTML: the payload and the compiled mini-app in one file.

The page makes no external request - no CDN, no font, no telemetry - so it opens on a cluster without a server and still reads in ten years from an archived file. The compiled bundle is built from report-app/ into this package’s assets/ directory; a checkout that never built it - or a wheel packaged without it - loses the report as a named degradation, never as a crash.

nunatak.report.html.assets_available(assets_dir: Path | None = None) → bool¶: Whether the compiled mini-app is present in this installation.

nunatak.report.html.render(payload: dict, assets_dir: Path | None = None) → str¶

The complete report page for one payload.

The payload is embedded as a JSON island the app reads at load time; </ is escaped so no payload content can close the script element. Assembled by concatenation: a format string would trip on the braces of the inlined CSS and JavaScript.

nunatak.report.html.write_report(directory: Path, run: Run, diagnostics: list[Diagnostic], assets_dir: Path | None = None, no_source: bool = False) → Path¶

Render the report of run into its directory and return its path.

The report is a product of the Run, regenerable at will: writing it next to the pivot keeps the directory self-sufficient without ever making the pivot depend on it. no_source produces the shareable variant under its own name - it never replaces the full report - and the payload is stripped before the page exists, so that file never contained a line of code.

Configuration¶

Configuration: three layers with increasing precedence - site, project, command-line flags.

TOML format, nunatak.toml at the repository root - never inside pyproject.toml, the profiled application being rarely written in Python. The effective configuration is recorded in the Provenance, thresholds included: a threshold can be tuned, it cannot be tuned silently.

class nunatak.config.Config(project_name: str | None = None, runs_dir: str = '.nunatak', tools: dict[str, str]=<factory>, source_map: dict[str, str]=<factory>, coverage_threshold: float = 0.8, sampling_frequency: int = 997)¶: Resolved configuration after the cascade.

nunatak.config.find_project_config(cwd: Path) → Path | None¶: Walk up from cwd to the filesystem root looking for nunatak.toml.

nunatak.config.load(cwd: Path, name: str | None = None, site_config: Path | None = None) → tuple[Config, dict[str, object]]¶

Load the configuration cascade and return (config, effective).

cwd anchors the project-file search, name is the –name flag (always winning), site_config overrides the site file location (used by tests; defaults to /etc/nunatak.toml, or $NUNATAK_SITE_CONFIG). effective is the flat mapping recorded in the Provenance.