Architecture

dag-ml-data is the data contract and planning layer. It describes what data is available and how it can be converted, but it does not decide ML phases or enforce OOF invariants.

Layers

Layer	Owned here	Not owned here
Schema	dataset id, sample ids, sources, targets, metadata	graph nodes and scheduler
Representation	axes, units, containers, dtype, ragged/sparse flags	model execution
Views	immutable selectors over samples/sources/columns	fold construction
Planning	materialize/adapt/align/join/collate data plans	selecting variants or models
Fingerprints	deterministic schema hashes for replay	artifact lineage graph
ABI	host data-provider vtable and schema helpers	controller/model ABI

Crates

Crate	Responsibility
dag-ml-data-core	Pure Rust serializable contracts and validation.
dag-ml-data	Stable Rust facade for downstream crates and bindings.
dag-ml-data-capi	C ABI helper functions and data provider vtable contracts.
dag-ml-data-cli	Local schema validation and fingerprint utility.

Data Planning Flow

DatasetSchema + ModelInputSpec + policy
  -> resolve representation path
  -> DataPlan(materialize -> adapt* -> align -> join -> collate)
  -> execute_fit / execute_transform in a host data provider
  -> host-owned data handle returned to dag-ml

The current scaffold reaches validated handles/views, source-scoped materialization relations, a typed numeric feature-buffer arena with manifests, data-handle-scoped feature-buffer bindings, and Arrow C Data smokes for identity, sample-level targets and observation-level features. Runtime adapters and production host buffer arenas are still outside the scaffold.

Boundary With dag-ml

dag-ml-data may expose:

• source descriptors and semantic axes;

• identity, group and origin relations;

• data plans and fitted adapter references;

• schema fingerprints;

• host handles for materialized views.

It must not expose:

• fold sets as a planning primitive;

• OOF prediction blocks;

• graph lineage records;

• model selection decisions.