# Capability Matrix

`dag-ml-data` supports the replacement of the current `nirs4all` data pipeline
surface by making data shape, identity and conversion explicit. It does not own
OOF or leakage decisions, but it exposes enough information for `dag-ml` to
enforce them and can validate externally supplied fold assignments against the
data identities it owns.

## Data Surface

| Capability | Data contract support | Enforcement owner |
|---|---|---|
| Multisource | `DatasetSchema`, `SourceDescriptor`, alignment policy, presence masks, planner-visible `Align` steps | `dag-ml` decides phase and accepted fusion policy |
| Repetitions | `SampleRelation` with observation/sample/target/group/origin ids plus optional `GroupSpec`/`FoldSpec` declarations and `FoldSet` validation helpers | `dag-ml` chooses split unit and aggregation; `dag-ml-data` rejects malformed/leaking fold contracts |
| Grouped samples | group ids exposed through sample relations and schema-level `GroupSpec` | `dag-ml` builds group-aware folds; `dag-ml-data` validates supplied fold boundaries |
| Augmentation | augmentation adapters declare output origin ids and optional `AugmentationMetadata` | `dag-ml` decides train-only use; `dag-ml-data` validates origin boundaries for supplied folds |
| Signal type | `RepresentationSpec.signal_type` records absorbance/reflectance/transmittance/log-reflectance/preprocessed/unknown | host loaders detect, `dag-ml-data` validates materialize/predict contracts |
| Shape contracts | `SourceDescriptor.shape_contract` pins rank and named axis sizes | providers must refuse materialization when source payload shape violates the contract |
| Metadata | `MetadataSchema` declares typed metadata fields and categorical vocabularies | bridge maps nirs4all metadata columns and refuses missing/invalid required fields |
| Multi-targets | `CoordinatorMultiTargetBlock` aligns multiple target tables on one sample axis and the C ABI exports nullable per-target Arrow columns | `dag-ml` decides target use per phase; providers preserve missing-target validity masks |
| Processings | representation adapters, fit scope, fitted adapter refs | `dag-ml` chooses fold/full-train scope |
| Splits | identity/group/origin inputs, JSON `FoldSet` validators and canonical fold fingerprints in Rust/Python/WASM/C ABI for exhaustive partition-style folds | `dag-ml` builds folds; `dag-ml-data` validates and fingerprints supplied folds |
| Models | `ModelInputSpec`, accepted representations/types, aux inputs | controller and `dag-ml` execute model phases |
| Refit | serialized `DataPlan`, schema fingerprints, fitted refs | `dag-ml` controls replay/refit phase |
| Branching | immutable views and source filters | `dag-ml` owns branch graph semantics |
| Merging | alignment, feature join, source join, repetition-preserving broadcast, null-filled missing sources, executable numeric collation contracts | `dag-ml` validates prediction joins and downstream use |
| Concatenation | namespace columns by default, duplicate-column refusal when unnamespaced, presence indicators, output representation | `dag-ml` decides whether the merge is legal in phase |
| Finetuning | stateful/supervised adapter declarations | `dag-ml` enforces fold-train fit boundaries |
| Generation | serializable adapter params and plugin versions | `dag-ml` owns variant enumeration |
| Tuning | dry-run shapes and deterministic data plans | `dag-ml` owns tuner phase and nested CV |

## Contract Requirements

1. Every source has stable sample identity.
2. Every representation carries semantic axes.
3. Every conversion path is explicit, costed, versioned and deterministic.
4. Lossy/stateful/supervised adapters are opt-in at planning time.
5. Presence masks and alignment choices are serializable and planned before
   multi-source joins.
6. Schema fingerprints are stable under irrelevant ordering changes.
7. No fold, OOF or prediction partition decision is made in this repo; supplied
   partition-style fold sets can be validated against relation group/origin
   boundaries.