Status

Current state: 0.2.0 release surface for the data-contract and provider conformance layer.

The release scope is intentionally bounded: schema/planning contracts, coordinator envelopes, relations/FoldSet validation, deterministic fingerprints, numeric buffers, N-D tensor transport, fitted adapter refs, in-memory provider conformance, C ABI helpers, Python/WASM JSON-contract bindings and cross-repo dag-ml fixtures are shipped and gated. Production host providers beyond the in-memory conformance backend, host-filtered branch views and the nirs4all connector remain explicit post-0.2.0 backlog.

Implemented:

• Rust workspace with core, facade, C ABI and CLI crates;

• identifier, axis, representation, source and schema types;

• data view, presence mask and data-plan structs;

• schema validation and deterministic fingerprinting;

• adapter registry and deterministic path solver;

• model-input data-plan planner with fixtures;

• sample alignment planner for inner, left and outer multi-source policies, plus planner-visible Align steps before multi-source joins;

• sample relation validation for groups, repetitions and augmentation origins;

• FoldSet validation against sample relation group/origin boundaries, exposed through Rust core, Python, WASM and C ABI JSON helpers;

• canonical FoldSet fingerprints for replay/lineage identity, exposed through Rust core, Python, WASM and C ABI JSON helpers;

• shared FoldSet fixture parity with dag-ml, checked by scripts/validate_contracts.py and smoke-tested through Python/WASM bindings;

• Python/WASM contract manifests for production/browser integrations to assert package version, supported contracts, exported helpers and shared fixture digests before running a pipeline;

• ADR-11 structured error descriptors in Rust core, Python exception attributes, WASM error payloads and C ABI validation error payloads, guarded by scripts/check_error_taxonomy.py;

• Python facade contract wrappers for host integrations to pass validated DatasetSchema, ModelInputSpec, AdapterRegistry, DataPlan, SampleRelationTable, FoldSet and CoordinatorDataPlanEnvelope values instead of raw JSON strings only;

• optional nirs4all integration schema fields: RepresentationSpec.signal_type, source ShapeContract, MetadataSchema, GroupSpec, FoldSpec and AugmentationMetadata, all additive for v1 readers and covered by the nirs4all-core Python/WASM smoke fixture;

• sample relation fingerprints;

• coordinator data-plan envelope export with schema, plan and relation fingerprints;

• explicit coordinator data-plan envelope schema version with unsupported version refusal;

• published JSON Schema artifact for coordinator data-plan envelopes, with a unit smoke that keeps its declared version aligned to the Rust contract;

• stdlib shared-contract validation script plus CI checkout of dag-ml so schema copies and coordinator fixtures cannot drift silently;

• CI documentation and packaging gates (cargo doc with rustdoc warnings denied, plus workspace cargo package --no-verify) for release readiness;

• Sphinx/MyST docs-site scaffold with a warnings-denied CI build and release metadata validation, so ADR-09 has a local build gate before hosted docs;

• ADR-14 managed-debt CI lint for production-path TODO/FIXME markers and Rust #[deprecated] attributes, with release metadata enforcing the gate;

• public Rust documentation coverage ratchet in CI, currently enforcing the measured baseline while leaving the 95% invariant-docstring target explicit for the hardening backlog;

• ADR-10 publish-plan CI check that validates internal Cargo dependency version pinning and dry-runs independently publishable root crates;

• CI MSRV gate with Rust 1.83.0 and cargo check --workspace --all-targets, matching the current PyO3 0.28.x floor;

• CI supply-chain audit gate with pinned cargo-audit and cargo audit --deny warnings;

• CI Python wheel metadata smoke for name/version, Requires-Python, CeCILL-2.1 OR AGPL-3.0-or-later license packaging, abi3, native extension, stubs and py.typed;

• CI WASM package tarball gate via wasm-pack pack for browser-target npm artifacts;

• CI npm tarball metadata gate via npm pack --dry-run --json for WASM package name/version, integrity, bundled-dependency absence and required published files;

• WASM smoke validation for generated npm package metadata, including package name, version, JS entry, typings, required emitted files and expected TypeScript exports;

• release metadata validator for Cargo workspace inheritance, internal crate version pins, Python PEP 440 package version, abi3-py311, MSRV/toolchain pins, MSRV-sensitive dependency pins, required governance files and the docs-site / managed-debt / publish-plan CI gates;

• public C ABI snapshot manifest for dag_ml_data.h, validated in CI so header changes cannot drift silently;

• shared parity-oracle handoff manifest for the first nirs4all-core parity cases, with fixture/gate references validated against dag-ml;

• shared conformance-pack manifest with canonical schema/fixture digests, C ABI requirements and required cross-repo checks, kept JSON-identical with dag-ml;

• conversion from SampleRelationTable to DAG-ML coordinator relation records;

• coordinator materialization request and handle-record contracts for validated data handles;

• in-memory coordinator handle arena that materializes envelopes into opaque handle records with run/node/phase/variant/fold/fingerprint traceability;

• materialized data handles scope coordinator relations to the request source_ids, so child views cannot expose observations from sources that were not materialized;

• identity-filtered view handles from DataView, preserving repeated observations while allowing sample/source/augmentation selection and honoring explicit requested sample order;

• sample-level target value alignment for view handles with deterministic de-duplication across repeated observations in view sample order;

• multi-target target alignment through CoordinatorMultiTargetBlock and dagmldata_coordinator_multi_target_arrow_json, preserving view sample order while emitting nullable per-target Arrow columns and validity masks;

• observation-level feature table alignment for view handles, preserving repeated observations, applying DataView.columns and keeping view sample order;

• executable observation-level feature fusion for aligned multi-source feature blocks, including namespaced columns, repetition-preserving reference rows, singleton-source broadcast, deterministic outer synthetic rows and explicit refusal of ambiguous repeated non-reference sources;

• feature exports now validate that the feature table representation matches the materialized data-plan output representation for the parent view handle;

• CLI materialize-envelope smoke command;

• Arrow C Data ABI structs, release helpers and coordinator identity-table export from validated envelopes;

• Arrow C Data numeric target-table export through materialized view handles;

• Arrow C Data numeric feature-table export through materialized view handles;

• Arrow C Data numeric feature-fusion export for aligned multi-source feature blocks;

• JSON and ABI-owned f64 numeric feature-collation exports for row-major tensor conformance;

• Rust-owned in-memory provider vtable with materialize, make-view, view-identity, target, feature, release and destroy callbacks;

• in-memory provider feature tables are converted once at provider creation into typed numeric buffers, so provider exports no longer re-parse JSON values on every feature_arrow call;

• typed numeric feature buffers now live in dag-ml-data-core as reusable column-major NumericFeatureBuffer contracts with projection tests, rather than being private C ABI fixture logic;

• typed row-major NumericFeatureMatrixF64 input with optional validity masks converts directly to column-major buffers without per-cell serde_json::Value parsing on the numeric conformance path;

• typed numeric feature buffers are grouped behind a core NumericFeatureBufferStore with deterministic manifests, row/feature/value counts, estimated value bytes and stable buffer fingerprints;

• materialized data handles now receive deterministic NumericFeatureBufferBinding records for the feature buffers whose representation and observation coverage match that handle’s scoped coordinator relations;

• the store and binding lifecycle now live behind the reusable core NumericFeatureBufferArena, so production providers can share the same manifest, bind, project and release contract instead of reimplementing it in the C ABI layer;

• provider feature_arrow accepts JSON fusion selectors and routes source-filtered provider-owned feature buffers through the core feature-fusion kernel;

• provider feature, fusion and collation exports validate that the selected buffer is bound to the parent data handle and requested source scope before exporting Arrow, JSON tensors or ABI-owned tensors;

• executable numeric late-collation kernel for feature blocks and ragged numeric rows, producing row-major tensor blocks with deterministic padding, truncation, presence masks and value-validity masks;

• provider-backed feature collation over in-memory provider typed buffers, including fusion selectors, returning deterministic JSON NumericTensorBlock or ABI-owned DagMlDataTensorF64 output without changing the provider vtable layout;

• provider-owned feature-buffer manifests are exported as JSON through the C ABI for binding conformance and lifecycle validation;

• provider construction accepts typed f64 feature matrices through the C ABI as the preferred conformance path for numeric feature buffers;

• provider construction also accepts borrowed C DagMlDataFeatureMatrixF64View descriptors, copying them into Rust-owned feature buffers during the constructor call so bindings can avoid JSON numeric value transport;

• provider construction also accepts borrowed C DagMlDataFeatureMatrixF64ColumnarView descriptors with per-column f64 slices and optional per-column validity bitmaps, mirroring production columnar layouts (Arrow IPC, Parquet, NumPy column ndarrays) and avoiding the row-major transpose copy paid by the row-major borrowed view path;

• coordinator and provider-backed feature collation now also return owned row-major DagMlDataTensorF32 blocks beside the existing DagMlDataTensorF64 exports. The collation kernel still operates in f64 to preserve canonical numeric semantics; values are cast to f32 at the ABI boundary and rejected with ValidationError if any padded value, finite input or padding fallback does not round-trip into a finite f32 (overflow to infinity, or non-finite input). The C ABI exposes DAG_ML_DATA_TENSOR_F32_ABI_VERSION, dagmldata_tensor_f32_free and the matching coordinator/inmemory_provider collation entry points;

• DataView now carries an optional branch_view field shaped as CoordinatorBranchView (view_id, branch_id, mode, selector, allow_overlap, metadata), mirroring dag-ml’s BranchViewPlan wire contract. make_view validates the mode↔selector field agreement and natively executes by_source branch views as an additional intersection over the existing source filter. separation mode passes through (the selector annotates branch identity and the host scheduler owns non-overlap), while by_metadata, by_tag and by_filter modes pass validation but are rejected for in-memory execution with a clear requires host-side filtering error so production providers can route them to native filter backends without breaking ABI compatibility;

• published coordinator_branch_view.schema.json for that contract, with matching COORDINATOR_BRANCH_VIEW_SCHEMA_ID Rust constant, the same digest pinned in both repos’ conformance_pack.v1.json and scripts/validate_contracts.py validating the published schema artifact shape + mode enum coverage in both dag-ml-data and the sibling dag-ml checkout;

• file-backed persistence for FittedAdapterManifest: FittedAdapterManifest::write_to_path writes pretty-printed JSON to disk after running the same validate/validate_portable gate that read_from_path enforces on load. The require_portable flag is honored symmetrically so manifests can never be persisted in a state that would not load cleanly on the other side, giving hosts a real cross-process persistence path without taking on a serialization dependency in core;

• runtime fitted-adapter store: RuntimeFittedAdapterStore trait plus the in-memory InMemoryFittedAdapterStore that registers FittedAdapterRef records, allocates deterministic opaque u64 handles, and materializes them on request after validating the request’s adapter_id and params_fingerprint against the registered ref. register_manifest accepts a FittedAdapterManifest for bulk registration; the store never reads, writes or deserializes adapter payloads — payload materialization stays host-side, behind the opaque handle, mirroring the dag-ml RuntimeArtifactStore lifecycle for model artifacts. The store is Send + Sync (backed by std::sync::Mutex), so the same handle can be shared across host threads through the C ABI without external locking; poisoned-mutex errors surface as validation failures rather than silent recoveries;

• C ABI for the in-memory fitted-adapter store: DagMlDataFittedAdapterStoreHandle opaque pointer with dagmldata_inmemory_fitted_adapter_store_new / destroy lifecycle, register_json / materialize_json for JSON round-trips through the Rust runtime store and release for adapter-id-keyed teardown. Errors carry an owned DagMlDataString. Allows non-Rust bindings to drive the full register/materialize/release cycle through the ABI without parsing JSON inside Rust;

• the in-memory provider can attach a fitted-adapter store through dagmldata_inmemory_provider_attach_fitted_adapter_store, after which host controllers running through the provider’s vtable can materialize fitted adapters via dagmldata_inmemory_provider_materialize_fitted_adapter_json. The provider holds a Mutex-guarded borrowed pointer to the store (the caller keeps ownership), passing a null handle detaches without freeing. Closes the gap where the fitted-adapter contract layer had no scheduler-time consumer: the provider is now the consumer, the same way the artifact store on dag-ml is consumed by REFIT-phase controllers going through the runtime provider;

• published fitted_adapter_ref.schema.json for the data-side fitted adapter contract, with matching FITTED_ADAPTER_REF_SCHEMA_ID Rust constant, the same digest pinned in both repos’ conformance_pack.v1.json and scripts/validate_contracts.py validating the published schema artifact in both repos. The C ABI exposes dagmldata_fitted_adapter_ref_validate_json and dagmldata_fitted_adapter_manifest_validate_json; both take a require_portable byte flag selecting between inline and portable validation modes and return ValidationError plus an error string on failure;

• fitted adapter serialization contract: FittedAdapterRef (adapter_id, adapter_version, params_fingerprint, optional backend/uri/content_fingerprint/size_bytes/plugin/plugin_version/ metadata) plus FittedAdapterBackend (joblib/pickle/json/numpy/onnx/raw) and FittedAdapterManifest (versioned collection of refs). Validation has two modes: validate() accepts inline refs while validate_portable() requires backend + safe relative URI + content fingerprint, refusing absolute paths, Windows drive prefixes, URI schemes, colons in the first path segment and .. traversal. The manifest enforces unique adapter ids and key-vs-ref consistency. This lets dag-ml replay restore stateful data-side adapters the same way ArtifactRef already covers fitted models, without serializing adapter binaries inside the core;

• data-handle-scoped feature-buffer bindings are exported as JSON through the C ABI and become invalid when the parent data handle is released;

• provider vtable release conformance, including parent data-handle release invalidating child view handles;

• C header and linked C runtime smokes for the provider vtable and Arrow signatures;

• dag_ml_data.h and dag_ml.h share a guarded data-provider vtable ABI version macro and are compiled together in both include orders when the sibling checkout is present;

• Python ctypes smoke for the same provider lifecycle;

• installable stdlib-only Python provider package (dag_ml_data_provider) in crates/dag-ml-data-capi/bindings/python, plus a reusable CLI smoke in examples/python that imports it;

• C ABI schema fingerprint entry point;

• example schema fixture;

• CI workflow;

• file-backed numeric feature buffer persistence: deterministic .n4d byte format (N4DF magic + LE u32 version/count + per-buffer typed column-major payload + SHA-256 trailer) implemented in buffer_file_store.rs. serialize_columnar_store / deserialize_columnar_store round-trip buffer fingerprints byte-identically across processes; write_store_to_path / read_store_from_path wrap the same logic with filesystem diagnostics. Rejects unknown magic, unsupported version, truncated payload, SHA-256 mismatch, malformed validity bytes;

• file-backed provider C ABI entry point dagmldata_inmemory_provider_new_from_file constructs the same InMemoryProvider from a .n4d file path. Lets hosts persist provider feature buffers across processes without re-ingesting the source data;

• AxisKind::Wavenumber added to dag-ml-data-core/src/model.rs. The axis kind serializes as "wavenumber" and is accepted by RepresentationSpec::validate. Unblocks IR and Raman spectroscopy representations without introducing NIRS-specific business logic;

• ADR-0001 publishes the official ownership of the SpectroDataset → CoordinatorDataPlanEnvelope bridge: nirs4all-io owns the bridge, dag-ml-data ROADMAP Phase 4 is descoped. Resolves the nirs4all-io PHASE2_GATE item 7 blocker;

• host-callback feature-buffer provider: new C ABI vtable DagMlDataBufferFetcherVTable (single fetch_columnar callback + destroy) plus DagMlDataBufferFetchRequest pair (feature_set_id, content_fingerprint) so hosts can fetch buffer bytes by id at construction time without packaging them as JSON or borrowed views. The constructor dagmldata_inmemory_provider_new_with_buffer_fetcher drives the fetcher for each declared request, copies returned views into Rust-owned buffers via the existing typed columnar path, and invokes the fetcher’s destroy callback exactly once before returning (success or failure). ABI version macro DAG_ML_DATA_BUFFER_FETCHER_ABI_VERSION = 1;

• Apache Arrow IPC feature-buffer reader: new optional crate dag-ml-data-arrow exposes read_buffers_from_ipc_stream/_file/_path that consume Arrow IPC streams or files and resolve each RecordBatch to a NumericFeatureMatrixF64Columnar. The mapping requires schema metadata dag_ml_data.feature_set_id + dag_ml_data.representation_id and an observation_id Utf8 column; Float64 feature columns translate validity-aware to Option<f64> masks. The capi crate exposes dagmldata_inmemory_provider_new_from_arrow_ipc behind the arrow-ipc feature flag so core consumers don’t pay the Arrow dependency by default;

• a dedicated dag-ml-data-provider crate factors the in-memory provider behind a DagMlDataProvider trait with a single Mutex<ProviderArenas> (linearizable materialize / make-view / release) and a shared JsonInMemoryProvider facade. Language shims consume the C ABI: a stdlib-only Python ctypes package (dag_ml_data_provider), a dag-ml-data-wasm binding, and a workspace-excluded extendr R package (dagmldata);

• borrowed N-D tensor transport: core NdTensor* (rank 1..=16, axis 0 = sample, keyed by observation_id) is materialized and bound under the provider lock. The C ABI accepts borrowed DagMlDataBorrowedTensorViews (strided input copied to canonical row-major) and returns view-filtered DagMlDataOwnedTensors via dagmldata_inmemory_provider_new_with_tensor_views / nd_tensor_manifest_json / nd_tensor_export_json / nd_tensor_free, with DAG_ML_DATA_BORROWED_TENSOR_VIEW_ABI_VERSION and DAG_ML_DATA_OWNED_TENSOR_ABI_VERSION mirrored in the conformance pack;

• typed axis coordinate contract: AxisSpec.coordinate: Option<CoordinateSpec> (numeric / categorical / datetime dtype, ordered, explicit list or numeric regular grid) replaces the untyped coordinates, with full validation and deny_unknown_fields;

• discriminated, host-routable DataError variants (ADR-11 granularity): FingerprintMismatch (compatibility), UnknownHandle (runtime), RelationBoundaryViolation and IncompatibleReducer (data), SignalTypeMismatch (compatibility) — each with a stable numeric code, so a host can route a fingerprint/leakage/handle/signal failure instead of one opaque validation code;

• ADR-07 aggregation reducer contract (aggregation.rs): flat deny_unknown_fields AggregationPolicy (mean / weighted_mean / median / vote / robust_mean / exclude_outliers / custom) with per-reducer parameter and cross-parameter validation and validate_for_task (vote is classification only). Exposed over the C ABI as dagmldata_aggregation_policy_validate_json. No reducer math in core (the bridge owns it);

• ADR-06 signal-type validation: SignalKind::as_str + require_signal_type_match(expected, actual, allow_unknown) reusable contract helper (caller-decided Unknown policy: tolerated at train, refused at predict);

• CI hardening: an AddressSanitizer lane over the C-ABI --lib unsafe surface, the capi arrow-ipc feature compiled+tested, the WASM provider feature built, and the stdlib-only ctypes provider package pip-installed + smoked. The public-doc ratchet floor is raised to track the current coverage (33% floor, measured 33.7%).

• frozen, published built-in representation registry (B-014 / DMD-001): representation_registry() derives a deterministic RepresentationRegistry from BUILTIN_DATA_MODELS (26 IDs, each with its frozen RepresentationSpec, BuiltinDataModel key, modality and an optional spectra+image MVP annotation), published as the docs/contracts/representation_registry.v1.json contract artifact and emitted by the CLI representation-registry command. A drift test (published_registry_matches_builtin_models) freezes the manifest against builtin_models.rs; no new representation vocabulary is introduced. The mvp block records the spectra+image MVP set (12 IDs: 8 emitted by the nirs4all-io bridge, 4 image IDs landed_pending_emit per IO-010). Cross-repo conformance_pack.v1.json / validate_contracts.py lockstep wiring with dag-ml is the next DMD-001 slice.

Post-0.2.0 backlog:

• production provider arenas for fused feature exports (the borrowed N-D tensor transport is now implemented; fused-feature exports remain in-memory-only);

• keep the documented ADR-07 aggregation mapping current; robust_mean / exclude_outliers remain data-side reducers unless a future shared aggregation_reducer_policy.v1 schema is added;

• wire require_signal_type_match into materialize/predict — blocked on a paired dag-ml lineage/bundle change that carries the expected signal type (no speculative expected_signal_type request field added);

• nirs4all connector (descoped — owned by nirs4all-io, see ADR-0001).

Next recommended post-0.2.0 task:

Write the honest host-adapter backlog in dag-ml/docs/HOST_ADAPTER_BACKLOG.md: process-adapter JSONL pattern is the only stable wire protocol for Python/R hosts, persistent pool infrastructure is already shipped, the existing sklearn smoke covers ~70% of a production sklearn controller, and SpectroChemPy/Orange-Spectroscopy are Python rather than R despite the user grouping them as such.