# C ABI

The data ABI lets a host runtime keep buffers and fitted data adapters in its own
memory while exposing deterministic descriptors and identity tables to the core.

Validation failures written to `error_out` use the ADR-11 descriptor JSON shape:
`category`, `code`, `severity`, `message`, `remediation_hint` and `context`.
Null-pointer preflight errors still use the legacy human string because callers
usually branch on `DagMlDataStatusCode` before parsing a payload.

## Current Scaffold

`crates/dag-ml-data-capi/include/dag_ml_data.h` exposes:

- version, string-free and tensor-free helpers;
- `dagmldata_schema_fingerprint_json`;
- `dagmldata_aggregation_policy_validate_json`, which validates an ADR-07
  aggregation-policy payload (reducer name plus its parameters);
- Arrow C Data `ArrowArray` and `ArrowSchema` structs plus release helpers;
- `DagMlDataTensorF64`, an owned row-major f64 tensor descriptor with identity,
  shape, values, optional masks and feature names;
- `DAG_ML_DATA_TENSOR_F64_ABI_VERSION`, the C-visible ABI version expected in
  each f64 tensor descriptor;
- `DagMlDataTensorF32`, an owned row-major f32 tensor descriptor mirroring the
  f64 descriptor for bindings that consume f32 inputs natively (deep-learning
  frameworks, GPU pipelines);
- `DAG_ML_DATA_TENSOR_F32_ABI_VERSION`, the C-visible ABI version expected in
  each f32 tensor descriptor;
- `DagMlDataFeatureMatrixF64View`, a borrowed C view over one row-major f64
  feature matrix;
- `DagMlDataFeatureMatrixF64ColumnarView` plus `DagMlDataF64ColumnView`, a
  borrowed C view over one column-major f64 feature matrix with per-column
  validity bitmaps, mirroring Arrow/Parquet/NumPy columnar layouts;
- `dagmldata_coordinator_identity_arrow_json` for identity-table smoke tests
  from a validated coordinator envelope;
- `dagmldata_coordinator_target_arrow_json` for numeric target-table smoke tests
  from a validated envelope, materialization request, `DataView` and target
  table;
- `dagmldata_coordinator_multi_target_arrow_json` for multi-output target
  export from the same envelope/view contracts, with one nullable f64 column per
  target and a per-target validity bitmap;
- `dagmldata_coordinator_feature_arrow_json` for numeric observation-level
  feature-table smoke tests from the same coordinator/view contracts;
- `dagmldata_coordinator_feature_fusion_arrow_json` for numeric multi-source
  fused feature-table smoke tests over already materialized coordinator feature
  blocks;
- `dagmldata_coordinator_feature_collation_json` for JSON row-major tensor
  collation smoke tests over coordinator feature blocks;
- `dagmldata_coordinator_feature_collation_tensor_f64_json` for ABI-owned
  row-major f64 tensor export over coordinator feature blocks;
- `dagmldata_coordinator_feature_collation_tensor_f32_json` for ABI-owned
  row-major f32 tensor export over the same coordinator feature blocks, with
  rejection of values that do not round-trip into a finite f32;
- `dagmldata_inmemory_provider_new_json` for a Rust-owned provider vtable that
  materializes data handles, creates view handles, exports view identity, exports
  numeric targets and supports release/destroy callbacks;
- `dagmldata_inmemory_provider_new_with_features_json` for the same provider
  plus JSON feature tables used by binding conformance tests;
- `dagmldata_inmemory_provider_new_with_f64_features_json` for the same provider
  plus typed row-major f64 feature matrices with an optional validity mask;
- `dagmldata_inmemory_provider_new_with_f64_feature_views` for the same provider
  plus borrowed C `DagMlDataFeatureMatrixF64View` descriptors, avoiding JSON
  value transport for numeric feature matrices;
- `dagmldata_inmemory_provider_new_with_f64_feature_columns` for the same
  provider plus borrowed C `DagMlDataFeatureMatrixF64ColumnarView` descriptors
  with per-column f64 slices and optional per-column validity bitmaps,
  avoiding the row-major transpose copy on the production columnar ingestion
  path;
- `dagmldata_inmemory_provider_feature_buffer_manifest_json` for deterministic
  JSON manifests of provider-owned numeric feature buffers;
- `dagmldata_inmemory_provider_data_feature_buffer_manifest_json` for
  data-handle-scoped feature-buffer bindings;
- `dagmldata_inmemory_provider_feature_collation_json` for JSON row-major
  tensor collation from feature buffers owned by the in-memory provider;
- `dagmldata_inmemory_provider_feature_collation_tensor_f64_json` for ABI-owned
  row-major f64 tensor export from provider-owned feature buffers;
- `dagmldata_inmemory_provider_feature_collation_tensor_f32_json` for ABI-owned
  row-major f32 tensor export from the same provider-owned feature buffers,
  with the same finite-f32 rejection contract as the coordinator entry point;
- `DataView.branch_view` (a `CoordinatorBranchView` carrying `view_id`,
  `branch_id`, `mode`, `selector`, `allow_overlap` and `metadata`), letting
  hosts pass `dag-ml` `BranchViewPlan` records straight through the existing
  `make_view` selector JSON. The in-memory provider executes `by_source`
  branch views natively and lets `separation` pass through; `by_metadata`,
  `by_tag` and `by_filter` are validated but require host-side filtering;
- `FittedAdapterRef` and `FittedAdapterManifest`, a JSON-serializable
  fitted-adapter persistence contract with `validate()` for inline refs and
  `validate_portable()` for refs carrying a backend + safe relative URI +
  content fingerprint (the same portability rules as `dag-ml`'s
  `ArtifactRef::validate_portable`). The manifest enforces unique adapter
  ids and key↔ref consistency;
- `dagmldata_fitted_adapter_ref_validate_json` and
  `dagmldata_fitted_adapter_manifest_validate_json` C ABI entry points that
  parse JSON, run the `validate()` or `validate_portable()` Rust check
  selected by the `require_portable` byte flag and return either `Ok` or
  `ValidationError` + an owned error string. The matching
  `fitted_adapter_ref.v1.schema.json` is published under `docs/contracts`
  and pinned in `conformance_pack.v1.json` for cross-repo digest equality;
- `DagMlDataFittedAdapterStoreHandle` opaque pointer plus
  `dagmldata_inmemory_fitted_adapter_store_new` / `destroy` /
  `register_json` / `materialize_json` / `release` entry points exposing
  the Rust `InMemoryFittedAdapterStore` to non-Rust bindings. Host callers
  receive a `u64` handle from register / materialize and own its release
  through `dagmldata_inmemory_fitted_adapter_store_release` (by adapter
  id) and the store-level `destroy` call;
- `DagMlDataVTable` with materialize/view/identity/target/feature/release hooks.
  The `feature_arrow` hook accepts either a plain feature-set id or a JSON
  feature-fusion selector. The vtable uses the shared
  `DAG_ML_DATA_PROVIDER_VTABLE_ABI_VERSION` macro and guarded
  `DagMlDataVTable` definition so `dag_ml_data.h` and `dag_ml.h` can be
  included together by bindings.

The coordinator envelope wire shape is versioned as
`CoordinatorDataPlanEnvelope` v1 and published at
`docs/contracts/coordinator_data_plan_envelope.schema.json`. Runtime validation
continues to check the stronger semantic contract: schema/data-plan/relation
fingerprints, identity consistency and materialization-request compatibility.

## Ownership Rules

| Object | Owner | Release path |
|---|---|---|
| Materialized data handle | Host | `DagMlDataVTable.release` |
| View handle | Host | `DagMlDataVTable.release` |
| Fitted adapter handle | Host | future fitted-adapter release hook |
| Rust error/fingerprint string | Rust allocation returned through ABI | `dagmldata_string_free` |
| Rust-owned f64 tensor descriptor and nested arrays | Rust allocation returned through ABI | `dagmldata_tensor_f64_free` |
| Rust-owned f32 tensor descriptor and nested arrays | Rust allocation returned through ABI | `dagmldata_tensor_f32_free` |
| Arrow arrays/schemas returned by Rust helpers | Rust allocation returned through ABI | `dagmldata_arrow_array_free`, `dagmldata_arrow_schema_free` |
| Arrow arrays produced by host vtables | Producer of the Arrow array | Arrow C Data Interface release callback |
| Rust-owned in-memory provider vtable | Rust allocation behind `user_data` | `DagMlDataVTable.destroy` or `dagmldata_inmemory_provider_destroy` |
| Borrowed `DagMlDataFeatureMatrixF64View` inputs | Caller | copied during constructor call; caller may release after return |
| Borrowed `DagMlDataFeatureMatrixF64ColumnarView` inputs (and nested `DagMlDataF64ColumnView` columns) | Caller | copied during constructor call; caller may release after return |

## Coordinator Identity Export

`dagmldata_coordinator_identity_arrow_json` is a narrow smoke helper, not the
final provider implementation. It validates a `CoordinatorDataPlanEnvelope` and
exports one Arrow struct row per coordinator relation with:

- `observation_id`, `sample_id`, `target_id`, `group_id`;
- `origin_sample_id`, `source_id`, `is_augmented`.

This is enough for ABI consumers to verify sample/repetition/group/augmentation
identity transfer before full buffer-backed provider lifecycles exist.

`dagmldata_coordinator_target_arrow_json` extends the smoke path to
sample-level targets. It materializes the envelope, creates a `DataView`, aligns
target values to the selected samples and emits `sample_id`, `target_id` and
numeric `value` columns. Repeated observations are intentionally de-duplicated
to one target value per sample.

`dagmldata_coordinator_multi_target_arrow_json` uses the same materialization
path for multi-output regression/classification targets. Its request carries
`target_tables` and its Arrow result emits `sample_id` plus one nullable f64
column per target id. Missing or explicit null values are represented by that
target column's validity bitmap rather than by dropping samples.

`dagmldata_coordinator_feature_arrow_json` is intentionally observation-level.
It materializes the same envelope/view, preserves repeated observations, applies
`DataView.columns`, and emits `observation_id`, `sample_id` plus one numeric
column per selected feature. This keeps the target aggregation rule separate
from feature row identity.

`dagmldata_coordinator_feature_fusion_arrow_json` exercises the pure Rust
feature-fusion kernel through the C ABI. It accepts a request shaped as
`{ feature_set_id, sources, alignment, source_layout?, policy? }`, where each
source contains a `source_id` and a `CoordinatorFeatureBlock`. When present,
`source_layout` is validated against those concrete blocks: source order,
per-source preprocessing output feature-set/representation, and contiguous
feature-axis concat spans must match. The exported Arrow table preserves
reference-source repeated observations, broadcasts singleton rows from secondary
sources, namespaces fused feature columns by default and refuses incoherent
presence masks or ambiguous repeated secondary sources.

`dagmldata_coordinator_feature_collation_json` exercises the pure Rust numeric
late-collation kernel through the C ABI. It accepts `{ feature_block, policy? }`
and returns a JSON `NumericTensorBlock` with observation/sample identity,
row-major shape and values, optional presence mask and optional value-validity
mask. It is a conformance helper, not a provider lifecycle.

`dagmldata_coordinator_feature_collation_tensor_f64_json` exports the same
result as an ABI-owned `DagMlDataTensorF64` instead of JSON. The tensor carries
`abi_version`, block/representation/container strings, observation ids, sample
ids, `shape`, contiguous row-major `values`, optional `presence_mask`, optional
`validity_mask` and optional `feature_names`. Masks are byte arrays with values
0 or 1. The caller must release the tensor with `dagmldata_tensor_f64_free`.

`dagmldata_coordinator_feature_collation_tensor_f32_json` mirrors that surface
with a `DagMlDataTensorF32` output. The kernel still runs in f64 to preserve
canonical numeric semantics; each value is cast to f32 at the ABI boundary and
the call returns `ValidationError` if any padded value, finite input or
padding fallback does not round-trip into a finite f32. The caller must
release the tensor with `dagmldata_tensor_f32_free`. The same f32 entry point
is available against provider-owned feature buffers through
`dagmldata_inmemory_provider_feature_collation_tensor_f32_json`.

`dagmldata_inmemory_provider_feature_collation_json` and
`dagmldata_inmemory_provider_feature_collation_tensor_f64_json` exercise the
same late-collation kernel against provider-owned typed numeric buffers. They
accept `{ feature_set_id, policy? }` for a single provider feature table or
`{ fusion, policy? }` where `fusion` is the provider feature-fusion selector.
The JSON export is a conformance/debug path; the `DagMlDataTensorF64` export is
the binding-oriented path. These helpers are specific to vtables created by
`dagmldata_inmemory_provider_new_with_features_json`; they do not change the
stable `DagMlDataVTable` layout.

`dagmldata_inmemory_provider_feature_buffer_manifest_json` returns an array of
`NumericFeatureBufferManifest` values for the provider-owned typed buffers. Each
manifest includes the feature-set id, representation id, feature and observation
ids, row/feature/value counts, estimated f64 storage bytes and a deterministic
buffer fingerprint. Bindings can use this before creating feature views or
tensors to verify that the provider loaded the expected data buffers.

`dagmldata_inmemory_provider_data_feature_buffer_manifest_json` returns
`NumericFeatureBufferBinding` values for one live materialized data handle. A
binding is created by the core `NumericFeatureBufferArena` during `materialize`
only when a provider buffer has the same output representation as the data
handle and covers the handle's scoped coordinator observations for one or more
source ids. The binding export refuses unknown or released data handles.

## In-Memory Provider VTable

The in-memory provider is the current ABI conformance target. It accepts one
validated coordinator envelope plus optional sample-level target tables and
observation-level feature tables, then implements:

- `materialize`: validates a coordinator materialization request and returns an
  opaque data handle whose coordinator relations are scoped to the requested
  `source_ids`;
- `make_view`: applies a `DataView` to a data handle and returns an opaque view
  handle;
- `view_identity`: returns the filtered relation table as Arrow C Data;
- `target_arrow`: returns sample-level numeric targets aligned to the view;
- `feature_arrow`: returns observation-level numeric features aligned to the
  view and filtered by `DataView.columns` when passed a plain feature-set id;
  when passed `{ feature_set_id, sources, alignment, source_layout?, policy? }`
  JSON, where each source is `{ source_id, feature_set_id, columns? }`, it fuses
  provider-owned source feature buffers through the core feature-fusion kernel;
- `release` and `destroy`: release handles and provider state.

The conformance provider can still receive small JSON fixture feature tables at
construction time, but the preferred numeric paths are typed row-major
`NumericFeatureMatrixF64` input with optional validity masks and direct borrowed
C `DagMlDataFeatureMatrixF64View` descriptors. The borrowed descriptors are
copied into Rust-owned buffers during construction; after the constructor
returns, callers may release their input arrays. All construction paths convert
once into column-major `NumericFeatureBuffer` values grouped by
`NumericFeatureBufferArena` in the provider state. At `materialize`, the arena
computes data-handle-scoped buffer bindings from the scoped coordinator
relations and materialized output representation. `feature_arrow` exports are
then view projections over bound buffers, not per-call JSON numeric parsing.
Fusion selectors reuse those typed buffers, filter each source by source
identity in the view, validate that each source buffer is bound to the parent
data handle through the arena, and then call the same pure Rust fusion kernel
used by the standalone ABI helper. Provider feature-collation selectors then
collate either a single feature table or the fused block into deterministic
row-major JSON or `DagMlDataTensorF64` tensors without reparsing feature values.
Full provider implementations will use the same vtable shape while keeping
production data buffers host-owned.

`tests/c_header_smoke.rs` has two C checks: a header syntax smoke with
`cc -fsyntax-only`, and a linked C program that loads the Rust `cdylib`, creates
the provider vtable, materializes a view, exports identity, target and feature
Arrow arrays, then releases all handles.
The syntax smoke also includes the sibling `dag_ml.h` in both include orders
when a `dag-ml` checkout is available, so shared data-provider vtable guards
cannot drift silently.

`tests/python_ctypes_smoke.rs` performs the same provider lifecycle from Python
using only `ctypes`. It also runs `examples/python/provider_smoke.py` against the
installable `dag_ml_data_provider` package
(`crates/dag-ml-data-capi/bindings/python`), a stdlib-only ctypes wrapper around
the provider vtable. The package is the binding-friendly conformance target for
materialize, view creation, identity export, target export, feature export,
release and destroy; it locates the cdylib via `library_path=` /
`DAG_ML_DATA_CAPI_LIB` / the Cargo target dir.

## ABI Roadmap

1. Freeze byte/string/status conventions.
2. Add C smoke test for schema fingerprinting.
3. Add path-solving and data-plan validation over canonical JSON.
4. Add native host provider conformance against the current
   identity/target/feature behavior.
5. Add ABI conformance for the core multi-source feature-fusion and numeric
   collation kernels.
6. Route in-memory provider `feature_arrow` fusion selectors through the same
   kernel.
7. Route in-memory provider feature-collation selectors through provider-owned
   typed buffers.
8. Expose provider-backed collation as `DagMlDataTensorF64` for bindings.
9. Expose provider-owned feature-buffer manifests for binding conformance.
10. Bind compatible feature buffers to live materialized data handles and
    validate those bindings before Arrow/tensor export.
11. Add typed f64 matrix provider construction, including borrowed C matrix
    views, avoiding per-cell JSON values on the numeric conformance path.
12. Replace in-memory typed fixture buffers with production feature-buffer
   lifecycles.