Datasets & Distributions

The core discovery object. A Dataset is the abstract metadata entity describing what the set of data is. A Distribution is a concrete physical access path describing how to retrieve it. A Dataset can exist without any distributions (metadata-only registration) and distributions can be added later via POST /datasets/{id}/distributions.

Dataset

Recording a dataset's provenance

activity_id says which run produced a dataset. derived_from says which data it was built from, and the two are independent: a run may consume several inputs while any one output depends on only some of them. FDS never guesses that link, so a derivation is only ever recorded because someone asserted it.

The upstream does not have to be registered in FDS. Identify it however you can, falling back to a free-text label and/or description only in the worst cases:

Give source_dataset_id or source_identifier, not both, since a registered dataset is already referenced by its own address. Every entry needs at least one of the three. A label and a description may accompany any of them.

A DOI is understood bare (10.5281/zenodo.123), prefixed (doi:10.5281/...), or as a full URL. Handle (hdl:), Software Heritage (swh:) and URNs are understood too. Anything else is recorded exactly as given, and simply not treated as resolvable.

"derived_from": [
  {"source_dataset_id": 42},
  {"source_identifier": "10.5281/zenodo.123", "source_label": "Upstream on Zenodo"},
  {"source_label": "Legacy tape", "source_description": "Recovered from a 1998 archive"}
]

Derivations can also be maintained afterwards:

POST   /v1/datasets/{id}/derivations
GET    /v1/datasets/{id}/derivations
DELETE /v1/datasets/{id}/derivations/{derivation_id}

Following the chain

derivations answers one hop. To get the whole ancestry in one request, ask for the lineage:

GET /v1/datasets/{id}/lineage

Each upstream is nested under the dataset that asserted it. Only registered datasets have upstreams to nest, so an external or described-only upstream is always a leaf.

{
  "dataset_id": 51,
  "name": "t_e_profile",
  "derived_from": [
    {
      "dataset_id": 42,
      "name": "raw_thomson",
      "derived_from": [
        {"identifier": "10.5281/zenodo.123", "label": "Upstream on Zenodo"}
      ]
    }
  ]
}

A dataset is expanded once per response. Reach it a second time, either because two paths lead to the same ancestor or because the derivations form a loop, and it appears as a stub marked seen carrying only its id. Look it up under its first occurrence for the full record. Two other flags mark a branch that stops early: restricted, where you do not have read access to the upstream, and missing, where the upstream has since been deleted. A node with none of the three is complete.

Registering a Dataset with a url and media_type creates its primary Distribution in the same call:

curl -X POST "$API/devices/mast/shots/30421/datasets" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"name": "equilibrium",
       "url": "s3://fds-data/shots/30421/equilibrium",
       "media_type": "application/x-zarr", "access_level": "public"}'

requests.post(
    f"{API}/devices/mast/shots/30421/datasets",
    headers=headers,
    json={
        "name": "equilibrium",
        "url": "s3://fds-data/shots/30421/equilibrium",
        "media_type": "application/x-zarr",
        "access_level": "public",
    },
).raise_for_status()

httpx.post(
    f"{API}/devices/mast/shots/30421/datasets",
    headers=headers,
    json={
        "name": "equilibrium",
        "url": "s3://fds-data/shots/30421/equilibrium",
        "media_type": "application/x-zarr",
        "access_level": "public",
    },
).raise_for_status()

await fetch(`${API}/devices/mast/shots/30421/datasets`, {
  method: "POST",
  headers: { Authorization: `Bearer ${TOKEN}`, "Content-Type": "application/json" },
  body: JSON.stringify({
    name: "equilibrium",
    url: "s3://fds-data/shots/30421/equilibrium",
    media_type: "application/x-zarr",
    access_level: "public",
  }),
});

Listing datasets

The device listing covers the whole device by default. Narrow it with scope:

curl "$API/devices/mast/datasets"                # everything mast hosts
curl "$API/devices/mast/datasets?scope=device"   # general device-level Datasets

A device-level Dataset describes the machine rather than a single experiment. Geometry and calibration data are examples of this.

Listings are paged with offset and limit (default 100, at most 1000) and returned in a stable order, so paging through a device covers it exactly once. Pages hold only datasets the caller can read, so a page shorter than limit is the last.

A Dataset can also link to reference geometry. A Dataset declares the geometry it needs via geometry_references, or a Device-level Dataset provides geometry via geometry_roles and applies_to. The same machinery carries reference calibration: a Dataset names the calibration it needs via calibration_references, and a Device-level Dataset provides it via calibration_roles, calibration_stage, and applies_to. Unlike geometry, calibration can resolve to an ordered chain of stages. See Reference Datasets.

Listing and filtering

The dataset lists accept a name filter, a property filter on the dataset's own scientific_metadata, and a shot_property filter on a property carried by its parent shot:

GET /v1/devices/mastu/datasets?name=equilibrium
GET /v1/devices/mastu/datasets?name=equilibrium&shot_property=elm

The second form answers a question spanning both levels ("equilibrium datasets from shots that had an ELM train") in one request. Datasets that don't belong to a shot never match a shot_property filter. Shot- and device-scoped dataset lists accept property on the same terms. See Finding annotated records.

Distribution

A Distribution is a physical access path for a Dataset, describing how to retrieve the data.

A Distribution has no access policy of its own: FDS applies the Dataset's to every distribution, for both metadata and credentials. Some copies may be harder to reach than that level suggests, such as a proprietary-format copy behind a site firewall or a data-access layer alongside an open-format copy in public storage. That restriction belongs to the infrastructure, not to FDS, so register at least one distribution that is as available as the Dataset's level says.

In most cases a Dataset will have exactly one Distribution, and you won't need to think about the distinction. The Dataset endpoints return the primary distribution's url, group, media_type, and format inlined directly on the Dataset response, so there is nothing extra to fetch.

How the data is opened

When FDS vends credentials it also renders them as storage_options, ready to pass to whichever library opens the data. storage_options_type says which library that is:

Set it when you know, and FDS infers it when you do not. An s3:// URL whose media_type mentions Icechunk is taken as icechunk_s3, any other s3:// URL as fsspec_s3, and anything else is left unset.

Azure and Google Cloud Storage leave it unset and still get storage_options, because each has one library and the URL scheme names it. gs:// and gcs:// get gcsfs options (token). az://, abfs:// and abfss:// get adlfs options: account_name with sas_token, or with anon for public data. For public data FDS takes the storage account from the URL when it names one, as in abfs://container@account.dfs.core.windows.net/path, and otherwise from the Azure provider configured for the distribution's endpoint_url.

The inference is right for the common cases and wrong in one that matters: an Icechunk store registered under a generic media type such as application/x-zarr is taken for plain Zarr, so the credentials come back in the wrong shape for the opener you are about to use. Setting storage_options_type explicitly removes the guess.

On any other URL, an unset value is also the reason include_storage_options=true can come back with nothing: FDS will not invent a shape for a distribution it cannot place.

Multiple distributions are supported when the same underlying data is available in more than one form, for example as both Zarr and HDF5, or through multiple access endpoints. All distributions of a given Dataset must be scientifically interchangeable; different data belongs in a separate Dataset. Additional distributions can be registered via POST /datasets/{id}/distributions.

A file holding several datasets

HDF5 and NetCDF files often hold several datasets as groups, for example one file per shot with a group for each IMAS IDS. Register each dataset's distribution with the file's url and the dataset's group:

{
  "url": "s3://fds-data/shots/30421.nc",
  "group": "equilibrium",
  "media_type": "application/x-netcdf",
  "format": "NetCDF-4"
}

The distribution is then the dataset's data and nothing more, so it can sit alongside a Zarr store of the same group as an interchangeable alternative. Leave group unset when url addresses the dataset on its own, as a Zarr group's URL does.

Credentials are vended for the whole file, since object storage cannot grant access to part of one. A reader still fetches little more than the group if it reads in small blocks. With s3fs's default 50 MB read-ahead, reading one group can transfer more than the whole file.

import s3fs
import xarray as xr

fs = s3fs.S3FileSystem(**storage_options)
f = fs.open(
    "s3://fds-data/shots/30421.nc", block_size=4 * 2**20, cache_type="blockcache"
)
ds = xr.open_dataset(f, engine="h5netcdf", group="equilibrium")