Metadata

Metadata#

Metadata for every shot, source and signal is published as Parquet files in the public S3 bucket. No credentials are needed.

Access	Location
S3	`s3://mast/fairmast-meta/`, endpoint `https://s3.echo.stfc.ac.uk`
HTTPS	`https://s3.echo.stfc.ac.uk/mast/fairmast-meta/`

Files#

File	Contents
`l1_shots.parquet`, `l2_shots.parquet`	One row per shot: shot metadata and CPF summary statistics
`l1_sources.parquet`, `l2_sources.parquet`	One row per source (group of signals) per shot
`l1_signals/signals_<shot_id>.parquet`, `l2_signals/signals_<shot_id>.parquet`	One row per signal for a single shot

l1_ files describe the level 1 data and l2_ files describe the level 2 data.

Loading and filtering#

Each file is read whole (the largest is around 15 MB) and filtered with pandas.

import pandas as pd

META_URL = "https://s3.echo.stfc.ac.uk/mast/fairmast-meta"

shots = pd.read_parquet(f"{META_URL}/l2_shots.parquet")

# One shot
shot = shots[shots.shot_id == 30421].iloc[0]

# Every shot in a campaign, with a few columns
m9 = shots[shots.campaign == "M9"]
m9[["shot_id", "timestamp", "preshot_description", "divertor_config"]]

# Number of shots per campaign
shots.groupby("campaign").size()

# Every signal in one shot
signals = pd.read_parquet(f"{META_URL}/l2_signals/signals_30421.parquet")

From metadata to data#

Each row’s url and endpoint_url locate the shot’s Zarr file. For signals, source is the group within it. Opening s3:// URLs needs s3fs.

import xarray as xr

ip = signals[(signals.source == "summary") & (signals.name == "ip")].iloc[0]

ds = xr.open_zarr(
    ip.url,
    group=ip.source,
    storage_options=dict(anon=True, endpoint_url=ip.endpoint_url),
)
ds["ip"]

The same file is available over HTTPS at https://s3.echo.stfc.ac.uk/mast/level2/shots/<shot_id>.zarr (level1 for level 1).

All signals#

Signals are stored as one file per shot: 15,969 files for level 1 and 11,573 for level 2. To search signals across all shots, download the folder with s5cmd and read it as a single DataFrame:

s5cmd --no-sign-request --endpoint-url https://s3.echo.stfc.ac.uk \
  cp "s3://mast/fairmast-meta/l2_signals/*" ./l2_signals/

signals = pd.read_parquet("./l2_signals")
signals[signals.name == "ip"]

See the Quickstart for a runnable introduction and Bulk Download for more on s5cmd.