Bulk Data Download

!pip install aiohttp requests s5cmd "xarray[io]"

Hide code cell output

 
Defaulting to user installation because normal site-packages is not writeable

Collecting aiohttp

  Downloading aiohttp-3.14.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (8.3 kB)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (2.31.0)
Collecting s5cmd

  Downloading s5cmd-0.3.3-py3-none-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (7.0 kB)
Collecting xarray[io]

  Downloading xarray-2026.9.0-py3-none-any.whl.metadata (12 kB)
Collecting aiohappyeyeballs>=2.5.0 (from aiohttp)
  Downloading aiohappyeyeballs-2.7.1-py3-none-any.whl.metadata (5.9 kB)

Collecting aiosignal>=1.4.0 (from aiohttp)
  Downloading aiosignal-1.4.0-py3-none-any.whl.metadata (3.7 kB)
Requirement already satisfied: attrs>=17.3.0 in /usr/lib/python3/dist-packages (from aiohttp) (23.2.0)

Collecting frozenlist>=1.1.1 (from aiohttp)

  Downloading frozenlist-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (20 kB)

Collecting multidict<8.0,>=4.5 (from aiohttp)
  Downloading multidict-7.2.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (6.1 kB)

Collecting propcache>=0.2.0 (from aiohttp)
  Downloading propcache-0.5.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (25 kB)

Requirement already satisfied: typing_extensions>=4.4 in /usr/lib/python3/dist-packages (from aiohttp) (4.10.0)

Collecting yarl<2.0,>=1.25.1 (from aiohttp)
  Downloading yarl-1.25.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (111 kB)
Requirement already satisfied: idna>=2.0 in /usr/lib/python3/dist-packages (from yarl<2.0,>=1.25.1->aiohttp) (3.6)

Collecting numpy>=1.26 (from xarray[io])
  Downloading numpy-2.5.4-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (6.6 kB)

Collecting packaging>=24.2 (from xarray[io])
  Downloading packaging-26.3-py3-none-any.whl.metadata (3.5 kB)

Collecting pandas>=2.2 (from xarray[io])

  Downloading pandas-3.1.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (18 kB)

Collecting netCDF4>=1.6.0 (from xarray[io])
  Downloading netcdf4-1.7.5.1.post1-cp311-abi3-manylinux_2_28_x86_64.whl.metadata (1.7 kB)

Collecting h5netcdf>=1.8.0 (from h5netcdf[h5py]>=1.8.0; extra == "io"->xarray[io])
  Downloading h5netcdf-1.8.1-py3-none-any.whl.metadata (15 kB)
Collecting pydap (from xarray[io])

  Downloading pydap-3.5.11-py3-none-any.whl.metadata (9.3 kB)

Collecting scipy>=1.15 (from xarray[io])
  Downloading scipy-1.18.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (62 kB)

Collecting zarr>=3.0 (from xarray[io])
  Downloading zarr-3.4.1-py3-none-any.whl.metadata (5.8 kB)

Collecting fsspec (from xarray[io])
  Downloading fsspec-2026.9.0-py3-none-any.whl.metadata (10 kB)

Collecting cftime (from xarray[io])
  Downloading cftime-1.6.6.1-cp311-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (8.8 kB)

Collecting pooch (from xarray[io])
  Downloading pooch-1.9.0-py3-none-any.whl.metadata (10 kB)

Collecting h5py (from h5netcdf[h5py]>=1.8.0; extra == "io"->xarray[io])
  Downloading h5py-3.16.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (3.0 kB)

Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from netCDF4>=1.6.0->xarray[io]) (2023.11.17)
Collecting python-dateutil>=2.9.0 (from pandas>=2.2->xarray[io])
  Downloading python_dateutil-3.0.0-py2.py3-none-any.whl.metadata (8.2 kB)
Requirement already satisfied: six>=1.6 in /usr/lib/python3/dist-packages (from python-dateutil>=2.9.0->pandas>=2.2->xarray[io]) (1.16.0)

Collecting tzdata (from python-dateutil>=2.9.0->pandas>=2.2->xarray[io])
  Downloading tzdata-2026.5-py2.py3-none-any.whl.metadata (1.4 kB)

Collecting donfig>=0.8 (from zarr>=3.0->xarray[io])
  Downloading donfig-0.8.1.post1-py3-none-any.whl.metadata (5.0 kB)

Collecting google-crc32c>=1.5 (from zarr>=3.0->xarray[io])
  Downloading google_crc32c-1.9.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.2 kB)

Collecting msgspec>=0.19 (from zarr>=3.0->xarray[io])
  Downloading msgspec-0.22.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (6.0 kB)

Collecting numcodecs>=0.16 (from zarr>=3.0->xarray[io])
  Downloading numcodecs-0.17.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (3.2 kB)

Collecting typing_extensions>=4.4 (from aiohttp)
  Downloading typing_extensions-4.16.0-py3-none-any.whl.metadata (3.3 kB)
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from donfig>=0.8->zarr>=3.0->xarray[io]) (6.0.1)
Requirement already satisfied: platformdirs>=2.5.0 in /usr/local/lib/python3.12/dist-packages (from pooch->xarray[io]) (4.13.0)

Collecting requests-cache (from pydap->xarray[io])
  Downloading requests_cache-1.3.3-py3-none-any.whl.metadata (9.4 kB)

Collecting webob>=1.8.10 (from pydap->xarray[io])
  Downloading webob-1.8.11-py2.py3-none-any.whl.metadata (13 kB)
Collecting beautifulsoup4 (from pydap->xarray[io])

  Downloading beautifulsoup4-4.15.0-py3-none-any.whl.metadata (3.8 kB)

Collecting lxml (from pydap->xarray[io])
  Downloading lxml-6.1.3-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl.metadata (3.3 kB)
Requirement already satisfied: Jinja2 in /usr/lib/python3/dist-packages (from pydap->xarray[io]) (3.1.2)

Collecting soupsieve>=1.6.1 (from beautifulsoup4->pydap->xarray[io])
  Downloading soupsieve-3.0.4-py3-none-any.whl.metadata (4.4 kB)
Collecting cattrs>=22.2 (from requests-cache->pydap->xarray[io])

  Downloading cattrs-26.2.1-py3-none-any.whl.metadata (8.5 kB)
Collecting url-normalize>=2.0 (from requests-cache->pydap->xarray[io])
  Downloading url_normalize-3.0.1-py3-none-any.whl.metadata (7.8 kB)
Requirement already satisfied: urllib3>=1.25.5 in /usr/lib/python3/dist-packages (from requests-cache->pydap->xarray[io]) (2.0.7)

Collecting attrs>=17.3.0 (from aiohttp)
  Downloading attrs-26.1.0-py3-none-any.whl.metadata (8.8 kB)
Downloading aiohttp-3.14.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (1.9 MB)
?25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.9 MB ? eta -:--:--

   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 33.1 MB/s  0:00:00
?25hDownloading multidict-7.2.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (516 kB)

Downloading yarl-1.25.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (117 kB)
Downloading s5cmd-0.3.3-py3-none-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (5.0 MB)
?25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/5.0 MB ? eta -:--:--

   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.0/5.0 MB 85.3 MB/s  0:00:00
?25hDownloading xarray-2026.9.0-py3-none-any.whl (1.4 MB)
?25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.4 MB ? eta -:--:--
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 327.7 MB/s  0:00:00
?25h

Downloading aiohappyeyeballs-2.7.1-py3-none-any.whl (15 kB)
Downloading aiosignal-1.4.0-py3-none-any.whl (7.5 kB)
Downloading frozenlist-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (242 kB)
Downloading h5netcdf-1.8.1-py3-none-any.whl (62 kB)

Downloading netcdf4-1.7.5.1.post1-cp311-abi3-manylinux_2_28_x86_64.whl (10.6 MB)
?25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/10.6 MB ? eta -:--:--

   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.6/10.6 MB 135.7 MB/s  0:00:00
?25h

Downloading numpy-2.5.4-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.7 MB)
?25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/16.7 MB ? eta -:--:--

   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.7/16.7 MB 144.0 MB/s  0:00:00
?25hDownloading packaging-26.3-py3-none-any.whl (129 kB)

Downloading pandas-3.1.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (12.0 MB)
?25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/12.0 MB ? eta -:--:--

   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.0/12.0 MB 159.7 MB/s  0:00:00
?25hDownloading propcache-0.5.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (250 kB)

Downloading python_dateutil-3.0.0-py2.py3-none-any.whl (94 kB)
Downloading scipy-1.18.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (35.3 MB)
?25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/35.3 MB ? eta -:--:--

   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 35.1/35.3 MB 695.3 MB/s eta 0:00:01

   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 35.3/35.3 MB 149.4 MB/s  0:00:00
?25hDownloading zarr-3.4.1-py3-none-any.whl (405 kB)
Downloading donfig-0.8.1.post1-py3-none-any.whl (21 kB)
Downloading google_crc32c-1.9.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (36 kB)

Downloading msgspec-0.22.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (230 kB)
Downloading numcodecs-0.17.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (1.4 MB)
?25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.4 MB ? eta -:--:--
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 261.4 MB/s  0:00:00
?25hDownloading typing_extensions-4.16.0-py3-none-any.whl (45 kB)

Downloading cftime-1.6.6.1-cp311-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (1.7 MB)
?25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.7 MB ? eta -:--:--
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 284.8 MB/s  0:00:00
?25hDownloading fsspec-2026.9.0-py3-none-any.whl (221 kB)

Downloading h5py-3.16.0-cp312-cp312-manylinux_2_28_x86_64.whl (5.4 MB)
?25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/5.4 MB ? eta -:--:--
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.4/5.4 MB 180.5 MB/s  0:00:00
?25hDownloading pooch-1.9.0-py3-none-any.whl (67 kB)

Downloading pydap-3.5.11-py3-none-any.whl (2.5 MB)
?25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.5 MB ? eta -:--:--
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 134.9 MB/s  0:00:00
?25hDownloading webob-1.8.11-py2.py3-none-any.whl (118 kB)

Downloading beautifulsoup4-4.15.0-py3-none-any.whl (109 kB)
Downloading soupsieve-3.0.4-py3-none-any.whl (42 kB)
Downloading lxml-6.1.3-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl (5.2 MB)
?25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/5.2 MB ? eta -:--:--

   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.2/5.2 MB 170.9 MB/s  0:00:00
?25hDownloading requests_cache-1.3.3-py3-none-any.whl (70 kB)
Downloading cattrs-26.2.1-py3-none-any.whl (74 kB)

Downloading attrs-26.1.0-py3-none-any.whl (67 kB)
Downloading url_normalize-3.0.1-py3-none-any.whl (18 kB)
Downloading tzdata-2026.5-py2.py3-none-any.whl (347 kB)

Installing collected packages: webob, url-normalize, tzdata, typing_extensions, soupsieve, s5cmd, propcache, packaging, numpy, multidict, msgspec, lxml, google-crc32c, fsspec, frozenlist, donfig, attrs, aiohappyeyeballs, yarl, scipy, python-dateutil, pooch, numcodecs, h5py, h5netcdf, cftime, cattrs, beautifulsoup4, aiosignal, zarr, requests-cache, pandas, netCDF4, aiohttp, xarray, pydap
?25l

?25h
Successfully installed aiohappyeyeballs-2.7.1 aiohttp-3.14.5 aiosignal-1.4.0 attrs-26.1.0 beautifulsoup4-4.15.0 cattrs-26.2.1 cftime-1.6.6.1 donfig-0.8.1.post1 frozenlist-1.8.0 fsspec-2026.9.0 google-crc32c-1.9.0 h5netcdf-1.8.1 h5py-3.16.0 lxml-6.1.3 msgspec-0.22.0 multidict-7.2.1 netCDF4-1.7.5.1.post1 numcodecs-0.17.1 numpy-2.5.4 packaging-26.3 pandas-3.1.0 pooch-1.9.0 propcache-0.5.4 pydap-3.5.11 python-dateutil-3.0.0 requests-cache-1.3.3 s5cmd-0.3.3 scipy-1.18.1 soupsieve-3.0.4 typing_extensions-4.16.0 tzdata-2026.5 url-normalize-3.0.1 webob-1.8.11 xarray-2026.9.0 yarl-1.25.1 zarr-3.4.1

import xarray as xr

Bulk Data Download#

This notebook shows how to perform bulk downloads with a S3 command line tool. This is useful if you want to have local access to a big subset of the data or event download the whole archive!

We can download data in bulk using any command line that supports the S3 protocol. We recommend using the s5cmd tool, which can be simply installed by running:

pip install s5cmd

Now we can download data using the cp command.

In this example, we are going to transfer the thompson scattering data for shot 30420 locally.

We need to set the endpoint of where the bucket is hosted (for now: https://s3.echo.stfc.ac.uk) and we need to set --no-sign-request for annonymous access.

%%capture --no-display
%%bash
s5cmd --no-sign-request --endpoint-url https://s3.echo.stfc.ac.uk cp s3://mast/level2/shots/30420.zarr/thomson_scattering/* ./30420.zarr/thomson_scattering;

Finally, we can open the file locally:

xr.open_zarr('30420.zarr/thomson_scattering', consolidated=False)

<xarray.Dataset> Size: 212kB
Dimensions:       (major_radius: 120, time: 73)
Coordinates:
  * major_radius  (major_radius) float64 960B 0.3 0.31 0.32 ... 1.47 1.48 1.49
  * time          (time) float32 292B 0.004166 0.008332 0.0125 ... 0.3 0.3042
    shot_id       int64 8B ...
Data variables:
    p_e           (major_radius, time) float64 70kB ...
    t_e_core      (time) float32 292B ...
    n_e_core      (time) float32 292B ...
    n_e           (major_radius, time) float64 70kB ...
    t_e           (major_radius, time) float64 70kB ...
Attributes:
    name:          thomson_scattering
    description:   Thomson scattering diagnostic
    imas:          thomson_scattering
    license_name:  Creative Commons 4.0 BY-SA
    license_url:   https://creativecommons.org/licenses/by-sa/4.0/
    ingested_at:   2026-09-24T17:12:34Z
    commit_url:    https://github.com/ukaea/fair-mast-ingestion/tree/ed6ea4e2...