!pip install xarray zarr
Show code cell output
Requirement already satisfied: xarray in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (2023.7.0)
Requirement already satisfied: zarr in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (2.15.0)
Requirement already satisfied: numpy>=1.21 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from xarray) (1.25.2)
Requirement already satisfied: pandas>=1.4 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from xarray) (2.0.3)
Requirement already satisfied: packaging>=21.3 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from xarray) (24.0)
Requirement already satisfied: asciitree in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from zarr) (0.3.3)
Requirement already satisfied: fasteners in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from zarr) (0.19)
Requirement already satisfied: numcodecs>=0.10.0 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from zarr) (0.12.1)
Requirement already satisfied: python-dateutil>=2.8.2 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from pandas>=1.4->xarray) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from pandas>=1.4->xarray) (2024.1)
Requirement already satisfied: tzdata>=2022.1 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from pandas>=1.4->xarray) (2024.1)
Requirement already satisfied: six>=1.5 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas>=1.4->xarray) (1.16.0)
Bulk Download#
This notebook shows how to perform bulk downloads with a S3 command line tool. This is useful if you want to have local access to a big subset of the data or event download the whole archive!
We can download data in bulk using any command line for that supports the S3 protocol. We recommend using the s5cmd tool.
Below we’re going to install s5cmd and then use it to download thompson scattering data from one shot.
First we install s5cmd:
%%bash
wget https://github.com/peak/s5cmd/releases/download/v2.2.2/s5cmd_2.2.2_Linux-64bit.tar.gz
tar -xvzf s5cmd_2.2.2_Linux-64bit.tar.gz
./s5cmd --help
Show code cell output
--2024-11-22 08:51:32-- https://github.com/peak/s5cmd/releases/download/v2.2.2/s5cmd_2.2.2_Linux-64bit.tar.gz
Resolving github.com (github.com)... 20.26.156.215
Connecting to github.com (github.com)|20.26.156.215|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/73909333/e095ae85-9acf-4dcc-b744-128b3311849c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241122%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241122T085133Z&X-Amz-Expires=300&X-Amz-Signature=04cf478f510de5300bae25acf3712a1f3621897191f118e10743629d53f63151&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Ds5cmd_2.2.2_Linux-64bit.tar.gz&response-content-type=application%2Foctet-stream [following]
--2024-11-22 08:51:33-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/73909333/e095ae85-9acf-4dcc-b744-128b3311849c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241122%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241122T085133Z&X-Amz-Expires=300&X-Amz-Signature=04cf478f510de5300bae25acf3712a1f3621897191f118e10743629d53f63151&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Ds5cmd_2.2.2_Linux-64bit.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4821452 (4.6M) [application/octet-stream]
Saving to: ‘s5cmd_2.2.2_Linux-64bit.tar.gz’
0K .......... .......... .......... .......... .......... 1% 3.19M 1s
50K .......... .......... .......... .......... .......... 2% 8.81M 1s
100K .......... .......... .......... .......... .......... 3% 7.17M 1s
150K .......... .......... .......... .......... .......... 4% 8.47M 1s
200K .......... .......... .......... .......... .......... 5% 7.90M 1s
250K .......... .......... .......... .......... .......... 6% 7.88M 1s
300K .......... .......... .......... .......... .......... 7% 6.82M 1s
350K .......... .......... .......... .......... .......... 8% 6.46M 1s
400K .......... .......... .......... .......... .......... 9% 9.92M 1s
450K .......... .......... .......... .......... .......... 10% 8.16M 1s
500K .......... .......... .......... .......... .......... 11% 8.13M 1s
550K .......... .......... .......... .......... .......... 12% 8.79M 1s
600K .......... .......... .......... .......... .......... 13% 7.85M 1s
650K .......... .......... .......... .......... .......... 14% 7.89M 1s
700K .......... .......... .......... .......... .......... 15% 6.48M 1s
750K .......... .......... .......... .......... .......... 16% 8.28M 1s
800K .......... .......... .......... .......... .......... 18% 7.79M 1s
850K .......... .......... .......... .......... .......... 19% 8.04M 1s
900K .......... .......... .......... .......... .......... 20% 6.31M 1s
950K .......... .......... .......... .......... .......... 21% 12.4M 0s
1000K .......... .......... .......... .......... .......... 22% 8.14M 0s
1050K .......... .......... .......... .......... .......... 23% 8.45M 0s
1100K .......... .......... .......... .......... .......... 24% 6.07M 0s
1150K .......... .......... .......... .......... .......... 25% 8.75M 0s
1200K .......... .......... .......... .......... .......... 26% 7.17M 0s
1250K .......... .......... .......... .......... .......... 27% 8.97M 0s
1300K .......... .......... .......... .......... .......... 28% 6.71M 0s
1350K .......... .......... .......... .......... .......... 29% 9.58M 0s
1400K .......... .......... .......... .......... .......... 30% 8.98M 0s
1450K .......... .......... .......... .......... .......... 31% 6.97M 0s
1500K .......... .......... .......... .......... .......... 32% 7.18M 0s
1550K .......... .......... .......... .......... .......... 33% 8.67M 0s
1600K .......... .......... .......... .......... .......... 35% 4.76M 0s
1650K .......... .......... .......... .......... .......... 36% 27.6M 0s
1700K .......... .......... .......... .......... .......... 37% 6.95M 0s
1750K .......... .......... .......... .......... .......... 38% 8.98M 0s
1800K .......... .......... .......... .......... .......... 39% 7.04M 0s
1850K .......... .......... .......... .......... .......... 40% 10.9M 0s
1900K .......... .......... .......... .......... .......... 41% 5.99M 0s
1950K .......... .......... .......... .......... .......... 42% 8.24M 0s
2000K .......... .......... .......... .......... .......... 43% 8.50M 0s
2050K .......... .......... .......... .......... .......... 44% 6.60M 0s
2100K .......... .......... .......... .......... .......... 45% 10.5M 0s
2150K .......... .......... .......... .......... .......... 46% 8.02M 0s
2200K .......... .......... .......... .......... .......... 47% 7.41M 0s
2250K .......... .......... .......... .......... .......... 48% 9.55M 0s
2300K .......... .......... .......... .......... .......... 49% 6.24M 0s
2350K .......... .......... .......... .......... .......... 50% 7.74M 0s
2400K .......... .......... .......... .......... .......... 52% 8.60M 0s
2450K .......... .......... .......... .......... .......... 53% 8.38M 0s
2500K .......... .......... .......... .......... .......... 54% 6.09M 0s
2550K .......... .......... .......... .......... .......... 55% 7.76M 0s
2600K .......... .......... .......... .......... .......... 56% 11.7M 0s
2650K .......... .......... .......... .......... .......... 57% 9.34M 0s
2700K .......... .......... .......... .......... .......... 58% 1.03M 0s
2750K .......... .......... .......... .......... .......... 59% 14.1M 0s
2800K .......... .......... .......... .......... .......... 60% 407M 0s
2850K .......... .......... .......... .......... .......... 61% 618M 0s
2900K .......... .......... .......... .......... .......... 62% 626M 0s
2950K .......... .......... .......... .......... .......... 63% 688M 0s
3000K .......... .......... .......... .......... .......... 64% 2.11M 0s
3050K .......... .......... .......... .......... .......... 65% 432M 0s
3100K .......... .......... .......... .......... .......... 66% 456M 0s
3150K .......... .......... .......... .......... .......... 67% 642M 0s
3200K .......... .......... .......... .......... .......... 69% 555M 0s
3250K .......... .......... .......... .......... .......... 70% 15.0M 0s
3300K .......... .......... .......... .......... .......... 71% 12.2M 0s
3350K .......... .......... .......... .......... .......... 72% 8.71M 0s
3400K .......... .......... .......... .......... .......... 73% 6.34M 0s
3450K .......... .......... .......... .......... .......... 74% 12.4M 0s
3500K .......... .......... .......... .......... .......... 75% 4.48M 0s
3550K .......... .......... .......... .......... .......... 76% 9.00M 0s
3600K .......... .......... .......... .......... .......... 77% 11.9M 0s
3650K .......... .......... .......... .......... .......... 78% 7.78M 0s
3700K .......... .......... .......... .......... .......... 79% 8.02M 0s
3750K .......... .......... .......... .......... .......... 80% 9.80M 0s
3800K .......... .......... .......... .......... .......... 81% 7.97M 0s
3850K .......... .......... .......... .......... .......... 82% 8.45M 0s
3900K .......... .......... .......... .......... .......... 83% 5.54M 0s
3950K .......... .......... .......... .......... .......... 84% 9.58M 0s
4000K .......... .......... .......... .......... .......... 86% 6.18M 0s
4050K .......... .......... .......... .......... .......... 87% 8.45M 0s
4100K .......... .......... .......... .......... .......... 88% 9.81M 0s
4150K .......... .......... .......... .......... .......... 89% 8.92M 0s
4200K .......... .......... .......... .......... .......... 90% 5.44M 0s
4250K .......... .......... .......... .......... .......... 91% 9.25M 0s
4300K .......... .......... .......... .......... .......... 92% 8.85M 0s
4350K .......... .......... .......... .......... .......... 93% 8.17M 0s
4400K .......... .......... .......... .......... .......... 94% 8.43M 0s
4450K .......... .......... .......... .......... .......... 95% 7.59M 0s
4500K .......... .......... .......... .......... .......... 96% 7.18M 0s
4550K .......... .......... .......... .......... .......... 97% 8.54M 0s
4600K .......... .......... .......... .......... .......... 98% 10.7M 0s
4650K .......... .......... .......... .......... .......... 99% 7.46M 0s
4700K ........ 100% 10.8M=0.6s
2024-11-22 08:51:34 (7.77 MB/s) - ‘s5cmd_2.2.2_Linux-64bit.tar.gz’ saved [4821452/4821452]
x CHANGELOG.md
x LICENSE
x README.md
x s5cmd
bash: line 4: ./s5cmd: cannot execute binary file
---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
Cell In[2], line 1
----> 1 get_ipython().run_cell_magic('bash', '', '\nwget https://github.com/peak/s5cmd/releases/download/v2.2.2/s5cmd_2.2.2_Linux-64bit.tar.gz\ntar -xvzf s5cmd_2.2.2_Linux-64bit.tar.gz\n./s5cmd --help\n')
File ~/miniconda3/envs/mast/lib/python3.11/site-packages/IPython/core/interactiveshell.py:2541, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
2539 with self.builtin_trap:
2540 args = (magic_arg_s, cell)
-> 2541 result = fn(*args, **kwargs)
2543 # The code below prevents the output from being displayed
2544 # when using magics with decorator @output_can_be_silenced
2545 # when the last Python token in the expression is a ';'.
2546 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):
File ~/miniconda3/envs/mast/lib/python3.11/site-packages/IPython/core/magics/script.py:155, in ScriptMagics._make_script_magic.<locals>.named_script_magic(line, cell)
153 else:
154 line = script
--> 155 return self.shebang(line, cell)
File ~/miniconda3/envs/mast/lib/python3.11/site-packages/IPython/core/magics/script.py:315, in ScriptMagics.shebang(self, line, cell)
310 if args.raise_error and p.returncode != 0:
311 # If we get here and p.returncode is still None, we must have
312 # killed it but not yet seen its return code. We don't wait for it,
313 # in case it's stuck in uninterruptible sleep. -9 = SIGKILL
314 rc = p.returncode or -9
--> 315 raise CalledProcessError(rc, cell)
CalledProcessError: Command 'b'\nwget https://github.com/peak/s5cmd/releases/download/v2.2.2/s5cmd_2.2.2_Linux-64bit.tar.gz\ntar -xvzf s5cmd_2.2.2_Linux-64bit.tar.gz\n./s5cmd --help\n'' returned non-zero exit status 126.
Now we can download data using the cp command.
In this example, we are going to transfer the thompson scattering data for shot 30420 locally.
We need to set the endpoint of where the bucket is hosted (for now: https://s3.echo.stfc.ac.uk) and we need to set --no-sign-request for annonymous access.
%%bash
./s5cmd --no-sign-request --endpoint-url https://s3.echo.stfc.ac.uk cp s3://mast/level1/shots/30420.zarr/ayc/* ./30420.zarr/ayc
cp s3://mast/level1/shots/30420.zarr/ayc/.zgroup 30420.zarr/ayc/.zgroup
cp s3://mast/level1/shots/30420.zarr/ayc/.zattrs 30420.zarr/ayc/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/version_poly/.zmetadata 30420.zarr/ayc/version_poly/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/laser_energy/.zattrs 30420.zarr/ayc/laser_energy/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/instrument_dr/.zattrs 30420.zarr/ayc/instrument_dr/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/interferometer_corr/0 30420.zarr/ayc/interferometer_corr/0
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_error/.zattrs 30420.zarr/ayc/spectra_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/ne_error/.zattrs 30420.zarr/ayc/ne_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectra/.zmetadata 30420.zarr/ayc/spectra/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/segment_number/.zattrs 30420.zarr/ayc/segment_number/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/scat_length/.zmetadata 30420.zarr/ayc/scat_length/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/ne_core/0 30420.zarr/ayc/ne_core/0
cp s3://mast/level1/shots/30420.zarr/ayc/chi2/.zattrs 30420.zarr/ayc/chi2/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/te_error/.zmetadata 30420.zarr/ayc/te_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/version_raman/.zmetadata 30420.zarr/ayc/version_raman/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_stray_light/.zmetadata 30420.zarr/ayc/spectra_stray_light/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/te_core/.zattrs 30420.zarr/ayc/te_core/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/acqiris_time/.zarray 30420.zarr/ayc/acqiris_time/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/r_error/.zattrs 30420.zarr/ayc/r_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/scat_length/0 30420.zarr/ayc/scat_length/0
cp s3://mast/level1/shots/30420.zarr/ayc/te_error/.zarray 30420.zarr/ayc/te_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/arb/.zattrs 30420.zarr/ayc/arb/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/acqiris_time/0 30420.zarr/ayc/acqiris_time/0
cp s3://mast/level1/shots/30420.zarr/ayc/angle/.zattrs 30420.zarr/ayc/angle/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/aspectra/.zattrs 30420.zarr/ayc/aspectra/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/aspectra/1.0.0 30420.zarr/ayc/aspectra/1.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/aspectra/.zmetadata 30420.zarr/ayc/aspectra/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/chi2/0.0 30420.zarr/ayc/chi2/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/arb/.zmetadata 30420.zarr/ayc/arb/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/aspectra/.zarray 30420.zarr/ayc/aspectra/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/radialindex/.zarray 30420.zarr/ayc/radialindex/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/time_/.zarray 30420.zarr/ayc/time_/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/ne_core/.zmetadata 30420.zarr/ayc/ne_core/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/yag_nelint/.zarray 30420.zarr/ayc/yag_nelint/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/laser_track/.zarray 30420.zarr/ayc/laser_track/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/status/.zmetadata 30420.zarr/ayc/status/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_error/1.0.0 30420.zarr/ayc/spectra_error/1.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/passnumber/.zattrs 30420.zarr/ayc/passnumber/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/passnumber/.zarray 30420.zarr/ayc/passnumber/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/.zmetadata 30420.zarr/ayc/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/segment_number/.zmetadata 30420.zarr/ayc/segment_number/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/radialindex/.zattrs 30420.zarr/ayc/radialindex/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/te_error/.zattrs 30420.zarr/ayc/te_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectral_index/.zattrs 30420.zarr/ayc/spectral_index/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectral_index/.zmetadata 30420.zarr/ayc/spectral_index/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/radialindex/0 30420.zarr/ayc/radialindex/0
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_stray_light/.zarray 30420.zarr/ayc/spectra_stray_light/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/interferometer_corr/.zattrs 30420.zarr/ayc/interferometer_corr/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/te_error/0.0 30420.zarr/ayc/te_error/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/angle/0 30420.zarr/ayc/angle/0
cp s3://mast/level1/shots/30420.zarr/ayc/ne_core/.zarray 30420.zarr/ayc/ne_core/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/laser_energy/0 30420.zarr/ayc/laser_energy/0
cp s3://mast/level1/shots/30420.zarr/ayc/te_core/0 30420.zarr/ayc/te_core/0
cp s3://mast/level1/shots/30420.zarr/ayc/spectral_index/0 30420.zarr/ayc/spectral_index/0
cp s3://mast/level1/shots/30420.zarr/ayc/poisson_error/.zarray 30420.zarr/ayc/poisson_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/instrument_dr/0.0 30420.zarr/ayc/instrument_dr/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/plasmalight_error/.zmetadata 30420.zarr/ayc/plasmalight_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/interferometer_corr/.zmetadata 30420.zarr/ayc/interferometer_corr/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/polyname/.zmetadata 30420.zarr/ayc/polyname/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/version_poly/.zarray 30420.zarr/ayc/version_poly/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/radialindex/.zmetadata 30420.zarr/ayc/radialindex/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/xyc_time/0 30420.zarr/ayc/xyc_time/0
cp s3://mast/level1/shots/30420.zarr/ayc/xyc_time/.zarray 30420.zarr/ayc/xyc_time/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/polyname/.zattrs 30420.zarr/ayc/polyname/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/pe/.zmetadata 30420.zarr/ayc/pe/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/pulse/.zattrs 30420.zarr/ayc/pulse/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/version_fibre/.zattrs 30420.zarr/ayc/version_fibre/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/r_core/.zattrs 30420.zarr/ayc/r_core/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectralindex/.zmetadata 30420.zarr/ayc/spectralindex/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/time_/.zattrs 30420.zarr/ayc/time_/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/angle/.zarray 30420.zarr/ayc/angle/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/instrument_dr/.zarray 30420.zarr/ayc/instrument_dr/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/poisson_error/0.0.0 30420.zarr/ayc/poisson_error/0.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_error/0.0.0 30420.zarr/ayc/spectra_error/0.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/poisson_error/1.0.0 30420.zarr/ayc/poisson_error/1.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/xyc_time/.zattrs 30420.zarr/ayc/xyc_time/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/chi2/.zmetadata 30420.zarr/ayc/chi2/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/covariance_ne_te/.zattrs 30420.zarr/ayc/covariance_ne_te/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/te_core/.zmetadata 30420.zarr/ayc/te_core/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/time_/.zmetadata 30420.zarr/ayc/time_/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/r/.zmetadata 30420.zarr/ayc/r/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_error/.zarray 30420.zarr/ayc/spectra_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/r_core/.zarray 30420.zarr/ayc/r_core/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/plasmalight_error/1.0.0 30420.zarr/ayc/plasmalight_error/1.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/laser_track/0 30420.zarr/ayc/laser_track/0
cp s3://mast/level1/shots/30420.zarr/ayc/plasmalight_error/.zarray 30420.zarr/ayc/plasmalight_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/poisson_error/.zmetadata 30420.zarr/ayc/poisson_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/yag_nelint/0 30420.zarr/ayc/yag_nelint/0
cp s3://mast/level1/shots/30420.zarr/ayc/version_fibre/0 30420.zarr/ayc/version_fibre/0
cp s3://mast/level1/shots/30420.zarr/ayc/r/.zattrs 30420.zarr/ayc/r/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectra/.zattrs 30420.zarr/ayc/spectra/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/radial_index/.zmetadata 30420.zarr/ayc/radial_index/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/ne/.zmetadata 30420.zarr/ayc/ne/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/scat_length/.zattrs 30420.zarr/ayc/scat_length/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_stray_light/1.0.0 30420.zarr/ayc/spectra_stray_light/1.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/acqiris_time/.zattrs 30420.zarr/ayc/acqiris_time/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/laser_track/.zmetadata 30420.zarr/ayc/laser_track/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/angle/.zmetadata 30420.zarr/ayc/angle/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/xyc_time/.zmetadata 30420.zarr/ayc/xyc_time/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/laser_track/.zattrs 30420.zarr/ayc/laser_track/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectral_index/.zarray 30420.zarr/ayc/spectral_index/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/radial_index/.zarray 30420.zarr/ayc/radial_index/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/time_/0 30420.zarr/ayc/time_/0
cp s3://mast/level1/shots/30420.zarr/ayc/spectralindex/.zattrs 30420.zarr/ayc/spectralindex/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/status/0 30420.zarr/ayc/status/0
cp s3://mast/level1/shots/30420.zarr/ayc/instrument_dr/.zmetadata 30420.zarr/ayc/instrument_dr/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/interferometer_corr/.zarray 30420.zarr/ayc/interferometer_corr/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/radial_index/0 30420.zarr/ayc/radial_index/0
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_error/.zmetadata 30420.zarr/ayc/spectra_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/passnumber/0 30420.zarr/ayc/passnumber/0
cp s3://mast/level1/shots/30420.zarr/ayc/version_fibre/.zarray 30420.zarr/ayc/version_fibre/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_stray_light/.zattrs 30420.zarr/ayc/spectra_stray_light/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/pe/.zattrs 30420.zarr/ayc/pe/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/r_core/0 30420.zarr/ayc/r_core/0
cp s3://mast/level1/shots/30420.zarr/ayc/r_error/.zmetadata 30420.zarr/ayc/r_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/yag_nelint/.zattrs 30420.zarr/ayc/yag_nelint/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/segment_number/.zarray 30420.zarr/ayc/segment_number/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/pulse/.zmetadata 30420.zarr/ayc/pulse/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/radial_index/.zattrs 30420.zarr/ayc/radial_index/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/r_core/.zmetadata 30420.zarr/ayc/r_core/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/covariance_ne_te/.zarray 30420.zarr/ayc/covariance_ne_te/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/te/.zmetadata 30420.zarr/ayc/te/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/pe/0.0 30420.zarr/ayc/pe/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/laser_energy/.zarray 30420.zarr/ayc/laser_energy/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/t/.zarray 30420.zarr/ayc/t/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/time/.zarray 30420.zarr/ayc/time/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/pulse/0 30420.zarr/ayc/pulse/0
cp s3://mast/level1/shots/30420.zarr/ayc/version_raman/0 30420.zarr/ayc/version_raman/0
cp s3://mast/level1/shots/30420.zarr/ayc/ne_error/0.0 30420.zarr/ayc/ne_error/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/spectra/1.0.0 30420.zarr/ayc/spectra/1.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/r/.zarray 30420.zarr/ayc/r/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/polyname/0 30420.zarr/ayc/polyname/0
cp s3://mast/level1/shots/30420.zarr/ayc/pe_error/.zattrs 30420.zarr/ayc/pe_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/te/0.0 30420.zarr/ayc/te/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/scat_length/.zarray 30420.zarr/ayc/scat_length/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/pulse/.zarray 30420.zarr/ayc/pulse/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/pe/.zarray 30420.zarr/ayc/pe/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/spectralindex/.zarray 30420.zarr/ayc/spectralindex/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/t/0 30420.zarr/ayc/t/0
cp s3://mast/level1/shots/30420.zarr/ayc/version_poly/.zattrs 30420.zarr/ayc/version_poly/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/ne_error/.zmetadata 30420.zarr/ayc/ne_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/laser_energy/.zmetadata 30420.zarr/ayc/laser_energy/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/r_error/0.0 30420.zarr/ayc/r_error/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/version_fibre/.zmetadata 30420.zarr/ayc/version_fibre/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/version_poly/0 30420.zarr/ayc/version_poly/0
cp s3://mast/level1/shots/30420.zarr/ayc/arb/0 30420.zarr/ayc/arb/0
cp s3://mast/level1/shots/30420.zarr/ayc/t/.zmetadata 30420.zarr/ayc/t/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/version_raman/.zarray 30420.zarr/ayc/version_raman/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/time/.zattrs 30420.zarr/ayc/time/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/te_core/.zarray 30420.zarr/ayc/te_core/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/time/0 30420.zarr/ayc/time/0
cp s3://mast/level1/shots/30420.zarr/ayc/polyname/.zarray 30420.zarr/ayc/polyname/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/poisson_error/.zattrs 30420.zarr/ayc/poisson_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/status/.zattrs 30420.zarr/ayc/status/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/acqiris_time/.zmetadata 30420.zarr/ayc/acqiris_time/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/yag_nelint/.zmetadata 30420.zarr/ayc/yag_nelint/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/pe_error/.zarray 30420.zarr/ayc/pe_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/ne/.zarray 30420.zarr/ayc/ne/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/status/.zarray 30420.zarr/ayc/status/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/segment_number/0 30420.zarr/ayc/segment_number/0
cp s3://mast/level1/shots/30420.zarr/ayc/te/.zarray 30420.zarr/ayc/te/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/version_raman/.zattrs 30420.zarr/ayc/version_raman/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/r/0.0 30420.zarr/ayc/r/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/plasmalight_error/.zattrs 30420.zarr/ayc/plasmalight_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/covariance_ne_te/.zmetadata 30420.zarr/ayc/covariance_ne_te/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/te/.zattrs 30420.zarr/ayc/te/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/pe_error/.zmetadata 30420.zarr/ayc/pe_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/spectralindex/0 30420.zarr/ayc/spectralindex/0
cp s3://mast/level1/shots/30420.zarr/ayc/chi2/.zarray 30420.zarr/ayc/chi2/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/r_error/.zarray 30420.zarr/ayc/r_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/ne_error/.zarray 30420.zarr/ayc/ne_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/t/.zattrs 30420.zarr/ayc/t/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/time/.zmetadata 30420.zarr/ayc/time/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_stray_light/0.0.0 30420.zarr/ayc/spectra_stray_light/0.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/pe_error/0.0 30420.zarr/ayc/pe_error/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/covariance_ne_te/0.0 30420.zarr/ayc/covariance_ne_te/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/ne_core/.zattrs 30420.zarr/ayc/ne_core/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/passnumber/.zmetadata 30420.zarr/ayc/passnumber/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/arb/.zarray 30420.zarr/ayc/arb/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/ne/0.0 30420.zarr/ayc/ne/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/ne/.zattrs 30420.zarr/ayc/ne/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectra/.zarray 30420.zarr/ayc/spectra/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/aspectra/0.0.0 30420.zarr/ayc/aspectra/0.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/spectra/0.0.0 30420.zarr/ayc/spectra/0.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/plasmalight_error/0.0.0 30420.zarr/ayc/plasmalight_error/0.0.0
Finally, we can open the file locally:
import xarray as xr
xr.open_zarr('30420.zarr', group='ayc')
<ipython-input-4-bcb7748acee7>:2: RuntimeWarning: Failed to open Zarr store with consolidated metadata, but successfully read with non-consolidated metadata. This is typically much slower for opening a dataset. To silence this warning, consider:
1. Consolidating metadata in this existing store with zarr.consolidate_metadata().
2. Explicitly setting consolidated=False, to avoid trying to read consolidate metadata, or
3. Explicitly setting consolidated=True, to raise an error in this case instead of falling back to try reading non-consolidated metadata.
xr.open_zarr('30420.zarr', group='ayc')
<xarray.Dataset>
Dimensions: (time: 146, arb: 130, radial_index: 130,
spectral_index: 4, radialindex: 130,
spectralindex: 4, t: 146)
Coordinates:
* arb (arb) float32 0.0 1.0 2.0 3.0 ... 127.0 128.0 129.0
* radial_index (radial_index) float32 1.0 2.0 3.0 ... 129.0 130.0
* radialindex (radialindex) float32 1.0 2.0 3.0 ... 128.0 129.0 130.0
* spectral_index (spectral_index) float32 1.0 2.0 3.0 4.0
* spectralindex (spectralindex) float32 1.0 2.0 3.0 4.0
* t (t) float32 0.0 0.004166 0.008332 ... 70.0 71.0 72.0
* time (time) float32 0.0 0.004166 0.008332 ... 70.0 71.0 72.0
Data variables: (12/37)
acqiris_time (time) float32 dask.array<chunksize=(146,), meta=np.ndarray>
angle (arb) float32 dask.array<chunksize=(130,), meta=np.ndarray>
aspectra (time, radial_index, spectral_index) float32 dask.array<chunksize=(73, 130, 4), meta=np.ndarray>
chi2 (time, radial_index) float32 dask.array<chunksize=(146, 130), meta=np.ndarray>
covariance_ne_te (time, radial_index) float32 dask.array<chunksize=(146, 130), meta=np.ndarray>
instrument_dr (time, radial_index) float32 dask.array<chunksize=(146, 130), meta=np.ndarray>
... ...
time_ (time) float32 dask.array<chunksize=(146,), meta=np.ndarray>
version_fibre float32 ...
version_poly float32 ...
version_raman float32 ...
xyc_time (time) float32 dask.array<chunksize=(146,), meta=np.ndarray>
yag_nelint (time) float32 dask.array<chunksize=(146,), meta=np.ndarray>
Attributes:
description: Core Thomson scattering data
file_name: ayc0304.20
format: IDA3
mds_name: None
name: ayc
quality: Not Checked
shot_id: 30420
signal_type: Analysed
source: ayc
uda_name: AYC
uuid: 8d043ece-8bf8-5af8-87e4-d2a1b01716fa
version: 0