Create a small sample RiOMar dataset

Context

Purpose

The goal is to create a smaller RiOMar dataset to test regridding to Healpix on Pangeo EOSC.

Description

In this notebook, we will:

• Open a RiOMar data file

• Select a few times to reduce the amount of data

• Save the transformed data in Zarr

Contributions

Notebook

• Tina Odaka (author), IFREMER (France), @tinaok

Bibliography and other interesting resources

• RiOMar

Warning

This notebook is designed to run on Datamor, the HPC cluster from IFREMER, where the RiOMar data currently resides. Running the notebook directly on Datamor is necessary because the dataset is large, and processing needs to occur close to the data for efficiency. However, the raw data is openly available online at https://data-fair2adapt.ifremer.fr/riomar/.

To enhance portability, we have included the URLs for the data online. However, executing it on the cloud would be very slow because the original data is in netCDF format, and we are simply reading it without leveraging chunks, for example, using tools like Kerchunk.

How to set up pangeo enviroment on datarmor for Fair2adapt riomar usecase:

ssh datarmor

micromamba create -n riomar python=3.12 xarray zarr  hdf5 ipykernel h5netcdf dask  netCDF4 bottleneck scipy  cftime numba healpy  matplotlib hvplot
pip install git+https://github.com/IAOCEA/xarray-healpy.git
python -m ipykernel install --user --name=riomar

Then connect to https://datarmor-jupyterhub.ifremer.fr/

## Import Libraries

import xarray as xr
import fsspec
from pathlib import Path

Open Croco grid file

• The grid file is either local if you are running on datamor or accessible via https is running elsewhere.

url = "/home/lops-oh-fair2adapt/riomar/misc/croco_grd.nc"
if Path(url).is_file():
    # file exists
    grid = xr.open_dataset(url, engine='netcdf4')
else:
    url = "https://data-fair2adapt.ifremer.fr/riomar//misc/croco_grd.nc"
    fs = fsspec.filesystem('https')
    grid = xr.open_dataset(fs.open(url))

#grid.to_netcdf('/home/lops-oh-fair2adapt/riomar/misc/croco_grd_hdf5.nc',format='NETCDF4')
grid

<xarray.Dataset> Size: 141MB
Dimensions:    (one: 1, eta_rho: 838, xi_rho: 727, bath: 1, eta_u: 838,
                xi_u: 726, eta_v: 837, xi_v: 727, eta_psi: 837, xi_psi: 726)
Dimensions without coordinates: one, eta_rho, xi_rho, bath, eta_u, xi_u, eta_v,
                                xi_v, eta_psi, xi_psi
Data variables: (12/34)
    xl         (one) float64 8B ...
    el         (one) float64 8B ...
    depthmin   (one) float64 8B ...
    depthmax   (one) float64 8B ...
    spherical  (one) |S1 1B ...
    angle      (eta_rho, xi_rho) float64 5MB ...
    ...         ...
    lat_v      (eta_v, xi_v) float64 5MB ...
    lat_psi    (eta_psi, xi_psi) float64 5MB ...
    mask_rho   (eta_rho, xi_rho) float64 5MB ...
    mask_u     (eta_u, xi_u) float64 5MB ...
    mask_v     (eta_v, xi_v) float64 5MB ...
    mask_psi   (eta_psi, xi_psi) float64 5MB ...
Attributes:
    title:    BOB1000 Model
    date:     09-Mar-2023
    type:     CROCO grid file

xarray.Dataset

• Dimensions:
  • one: 1
  • eta_rho: 838
  • xi_rho: 727
  • bath: 1
  • eta_u: 838
  • xi_u: 726
  • eta_v: 837
  • xi_v: 727
  • eta_psi: 837
  • xi_psi: 726
• Coordinates: (0)
• Data variables: (34)
  • xl
    (one)
    float64
    ...

    long_name :

    domain length in the XI-direction

    units :

    meter

    [1 values with dtype=float64]

  • el
    (one)
    float64
    ...

    long_name :

    domain length in the ETA-direction

    units :

    meter

    [1 values with dtype=float64]

  • depthmin
    (one)
    float64
    ...

    long_name :

    Shallow bathymetry clipping depth

    units :

    meter

    [1 values with dtype=float64]

  • depthmax
    (one)
    float64
    ...

    long_name :

    Deep bathymetry clipping depth

    units :

    meter

    [1 values with dtype=float64]

  • spherical
    (one)
    |S1
    ...

    long_name :

    Grid type logical switch

    option_T :

    spherical

    [1 values with dtype=|S1]

  • angle
    (eta_rho, xi_rho)
    float64
    ...

    long_name :

    angle between xi axis and east

    units :

    radian

    [609226 values with dtype=float64]

  • h
    (eta_rho, xi_rho)
    float64
    ...

    long_name :

    Final bathymetry at RHO-points

    units :

    meter

    [609226 values with dtype=float64]

  • hraw
    (bath, eta_rho, xi_rho)
    float64
    ...

    long_name :

    Working bathymetry at RHO-points

    units :

    meter

    [609226 values with dtype=float64]

  • alpha
    (eta_rho, xi_rho)
    float64
    ...

    long_name :

    Weights between coarse and fine grids at RHO-points

    [609226 values with dtype=float64]

  • f
    (eta_rho, xi_rho)
    float64
    ...

    long_name :

    Coriolis parameter at RHO-points

    units :

    second-1

    [609226 values with dtype=float64]

  • pm
    (eta_rho, xi_rho)
    float64
    ...

    long_name :

    curvilinear coordinate metric in XI

    units :

    meter-1

    [609226 values with dtype=float64]

  • pn
    (eta_rho, xi_rho)
    float64
    ...

    long_name :

    curvilinear coordinate metric in ETA

    units :

    meter-1

    [609226 values with dtype=float64]

  • dndx
    (eta_rho, xi_rho)
    float64
    ...

    long_name :

    xi derivative of inverse metric factor pn

    units :

    meter

    [609226 values with dtype=float64]

  • dmde
    (eta_rho, xi_rho)
    float64
    ...

    long_name :

    eta derivative of inverse metric factor pm

    units :

    meter

    [609226 values with dtype=float64]

  • x_rho
    (eta_rho, xi_rho)
    float64
    ...

    long_name :

    x location of RHO-points

    units :

    meter

    [609226 values with dtype=float64]

  • x_u
    (eta_u, xi_u)
    float64
    ...

    long_name :

    x location of U-points

    units :

    meter

    [608388 values with dtype=float64]

  • x_v
    (eta_v, xi_v)
    float64
    ...

    long_name :

    x location of V-points

    units :

    meter

    [608499 values with dtype=float64]

  • x_psi
    (eta_psi, xi_psi)
    float64
    ...

    long_name :

    x location of PSI-points

    units :

    meter

    [607662 values with dtype=float64]

  • y_rho
    (eta_rho, xi_rho)
    float64
    ...

    long_name :

    y location of RHO-points

    units :

    meter

    [609226 values with dtype=float64]

  • y_u
    (eta_u, xi_u)
    float64
    ...

    long_name :

    y location of U-points

    units :

    meter

    [608388 values with dtype=float64]

  • y_v
    (eta_v, xi_v)
    float64
    ...

    long_name :

    y location of V-points

    units :

    meter

    [608499 values with dtype=float64]

  • y_psi
    (eta_psi, xi_psi)
    float64
    ...

    long_name :

    y location of PSI-points

    units :

    meter

    [607662 values with dtype=float64]

  • lon_rho
    (eta_rho, xi_rho)
    float64
    ...

    long_name :

    longitude of RHO-points

    units :

    degree_east

    [609226 values with dtype=float64]

  • lon_u
    (eta_u, xi_u)
    float64
    ...

    long_name :

    longitude of U-points

    units :

    degree_east

    [608388 values with dtype=float64]

  • lon_v
    (eta_v, xi_v)
    float64
    ...

    long_name :

    longitude of V-points

    units :

    degree_east

    [608499 values with dtype=float64]

  • lon_psi
    (eta_psi, xi_psi)
    float64
    ...

    long_name :

    longitude of PSI-points

    units :

    degree_east

    [607662 values with dtype=float64]

  • lat_rho
    (eta_rho, xi_rho)
    float64
    ...

    long_name :

    latitude of RHO-points

    units :

    degree_north

    [609226 values with dtype=float64]

  • lat_u
    (eta_u, xi_u)
    float64
    ...

    long_name :

    latitude of U-points

    units :

    degree_north

    [608388 values with dtype=float64]

  • lat_v
    (eta_v, xi_v)
    float64
    ...

    long_name :

    latitude of V-points

    units :

    degree_north

    [608499 values with dtype=float64]

  • lat_psi
    (eta_psi, xi_psi)
    float64
    ...

    long_name :

    latitude of PSI-points

    units :

    degree_north

    [607662 values with dtype=float64]

  • mask_rho
    (eta_rho, xi_rho)
    float64
    ...

    long_name :

    mask on RHO-points

    option_0 :

    land

    option_1 :

    water

    [609226 values with dtype=float64]

  • mask_u
    (eta_u, xi_u)
    float64
    ...

    long_name :

    mask on U-points

    option_0 :

    land

    option_1 :

    water

    [608388 values with dtype=float64]

  • mask_v
    (eta_v, xi_v)
    float64
    ...

    long_name :

    mask on V-points

    option_0 :

    land

    option_1 :

    water

    [608499 values with dtype=float64]

  • mask_psi
    (eta_psi, xi_psi)
    float64
    ...

    long_name :

    mask on PSI-points

    option_0 :

    land

    option_1 :

    water

    [607662 values with dtype=float64]

• Indexes: (0)
• Attributes: (3)

  title :

  BOB1000 Model

  date :

  09-Mar-2023

  type :

  CROCO grid file

Open one RiOMar model file

• again the file is either available from datamor if you are running on datamor or accessible via https elsewhere.

url = "/home/lops-oh-fair2adapt/riomar/GAMAR/GAMAR_1h_inst_Y2006M01.nc"

if Path(url).is_file():
    # file exists
    ds = xr.open_dataset(url, engine='h5netcdf')[["temp"]]
else:
    url = "https://data-fair2adapt.ifremer.fr/riomar/GAMAR/GAMAR_1h_inst_Y2006M01.nc"
    fs = fsspec.filesystem('https')
    ds = xr.open_dataset(fs.open(url), engine='h5netcdf')[["temp"]]

ds

<xarray.Dataset> Size: 73GB
Dimensions:       (time_counter: 744, s_rho: 40, y_rho: 838, x_rho: 727)
Coordinates:
  * s_rho         (s_rho) float32 160B -0.9875 -0.9625 ... -0.0375 -0.0125
    nav_lat_rho   (y_rho, x_rho) float32 2MB ...
    nav_lon_rho   (y_rho, x_rho) float32 2MB ...
    time_instant  (time_counter) datetime64[ns] 6kB ...
  * time_counter  (time_counter) datetime64[ns] 6kB 2006-01-01T00:57:45 ... 2...
Dimensions without coordinates: y_rho, x_rho
Data variables:
    temp          (time_counter, s_rho, y_rho, x_rho) float32 73GB ...
Attributes: (12/45)
    name:           GAMAR_GLORYS_1h_inst
    description:    Created by xios
    Conventions:    CF-1.6
    timeStamp:      2024-Apr-02 09:15:02 GMT
    uuid:           1563e80a-8c72-4739-a6b5-424221c7cf2b
    title:          GAMAR_GLORYS
    ...             ...
    gamma2_expl:    Slipperiness parameter
    x_sponge:       0.0
    v_sponge:       0.0
    sponge_expl:    Sponge parameters : extent (m) & viscosity (m2.s-1)
    SRCS:           main.F step.F read_inp.F timers_roms.F init_scalars.F ini...
    CPP-options:    REGIONAL GAMAR MPI TIDES OBC_WEST OBC_NORTH XIOS USE_CALE...

xarray.Dataset

• Dimensions:
  • time_counter: 744
  • s_rho: 40
  • y_rho: 838
  • x_rho: 727
• Coordinates: (5)
  • s_rho
    (s_rho)
    float32
    -0.9875 -0.9625 ... -0.0375 -0.0125

    name :

    s_rho

    array([-0.9875, -0.9625, -0.9375, -0.9125, -0.8875, -0.8625, -0.8375, -0.8125,
           -0.7875, -0.7625, -0.7375, -0.7125, -0.6875, -0.6625, -0.6375, -0.6125,
           -0.5875, -0.5625, -0.5375, -0.5125, -0.4875, -0.4625, -0.4375, -0.4125,
           -0.3875, -0.3625, -0.3375, -0.3125, -0.2875, -0.2625, -0.2375, -0.2125,
           -0.1875, -0.1625, -0.1375, -0.1125, -0.0875, -0.0625, -0.0375, -0.0125],
          dtype=float32)

  • nav_lat_rho
    (y_rho, x_rho)
    float32
    ...

    standard_name :

    latitude

    long_name :

    Latitude

    units :

    degrees_north

    [609226 values with dtype=float32]

  • nav_lon_rho
    (y_rho, x_rho)
    float32
    ...

    standard_name :

    longitude

    long_name :

    Longitude

    units :

    degrees_east

    [609226 values with dtype=float32]

  • time_instant
    (time_counter)
    datetime64[ns]
    ...

    standard_name :

    time

    long_name :

    Time axis

    time_origin :

    2000-01-01 00:00:00

    bounds :

    time_instant_bounds

    [744 values with dtype=datetime64[ns]]

  • time_counter
    (time_counter)
    datetime64[ns]
    2006-01-01T00:57:45 ... 2006-01-...

    axis :

    T

    standard_name :

    time

    long_name :

    Time axis

    time_origin :

    2000-01-01 00:00:00

    bounds :

    time_counter_bounds

    array(['2006-01-01T00:57:45.000000000', '2006-01-01T01:57:45.000000000',
           '2006-01-01T02:57:45.000000000', ..., '2006-01-31T21:57:45.000000000',
           '2006-01-31T22:57:45.000000000', '2006-01-31T23:57:45.000000000'],
          dtype='datetime64[ns]')

• Data variables: (1)
  • temp
    (time_counter, s_rho, y_rho, x_rho)
    float32
    ...

    long_name :

    potential temperature

    units :

    Celsius

    online_operation :

    instant

    interval_operation :

    1 h

    interval_write :

    1 h

    cell_methods :

    time: point

    [18130565760 values with dtype=float32]

• Indexes: (2)
  • s_rho
    PandasIndex

    PandasIndex(Index([   -0.987500011920929,   -0.9624999761581421,               -0.9375,
             -0.9125000238418579,    -0.887499988079071,    -0.862500011920929,
             -0.8374999761581421,               -0.8125,   -0.7875000238418579,
              -0.762499988079071,    -0.737500011920929,   -0.7124999761581421,
                         -0.6875,   -0.6625000238418579,    -0.637499988079071,
              -0.612500011920929,   -0.5874999761581421,               -0.5625,
             -0.5375000238418579,    -0.512499988079071,  -0.48750001192092896,
             -0.4625000059604645,               -0.4375,   -0.4124999940395355,
            -0.38749998807907104,  -0.36250001192092896,   -0.3375000059604645,
                         -0.3125,   -0.2874999940395355,  -0.26249998807907104,
            -0.23749999701976776,  -0.21250000596046448,               -0.1875,
            -0.16249999403953552,  -0.13750000298023224,  -0.11249999701976776,
            -0.08749999850988388,               -0.0625,  -0.03750000149011612,
           -0.012500000186264515],
          dtype='float32', name='s_rho'))

  • time_counter
    PandasIndex

    PandasIndex(DatetimeIndex(['2006-01-01 00:57:45', '2006-01-01 01:57:45',
                   '2006-01-01 02:57:45', '2006-01-01 03:57:45',
                   '2006-01-01 04:57:45', '2006-01-01 05:57:45',
                   '2006-01-01 06:57:45', '2006-01-01 07:57:45',
                   '2006-01-01 08:57:45', '2006-01-01 09:57:45',
                   ...
                   '2006-01-31 14:57:45', '2006-01-31 15:57:45',
                   '2006-01-31 16:57:45', '2006-01-31 17:57:45',
                   '2006-01-31 18:57:45', '2006-01-31 19:57:45',
                   '2006-01-31 20:57:45', '2006-01-31 21:57:45',
                   '2006-01-31 22:57:45', '2006-01-31 23:57:45'],
                  dtype='datetime64[ns]', name='time_counter', length=744, freq=None))

• Attributes: (45)

  name :

  GAMAR_GLORYS_1h_inst

  description :

  Created by xios

  Conventions :

  CF-1.6

  timeStamp :

  2024-Apr-02 09:15:02 GMT

  uuid :

  1563e80a-8c72-4739-a6b5-424221c7cf2b

  title :

  GAMAR_GLORYS

  rst_file :

  croco_rst.nc

  grd_file :

  croco_grd.nc

  ini_file :

  croco_ini.nc

  frc_file :

  croco_frc.nc

  VertCoordType :

  NEW

  theta_s :

  6.0

  theta_s_expl :

  S-coordinate surface control parameter

  theta_b :

  4.0

  theta_b_expl :

  S-coordinate bottom control parameter

  Tcline :

  15.0

  Tcline_expl :

  S-coordinate surface/bottom layer width

  Tcline_units :

  meter

  hc :

  15.0

  hc_expl :

  S-coordinate parameter, critical depth

  hc_units :

  meter

  sc_w :

  have a look at variable sc_w in this file

  Cs_w :

  have a look at variable Cs_w in this file

  sc_r :

  have a look at variable sc_r in this file

  Cs_r :

  have a look at variable Cs_r in this file

  ntimes :

  4268161

  ndtfast :

  60

  dt :

  45.0

  dtfast :

  0.75

  nwrt :

  9600

  tnu4_expl :

  biharmonic mixing coefficient for tracers

  units :

  meter4 second-1

  rdrg :

  0.0

  rdrg_expl :

  linear drag coefficient

  rdrg_units :

  meter second-1

  rho0 :

  1025.0

  rho0_expl :

  Mean density used in Boussinesq approximation

  rho0_units :

  kilogram meter-3

  gamma2 :

  1.0

  gamma2_expl :

  Slipperiness parameter

  x_sponge :

  0.0

  v_sponge :

  0.0

  sponge_expl :

  Sponge parameters : extent (m) & viscosity (m2.s-1)

  SRCS :

  main.F step.F read_inp.F timers_roms.F init_scalars.F init_arrays.F set_weights.F set_scoord.F ana_grid.F setup_grid1.F setup_grid2.F set_nudgcof.F ana_initial.F analytical.F zonavg.F step2d.F u2dbc.F v2dbc.F zetabc.F obc_volcons.F pre_step3d.F step3d_t.F step3d_uv1.F step3d_uv2.F prsgrd.F rhs3d.F set_depth.F omega.F uv3dmix.F uv3dmix_spg.F t3dmix.F t3dmix_spg.F hmix_coef.F wetdry.F wavedry.F u3dbc.F v3dbc.F t3dbc.F step3d_fast.F step3d_w.F rhs3d_w_nh.F initial_nbq.F grid_nbq.F unbq_bc.F vnbq_bc.F wnbq_bc.F rnbq_bc.F w3dbc.F nbq_bry_store.F rho_eos.F ab_ratio.F alfabeta.F alfabeta_k.F ana_vmix.F bvf_mix.F lmd_vmix.F gls_mixing.F lmd_skpp.F lmd_bkpp.F lmd_swfrac.F lmd_wscale.F diag.F wvlcty.F checkdims.F grid_stiffness.F bio_diag.F setup_kwds.F check_kwds.F check_srcs.F check_switches1.F check_switches2.F debug.F param.F ncscrum.F scalars.F output.F put_global_atts.F nf_fread.F nf_fread_x.F nf_fread_y.F nf_read_bry.F get_date.F lenstr.F closecdf.F insert_node.F fillvalue.F nf_add_attribute.F set_cycle.F def_grid_2d.F def_grid_3d.F def_his.F def_rst.F def_diags.F def_diagsM.F def_bio_diags.F wrt_grid.F wrt_his.F wrt_avg.F wrt_rst.F wrt_diags.F wrt_diags_avg.F wrt_diagsM.F wrt_diagsM_avg.F wrt_bio_diags.F wrt_bio_diags_avg.F set_avg.F set_diags_avg.F set_diagsM_avg.F set_bio_diags_avg.F def_diags_vrt.F wrt_diags_vrt.F set_diags_vrt.F set_diags_vrt_avg.F wrt_diags_vrt_avg.F def_diags_ek.F wrt_diags_ek.F set_diags_ek.F set_diags_ek_avg.F wrt_diags_ek_avg.F def_diags_pv.F wrt_diags_pv.F set_diags_pv.F set_diags_pv_avg.F wrt_diags_pv_avg.F def_diags_eddy.F set_diags_eddy_avg.F wrt_diags_eddy_avg.F def_surf.F wrt_surf.F set_surf_avg.F wrt_surf_avg.F get_grid.F get_initial.F get_vbc.F get_wwave.F get_tclima.F get_uclima.F get_ssh.F get_sss.F get_smflux.F get_stflux.F get_srflux.F get_sst.F mod_tides_mas.F tidedata.F mas.F get_tides.F clm_tides.F get_bulk.F bulk_flux.F get_bry.F get_bry_bio.F sstskin.F get_psource.F get_psource_ts.F get_btflux.F mrl_wci.F wkb_wwave.F wkbbc.F get_bry_wkb.F online_bulk_var.F online_get_bulk.F online_interp.F online_interpolate_bulk.F online_set_bulk.F online_interpolate_3d_bulk.F init_floats.F wrt_floats.F step_floats.F rhs_floats.F interp_rho.F def_floats.F init_arrays_floats.F random_walk.F get_initial_floats.F init_sta.F wrt_sta.F step_sta.F interp_sta.F def_sta.F init_arrays_sta.F biology.F o2sato.F sediment.F bbl.F MPI_Setup.F MessPass2D.F MessPass3D.F exchange.F autotiling.F zoom.F update2D.F set_nudgcof_fine.F zoombc_2D.F zoombc_3D.F uv3dpremix.F t3dpremix.F update3D.F zoombc_3Dfast.F Agrif2Model.F send_xios_diags.F cpl_prism_define.F cpl_prism_put.F cpl_prism_init.F cpl_prism_get.F cpl_prism_getvar.F cpl_prism_grid.F module_substance.F module_MUSTANG.F abl_step.F abl_ini.F abl_tke.F wrt_abl_his.F def_abl_his.F wrt_abl_avg.F set_abl_avg.F online_spectral_diags.F var3d_oa_out.F var2d_oa_out.F scal0d_oa_out_loc.F scal0d_oa_out_full.F output_oa.F par_pisces.F90 ocean2pisces.F90 trc.F90 sms_pisces.F90 p4zche.F90 p4zint.F90 p4zlys.F90 p4zflx.F90 p4zlim.F90 p4zsink.F90 p4zmicro.F90 p4zmeso.F90 p4zmort.F90 p4zopt.F90 p4zprod.F90 p4zrem.F90 p4zsbc.F90 p4zsed.F90 trcwri_pisces.F90 p4zagg.F90 p4zfechem.F90 p4zligand.F90 p4zpoc.F90 p2zlim.F90 p2zmicro.F90 p2zmort.F90 p2zprod.F90 p5zmicro.F90 p5zmort.F90 p5zprod.F90 p5zlim.F90 p5zmeso.F90 p4zbio.F90 p4zsms.F90 trcsms_pisces.F90 trcini_pisces.F90 pisces_ini.F90 oce_sed.F90 par_sed.F90 sedadv.F90 sedarr.F90 sedbtb.F90 sedchem.F90 sedco3.F90 seddiff.F90 seddsr.F90 seddta.F90 sed.F90 sedini.F90 sedinitrc.F90 sedinorg.F90 sedmat.F90 sedmbc.F90 sedmodel.F90 sed_oce.F90 sedorg.F90 sedrst.F90 sedsfc.F90 sedstp.F90 sedwri.F90 setavg_sed.F90 module_parameter_oa.F90 module_grd_oa.F90 module_oa_time.F90 module_oa_space.F90 module_oa_periode.F90 module_oa_variables.F90 module_oa_type.F90 module_tile_oa.F90 module_oa_level.F90 module_oa_interface.F90 var_oa.F90 tooldatosec.F90 toolsectodat.F90 tooldecompdat.F90 tooldatetosec.F90 toolorigindate.F90 flocmod.F90 comsubstance.F90 substance.F90 comMUSTANG.F90 coupler_MUSTANG.F90 sed_MUSTANG_CROCO.F90 sed_MUSTANG.F90 initMUSTANG.F90 plug_MUSTANG_CROCO.F90

  CPP-options :

  REGIONAL GAMAR MPI TIDES OBC_WEST OBC_NORTH XIOS USE_CALENDAR NC4PAR MPI_NOLAND CURVGRID SPHERICAL MASKING WET_DRY NEW_S_COORD SOLVE3D UV_COR UV_ADV SALINITY NONLIN_EOS BULK_FLUX BULK_ECUMEV6 BULK_GUSTINESS BULK_LW ONLINE ERA_ECMWF BULK_MONTH_1DIGIT FRC_BRY Z_FRC_BRY M2_FRC_BRY M3_FRC_BRY T_FRC_BRY UV_HADV_UP3 UV_VADV_SPLINES TS_HADV_RSUP3 TS_DIF4 TS_VADV_SPLINES SPONGE LIMIT_BSTRESS GLS_MIXING ANA_BSFLUX ANA_BTFLUX PSOURCE PSOURCE_NCFILE SSH_TIDES UV_TIDES POT_TIDES TIDES_MAS OBC_M2CHARACT OBC_M3ORLANSKI OBC_TORLANSKI AVERAGES AVERAGES_K OUTPUTS_SURFACE HOURLY_VELOCITIES EXACT_RESTART ANA_PSOURCE TS_HADV_C4 MPI_COMM_WORLD M2FILTER_POWER TRACERS TEMPERATURE HZR VAR_RHO_2D SPLIT_EOS UV_MIX_S DIF_COEF_3D TS_MIX_ISO TS_MIX_IMP TS_MIX_ISO_FILT NTRA_T3DMIX SPONGE_GRID SPONGE_DIF2 SPONGE_VIS2 GLS_KEPSILON CANUTO_A CUBIC_INTERP NF_CLOBBER

ds["nav_lon_rho"]=ds["nav_lon_rho"] * 0 + grid.lon_rho.data
ds["nav_lat_rho"]=ds["nav_lat_rho"] * 0 + grid.lat_rho.data
ds["ocean_mask"]=ds.temp.isel(time_counter=0,s_rho=0).notnull()

Save to Zarr

smallpath = "/home/lops-oh-fair2adapt/riomar/small.zarr"

if not Path(smallpath).exists():
    smallpath = "./small.zarr"
    

small = ds[['temp','ocean_mask']].chunk({'time_counter':1}).isel(time_counter=slice(0,5))
small.to_zarr(smallpath, mode='w')

<xarray.backends.zarr.ZarrStore at 0x74922df79740>

Open the newly created Zarr to check it

ds = xr.open_dataset(smallpath)
ds

/srv/conda/envs/notebook/lib/python3.12/site-packages/xarray/backends/plugins.py:159: RuntimeWarning: 'netcdf4' fails while guessing
  warnings.warn(f"{engine!r} fails while guessing", RuntimeWarning)
/srv/conda/envs/notebook/lib/python3.12/site-packages/xarray/backends/plugins.py:159: RuntimeWarning: 'h5netcdf' fails while guessing
  warnings.warn(f"{engine!r} fails while guessing", RuntimeWarning)
/srv/conda/envs/notebook/lib/python3.12/site-packages/xarray/backends/plugins.py:159: RuntimeWarning: 'scipy' fails while guessing
  warnings.warn(f"{engine!r} fails while guessing", RuntimeWarning)

<xarray.Dataset> Size: 498MB
Dimensions:       (y_rho: 838, x_rho: 727, s_rho: 40, time_counter: 5)
Coordinates:
    nav_lat_rho   (y_rho, x_rho) float64 5MB ...
    nav_lon_rho   (y_rho, x_rho) float64 5MB ...
  * s_rho         (s_rho) float32 160B -0.9875 -0.9625 ... -0.0375 -0.0125
  * time_counter  (time_counter) datetime64[ns] 40B 2006-01-01T00:57:45 ... 2...
    time_instant  (time_counter) datetime64[ns] 40B ...
Dimensions without coordinates: y_rho, x_rho
Data variables:
    ocean_mask    (y_rho, x_rho) bool 609kB ...
    temp          (time_counter, s_rho, y_rho, x_rho) float32 487MB ...
Attributes: (12/45)
    CPP-options:    REGIONAL GAMAR MPI TIDES OBC_WEST OBC_NORTH XIOS USE_CALE...
    Conventions:    CF-1.6
    Cs_r:           have a look at variable Cs_r in this file
    Cs_w:           have a look at variable Cs_w in this file
    SRCS:           main.F step.F read_inp.F timers_roms.F init_scalars.F ini...
    Tcline:         15.0
    ...             ...
    title:          GAMAR_GLORYS
    tnu4_expl:      biharmonic mixing coefficient for tracers
    units:          meter4 second-1
    uuid:           1563e80a-8c72-4739-a6b5-424221c7cf2b
    v_sponge:       0.0
    x_sponge:       0.0

xarray.Dataset

• Dimensions:
  • y_rho: 838
  • x_rho: 727
  • s_rho: 40
  • time_counter: 5
• Coordinates: (5)
  • nav_lat_rho
    (y_rho, x_rho)
    float64
    ...

    [609226 values with dtype=float64]

  • nav_lon_rho
    (y_rho, x_rho)
    float64
    ...

    [609226 values with dtype=float64]

  • s_rho
    (s_rho)
    float32
    -0.9875 -0.9625 ... -0.0375 -0.0125

    name :

    s_rho

    array([-0.9875, -0.9625, -0.9375, -0.9125, -0.8875, -0.8625, -0.8375, -0.8125,
           -0.7875, -0.7625, -0.7375, -0.7125, -0.6875, -0.6625, -0.6375, -0.6125,
           -0.5875, -0.5625, -0.5375, -0.5125, -0.4875, -0.4625, -0.4375, -0.4125,
           -0.3875, -0.3625, -0.3375, -0.3125, -0.2875, -0.2625, -0.2375, -0.2125,
           -0.1875, -0.1625, -0.1375, -0.1125, -0.0875, -0.0625, -0.0375, -0.0125],
          dtype=float32)

  • time_counter
    (time_counter)
    datetime64[ns]
    2006-01-01T00:57:45 ... 2006-01-...

    axis :

    T

    bounds :

    time_counter_bounds

    long_name :

    Time axis

    standard_name :

    time

    time_origin :

    2000-01-01 00:00:00

    array(['2006-01-01T00:57:45.000000000', '2006-01-01T01:57:45.000000000',
           '2006-01-01T02:57:45.000000000', '2006-01-01T03:57:45.000000000',
           '2006-01-01T04:57:45.000000000'], dtype='datetime64[ns]')

  • time_instant
    (time_counter)
    datetime64[ns]
    ...

    bounds :

    time_instant_bounds

    long_name :

    Time axis

    standard_name :

    time

    time_origin :

    2000-01-01 00:00:00

    [5 values with dtype=datetime64[ns]]

• Data variables: (2)
  • ocean_mask
    (y_rho, x_rho)
    bool
    ...

    [609226 values with dtype=bool]

  • temp
    (time_counter, s_rho, y_rho, x_rho)
    float32
    ...

    cell_methods :

    time: point

    interval_operation :

    1 h

    interval_write :

    1 h

    long_name :

    potential temperature

    online_operation :

    instant

    units :

    Celsius

    [121845200 values with dtype=float32]

• Indexes: (2)
  • s_rho
    PandasIndex

    PandasIndex(Index([   -0.987500011920929,   -0.9624999761581421,               -0.9375,
             -0.9125000238418579,    -0.887499988079071,    -0.862500011920929,
             -0.8374999761581421,               -0.8125,   -0.7875000238418579,
              -0.762499988079071,    -0.737500011920929,   -0.7124999761581421,
                         -0.6875,   -0.6625000238418579,    -0.637499988079071,
              -0.612500011920929,   -0.5874999761581421,               -0.5625,
             -0.5375000238418579,    -0.512499988079071,  -0.48750001192092896,
             -0.4625000059604645,               -0.4375,   -0.4124999940395355,
            -0.38749998807907104,  -0.36250001192092896,   -0.3375000059604645,
                         -0.3125,   -0.2874999940395355,  -0.26249998807907104,
            -0.23749999701976776,  -0.21250000596046448,               -0.1875,
            -0.16249999403953552,  -0.13750000298023224,  -0.11249999701976776,
            -0.08749999850988388,               -0.0625,  -0.03750000149011612,
           -0.012500000186264515],
          dtype='float32', name='s_rho'))

  • time_counter
    PandasIndex

    PandasIndex(DatetimeIndex(['2006-01-01 00:57:45', '2006-01-01 01:57:45',
                   '2006-01-01 02:57:45', '2006-01-01 03:57:45',
                   '2006-01-01 04:57:45'],
                  dtype='datetime64[ns]', name='time_counter', freq=None))

• Attributes: (45)

  CPP-options :

  REGIONAL GAMAR MPI TIDES OBC_WEST OBC_NORTH XIOS USE_CALENDAR NC4PAR MPI_NOLAND CURVGRID SPHERICAL MASKING WET_DRY NEW_S_COORD SOLVE3D UV_COR UV_ADV SALINITY NONLIN_EOS BULK_FLUX BULK_ECUMEV6 BULK_GUSTINESS BULK_LW ONLINE ERA_ECMWF BULK_MONTH_1DIGIT FRC_BRY Z_FRC_BRY M2_FRC_BRY M3_FRC_BRY T_FRC_BRY UV_HADV_UP3 UV_VADV_SPLINES TS_HADV_RSUP3 TS_DIF4 TS_VADV_SPLINES SPONGE LIMIT_BSTRESS GLS_MIXING ANA_BSFLUX ANA_BTFLUX PSOURCE PSOURCE_NCFILE SSH_TIDES UV_TIDES POT_TIDES TIDES_MAS OBC_M2CHARACT OBC_M3ORLANSKI OBC_TORLANSKI AVERAGES AVERAGES_K OUTPUTS_SURFACE HOURLY_VELOCITIES EXACT_RESTART ANA_PSOURCE TS_HADV_C4 MPI_COMM_WORLD M2FILTER_POWER TRACERS TEMPERATURE HZR VAR_RHO_2D SPLIT_EOS UV_MIX_S DIF_COEF_3D TS_MIX_ISO TS_MIX_IMP TS_MIX_ISO_FILT NTRA_T3DMIX SPONGE_GRID SPONGE_DIF2 SPONGE_VIS2 GLS_KEPSILON CANUTO_A CUBIC_INTERP NF_CLOBBER

  Conventions :

  CF-1.6

  Cs_r :

  have a look at variable Cs_r in this file

  Cs_w :

  have a look at variable Cs_w in this file

  SRCS :

  main.F step.F read_inp.F timers_roms.F init_scalars.F init_arrays.F set_weights.F set_scoord.F ana_grid.F setup_grid1.F setup_grid2.F set_nudgcof.F ana_initial.F analytical.F zonavg.F step2d.F u2dbc.F v2dbc.F zetabc.F obc_volcons.F pre_step3d.F step3d_t.F step3d_uv1.F step3d_uv2.F prsgrd.F rhs3d.F set_depth.F omega.F uv3dmix.F uv3dmix_spg.F t3dmix.F t3dmix_spg.F hmix_coef.F wetdry.F wavedry.F u3dbc.F v3dbc.F t3dbc.F step3d_fast.F step3d_w.F rhs3d_w_nh.F initial_nbq.F grid_nbq.F unbq_bc.F vnbq_bc.F wnbq_bc.F rnbq_bc.F w3dbc.F nbq_bry_store.F rho_eos.F ab_ratio.F alfabeta.F alfabeta_k.F ana_vmix.F bvf_mix.F lmd_vmix.F gls_mixing.F lmd_skpp.F lmd_bkpp.F lmd_swfrac.F lmd_wscale.F diag.F wvlcty.F checkdims.F grid_stiffness.F bio_diag.F setup_kwds.F check_kwds.F check_srcs.F check_switches1.F check_switches2.F debug.F param.F ncscrum.F scalars.F output.F put_global_atts.F nf_fread.F nf_fread_x.F nf_fread_y.F nf_read_bry.F get_date.F lenstr.F closecdf.F insert_node.F fillvalue.F nf_add_attribute.F set_cycle.F def_grid_2d.F def_grid_3d.F def_his.F def_rst.F def_diags.F def_diagsM.F def_bio_diags.F wrt_grid.F wrt_his.F wrt_avg.F wrt_rst.F wrt_diags.F wrt_diags_avg.F wrt_diagsM.F wrt_diagsM_avg.F wrt_bio_diags.F wrt_bio_diags_avg.F set_avg.F set_diags_avg.F set_diagsM_avg.F set_bio_diags_avg.F def_diags_vrt.F wrt_diags_vrt.F set_diags_vrt.F set_diags_vrt_avg.F wrt_diags_vrt_avg.F def_diags_ek.F wrt_diags_ek.F set_diags_ek.F set_diags_ek_avg.F wrt_diags_ek_avg.F def_diags_pv.F wrt_diags_pv.F set_diags_pv.F set_diags_pv_avg.F wrt_diags_pv_avg.F def_diags_eddy.F set_diags_eddy_avg.F wrt_diags_eddy_avg.F def_surf.F wrt_surf.F set_surf_avg.F wrt_surf_avg.F get_grid.F get_initial.F get_vbc.F get_wwave.F get_tclima.F get_uclima.F get_ssh.F get_sss.F get_smflux.F get_stflux.F get_srflux.F get_sst.F mod_tides_mas.F tidedata.F mas.F get_tides.F clm_tides.F get_bulk.F bulk_flux.F get_bry.F get_bry_bio.F sstskin.F get_psource.F get_psource_ts.F get_btflux.F mrl_wci.F wkb_wwave.F wkbbc.F get_bry_wkb.F online_bulk_var.F online_get_bulk.F online_interp.F online_interpolate_bulk.F online_set_bulk.F online_interpolate_3d_bulk.F init_floats.F wrt_floats.F step_floats.F rhs_floats.F interp_rho.F def_floats.F init_arrays_floats.F random_walk.F get_initial_floats.F init_sta.F wrt_sta.F step_sta.F interp_sta.F def_sta.F init_arrays_sta.F biology.F o2sato.F sediment.F bbl.F MPI_Setup.F MessPass2D.F MessPass3D.F exchange.F autotiling.F zoom.F update2D.F set_nudgcof_fine.F zoombc_2D.F zoombc_3D.F uv3dpremix.F t3dpremix.F update3D.F zoombc_3Dfast.F Agrif2Model.F send_xios_diags.F cpl_prism_define.F cpl_prism_put.F cpl_prism_init.F cpl_prism_get.F cpl_prism_getvar.F cpl_prism_grid.F module_substance.F module_MUSTANG.F abl_step.F abl_ini.F abl_tke.F wrt_abl_his.F def_abl_his.F wrt_abl_avg.F set_abl_avg.F online_spectral_diags.F var3d_oa_out.F var2d_oa_out.F scal0d_oa_out_loc.F scal0d_oa_out_full.F output_oa.F par_pisces.F90 ocean2pisces.F90 trc.F90 sms_pisces.F90 p4zche.F90 p4zint.F90 p4zlys.F90 p4zflx.F90 p4zlim.F90 p4zsink.F90 p4zmicro.F90 p4zmeso.F90 p4zmort.F90 p4zopt.F90 p4zprod.F90 p4zrem.F90 p4zsbc.F90 p4zsed.F90 trcwri_pisces.F90 p4zagg.F90 p4zfechem.F90 p4zligand.F90 p4zpoc.F90 p2zlim.F90 p2zmicro.F90 p2zmort.F90 p2zprod.F90 p5zmicro.F90 p5zmort.F90 p5zprod.F90 p5zlim.F90 p5zmeso.F90 p4zbio.F90 p4zsms.F90 trcsms_pisces.F90 trcini_pisces.F90 pisces_ini.F90 oce_sed.F90 par_sed.F90 sedadv.F90 sedarr.F90 sedbtb.F90 sedchem.F90 sedco3.F90 seddiff.F90 seddsr.F90 seddta.F90 sed.F90 sedini.F90 sedinitrc.F90 sedinorg.F90 sedmat.F90 sedmbc.F90 sedmodel.F90 sed_oce.F90 sedorg.F90 sedrst.F90 sedsfc.F90 sedstp.F90 sedwri.F90 setavg_sed.F90 module_parameter_oa.F90 module_grd_oa.F90 module_oa_time.F90 module_oa_space.F90 module_oa_periode.F90 module_oa_variables.F90 module_oa_type.F90 module_tile_oa.F90 module_oa_level.F90 module_oa_interface.F90 var_oa.F90 tooldatosec.F90 toolsectodat.F90 tooldecompdat.F90 tooldatetosec.F90 toolorigindate.F90 flocmod.F90 comsubstance.F90 substance.F90 comMUSTANG.F90 coupler_MUSTANG.F90 sed_MUSTANG_CROCO.F90 sed_MUSTANG.F90 initMUSTANG.F90 plug_MUSTANG_CROCO.F90

  Tcline :

  15.0

  Tcline_expl :

  S-coordinate surface/bottom layer width

  Tcline_units :

  meter

  VertCoordType :

  NEW

  description :

  Created by xios

  dt :

  45.0

  dtfast :

  0.75

  frc_file :

  croco_frc.nc

  gamma2 :

  1.0

  gamma2_expl :

  Slipperiness parameter

  grd_file :

  croco_grd.nc

  hc :

  15.0

  hc_expl :

  S-coordinate parameter, critical depth

  hc_units :

  meter

  ini_file :

  croco_ini.nc

  name :

  GAMAR_GLORYS_1h_inst

  ndtfast :

  60

  ntimes :

  4268161

  nwrt :

  9600

  rdrg :

  0.0

  rdrg_expl :

  linear drag coefficient

  rdrg_units :

  meter second-1

  rho0 :

  1025.0

  rho0_expl :

  Mean density used in Boussinesq approximation

  rho0_units :

  kilogram meter-3

  rst_file :

  croco_rst.nc

  sc_r :

  have a look at variable sc_r in this file

  sc_w :

  have a look at variable sc_w in this file

  sponge_expl :

  Sponge parameters : extent (m) & viscosity (m2.s-1)

  theta_b :

  4.0

  theta_b_expl :

  S-coordinate bottom control parameter

  theta_s :

  6.0

  theta_s_expl :

  S-coordinate surface control parameter

  timeStamp :

  2024-Apr-02 09:15:02 GMT

  title :

  GAMAR_GLORYS

  tnu4_expl :

  biharmonic mixing coefficient for tracers

  units :

  meter4 second-1

  uuid :

  1563e80a-8c72-4739-a6b5-424221c7cf2b

  v_sponge :

  0.0

  x_sponge :

  0.0