Combining multiple `Datasets`¶

Objectives:¶

Show how to combine multiple ECCO v4 state estimate Datasets after loading.

Opening multiple `Datasets` centered on different coordinates¶

In previous tutorials we’ve loaded single lat-lon-cap NetCDF tile files (granules) for ECCO state estimate variables and model grid parameters. Here we will demonstrate merging Datasets together. Some benefits of merging Datasets include having a tidier workspace and simplifying subsetting operations (e.g., using xarray.isel or xarray.sel as shown in the previous tutorial).

First, we’ll load three ECCOv4 NetCDF state estimate variables (each centered on different coordinates) as well as the model grid file. For this, you will need to download 2 datasets of monthly mean fields for the year 2010. The ShortNames for the 2 datasets are:

ECCO_L4_SSH_LLC0090GRID_MONTHLY_V4R4
ECCO_L4_OCEAN_3D_TEMPERATURE_FLUX_LLC0090GRID_MONTHLY_V4R4

If you did the previous tutorial you already have the SSH files.

Once you have the required ECCOv4 output downloaded, let’s define our environment.

[1]:

import numpy as np
import xarray as xr
import sys
import matplotlib.pyplot as plt
import json

[2]:

## Import the ecco_v4_py library into Python
## =========================================
#import ecco_v4_py as ecco

## -- If ecco_v4_py is not installed in your local Python library,
##    tell Python where to find it.  For example, if your ecco_v4_py
##    files are in /Users/ifenty/ECCOv4-py/ecco_v4_py, then use:

from os.path import join,expanduser
user_home_dir = expanduser('~')

sys.path.append(join(user_home_dir,'ECCOv4-py'))

import ecco_v4_py as ecco

[3]:

## Set top-level file directory for the ECCO NetCDF files
## =================================================================

## currently set to ~/Downloads/ECCO_V4r4_PODAAC,
## the default if ecco_podaac_download was used to download dataset granules
ECCO_dir = join(user_home_dir,'Downloads','ECCO_V4r4_PODAAC')

Open c point variable: `SSH`¶

[4]:

# load dataset containing monthly SSH in 2010
ecco_dataset_A = xr.open_mfdataset(join(ECCO_dir,'*SSH*MONTHLY*','*_2010-??_*.nc'))

to see the data variables in a dataset, use .data_vars:

[5]:

ecco_dataset_A.data_vars

[5]:

Data variables:
    SSH       (time, tile, j, i) float32 dask.array<chunksize=(1, 13, 90, 90), meta=np.ndarray>
    SSHIBC    (time, tile, j, i) float32 dask.array<chunksize=(1, 13, 90, 90), meta=np.ndarray>
    SSHNOIBC  (time, tile, j, i) float32 dask.array<chunksize=(1, 13, 90, 90), meta=np.ndarray>
    ETAN      (time, tile, j, i) float32 dask.array<chunksize=(1, 13, 90, 90), meta=np.ndarray>

ecco_dataset_A has four data variables, all having dimensions i, j, tile, and time, which mean that they are centered with respect to the grid cells of the model. The coordinates associated with the SSH variable are:

[6]:

ecco_dataset_A.SSH.coords

[6]:

Coordinates:
  * i        (i) int32 0 1 2 3 4 5 6 7 8 9 10 ... 80 81 82 83 84 85 86 87 88 89
  * j        (j) int32 0 1 2 3 4 5 6 7 8 9 10 ... 80 81 82 83 84 85 86 87 88 89
  * tile     (tile) int32 0 1 2 3 4 5 6 7 8 9 10 11 12
  * time     (time) datetime64[ns] 2010-01-16T12:00:00 ... 2010-12-16T12:00:00
    XC       (tile, j, i) float32 dask.array<chunksize=(13, 90, 90), meta=np.ndarray>
    YC       (tile, j, i) float32 dask.array<chunksize=(13, 90, 90), meta=np.ndarray>

You can see the coordinates that are also dimensions (dimensional coordinates) have an asterisk. The non-dimensional coordinates XC and YC are not used for indexing, but are very important as they indicate the longitude and latitude respectively associated with the dimensional coordinates.

Open u and v point variables: `ADVx_TH` and `ADVy_TH`¶

Now let’s open the ECCOv4 output files containing the horizontal advective fluxes of potential temperature, ADVx_TH and ADVy_TH.

[7]:

# load dataset containing monthly mean 3D temperature fluxes in 2010
ecco_dataset_B = xr.open_mfdataset(join(ECCO_dir,'*3D_TEMPERATURE_FLUX_LLC0090GRID_MONTHLY*','*_2010-??_*.nc'))

ecco_dataset_B.data_vars

[7]:

Data variables:
    ADVx_TH  (time, k, tile, j, i_g) float32 dask.array<chunksize=(1, 50, 13, 90, 90), meta=np.ndarray>
    DFxE_TH  (time, k, tile, j, i_g) float32 dask.array<chunksize=(1, 50, 13, 90, 90), meta=np.ndarray>
    ADVy_TH  (time, k, tile, j_g, i) float32 dask.array<chunksize=(1, 50, 13, 90, 90), meta=np.ndarray>
    DFyE_TH  (time, k, tile, j_g, i) float32 dask.array<chunksize=(1, 50, 13, 90, 90), meta=np.ndarray>
    ADVr_TH  (time, k_l, tile, j, i) float32 dask.array<chunksize=(1, 50, 13, 90, 90), meta=np.ndarray>
    DFrE_TH  (time, k_l, tile, j, i) float32 dask.array<chunksize=(1, 50, 13, 90, 90), meta=np.ndarray>
    DFrI_TH  (time, k_l, tile, j, i) float32 dask.array<chunksize=(1, 50, 13, 90, 90), meta=np.ndarray>

ecco_dataset_B has seven data variables! These include three variables (starting with AD) that quantify 3D advection fluxes, and four variables (starting with DF) that quantify 3D diffusive fluxes, including one DFrI_TH that is an implicit flux from the vertical mixing parameterization of the model. In this tutorial we will focus on the two variables that quantify horizontal advective fluxes.

Let’s look at one of these variables, ADVx_TH

[8]:

ecco_dataset_B.ADVx_TH

[8]:

<xarray.DataArray 'ADVx_TH' (time: 12, k: 50, tile: 13, j: 90, i_g: 90)>
dask.array<concatenate, shape=(12, 50, 13, 90, 90), dtype=float32, chunksize=(1, 50, 13, 90, 90), chunktype=numpy.ndarray>
Coordinates:
  * i_g      (i_g) int32 0 1 2 3 4 5 6 7 8 9 ... 80 81 82 83 84 85 86 87 88 89
  * j        (j) int32 0 1 2 3 4 5 6 7 8 9 10 ... 80 81 82 83 84 85 86 87 88 89
  * k        (k) int32 0 1 2 3 4 5 6 7 8 9 10 ... 40 41 42 43 44 45 46 47 48 49
  * tile     (tile) int32 0 1 2 3 4 5 6 7 8 9 10 11 12
  * time     (time) datetime64[ns] 2010-01-16T12:00:00 ... 2010-12-16T12:00:00
    Z        (k) float32 dask.array<chunksize=(50,), meta=np.ndarray>
Attributes:
    long_name:              Lateral advective flux of potential temperature i...
    units:                  degree_C m3 s-1
    mate:                   ADVy_TH
    coverage_content_type:  modelResult
    direction:              >0 increases potential temperature (THETA)
    comment:                Lateral advective flux of potential temperature (...
    valid_min:              -28231902.0
    valid_max:              36523468.0

xarray.DataArray

'ADVx_TH'

time: 12
k: 50
tile: 13
j: 90
i_g: 90

dask.array<chunksize=(1, 50, 13, 90, 90), meta=np.ndarray>

	Array	Chunk
Bytes	241.01 MiB	20.08 MiB
Shape	(12, 50, 13, 90, 90)	(1, 50, 13, 90, 90)
Count	36 Tasks	12 Chunks
Type	float32	numpy.ndarray

Coordinates: (6)

i_g

(i_g)

int32

0 1 2 3 4 5 6 ... 84 85 86 87 88 89

axis :: X
long_name :: grid index in x for variables at 'u' and 'g' locations
c_grid_axis_shift :: -0.5
swap_dim :: XG
comment :: In the Arakawa C-grid system, 'u' (e.g., UVEL) and 'g' variables (e.g., XG) have the same x coordinate on the model grid.
coverage_content_type :: coordinate

array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
       36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
       54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
       72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89])

j

(j)

int32

0 1 2 3 4 5 6 ... 84 85 86 87 88 89

axis :: Y
long_name :: grid index in y for variables at tracer and 'u' locations
swap_dim :: YC
comment :: In the Arakawa C-grid system, tracer (e.g., THETA) and 'u' variables (e.g., UVEL) have the same y coordinate on the model grid.
coverage_content_type :: coordinate

array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
       36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
       54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
       72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89])

k

(k)

int32

0 1 2 3 4 5 6 ... 44 45 46 47 48 49

axis :: Z
long_name :: grid index in z for tracer variables
swap_dim :: Z
coverage_content_type :: coordinate

array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
       36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49])

tile
(tile)
int32
0 1 2 3 4 5 6 7 8 9 10 11 12
long_name :
lat-lon-cap tile index
comment :
The ECCO V4 horizontal model grid is divided into 13 tiles of 90x90 cells for convenience.
coverage_content_type :
coordinate
```
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12])
```

time

(time)

datetime64[ns]

2010-01-16T12:00:00 ... 2010-12-...

long_name :: center time of averaging period
axis :: T
bounds :: time_bnds
coverage_content_type :: coordinate
standard_name :: time

array(['2010-01-16T12:00:00.000000000', '2010-02-15T00:00:00.000000000',
       '2010-03-16T12:00:00.000000000', '2010-04-16T00:00:00.000000000',
       '2010-05-16T12:00:00.000000000', '2010-06-16T00:00:00.000000000',
       '2010-07-16T12:00:00.000000000', '2010-08-16T12:00:00.000000000',
       '2010-09-16T00:00:00.000000000', '2010-10-16T12:00:00.000000000',
       '2010-11-16T00:00:00.000000000', '2010-12-16T12:00:00.000000000'],
      dtype='datetime64[ns]')

Z

(k)

float32

dask.array<chunksize=(50,), meta=np.ndarray>

long_name :: depth of tracer grid cell center
units :: m
positive :: up
bounds :: Z_bnds
comment :: Non-uniform vertical spacing.
coverage_content_type :: coordinate
standard_name :: depth

	Array	Chunk
Bytes	200 B	200 B
Shape	(50,)	(50,)
Count	55 Tasks	1 Chunks
Type	float32	numpy.ndarray

Attributes: (8)
long_name :
Lateral advective flux of potential temperature in the model +x direction
units :
degree_C m3 s-1
mate :
ADVy_TH
coverage_content_type :
modelResult
direction :
>0 increases potential temperature (THETA)
comment :
Lateral advective flux of potential temperature (THETA) in the +x direction through the 'u' face of the tracer cell on the native model grid. Note: in the Arakawa-C grid, horizontal flux quantities are staggered relative to the tracer cells with indexing such that +ADVx_TH(i_g,j,k) corresponds to +x fluxes through the 'u' face of the tracer cell at (i,j,k). Also, the model +x direction does not necessarily correspond to the geographical east-west direction because the x and y axes of the model's lat-lon-cap (llc) curvilinear lat-lon-cap (llc) grid have arbitrary orientations which vary within and across tiles.
valid_min :
-28231902.0
valid_max :
36523468.0

The long_name and the comment tell us that this variable is the advective flux of potential temperature in the model + $x$ direction. It has dimensional coordinates i_g, j, k, tile, and time. The k dimension indexes depth (which was not a part of the SSH dataset), and the i_g dimension has replaced the i dimension that SSH had.

Since ADVx_TH has a i_g dimensional coordinate instead of i, we know that the flux is quantified not at the center of each cell, but is offset in the model $x$ direction. Specifically, the flux is quantified on the left or “west” face of each grid cell. In this context “west” may not actually refer to geographical west, but rather the left side of a grid cell on an axis where i and i_g increase to the right.

Now consider ADVy_TH:

[9]:

ecco_dataset_B.ADVy_TH

[9]:

<xarray.DataArray 'ADVy_TH' (time: 12, k: 50, tile: 13, j_g: 90, i: 90)>
dask.array<concatenate, shape=(12, 50, 13, 90, 90), dtype=float32, chunksize=(1, 50, 13, 90, 90), chunktype=numpy.ndarray>
Coordinates:
  * i        (i) int32 0 1 2 3 4 5 6 7 8 9 10 ... 80 81 82 83 84 85 86 87 88 89
  * j_g      (j_g) int32 0 1 2 3 4 5 6 7 8 9 ... 80 81 82 83 84 85 86 87 88 89
  * k        (k) int32 0 1 2 3 4 5 6 7 8 9 10 ... 40 41 42 43 44 45 46 47 48 49
  * tile     (tile) int32 0 1 2 3 4 5 6 7 8 9 10 11 12
  * time     (time) datetime64[ns] 2010-01-16T12:00:00 ... 2010-12-16T12:00:00
    Z        (k) float32 dask.array<chunksize=(50,), meta=np.ndarray>
Attributes:
    long_name:              Lateral advective flux of potential temperature i...
    units:                  degree_C m3 s-1
    mate:                   ADVx_TH
    coverage_content_type:  modelResult
    direction:              >0 increases potential temperature (THETA)
    comment:                Lateral advective flux of potential temperature (...
    valid_min:              -31236064.0
    valid_max:              43466144.0

xarray.DataArray

'ADVy_TH'

time: 12
k: 50
tile: 13
j_g: 90
i: 90

dask.array<chunksize=(1, 50, 13, 90, 90), meta=np.ndarray>

	Array	Chunk
Bytes	241.01 MiB	20.08 MiB
Shape	(12, 50, 13, 90, 90)	(1, 50, 13, 90, 90)
Count	36 Tasks	12 Chunks
Type	float32	numpy.ndarray

Coordinates: (6)

i

(i)

int32

0 1 2 3 4 5 6 ... 84 85 86 87 88 89

axis :: X
long_name :: grid index in x for variables at tracer and 'v' locations
swap_dim :: XC
comment :: In the Arakawa C-grid system, tracer (e.g., THETA) and 'v' variables (e.g., VVEL) have the same x coordinate on the model grid.
coverage_content_type :: coordinate

array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
       36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
       54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
       72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89])

j_g

(j_g)

int32

0 1 2 3 4 5 6 ... 84 85 86 87 88 89

axis :: Y
long_name :: grid index in y for variables at 'v' and 'g' locations
c_grid_axis_shift :: -0.5
swap_dim :: YG
comment :: In the Arakawa C-grid system, 'v' (e.g., VVEL) and 'g' variables (e.g., XG) have the same y coordinate.
coverage_content_type :: coordinate

array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
       36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
       54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
       72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89])

k

(k)

int32

0 1 2 3 4 5 6 ... 44 45 46 47 48 49

axis :: Z
long_name :: grid index in z for tracer variables
swap_dim :: Z
coverage_content_type :: coordinate

array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
       36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49])

tile
(tile)
int32
0 1 2 3 4 5 6 7 8 9 10 11 12
long_name :
lat-lon-cap tile index
comment :
The ECCO V4 horizontal model grid is divided into 13 tiles of 90x90 cells for convenience.
coverage_content_type :
coordinate
```
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12])
```

time

(time)

datetime64[ns]

2010-01-16T12:00:00 ... 2010-12-...

long_name :: center time of averaging period
axis :: T
bounds :: time_bnds
coverage_content_type :: coordinate
standard_name :: time

array(['2010-01-16T12:00:00.000000000', '2010-02-15T00:00:00.000000000',
       '2010-03-16T12:00:00.000000000', '2010-04-16T00:00:00.000000000',
       '2010-05-16T12:00:00.000000000', '2010-06-16T00:00:00.000000000',
       '2010-07-16T12:00:00.000000000', '2010-08-16T12:00:00.000000000',
       '2010-09-16T00:00:00.000000000', '2010-10-16T12:00:00.000000000',
       '2010-11-16T00:00:00.000000000', '2010-12-16T12:00:00.000000000'],
      dtype='datetime64[ns]')

Z

(k)

float32

dask.array<chunksize=(50,), meta=np.ndarray>

long_name :: depth of tracer grid cell center
units :: m
positive :: up
bounds :: Z_bnds
comment :: Non-uniform vertical spacing.
coverage_content_type :: coordinate
standard_name :: depth

	Array	Chunk
Bytes	200 B	200 B
Shape	(50,)	(50,)
Count	55 Tasks	1 Chunks
Type	float32	numpy.ndarray

Attributes: (8)
long_name :
Lateral advective flux of potential temperature in the model +y direction
units :
degree_C m3 s-1
mate :
ADVx_TH
coverage_content_type :
modelResult
direction :
>0 increases potential temperature (THETA)
comment :
Lateral advective flux of potential temperature (THETA) in the +y direction through the 'v' face of the tracer cell on the native model grid. Note: in the Arakawa-C grid, horizontal flux quantities are staggered relative to the tracer cells with indexing such that +ADVy_TH(i,j_g,k) corresponds to +y fluxes through the 'v' face of the tracer cell at (i,j,k). Also, the model +y direction does not necessarily correspond to the geographical north-south direction because the x and y axes of the model's curvilinear lat-lon-cap (llc) grid have arbitrary orientations which vary within and across tiles.
valid_min :
-31236064.0
valid_max :
43466144.0

ADVy_TH is the horizontal advective flux of potential temperature in each tile’s $y$ direction. The dimensional coordinates are i, j_g, k, tile, and time. In this case we have the centered $x$ coordinate i but the off-center (shifted) $y$ coordinate j_g, which indicates that these fluxes are located on the lower or “south” face of each grid cell—again with the caveat that this does not always correspond to geographical south.

Examining the dimensions and coordinates of these `Datasets`¶

Each of the three variables we have discussed comprises an xarray DataArray, and each of these DataArray objects has different horizontal dimension labels.

i and j for SSH
i_g and j for ADVx_TH
i and j_g for ADVy_TH

[10]:

# print just the first line of each DataArray's information
print((str(ecco_dataset_A.SSH)).split('\n')[0])
print((str(ecco_dataset_B.ADVx_TH)).split('\n')[0])
print((str(ecco_dataset_B.ADVy_TH)).split('\n')[0])

<xarray.DataArray 'SSH' (time: 12, tile: 13, j: 90, i: 90)>
<xarray.DataArray 'ADVx_TH' (time: 12, k: 50, tile: 13, j: 90, i_g: 90)>
<xarray.DataArray 'ADVy_TH' (time: 12, k: 50, tile: 13, j_g: 90, i: 90)>

Merging and memory¶

Merging Datasets together does not make copies of the data in memory. Instead, merged Datasets are in fact just a reorganized collection of pointers. You may want to delete the original variables to clear your namespace, but it is not necessary.

Summary¶

Now you know how to merge multiple Datasets using the merge command. We demonstrated merging of Datasets constructed from three different variables types and the model grid parameters.

Combining multiple `Datasets`¶

Objectives:¶

Opening multiple `Datasets` centered on different coordinates¶

Open c point variable: `SSH`¶

Open u and v point variables: `ADVx_TH` and `ADVy_TH`¶

Examining the dimensions and coordinates of these `Datasets`¶

Merging multiple `Datasets` from state estimate variables¶

Examining the merged `Dataset`¶

1. Dimensions¶

2. Dimension Coordinates¶

3. Non-Dimension Coordinates¶

4. Attributes¶

Adding the model grid `Dataset`¶

Load the model grid parameters¶

Merge `grid_all_tiles` with `output_merged`¶

Examining the merged `Dataset`¶

Merging and memory¶

Summary¶

Combining multiple Datasets¶

Objectives:¶

Opening multiple Datasets centered on different coordinates¶

Open c point variable: SSH¶

Open u and v point variables: ADVx_TH and ADVy_TH¶

Examining the dimensions and coordinates of these Datasets¶

Merging multiple Datasets from state estimate variables¶

Examining the merged Dataset¶

1. Dimensions¶

2. Dimension Coordinates¶

3. Non-Dimension Coordinates¶

4. Attributes¶

Adding the model grid Dataset¶

Load the model grid parameters¶

Merge grid_all_tiles with output_merged¶

Examining the merged Dataset¶

Merging and memory¶

Summary¶

Combining multiple `Datasets`¶

Opening multiple `Datasets` centered on different coordinates¶

Open c point variable: `SSH`¶

Open u and v point variables: `ADVx_TH` and `ADVy_TH`¶

Examining the dimensions and coordinates of these `Datasets`¶

Merging multiple `Datasets` from state estimate variables¶

Examining the merged `Dataset`¶

Adding the model grid `Dataset`¶

Merge `grid_all_tiles` with `output_merged`¶

Examining the merged `Dataset`¶