python-xarray: open_mfdataset concat along two dimensions

Question

I have files which are made of 10 ensembles and 35 time files. One of these files looks like:

>>> xr.open_dataset(\'ens1/CCSM4_ens1_07ic_19820701-19         


        

Answer 1

xarray does now support N-D concatenation. As your data has 1-D dimension coordinates, you can simply do

ds = xr.open_mfdataset('ens*/*NPac*.nc', combine='by_coords')

and it should combine them in order automatically! It should even work for the ensemble dimension, as you gave that a coordinate too.

Also see this answer to a very similar question.

Answer 2

xarray.open_mfdataset does not support 2d merges. What you will need to do is use concat along the second dimension:

import os
import xarray as xr

ens_list = []
for num in range(1, 11):
     ens = 'ens%d' % num
     ens_list.append(xr.open_mfdataset(os.path.join(ens, '*NPac*')))
ds = xr.concat(ens_list, dim='ensemble')

This is a common problem that xarray users run into. It is quite difficult, however, to write a generalized ND concat routine.

Answer 3

I wrote the following function as a workaround for my own use case: https://gist.github.com/jnhansen/fa474a536201561653f60ea33045f4e2

It works with arbitrary dimensions, but currently requires that the same variables exist in each file/dataset.

In my case I have a number of tiles (split along e.g. lat, lon, and time):

ds = auto_merge('data/part*.nc')

This will execute immediately as it returns only a view of the data (just like xarray.open_mfdataset would do).