Is there a way to handle concatenating dataset if data_vars differ?

[ahuang11]

import xarray as xr
import pandas as pd
ds1 = xr.tutorial.open_dataset('air_temperature')
ds2 = ds1.copy()
ds2['airx2'] = ds1['air'] * 2
ds2['time'] = ds2['time'] + pd.Timedelta(days=1000)
ds2 = ds2.drop('air')
ds3 = xr.concat([ds1, ds2], 'time')

ValueError: 'air' is not present in all datasets.

Is there a way to get around this? I tried setting data_vars='different' but still errors.

[ahuang11]

I guess combine_by_coords seems to work, but I wonder if concat can have a keyword to handle this automatically?
ds3 = xr.combine_by_coords([ds1, ds2])

[keewis]

Unfortunately, the documentation of the data_vars parameter is a bit unclear, so we need to look at the code to know its purpose. The variables to be concatenated are calculated in _calc_concat_over (which could use some refactoring, by the way):

xarray/xarray/core/concat.py

Lines 220 to 333 in dd9fe2a

	def _calc_concat_over(datasets, dim, dim_names, data_vars, coords, compat):
	"""
	Determine which dataset variables need to be concatenated in the result,
	"""
	# Return values
	concat_over = set()
	equals = {}
	if dim in dim_names:
	concat_over_existing_dim = True
	concat_over.add(dim)
	else:
	concat_over_existing_dim = False
	concat_dim_lengths = []
	for ds in datasets:
	if concat_over_existing_dim:
	if dim not in ds.dims:
	if dim in ds:
	ds = ds.set_coords(dim)
	concat_over.update(k for k, v in ds.variables.items() if dim in v.dims)
	concat_dim_lengths.append(ds.dims.get(dim, 1))
	def process_subset_opt(opt, subset):
	if isinstance(opt, str):
	if opt == "different":
	if compat == "override":
	raise ValueError(
	"Cannot specify both %s='different' and compat='override'."
	% subset
	)
	# all nonindexes that are not the same in each dataset
	for k in getattr(datasets[0], subset):
	if k not in concat_over:
	equals[k] = None
	variables = []
	for ds in datasets:
	if k in ds.variables:
	variables.append(ds.variables[k])
	if len(variables) == 1:
	# coords="different" doesn't make sense when only one object
	# contains a particular variable.
	break
	elif len(variables) != len(datasets) and opt == "different":
	raise ValueError(
	f"{k!r} not present in all datasets and coords='different'. "
	f"Either add {k!r} to datasets where it is missing or "
	"specify coords='minimal'."
	)
	# first check without comparing values i.e. no computes
	for var in variables[1:]:
	equals[k] = getattr(variables[0], compat)(
	var, equiv=lazy_array_equiv
	)
	if equals[k] is not True:
	# exit early if we know these are not equal or that
	# equality cannot be determined i.e. one or all of
	# the variables wraps a numpy array
	break
	if equals[k] is False:
	concat_over.add(k)
	elif equals[k] is None:
	# Compare the variable of all datasets vs. the one
	# of the first dataset. Perform the minimum amount of
	# loads in order to avoid multiple loads from disk
	# while keeping the RAM footprint low.
	v_lhs = datasets[0].variables[k].load()
	# We'll need to know later on if variables are equal.
	computed = []
	for ds_rhs in datasets[1:]:
	v_rhs = ds_rhs.variables[k].compute()
	computed.append(v_rhs)
	if not getattr(v_lhs, compat)(v_rhs):
	concat_over.add(k)
	equals[k] = False
	# computed variables are not to be re-computed
	# again in the future
	for ds, v in zip(datasets[1:], computed):
	ds.variables[k].data = v.data
	break
	else:
	equals[k] = True
	elif opt == "all":
	concat_over.update(
	set(getattr(datasets[0], subset)) - set(datasets[0].dims)
	)
	elif opt == "minimal":
	pass
	else:
	raise ValueError(f"unexpected value for {subset}: {opt}")
	else:
	invalid_vars = [k for k in opt if k not in getattr(datasets[0], subset)]
	if invalid_vars:
	if subset == "coords":
	raise ValueError(
	"some variables in coords are not coordinates on "
	"the first dataset: %s" % (invalid_vars,)
	)
	else:
	raise ValueError(
	"some variables in data_vars are not data variables "
	"on the first dataset: %s" % (invalid_vars,)
	)
	concat_over.update(opt)
	process_subset_opt(data_vars, "data_vars")
	process_subset_opt(coords, "coords")
	return concat_over, equals, concat_dim_lengths

If I read that correctly, all variables with dim are added to the set of variables to be concatenated. Every value other than data_vars="minimal" extends that with variables that don't have that dimension.

As a summary, there are two documentation bugs: we don't explicitly state that concatenated variables have to be present in all objects (we currently only learn about that from the error message), and the documentation of data_vars could use some polishing.

I guess the conclusion is that xr.concat is just not the right tool for your use case, and that it would probably be better to use xr.merge or a related function like xr.combine_by_coords?

[ahuang11]

Thanks! I didn't know xr.merge could concatenate coordinates automatically too.

[spencerkclark]

This is indeed often awkward. There's been some work and discussion around this before, but nothing yet merged; see #3545 and #508.