Further documentation on creating fields and writing fields to file

[sadielbartholomew]

As stated in an email thread:

An overview of creation and writing near the top of the tutorial (where there is already a read overview) would be beneficial.

This is in response to a(n expert) user request:

a couple of tutorials on "how to create CF compliant data" would be really useful. We could tie them to useful examples, like: creating a multimodel ensemble dataset and "converting a grib file to cf compliant netcdf"

[kurtsansom]

This would be a great help!

Lets say I already have a compliant file, can I extract the schema in a sensible in order to populate a new file with similar but potentially different data. An example like this would be really helpful.

Is there a way to create a "skeleton" of the file and then populate it with data?

[sadielbartholomew]

Hi @kurtsansom, thanks for registering that you would also find such further documentation useful. It helps us to know multiple people would benefit from some improvement, as in this case, so we can make it a higher priority.

Your particular questions are interesting because (if I understand your requests correctly) both are indeed possible and also documented quite well, just not with the cfdm library directly, rather in cf-python, our data analysis library which extends cfdm in functionality and scope. So, if you are willing to use cf-python instead of cfdm standalone (see here for the installation guide), I can point you directly to documentation which covers in detail how to do what you wish to, programmatically:

Lets say I already have a compliant file, can I extract the schema in a sensible in order to populate a new file with similar but potentially different data.

Is there a way to create a "skeleton" of the file and then populate it with data?

Yes, this can be achieved by two different approaches in cf-python:

1. reading in the compliant file (with cf.read) to produce a corresponding list of fields;
2. then either:
   • overwriting the data of these read-in fields (or copies of them if preferred) using the set_data methods on the metadata constructs or entire field; or
   • running the creation_commands method on each read-in field to produce the exact sequence of calls required to generate the fields from scratch (see this sub-section with an example), then running those but with the arguments to the set_data call changed as applicable to set different data on the fields;
3. finally writing the fields from step (2) out to file via the cf.write function).

An example like this would be really helpful.

We do have detailed examples for each the three stages I list above in our documentation, but not all three stages at once to create an end-to-end example for what you specifically want to do. Now I have pointed out relevant sections, do you think it could still be helpful to have an end-to-end example, or do you think what we already have, as covered separately, is sufficient? I am asking because I personally think it could be "overkill" to have an additional section, but what is obvious to me about how to link various functionality together might not be obvious to others.

In general, for documentation on using cf-python to write out new files modified in a CF-compliant way relative to existing ones, I suggest consulting the following sections which detail the two steps necessary:

1. first to create the appropriate field construct (which does not have to be done by reading in existing fields, as in my above list, but can be done from scratch), see this section for overall approaches and examples;
2. then to write the field construct to file see this section or the cf.write API reference.

If instead you want or need to use cfdm and not cf-python, note that in the near future we plan to port the so-called 'creation commands' which enable such constructions from cf-python into cfdm, such that they can be used in both libraries and therefore will be available directly in cfdm too. Of course we will update the documentation accordingly to cover the extended capability.

I hope this helps. If it is not quite what you were asking for, or you have further questions, please let us know.

[davidhassell]

The " creation commands" methods will be available in cfdm at the next release (1.8.7.0). They'll behave exactly the same as they do in cf-python (i.e. https://ncas-cms.github.io/cf-python/tutorial.html#command-modification)

Thanks.

[kurtsansom]

@sadielbartholomew
This is really helpful information. I am trying to figure out the cfdm to better understand what I need to modify.

I think I was thinking along the lines of the metadata as well. pyart, uses something like a simple dictionary of dictionaries to define the attributes that map to the actual variables in the code. Or something like ncdump except it generates a dictionary with the fields and attributes.

add_attrs = {
    'latitude': {
        'long_name': 'Latitude',
        'standard_name': 'Latitude',
        'units': 'degrees_north',
        'comment': 'Latitude of instrument, using WGS84.',
    },
}

[davidhassell]

Hi @kurtsansom, creating such a dictionary should be straightforward, using the properties method of the constructs, for example:

In [8]: import cfdm

In [9]: f = cfdm.example_field(0)

In [10]: f.properties()
Out[10]: 
{'Conventions': 'CF-1.8',
 'project': 'research',
 'standard_name': 'specific_humidity',
 'units': '1'}

In [11]: for c in f.constructs.values():
    ...:     try:
    ...:         print(repr(c), c.properties())
    ...:     except AttributeError:
    ...:         pass
    ...:     
<DimensionCoordinate: latitude(5) degrees_north> {'units': 'degrees_north', 'standard_name': 'latitude'}
<DimensionCoordinate: longitude(8) degrees_east> {'units': 'degrees_east', 'standard_name': 'longitude'}
<DimensionCoordinate: time(1) days since 2018-12-01 > {'units': 'days since 2018-12-01', 'standard_name': 'time'}

Does that help? Do let me know if I've missed anything.