Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Netcdf file types at NERSC #1970

Closed
rljacob opened this issue Dec 7, 2017 · 7 comments
Closed

Netcdf file types at NERSC #1970

rljacob opened this issue Dec 7, 2017 · 7 comments
Labels
Documentation

Comments

@rljacob
Copy link
Member

rljacob commented Dec 7, 2017 •
edited
Loading

Recording this observation from @ndkeen

When I walk the entire inputdata directory at NERSC, I counted: Number of files= 44566 Total GB= 8173.320 Of those files, 36837 are netcdf files (identified by .nc extension). I then (after loading default pnetcdf module on Cori) used ncdump -k on each file, and here are results:

classic               33103
64-bit offset          3619
netCDF-4 classic model   55
netCDF-4                 25
Permission denied        27
Unknown file format       4
cdf5                      4
total                 36837
@rljacob
Copy link
Member Author

rljacob commented Dec 7, 2017

Which one of these can E3SM not read?

@ndkeen
Copy link
Contributor

ndkeen commented Dec 7, 2017 •
edited
Loading

Thanks Rob, I didn't think this was a code issue? Now I'm not sure if our code is unable to read them or not, I just know that the SMS_R test failed when trying to open a file that was classified as netCDF-4 classic model. Copying over this file with one of type classic fixed that particular issue. And I'm wondering if we need cleanup? Or occasional checks?
I can provide a list of all of our netcdf files, the size, and the netcdf type.

@jayeshkrishna
Copy link
Contributor

jayeshkrishna commented Dec 7, 2017 •
edited
Loading

The PIO iotype needs to be changed from netcdf to netcdf4c or netcdf4p to read the netcdf4 files.
The case that failed required an input file that was in netcdf4 format and was being read using pnetcdf/netcdf. @bogensch confirmed that he does not need the "netcdf4 classic" format for the input file and will check in the same file after converting it to "netcdf classic" so that all pio iotypes can read them.

@rljacob
Copy link
Member Author

rljacob commented Dec 7, 2017

So we could read the "netcdf4 classic" files but it means changing the iotype and that can't be done per-file?

@jayeshkrishna
Copy link
Contributor

The PIO interface allows changing the iotype per file. However the E3SM application relies on iotypes specified via env_run.xml:PIO_TYPENAME value.
The netcdf4 interfaces in PIO1 work but are not very well tested (used frequently).

@ndkeen ndkeen mentioned this issue Jan 23, 2018
Open
@PeterCaldwell
Copy link
Contributor

I'm pretty sure there's no scientific benefit to using any particular netcdf format. In other words, I don't think anyone creating netcdf files cares what format they use... they are basically just choosing a format at random. It would be useful for the people creating these files (like @cameronsmith1 and @golaz) to know which format is best performance-wise. @jayeshkrishna - do you have any suggestions in that regard? Note that reformatting existing netcdf files would also probably be fine.

@ndkeen ndkeen mentioned this issue Feb 1, 2018
Closed
@amametjanov amametjanov mentioned this issue Apr 18, 2018
Closed
@cameronsmith1 cameronsmith1 mentioned this issue Jul 10, 2019
Closed
@rljacob
Copy link
Member Author

rljacob commented Jul 19, 2019

Closing this. See #2292 for more.

@rljacob rljacob closed this as completed Jul 19, 2019
rljacob pushed a commit that referenced this issue Apr 12, 2021
@jgfouca
Merge pull request #1970 from ESMCI/jgfouca/branch-for-acme-split-10-…
5fc342d 
…17-2017-pr

Bring in E3SM cime changes for past 3 months

Major change list:

Python

Minor improvements to bless_test_results.py
Add mach-specific throughput checking to test machines
Some tools were not loading environment before running which was causing problems on some platforms
Improvements in robustness, error msgs of check_input_data
configure.py new FakeCase class
add FakeCase to fortran_unit_testing/run_tests.py
system_tests_common.py change to tput test
env_batch.py add new CHARGE_ACCOUNT
env_mach_specific.py - Allow config_machines.xml to have any case variable as an attribute,
pes.py - also output threads, root
generic_xml.py - remove minidom
build.py - add USE_MOAB option
case.py - use new CHARGE_ACCOUNT. new SMP_PRESENT
provenancy.py - update E3SM provenance.
test_scheduler.py - tput tolerance. new "R" modifier for single column tests.
utils.py new get_charge_account method
update_acme_tests.py - more tests and test suites.
schema changes

config_compilers_v2.xsd: Add MOAB
config_machines.xsd: Add CHARGE_ACCOUNT for charging to a different name then PROJECT used for directory paths
config_machines_template.xml: Also add CHARGE_ACCOUNT
env_mach_specific.xsd: Change debug to upperBoolean
Tools

new acme_cime_split to help with subtree splitting
new tests for the run_acme script in scripts/test/run_acme_test
xml changes

Change "debug" to "DEBUG" in all config_machines.xml
** data models **

datm/cime_config/namelist_definition_datm.xml new 6hourly forcing data set
** mct driver **

config_component.xml new SMP_PRESENT, USE_MOAB, TEST_TPUT_TOLERANCE, CHARGE_ACCOUNT
cime_comp_mod.F90 new option to shr_mem_getusage to also print.
seq_diag_mct.F90 fix an E3SM diag
timing

fix for BGQ
shr_reprosum_mod.F90 - optionally check for nans.

Rename some man3 pages in PIO1

Many E3SM config changes.

Test suite: scripts_regression_tests
Test baseline:
Test namelist changes:
Test status: bit for bit

Fixes ESMCI/cime#1332
Fixes ESMCI/cime#1626

User interface changes?:

Update gh-pages html N

Code review:
wlin7 pushed a commit that referenced this issue Feb 26, 2023
@E3SM-Bot
Merge Pull Request #1970 from E3SM-Project/scream/brhillman/eamxx/add…
bb01ade 
…-cldtot-output

Automatically Merged using E3SM Pull Request AutoTester
PR Title: Add vertically-projected cloud area output from radiation
PR Author: brhillman
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Documentation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rljacob @ndkeen @jayeshkrishna @PeterCaldwell