Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

File name conversion tool #438

Merged
merged 11 commits into from
Dec 9, 2022
Merged

File name conversion tool #438

merged 11 commits into from
Dec 9, 2022

Conversation

wrongkindofdoctor
Copy link
Collaborator

Description
Add a helper tool rename_input_files.py to convert file names and paths to symlinks that adhere to the LocalFile format. The script and config file template are located in a subdirectory in a new tools folder for scripts to aid in testing and data handling that are separate from the main framework code.

Associated issue #436

How Has This Been Tested?
Tested on centos8 system with Python 3.10

Checklist:

  • I have reviewed my own code to ensure that if follows the POD development guidelines
  • My branch is up-to-date with the NOAA-GFDL main branch, and all merge conflicts are resolved
  • The scripts are written in Python 3.10 or above (preferred; required if funded by a CPO grant), NCL, or R
  • All of my scripts are in the diagnostics/[POD short name] subdirectory, and include a main_driver script, template html, and settings.jsonc file
  • I have made corresponding changes to the documentation in the POD's doc/ subdirectory
  • I have requested that the framework developers add packages required by my POD to the python3, NCL, or R environment yaml file if necessary, and my environment builds with conda_env_setup.sh
  • I have added any necessary data to input_data/obs_data/[pod short name] and/or input_data/model/[pod short name]
  • My code is portable; it uses MDTF environment variables, and does not contain hard-coded file or directory paths
  • I have provided the code to generate digested data files from raw data files
  • Each digested data file generated by the script contains numerical data (no figures), and is 3 GB or less in size
  • The repository contains no extra test scripts or data files

@wrongkindofdoctor
add tools directory with script to rename input files to local file s…
72519b6 
…tandard and configuration template file

add the click package to the base conda environment file
@wrongkindofdoctor
add functions to make new file names and link files to rename_input_f…
6d59009 
…iles.py
@wrongkindofdoctor
moved rename_input_files.py and config template to designated subdire…
0fd7648 
…ctory in tools

finalized rename_input_files.py
modified config_template parameters and added comments
@wrongkindofdoctor
remove personal directories from template file and make filenames non…
67d80b6 
…standard
@wrongkindofdoctor wrongkindofdoctor added feature-request New feature or request tool Scripts or features that assist with data or code management that are not part of the framework src labels Dec 9, 2022
@wrongkindofdoctor wrongkindofdoctor self-assigned this Dec 9, 2022
github-advanced-security[bot]
github-advanced-security bot found potential problems Dec 9, 2022
View reviewed changes
tools/rename_input_files/rename_input_files.py
# add new_name attribute to the original file name
# with the format <CASENAME>.<frequency>.<variable name>.nc
def append_new_filenames(file_list=list, casename=str):
for f in file_list:

Check failure

Code scanning / CodeQL

Non-iterable used in for loop

This for-loop may attempt to iterate over a [non-iterable instance](1) of class [type](2).
tools/rename_input_files/rename_input_files.py
except yaml.YAMLError as exc:
print(exc)

casename = case_info['CASENAME']

Check failure

Code scanning / CodeQL

Potentially uninitialized local variable

Local variable 'case_info' may be used before it is initialized.
tools/rename_input_files/rename_input_files.py Fixed Show fixed Hide fixed
tools/rename_input_files/rename_input_files.py Fixed Show fixed Hide fixed
@wrongkindofdoctor wrongkindofdoctor merged commit 00766e7 into NOAA-GFDL:main Dec 9, 2022
@wrongkindofdoctor wrongkindofdoctor deleted the add_developer_mode branch December 12, 2022 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jkrasting jkrasting Awaiting requested review from jkrasting jkrasting is a code owner

@aradhakrishnanGFDL aradhakrishnanGFDL Awaiting requested review from aradhakrishnanGFDL aradhakrishnanGFDL is a code owner

Assignees

@wrongkindofdoctor wrongkindofdoctor

Labels
feature-request New feature or request tool Scripts or features that assist with data or code management that are not part of the framework src
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@wrongkindofdoctor