Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Complete support for Python Xarray embedding. #1534

Closed
19 tasks
JohnHalleyGotway opened this issue Oct 21, 2020 · 4 comments · Fixed by #1614
Closed
19 tasks

Complete support for Python Xarray embedding. #1534

JohnHalleyGotway opened this issue Oct 21, 2020 · 4 comments · Fixed by #1614
Assignees
Labels
MET: Library Code MET: Python Embedding reporting: DTC NOAA R2O NOAA Research to Operations DTC Project requestor: NCAR National Center for Atmospheric Research type: enhancement Improve something that it is currently doing
Milestone

Comments

@JohnHalleyGotway
Copy link
Collaborator

JohnHalleyGotway commented Oct 21, 2020

Describe the Problem

I am splitting issue MET #1472 down into 2 pieces: 1 bugfix (#1472) and 1 enhancement (this issue). By the time we work on this issue, #1472 should have already been resolved.

Here:

we exit(1) without printing an error message to the user! We should issue a bug fix to print an error message, and then implement the support for this section (Xarray Python embedding with pickling) for v10.0.

Expected Behavior

I expect a warning message instead of MET silently exiting and producing no output.

Environment

Describe your runtime environment:
*1. Machine: kiowa

To Reproduce

To force pickling, set the MET_PYTHON_EXE environment variable. This can be set to any version of Python I think, which just forces invocation of the pickling logic in MET which is necessary to reproduce the error.
export MET_PYTHON_EXE=/path/to/python

Call a test script I wrote for Xarray Python embedding
/usr/local/met-9.1/bin/plot_data_plane PYTHON_XARRAY testXR.ps 'name="/home/dadriaan/projects/remote/read_ascii_xarray.py /home/dadriaan/git/met/met/data/python/fcst.txt FCST";' -title "Testing" -v 10

Relevant Deadlines

NONE

Funding Source

2791541

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Review projects and select relevant Repository and Organization ones
  • Select milestone

Define Related Issue(s)

Consider the impact to the other METplus components.

Bugfix Checklist

See the METplus Workflow for details.

  • Complete the issue definition above.
  • Fork this repository or create a branch of master_<Version>.
    Branch name: bugfix_<Issue Number>_master_<Version>_<Description>
  • Fix the bug and test your changes.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into master_<Version> and link the pull request to this issue.
    Pull request: bugfix <Issue Number> master_<Version> <Description>
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Complete the steps above to fix the bug on the develop branch and link the pull request to this issue.
    Branch name: bugfix_<Issue Number>_develop_<Description>
    Pull request: bugfix <Issue Number> develop <Description>
  • Close this issue.
@JohnHalleyGotway JohnHalleyGotway added type: enhancement Improve something that it is currently doing component: library code requestor: NCAR National Center for Atmospheric Research alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle labels Oct 21, 2020
@JohnHalleyGotway JohnHalleyGotway added this to the MET 10.0 milestone Oct 21, 2020
@JohnHalleyGotway JohnHalleyGotway removed the alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle label Nov 5, 2020
@DanielAdriaansen
Copy link
Contributor

@davidfillmore while doing some other work, I prototyped a "read_xarray_python.py" script based on the "read_numpy_python.py" script already in MET. You can view that script on kiowa:/home/dadriaan/projects/remote/read_xarray_python.py. There's also some information in kiowa:/home/dadriaan/projects/remote/remote_demo.py near the bottom where I conform the attributes of the Xarray DataArray object to the MET standards which may be useful and is related to #1470.

@dwfncar
Copy link
Contributor

dwfncar commented Dec 21, 2020

write_pickle_dataplane.py fails on system module imort
import importlib.util
System path issue?

@dwfncar
Copy link
Contributor

dwfncar commented Dec 21, 2020

write_pickle_dataplane is run by the user's python,
designated by MET_PYTHON_EXE.
This must be python 3 and have the importlib module available.

@dwfncar
Copy link
Contributor

dwfncar commented Dec 21, 2020

Added read_ascii_xarray to met/scripts/python.
This script currently fails within write_pickle_dataplane with module 'read_ascii_xarray' has no attribute 'atttrs'.

@JohnHalleyGotway JohnHalleyGotway linked a pull request Jan 6, 2021 that will close this issue
8 tasks
JohnHalleyGotway added a commit that referenced this issue Jan 6, 2021
* Added sample script to read ascii data and create an xarray.

* Disabled use_xarray exit for testing.

* Get attrs from DataArray if using xarray.

* Removing commented out code.

* Minor fix to error message.

* Added plot_dataplane unit test for xarray.

* Fixed letter numpy and xarray unit tests.

* Modified Makefile.am to install read_ascii_xarray.py in share/met/python.

* Fixed unit_python.xml xarray letter test.

* Update read_ascii_xarray.py

Removing commented out code to avoid future confusion.

* More cleanup.

* Per #1534, fixing the issue that I introduced in this python script.'

Co-authored-by: David Fillmore <fillmore@kiowa.rap.ucar.edu>
Co-authored-by: johnhg <johnhg@ucar.edu>
JohnHalleyGotway added a commit that referenced this issue Jan 7, 2021
MET PR #1614 for Issue #1534 changed the output of the NB. And the NB on 20210107 failed as expected with the errors listed below:
ERROR: folder MET-develop-ref/test_output missing 2 files
    python/letter_numpy.ps
    python/letter_xarray.ps
ERROR: folder MET-develop/test_output missing 1 files
    python/letter.ps

Updated the develop-ref branch to get past these differences.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
MET: Library Code MET: Python Embedding reporting: DTC NOAA R2O NOAA Research to Operations DTC Project requestor: NCAR National Center for Atmospheric Research type: enhancement Improve something that it is currently doing
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants