State of ulmo

[jfburkhart]

This is a very valuable resource, but it seems to be abandoned.

Builds are currently failing, the code doesn't seem to be getting updated, and there are some deprecation warnings coming in (e.g. #156 )

@kkremitzki wants to package it for Ubuntu (#158 ), which would be great, but there should be some assurance the package is being maintained.

Maybe @solomon-negusse or @dharhas could comment?

[evanleeturner]

I'm also curious to see what plans @dharhas has for this package in the long term. I sat through his most recent presentation at Scipy and I think he is working toward more unified data standards. I use this package as well to access water data - but i'm not interested in inheriting it!!

In a larger conversation, it would be nice for the world if data warehouses agree on standards of communication - so ulmo would be unnecessary!

[kkremitzki]

I've actually already packaged it for Debian, and so now it should be available in Ubuntu 18.10 and beyond: https://tracker.debian.org/pkg/python-ulmo

[sdc50]

Our current development efforts have been on a new package called Quest, which uses Ulmo as a dependency. We are planning to refactor Ulmo soon to incorporate some of the changes that we've included in Quest. That said, we are still planning to get the builds working and get some patches in before starting the refactor.

(@dharhas please correct me as necessary)

[dharhas]

@sdc50 is correct. We have been working on another package called Quest that we were recently able to open source. We hope to use some of the lessons learned in Quest to refactor and cleanup the ulmo codebase. Ulmo still builds correctly locally but the CI is broken because something changed on the CI side, I think it is the url we retrieve miniconda from, but I haven't been able to check. There is a pull request that fixes #156 but I'm not keen on merging that until we get the CI stuff working again.

Unfortunately, I haven't had the cycles recently to work on any of this, and probably won't in the near future. It is on @sdc50 roadmap, but we don't have a concrete timeline for it.

[dharhas]

So I fixed the CI system and merged it in. I'm dropping testing for Python 2 and adding 3.7.

Unfortunately, there are currently many test failures due to changes in various upstream websites. I'm not sure what our strategy will be going forward. If there is an easy fix we can probably fix it. If it is a complete rewrite of a module and it isn't one we use we may have to drop support unless someone steps up with a pull request.

@sdc50 @solomon-negusse any thoughts?

[ColinTalbert]

I just happened to have found a bug caused by a change in the service for NASA ORNL's Daymet. It's an easy fix. If Ulmo will be continuing I can submit a pr.

If I should be moving to quest, I can do that as well.

I think this is a useful package but I don't get a chance to use it frequently.

[dharhas]

@talbertc-usgs please submit a PR for ulmo. Quest and Ulmo have a lot of overlap but Quest is not yet a replacement for ulmo. Currently my plan when I have some cycles to spare is to do one more release of ulmo with a bunch on the recent fixes and then think about the long term.

[solomon-negusse]

@dharhas i'm sorry for the long delay - i'm just seeing your message here now. dropping test support for 2.7 will be fine with me. we'll make sure that tests pass on py3 when we push any updates. in terms of maintenance, i'm on the same boat... we have only had enough bandwidth to update the services we use here when the need arises and we'll continue to do that.

[emiliom]

@dharhas I know "cycles to spare" are in short supply. Just wondering what your current thinking is regarding merging PR's and possibly issuing a new release. Selfishly, I'm most interested in the PR I submitted a couple of weeks ago (#165), plus #157. I may be able to help, if you'd like, after mid February. Thanks!

[dharhas]

@emiliom I have a work related deadline Feb 1st. Once I'm past that, I should be able to devote some time to merging some PR's and making a new release. My current thinking is to disable any services that no longer work and make a new release. I'll try and think through the best way to do that next week.

[emiliom]

Regarding #156, I don't see an open PR that fixes it. You might be thinking of #157?

#156 involves https://github.com/ulmo-dev/ulmo/blob/master/ulmo/util/misc.py#L52. The fix is simple, changing that line to return pandas.Timestamp(datetime).to_pydatetime(). I'll be testing this over the next couple of days and would be happy to submit a PR afterwards.

FYI, I've created a branch on my fork which merges that fix, together with PRs #157, #164 and #165. That's what I (and someone I'm working with) will be testing over the next week. We'll be using ulmo heavily at a webinar on 2/7 next week, and wanted to be able to use the latest Pandas as well as have some of these fixes in place.

[emiliom]

@talbertc-usgs can you go ahead and submit a PR for the NASA ORNL's Daymet fix you mentioned above at #161 (comment)? That way we can fold it into a next release fairly soon. Thanks!

[dharhas]

I'm working on this today (02/04/2019). I should be able to get a release out before @emiliom webinar.

[emiliom]

Thanks @dharhas! But no pressure. I'm good for my webinar with the temporary fork I've already created. I've got a conda environment file that installs that version of ulmo directly from my fork via pip install, and that's meeting our needs this week.

[dharhas]

Ok I just pushed a bunch of fixes to bring things up to date and compatible with modern pandas and updates to various source web services. Tests now pass on my local machine. CI is running now, hopefully my fixes are cross platform.

I attempted to fix all the services that I could but wasn't able to get all of them.

• eros: I've removed the 'eros' service, the usgs server for that no longer exists.
• lcra: The two LCRA services are completely broken and from a quick look at the LCRA website, I think they would require a complete rewrite.
• nwis: The basic USGS NWIS service is working now and passes all tests but the hdf5 and pytables backends are completely broken for recent versions of Pandas since so much has changed since they were originally written.

I'm thinking of removing the lcra service completely unless someone wants to rewrite it. I'll leave the hdf5 and pytables in with a warning that they don't work on recent pandas versions

[dharhas]

@emiliom @solomon-negusse

CI Tests are finally passing again. I will create a 0.8.5 conda & pip package tomorrow. If there is anything else folks want to add before I do the release, please let me know.

[kkremitzki]

Great, as soon as it's tagged I'll be ready to prepare a new package upload for Debian (just in time for Debian 10 release too! thanks for working on a new release.)

[solomon-negusse]

@evanleeturner we are still pulling data for the LCRA water quality stations, right? if so, i think it'd be great to fix it here on ulmo instead of internal scripts.. i'll check if LCRA has a better api to get the data from

[evanleeturner]

@solomon-negusse Our LCRA script was using ulmo and that broke at some point in late 2018. They had made some sort of change because their data is no longer live reporting, so that may have altered their internal API. The website system they have is still functional: https://waterquality.lcra.org/parameterbay.aspx?qrySite=MB02 I haven't been able to get in touch with their people but I agree that I could do a fix for their data here.

[emiliom]

@dharhas thanks for all that work! That's terrific.

I don't have anything else to suggest, beyond my prodding @talbertc-usgs to see if he can submit soonish the ORNL Daymet PR he offered. While I don't use Daymet, I think it'd be good to wait to see if he has time to submit that bug fix so it can be folded into this release (assuming that's not too disruptive to you).

If you'd like a second pair of eyes and tests on your recent fixes before issuing the release, I can offer to help; but it'd have to be next week.

[dharhas]

@evanleeturner @solomon-negusse if you plan to get the LCRA fix in, I'll wait a few days before making a new release. Otherwise, I'll plan to remove those services.

[evanleeturner]

Just do whatever you feel like. I don't have time to look at this until maybe late March at the earliest when we get our vacant positions filled.

[emiliom]

@dharhas, I'm assuming you lost your momentum for issuing a new ulmo release? It looked like you were practically ready. Let me know if I can help pushing it out the door soon -- say, by helping draft release notes.

Looking longer term, I've been thinking of trying to persuade CUAHSI (technical staff in the Boston office) to play a role in helping to maintain ulmo ... Thoughts?

[dharhas]

@emiliom I got busy. Drafting release notes would help. Long term, I'm happy to hand over the reigns to someone more invested in ulmo going forward. I don't use ulmo as part of my day job any longer so its difficult to find time to push it forward.

[emiliom]

Drafting release notes would help.

Alright, I'll try to start on Monday.

Long term, I'm happy to hand over the reigns to someone more invested in ulmo going forward. I don't use ulmo as part of my day job any longer so its difficult to find time to push it forward.

Yeah, that's why I've been thinking about sustainability. I'll reach out to CUAHSI in the next couple of weeks (I'm on its informatics committee).

[emiliom]

@dharhas I looked at the commits listing since the last release, and came up with the release notes below. Edit as you see fit. I assume the release version tag will be v0.8.5? If you could issue the release by Friday, that'd be really helpful. Once the release is ready, I can follow up on the conda-forge feedstock to get the new conda package built.

FYI, I wrote to the CUAHSI Informatics committee to propose having CUAHSI adopt ulmo. We'll probably discuss this at our next meeting, next week, and hopefully make a recommendation to CUAHSI right after it. I'll report back.

Thanks!

---

This release includes commits from 8 contributors. It includes many bug fixes and changes to plugins, as well as to the CI, test, and package build infrastructures.

Bug Fixes and Changes to Plugins

• Bug fixes and some small enhancements for multiple plugins: CDEC, CPC, CUAHSI WaterML/WOF, NASA ORNL Daymet, NCDC GSOD and GHCN, USACE, and USGS EDDN
• Removed USGS EROS plugin (service is no longer available)
• LCRA plugins are experiencing problems. Removed from CI testing, until fixes are applied.

Other Bug Fixes

• Pandas API updates and error handling
• Continuous Integration and test infrastructure, including Python 3.6 and 3.7 builds

[emiliom]

A new ulmo release (0.8.5) is now out, and a available via pypi and conda-forge. Thanks @dharhas for your help getting set up!

Also, on Wednesday I'll be making a pitch to CUAHSI to adopt ulmo

[ChristinaB]

@emiliom How did that pitch go? What does 'adopt' mean?

[emiliom]

@ChristinaB thanks for asking. I made the pitch back then, and the reception was positive but non-committal. What "adoption" would entail was TBD, but it probably would focus around ulmo readers for CUAHSI data.

I was asked to send them a justification with background on ulmo. I started putting that written pitch together, including statistics about usage and contributions, but I never completed it. I still would like to do it. Also, my momentum was deflated by not getting any responses to my request (April) for testimonials from users, over on the ulmo Google groups.

I'll try to get back to this next week, once and for all. There's also a PR waiting for my approval and merge.

[mroberge]

@emiliom Hey I've been lurking for a while and I just wanted to check in to say thank you to everyone for their work on this project. It is not easy to maintain open source software, but it can make a big difference.
I have my own OSS project (hydrofunctions) that I created to make it easier for my students to work with USGS stream gauge data. I've long been considering Ulmo to provide access to more data sources.

[emiliom]

Thanks @mroberge! And thanks also for mentioning hydrofunctions. I didn't know about it, and it looks like a really nice tool for NWIS access. I'll try to check it out in more detail later on.

[dharhas]

I think this issue can be closed now.

In summary, @emiliom has taken over active maintenance of ulmo and has also proposed to the CUAHSI Informatics group that they take the project under their umbrella (that has not happened yet, but is still a possibility). I still lurk and am available to answer historical questions about the codebase but am no longer actively involved in the project.