remove upper pins on spacetelescope projects

[zacharyburnett]

This PR addresses issues with testing that arise from upper pins on dependencies.

Checklist for maintainers

• added entry in CHANGES.rst within the relevant release section
• updated or added relevant tests
• updated relevant documentation
• added relevant milestone
• added relevant label(s)
• ran regression tests, post a link to the Jenkins job below.
  How to run regression tests on a PR
• Make sure the JIRA ticket is resolved properly

[codecov]

Codecov Report

Patch and project coverage have no change.

Comparison is base (40f282e) 77.60% compared to head (9f98b41) 77.60%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #7480   +/-   ##
=======================================
  Coverage   77.60%   77.60%           
=======================================
  Files         452      452           
  Lines       36087    36087           
=======================================
  Hits        28005    28005           
  Misses       8082     8082           

Flag	Coverage Δ		*Carryforward flag
nightly	77.61% <ø> (ø)		Carriedforward from 40f282e
unit	49.57% <ø> (-0.08%)	⬇️

*This pull request uses carry forward flags. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[nden]

Is this going to create issues to external users the next time e.g. stcal releases incompatible changes? Or is this meant to be temporary while we are testing and developing and then pinning again before the next release?

[braingram]

Thanks!

This should unblock the development on stdatamodels by allowing the CI there to test against the development version of jwst and stdatamodels (which has a version above the upper pin that is removed in this PR).

For more context see the stpipe PR (stpipe was checking the jwst version requirements at runtime which made a forced install of incompatible dependencies not possible): spacetelescope/stpipe#84

[WilliamJamieson]

Is this going to create issues to external users the next time e.g. stcal releases incompatible changes? Or is this meant to be temporary while we are testing and developing and then pinning again before the next release?

Adding upper pins to dependencies for libraries causes all sorts of issues downstream and causes all sorts of unintended consequences, see https://iscinumpy.dev/post/bound-version-constraints/ for a better analysis than I can give.

In general, it is up to external users to manage their dependencies not us. In particular, downstream packages to JWST should in fact be testing regularly against the development version of JWST. As for individual users, we should give clear guidance on to how to update their environments. Namely, pip install --upgrade does not work as expected, though we can limit some issues via changing the lower pin on say stcal as needed.

[nden]

I only quickly skimmed that link and I agree with what it says. I'm not sure it fully applies for the jwst package, since it's not exactly a library.
The issues arising from releasing unpinned jwst are listed and were discussed here
https://innerspace.stsci.edu/display/SCSB/Development+Workflow+-+stcal%2C+stpipe%2C+jwst%2C+romancal

We could go over them and discuss them again if you htink it's necessary. OTH, I think it's OK to unpin while in development unless there's something that needs to be avoid in a specific dependency version.

[nden]

My impression is that with pinned dependencies in jwst we have had fewer help calls or issues to resolve (the current testing one being the only one, actually) than with unpinned dependencies. Others should correct me on this one if I'm wrong.

[WilliamJamieson]

Given that our release style guilde https://github.com/spacetelescope/style-guides/blob/master/guides/release-workflow.md calls for releases to occur from their own distinct branches, I think it a reasonable compromise for JWST would be to pin dependencies only on these release branches only.

This would limit help calls in the same way the current pins do, while making it much simpler to do development (running CI) across multiple packages, which is something that is often required for JWST. This would fix both issues at the expense of making the release process slightly more complicated (CI checks and automations could limit these issues).

[zacharyburnett]

... I think it a reasonable compromise for JWST would be to pin dependencies only on these release branches only.

That's a good idea, we could definitely incorporate that into the process (and also consistently release from dedicated branches)