Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RLS: 2.0 #46776

Closed
1 task done
simonjayhawkins opened this issue Apr 14, 2022 · 108 comments
Closed
1 task done

RLS: 2.0 #46776

simonjayhawkins opened this issue Apr 14, 2022 · 108 comments
Labels
Milestone

Comments

@simonjayhawkins
Copy link
Member

simonjayhawkins commented Apr 14, 2022

Tracking issue for the 2.0 release.

Note: pandas 2.0 is the next major release in the pandas semver-like release cycle and different from some historical discussion about the pandas future also named pandas 2.

Blockers

Scheduled

Feb 2023

References

@simonjayhawkins simonjayhawkins added this to the 2.0 milestone Apr 14, 2022
@simonjayhawkins
Copy link
Member Author

we don't have a master tracker for deprecations that should be in a released pandas before we release 2.0.

There are currently 91 issues labeled deprecate https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3ADeprecate, so need to start whittling this list down.

@jbrockmendel
Copy link
Member

@mroeschke
Copy link
Member

we don't have a master tracker for deprecations that should be in a released pandas before we release 2.0.

There are currently 91 issues labeled deprecate https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3ADeprecate, so need to start whittling this list down.

My interpretation is that issues labeled deprecate doesn't necessarily mean they need to all be decided upon (yes & implemented or no) by 2.0; just that they are up for consideration anytime.

@rhshadrach
Copy link
Member

Agreed @mroeschke. I've been tagging deprecation issues where the conversation makes it clear there is no consensus as "Needs Discussion". There are currently only 46 issues that are tagged deprecation but not needs discussion. Maybe it'd be reasonable to whittle down this list by either (a) implementing or (b) tagging as needs discussion.

https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3ADeprecate+-label%3A%22Needs+Discussion%22

@lithomas1
Copy link
Member

@pandas-dev/pandas-core Are there any objections for the next release to be 2.0?

@phofl
Copy link
Member

phofl commented Sep 21, 2022

I think the agreement on the august call was to wait with the final decision till 1.5.1 is out in case anything unexpected happens

@mroeschke
Copy link
Member

As mentioned by @lithomas1 on Slack, do folks think we should push a 2.0.0.dev0 tag now that 2.0 is agreed to be the new version? The reasoning is to help downstream nightly/main users pin as we start enforcing deprecations.

This was referenced Jan 4, 2023
@datapythonista
Copy link
Member

I updated this issue description with the list of blockers for 2.0 I'm aware of (the known deprecations, plus a 1.4 deprecation that was not track or enforced). If there are other blockers to be considered, it'd be good to add them to the list before the call next week, so we can discuss them.

@mroeschke
Copy link
Member

Updated the issue description after the outcome of the 2023-01-11 dev meeting (only blocker was the Int/Uint/FloatIndex removal)

@mroeschke
Copy link
Member

Updated the issue description after the outcome of the 2023-02-08 dev meeting

@datapythonista would you still be willing to do a 2.0rc release sometime next week assuming the above PRs get merged? @lithomas1 expressed willingness to do a release if you are unable

@datapythonista
Copy link
Member

Sorry, I have a conflict, and I'm teaching at the time of the pandas dev call, I won't be able to join them for at least couple of months. But I can do the release when things are ready, and also happy to hand over to @lithomas1 if he prefers to do them. But if needed, no problem to continue doing the releases myself.

@jbrockmendel
Copy link
Member

There's a handful of comments # TODO(1.4): Change me to xfails at release time. i guess we should make that change for the 2.0 release?

@mroeschke
Copy link
Member

I don't think that should necessarily block a 2.0rc release next week

@lithomas1
Copy link
Member

There's a handful of comments # TODO(1.4): Change me to xfails at release time. i guess we should make that change for the 2.0 release?

Those are for the pyarrow csv tests, should be OK to ignore, but I'm getting around to fix them.

@rhshadrach
Copy link
Member

It'd be good to get in #51335 for a few reasons (see the linked issues), but not a blocker in my opinion.

@datapythonista
Copy link
Member

I think all blockers are now merged. There seems to be a problem with pylint on master. I'll start preparing for the release and fix it if nobody does before, and I'm aiming to tag and release 2.0 RC on Thursday. Feel free to continue merging things until then, but probably worth trying to get PRs rebased more often than usual to avoid problems in master.

@lithomas1
Copy link
Member

Wheel builders are currently broken on Windows and uploads don't work on aarch. Can you give till Monday to work things out?

@lithomas1
Copy link
Member

I have a PR open for aarch uploads but it'll take until the next nightly to verify it works.

@datapythonista
Copy link
Member

Absolutely, I didn't see that, thanks for the heads up. Let's aim next week then, let me know if you need help.

@lithomas1
Copy link
Member

aarch nightly uploads are successful (I haven't tested that it triggers on a tag, though. We'd have to do the RC to figure out if that functionality works).

Now, just the Windows failing tests. If we can worked out quickly, then maybe we can release on Friday.

@lithomas1
Copy link
Member

@phofl
It looks like #50764 is the cause.
(I dunno why this is not failing on the other Windows jobs, only difference here is that the Windows jobs for wheel builders are run in a Docker image)

(The wheel builders output dtype of int64 looks more correct)
https://github.com/pandas-dev/pandas/blame/main/pandas/tests/frame/test_query_eval.py#L1325

@lithomas1
Copy link
Member

All build blockers should be cleared for 2.0.

We could technically release on Thursday as originally planned, but I'd like the nightlies to do another full green run to flush out issues(which will be caught in downstream project's CI).

@phofl
Copy link
Member

phofl commented Feb 15, 2023

Yeah I think this is a good idea

@mroeschke
Copy link
Member

Yeah releasing Friday (or even Monday) sounds good

@phofl
Copy link
Member

phofl commented Feb 19, 2023

Plan is still to release tomorrow?

@mroeschke
Copy link
Member

It's an unclosed File warning that happened on the last 2, the 5 before that were green.

Just to note, these unclosed files are from matplotlib and not related to us #44844 (comment)

@datapythonista
Copy link
Member

PyPI packages already available. Tested for my platform, and everything seems fine. Still waiting for the conda-forge CI to finish (arm builds take longer). If all is green the conda-forge PR will automerge and packages for most architectures should be available in less than one hour.

I'll send an update here and announce the release once most conda-forge packages are available.

Are we OK to start merging now?

Yep, it's fine now. Not sure if I was clear, but it was only backporting to 2.0.x which was a problem. It's fine now, but as expected just for things that should go into the final 2.0.0 release.

@WillAyd
Copy link
Member

WillAyd commented Mar 16, 2023

Just to note, these unclosed files are from matplotlib and not related to us #44844 (comment)

The Python standard documentation has an explicit example of using tracemalloc to hunt these down. Stumbled across this and never personally used it, but might help someone with a passion for troubleshooting this one

https://docs.python.org/3/library/devmode.html#resourcewarning-example

@datapythonista
Copy link
Member

ppc64 still failing in conda forge, moving forward with the rest

@datapythonista
Copy link
Member

All conda-forge packages published (except ppc64), and release announced.

Unless there is something unexpected with the second release candidate, the plan is to release the final 2.0.0 in around two weeks.

We should do something with ppc64 before that. If the problem isn't easy to fix, should we xfail the date mismatches for that architecture, or should we remove ppc64 from conda-forge builds? No strong preference from my side.

dcherian added a commit to pydata/xarray that referenced this issue Mar 20, 2023
Pandas is expecting to release v2 in two weeks (pandas-dev/pandas#46776 (comment)). But we are still incompatible with their main branch: 
- #7441 
- #7420

This PR pins pandas to `<2`
dcherian added a commit to pydata/xarray that referenced this issue Mar 22, 2023
* Pin pandas  < 2

Pandas is expecting to release v2 in two weeks (pandas-dev/pandas#46776 (comment)). But we are still incompatible with their main branch: 
- #7441 
- #7420

This PR pins pandas to `<2`

* modify requirements.txt

* Update requirements
@MarcoGorelli
Copy link
Member

there's quite a few issues / PRs with the 2.0 milestone

it's hard to tell which are blockers - I'm trying to remove ones which clearly aren't blockers, but for others, could people please check whether the milestone of their issues/PRs should be updated?

@phofl
Copy link
Member

phofl commented Mar 29, 2023

Thoughts on releasing later this week or early next week?

@MarcoGorelli
Copy link
Member

are your open PRs milestoned to 2.0 not blockers? if so, let's do it!

probably best over the weekend or on monday, or avoid breaking code on a thursday/friday

@phofl
Copy link
Member

phofl commented Mar 29, 2023

The ones that should get in are ready, the others are not blockers, should have cleared this out by tomorrow (hopefully)

@datapythonista
Copy link
Member

I'm busy this week, but I should be able to release on Monday of everything is ready. One thing that we should fix if it hasn't been addressed is the ppc64 builds. We can skip the tests for that platform if we're happy to release with the bug with wrong dates on that platform. Or we can decide to not support ppc64 and remove the wheela jobs too.

@jbrockmendel could you have a look at the problem?

@jbrockmendel
Copy link
Member

Sure, is there a link?

@datapythonista
Copy link
Member

Sure, is there a link?

I don't think we created an issue, and the log is gone now, but I opened a PR to regenerate it. You should have the failures in like 3 hours: conda-forge/pandas-feedstock#158

@datapythonista
Copy link
Member

Starting the release. Please do not merged things with the 2.0 milestone (I rolled the milestone of outstanding issues/PRs), and do not tag things to that milestone.

You can continue merging to main as usual.

@datapythonista
Copy link
Member

datapythonista commented Apr 3, 2023

I created the 2.0.1 milestone, which is backporting to the 2.0.x branch. Feel free to add any regression issue (or anything worth backporting, but as with the 2.0 milestone, please do not merge issues with that milestone before the release is ready (it's not a big deal, but adds a bit of confusion if things are added to the 2.0.x branch at this point).

https://github.com/pandas-dev/pandas/milestone/104

@datapythonista
Copy link
Member

Tag created and pushed

@MarcoGorelli
Copy link
Member

awesome! thanks a tonne for doing this

OK to merge to 2.0.x (e.g. backporting docs corrections)?

@datapythonista
Copy link
Member

OK to merge to 2.0.x (e.g. backporting docs corrections)?

Yes, I don't think merging to 2.0.x should cause any problem with the release now.

@datapythonista
Copy link
Member

@datapythonista
Copy link
Member

Seems like we added tzdata to the dependencies since the RC, the conda-forge builds have failed. I'll add the dependency there, and rerun.

@datapythonista
Copy link
Member

pip packages are now ready: https://pypi.org/project/pandas/#history

Looks like simply adding the tzdata dependency didn't fix the conda-forge problem, I'm trying to understand what's going on, it's not immediately clear to me.

@datapythonista
Copy link
Member

The problem with tzdata is fixed. We had the same ppc errors we had for the RCs, and also a failure in one of the windows builds that hopefully has been something temporary and is fixed after reattempting.

Besides that, seems like there is an issue with conda-forge: conda-forge/status#143

It starts to be late in my timezone, but if someone wants to have a look at the feedstock PR in couple of hours, and if both the windows failure and the conda-forge problem are fixed, feel free to merge. Otherwise I'll continue with this tomorrow morning UAE time.

@datapythonista
Copy link
Member

All good now with conda-forge, PR merged, packages should be available in one or two hours for most architectures: conda-forge/pandas-feedstock#159

@datapythonista
Copy link
Member

Most packages for conda-forge are already available, and many others should be available soon: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=683592&view=results

I'm going to announce the release now.

@datapythonista
Copy link
Member

Release complete (pending few conda-forge ARM builds) and announced. Closing this, issue for the 2.0.1 release: #52383.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests