RLS: 1.5

[simonjayhawkins]

Tracking issue for the 1.5 release.

https://github.com/pandas-dev/pandas/milestone/92

Currently scheduled for June 30, 2022 (with a release candidate around 2 weeks before), although I believe that we decided to do a 1.5 since we still had some deprecations to do and did not want to delay the 1.4 release (xref #41957 (comment) and subsequent comments). If that being the case, I see no reason why 1.5 should not be released earlier, once the necessary deprecations are in place.

List of open regressions: https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3ARegression

List of proposed deprecations #41957 (comment)

---

Also it occurred to me that maybe we can branch earlier for the next release, i.e. once we feel we are close, and then tag the rc from 1.5.x. While preparing for the release, master is a moving target.

[YarShev]

HI, I am wondering when this release is going to happen. Is there an estimate date?

[jbrockmendel]

#47485 says july 31 is the expected date for 1.4.4. Should we update the estimate for 1.5?

[simonjayhawkins]

Should we update the estimate for 1.5?

v1.4.0rc0 was on Jan 5 and the release on Jan 22, so ideally add 6 months to those dates.

1.4.4 will probably be the last in the 1.4.x series (we had bigger gaps than the recent releases between 1.4.1 and 1.4.2 and between 1.4.2 and 1.4.3). So can expect 1.4.4 to be released somewhere around the 1.5.0rc0

given approx 2 weeks for the rc period. How does mid July for the rc and end of July for 1.5 sound?

[jbrockmendel]

Seems reasonable.

[kushalkolar]

Is the RC still happening in mid July?

[kushalkolar]

any update on when we can expect the RC?

[vnlitvinov]

Any estimates on release date? Interesting per the Modin side...

[mroeschke]

@pandas-dev/pandas-core I propose we cut 1.5rc next week, let's say Monday, August 22nd (give or take people's availability).

Remaining items for the 1.5rc discussed during this month's dev call: (I can help push items over the finish line)

1. Arrow PRs

• ENH: Add ArrowDype and .array.ArrowExtensionArray to top level #47818
• DOC/ENH: Add documentation and whatsnew entry for ArrowDtype and ArrowExtensionArray #47854

2. Copy on write flag

• API: New copy / view semantics using Copy-on-Write #46958

3. Groupby with nan fix:

• BUG: algorithms.factorize moves null values when sort=False #46601

4. Reverting Interval.closed -> Interval.inclusive deprecation (BUG: default for pd.Interval of closed/inclusive changed on main from right to both #47365).

• Revert Interval/IntervalIndex/interval_range.inclusive deprecation #48116

5. Setting numeric value in Categorical Series with enlargement

• REGR: setting numeric value in Categorical Series with enlargement raise internal error #48106

6. Restore old NaT/Timestamp type annotation (TYP: remove NaTType as possible result of Timestamp and Timedelta constructor #46171 (comment))

• revert Timestamp and Timedelta constructors typing allowing NaTType return #48112

[phofl]

#47751 should get merged before the rc too

[simonjayhawkins]

@mroeschke #47933 has been re-milestoned 1.4.4 but is equally a blocker for 1.5. This should be fixed and (imo) release 1.5 not released for at least a few days later to ensure any issues with downstream packages are identified with the nightly builds.

( a couple of caveats, I think Geopandas is currently testing against main not nightlies, @jorisvandenbossche? and maybe could be argued that it is not so important for the rc, so just needs to be fixed to actually build the wheels or skip the failing tests so that the ci progresses to the wheel upload step)

[mroeschke]

@simonjayhawkins does the hypothesis pinning "unblock" #47933 enough to potentially move forward with 1.5?

It's not entirely clear from that issue what test is failing & need addressing. From a recent build (https://dev.azure.com/pandas-dev/pandas-wheels/_build/results?buildId=77419&view=logs&j=9d2a1335-31ce-57a0-ed89-d15824b44ca5&t=1c1db528-835d-5133-121a-4315ad351113), it appears some pytz version parsing is failing in pandas/tests/tseries/offsets/test_dst.py which was fixed in #48065

[simonjayhawkins]

we'll know if nightlies are green overnight than the issue can be effectively closed wrt 1.5.0rc0. (may need to open another to unpin hypothesis, #47952 is investigating that)

so within a few days should know if any changes in the last few weeks are causing any issues downstream.

(the pytz issue will only affect the 1.4.x wheel builds and will probably be fixed by backporting #48065)

[jorisvandenbossche]

I think we discussed this at the meeting last week (but there wasn't a clear decision noted down, and I don't exactly remember), but related to the 1.4.x and 1.5 releases: what do you think about moving the 1.5rc some days ahead, and first concentrate some effort next week on getting 1.4.4 out?

The 1.4.x branch already has some regressions merged, and there are a bunch of open PRs fixing more of them.

Similarly as @lithomas1 mentioned at #47485 (comment), I think it would be helpful that we can first do 1.4.4, before cutting 1.5.0rc (i.e. branching 1.5.x). That would also avoid that we have commits that would have to be backported to two other branches for some time.
(if there come up other major issues with 1.4, we can still decide to do a 1.4.5 after 1.5.x has branched if needed; but until that arises, we could avoid two active maintenance branches for now)

[jorisvandenbossche]

(and also if we don't do that, it would still be nice to get as much as possible merged for 1.4.x, before doing 1.5.0rc, to avoid double backporting)

[mroeschke]

I guess my original impression from the dev meeting last week was that since 1.5 could be released independently from 1.4.4, albeit with a little more maintenance overhead, it was worth moving forward with either despite the maintenance overhead since both releases are fairly delayed already. I decided to focus more effort of 1.5 tasks this week since it appears there more user interest in 1.5 from this issue.

On the mailing list, I already sent out a Doodle poll to inquire when others would be interested in shadowing the 1.5 release (I will be shadowing as well) hoping to make a decision by Monday, August 22nd.

But if double back porting is a significant concern to others (personally I think it's manageable given only related branch bug fixes should be merged in the meantime), we could ensure 1.4.4 is released before 1.5.0rc cc @pandas-dev/pandas-core

[simonjayhawkins]

on the 1.4.x front, we can release 1.4.4 anytime and then could start 1.4.5.

that does not commit us to a 1.4.5. if not doing a 1.4.5, we would move anything in the 1.4.5 release notes into 1.5

I'm monitoring this discussion here and we are ready to release 1.4.4 anytime (MacPython/pandas-wheels#173)

lets keep the discussion on what's included in 1.4.x and release timing that's not influenced by 1.5 in #47485

[simonjayhawkins]

(and also if we don't do that, it would still be nice to get as much as possible merged for 1.4.x, before doing 1.5.0rc, to avoid double backporting)

backporting to 1.5.x is expected to be trivial as those branches won't diverge too much to start (the caveat here is that any global style changes should be merged before branching or deferred till maybe after 1.5 or 1.5.1)

[jorisvandenbossche]

I decided to focus more effort of 1.5 tasks this week since it appears there more user interest in 1.5 from this issue.

And to be clear, that was super useful! Also if we would release 1.5rc a few days later, we would still have needed that effort.

On the mailing list, I already sent out a Doodle poll to inquire when others would be interested in shadowing the 1.5 release

As far as I understand, it's mostly for seeing how the release process works? That could also be done with a 1.4.4 release? (the mechanical parts shouldn't differ much between both?)

I don't think that the double backporting will be a big trouble, but it is creating more work, that could be avoided by ensuring we merge as many PRs as possible that are targetted for 1.4.4 before cutting 1.5.0rc

(and to be clear: I am fine with doing a 1.5rc early next week if all blockers are merged, I just wanted to raise the possibility of doing 1.4.4 first)

[jorisvandenbossche]

BTW, does the bot handle backporting to two branches, or will that be manual work? I assume it's a matter of adding an additional explicit comment to ask to backport on the PR?

[simonjayhawkins]

once we fork, we will add a on-merge: backport to 1.5.x to the 1.5 milestone. and the bot should succeed in most cases to start with (the bot can also fail when batch merging PR to main, and the backports have not been merged to the backport branch (1.5.x) causing merge conflicts, normally in the release notes, which can be fixed easily by anyone with permissions (i.e. the maintainers) in the github web interface - the only issue here is that sometimes the release notes get out of sync on main and backport, but again not difficult to fix.

for the 1.4.x PRs, the bot will backport to 1.4.x and we will need to ensure we manually trigger the backport to 1.5.x. Again comparing releases notes on both branches should catch most things.

[simonjayhawkins]

the main branch looks moreless ready https://github.com/simonjayhawkins/pandas-release/actions/runs/2882295370, the failures there have been seen before, but not seeing on the nightly build so maybe not an issue (nighlties succeeded for azure https://github.com/MacPython/pandas-wheels/runs/7911638968, fix for Travis MacPython/pandas-wheels#191 currently being tested)

(1.4.x is also ready - https://github.com/simonjayhawkins/pandas-release/actions/runs/2880754690)

[simonjayhawkins]

(and to be clear: I am fine with doing a 1.5rc early next week if all blockers are merged, I just wanted to raise the possibility of doing 1.4.4 first)

and to be clear, some of those issues on the 1.4 milestone have been open for months. So there has been plenty of opportunity for anyone to fix. So I see no reason not to push forward with the 1.5 release.

[jorisvandenbossche]

some of those issues on the 1.4 milestone have been open for months. So there has been plenty of opportunity for anyone to fix. So I see no reason not to push forward with the 1.5 release.

This becomes more a discussion for #47485 (so will move my comment also there), but there are already 8 fixes in and 10 open PRs for 1.4.4. So I also don't see a reason not to push forward with a 1.4.4 release ;)
(and of course there are reasons: someone needs to take the time to review and merge those PRs, and actually do the release. And I know we will do that anyway at some point)

Since I am not the one actually going to do the release, I will stop arguing about what might be easier ;)

[simonjayhawkins]

some of those issues on the 1.4 milestone have been open for months. So there has been plenty of opportunity for anyone to fix. So I see no reason not to push forward with the 1.5 release.

This becomes more a discussion for #47485 (so will move my comment also there), but there are already 8 fixes in and 10 open PRs for 1.4.4. So I also don't see a reason not to push forward with a 1.4.4 release ;) (and of course there are reasons: someone needs to take the time to review and merge those PRs, and actually do the release. And I know we will do that anyway at some point)

sure. In that issue, I said that the release timing of 1.4.x would be reviewed once the 1.5.0rc0 date is firmed up #47485 (comment)

so getting those open PRs merged is now the priority for 1.4.x and push forward with the 1.4.4 release also.

Since I am not the one actually going to do the release, I will stop arguing about what might be easier ;)

yep, we decided to manage the releases independently so that more progress could be made on getting the 1.5 release cycle started. So I am reluctant to block that pending 1.4.x issue resolution.

[zaneselvans]

Maybe it's too late for 1.5 but I just opened #48574 which seems like it might be a regression related to #47844 / #48555 ?

[datapythonista]

Opened #48627 with the pending changes to the release notes before the release. I'll start working on the release of pandas 1.5.0 tomorrow morning (Europe timezone). Once the wheels are ready, I'll need someone with access to our pypi to upload them, I don't have permissions. Let me know if anything.

[lithomas1]

@datapythonista Can you put your email in the permissions thread on the core mailing list? I think Jeff can add us right now.

[datapythonista]

Everything seems ready and backported, starting the release of 1.5.0 now.

[datapythonista]

I've got one failure during final pip package validation. I guess the crash is caused by a memory error (even I think I should have plenty of memory for it in my system, but I was running the conda validation in parallel, maybe it consumes a lot). I'll rerun tests again and see if the problem persists, before continuing with the release.

____________________________________________________________________________ tests/io/parser/test_c_parser_only.py ____________________________________________________________________________
[gw0] linux -- Python 3.8.13 /opt/conda/envs/pip-test/bin/python3
worker 'gw0' crashed while running 'tests/io/parser/test_c_parser_only.py::test_bytes_exceed_2gb[c_high]'

[datapythonista]

Looks like the test has a peak heap allocation of 12.2Gb, and I'm testing on my laptop with 16Gb RAM. I managed to get the test passing when nothing else runs in parallel. Assuming the failures are caused by not having enough memory available in the system when running the tests in parallel, and moving forward with the release.

Pushing the tag now, no going back. ;)

[datapythonista]

I opened the PR for the wheels a while ago, but seems like the arm wheels are built sequentially, and the CI for the PR will probably take couple of hours more: MacPython/pandas-wheels#196

I'd probably move forward and merge, before the last arm builds finish, I don't think it makes a difference. I don't have write permissions on the MacPython repo. @jreback, @lithomas1 (or whoever else have permissions there) if you want to take care of merging it, whenever you consider (now, or when the CI finishes), that would be great. And if you have rights to give me commit access to the repo that would also be great. Thanks!

[lithomas1]

Yeah, I added you. I think its probably fine to skip the arm wheels if they are taking too long and upload them later.

[datapythonista]

Thanks @lithomas1. I meant not waiting for the PR builds, that are only to test that everything is fine. But I think you're right, all the other builds are now ready, and the arm ones will still take 3 more hours. I think I'll continue the release without them for now, so we can get the conda packages building, and will add the arm wheels later to the pypi.

[datapythonista]

An update:

• Docs for 1.5.0 are available: https://pandas.pydata.org/pandas-docs/version/1.5.0/
• Link for stable docs created: https://pandas.pydata.org/docs/
• Updates to the version switcher in Sphinx and the website will happen when merging DOC: Fix version switcher in old docs #48600 (the website will be updated with any PR merged really)
• PyPI packages already available for 1.5.0 (except for arm): Checked that pip install pandas installs 1.5.0 and seems to work with the last features
• Something seems to be wrong with TravisCI, one arm build finished, but the others have been in the queue for a long time and don't seem to start. Not sure if there may be a quote issue, I'll give it some more time and rerun the whole job to see if that helps: https://app.travis-ci.com/github/MacPython/pandas-wheels/builds/255721398
• Waiting for conda-forge to automatically detect the new release, and automatically create the PR to build the conda-forge packages. I think it can take few hours

[datapythonista]

Conda PR now created: conda-forge/pandas-feedstock#142

[datapythonista]

Seems like there are some problems with arm in conda-forge, this time a failure:

pandas/_libs/tslibs/np_datetime.pypy38-pp73-aarch64-linux-gnu.so: undefined symbol: cmp_npy_datetimestruct

Not quite sure what's the problem, having a look.

[mroeschke]

If pandas is being built with -j> 1, it may the parallel build race bug in #47305. All of our other CI builds specify -j 1

[datapythonista]

I don't fully understand how conda-forge builds the packages. Seems like it's using mambabuild (replacement for conda build), which I guess will end up using this to know what to build: https://github.com/conda-forge/pandas-feedstock/blob/main/recipe/meta.yaml#L16

I'll update -j to be 1 and see if that fixes the problem.

[TheNeuralBit]

1,5 release planned for Monday, September 19. Main issues to discuss in the dev meeting on the 14th for the release are (please let me know if anything else):

• ENH/PERF: copy=True keyword in methods where copy=False can improve perf #48141
• ENH: Warn when dtype is not passed to get_dummies #48022
• BUG: RecursionError when attempting to replace np.nan values under main branch #45725
• DOCS: series.astype return error when a datetime64 (unitless) series astype numpy.datetime64 #47844

Are there notes from this meeting? I'm curious what was decided regarding #45725

[phofl]

We will fix this but it was not ruled a blocker for the release

[datapythonista]

Conda-forge CI finally finished. It takes more than 3 hours to complete some of the jobs. PR is now merged, I guess we should have conda packages in something more than 3 hours from now.

I also emailed TravisCI support to see if they can provide more information about the arm build problems. I'll post here if I hear back from them.

[astrojuanlu]

Comment from the peanut gallery: the SciPy folks are experimenting with Cirrus CI for ARM builds and it's quite fast, see https://github.com/FFY00/meson-python/issues/131 for discussion and pointers.

[datapythonista]

All conda-forge packages except PowerPC should be now published. I manually tested the one for my architecture and python version (py39/linux64) and seems to be working fine.

No answer from TravisCI yet, but looks like some other users are experiencing the same problem, and TravisCI is aware: https://travis-ci.community/t/arm64-graviton2-not-getting-queued/13299

I'll go ahead and make the announcements to the different channels.

[datapythonista]

Packages for all supported architectures now published, release complete.

[pkanavos]

We will fix this but it was not ruled a blocker for the release

I just found out about #45725 because upgrading broke a job I was about to deploy. I have feelings about this

[phofl]

Prs are welcome