[BEAM-11167] Updates dill package to version 0.3.5.1

[ryanthompson591]

Dill hasn't been upgraded in 3 years. We need to keep the dependency tight. Because dill doesn't have backwards compatibility between minor versions workers and runners MUST be on the same minor version (in this case 0.3.5.1).

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[asf-ci]

Can one of the admins verify this patch?

[asf-ci]

Can one of the admins verify this patch?

[asf-ci]

Can one of the admins verify this patch?

[asf-ci]

Can one of the admins verify this patch?

[asf-ci]

Can one of the admins verify this patch?

[ryanthompson591]

Run Postcommit 3.8

[ryanthompson591]

Run Python 3.8 PostCommit

[ryanthompson591]

R: @tvalentyn @y1chi

[ryanthompson591]

Run Python 3.8 PostCommit

[codecov]

Codecov Report

Merging #17669 (fee88f2) into master (999bcea) will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #17669      +/-   ##
==========================================
- Coverage   74.09%   74.06%   -0.03%     
==========================================
  Files         697      697              
  Lines       91980    91980              
==========================================
- Hits        68148    68129      -19     
- Misses      22583    22602      +19     
  Partials     1249     1249              

Flag	Coverage Δ
python	83.72% <ø> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/utils/interactive_utils.py	95.12% <0.00%> (-2.44%)	⬇️
sdks/python/apache_beam/internal/dill_pickler.py	84.89% <0.00%> (-1.44%)	⬇️
...hon/apache_beam/runners/direct/test_stream_impl.py	93.28% <0.00%> (-0.75%)	⬇️
...python/apache_beam/runners/worker/worker_status.py	78.98% <0.00%> (-0.73%)	⬇️
...eam/runners/portability/fn_api_runner/execution.py	92.44% <0.00%> (-0.65%)	⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py	88.46% <0.00%> (-0.64%)	⬇️
sdks/python/apache_beam/runners/direct/executor.py	96.46% <0.00%> (-0.55%)	⬇️
sdks/python/apache_beam/transforms/core.py	92.14% <0.00%> (-0.16%)	⬇️
sdks/python/apache_beam/coders/coders.py	88.22% <0.00%> (-0.13%)	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	93.54% <0.00%> (-0.13%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 999bcea...fee88f2. Read the comment docs.

[tvalentyn]

Looks like the tool that pulls licences is having trouble to get dill's license now:

RuntimeError: Could not retrieve licences for packages ['dill'] in Python3.8 environment. 
 These licenses were not able to be pulled automatically. Please search code source of the dependencies on the internet and add urls to RAW license file at sdks/python/container/license_scripts/dep_urls_py.yaml for each missing license and rerun the test. If no such urls can be found, you need to manually add LICENSE and NOTICE (if available) files at sdks/python/container/license_scripts/manual_licenses/{dep}/ and add entries to sdks/python/container/license_scripts/dep_urls_py.yaml.
�[0mThe command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then       pip install 'pip-licenses<4.0.0' pyyaml tenacity &&       python /tmp/license_scripts/pull_licenses_py.py ;     fi' returned a non-zero code: 1

> Task :sdks:python:container:py38:docker FAILED

[tvalentyn]

you'd have to fix this in the generator script

[ryanthompson591]

Done.

[ryanthompson591]

@tvalentyn do you think this needs a note in changes.md since this upgrade hasn't happened in 3 years? Or do you think library version upgrades is such a common event it's not worth noting?

[tvalentyn]

@tvalentyn do you think this needs a note in changes.md since this upgrade hasn't happened in 3 years? Or do you think library version upgrades is such a common event it's not worth noting?

It's worth noting the implications. Dill is an implementation detail that users don't care about. But there is a risk that some pipeline would fail to launch because something suddenly fails to pickle after upgrading to a new SDK. I am not very worried about 0.3.4, as it's been around for a while. But 0.3.5 (which may happen sometime soon), might include rough edges due to uqfoundation/dill#443.

[ryanthompson591]

OK, I made a note in change.md. It doesn't go into details or implications deeply. Is this adequate?

[tvalentyn]

Let's come back to this when the 2.38.0 release is finalized.

[ryanthompson591]

I guess I don't understand why we should come back to this waiting on a release finalization. Isn't the release already cut at a give PR, and future PR's (like this one) shouldn't effect the release?

[tvalentyn]

It's just for the benefit of managing the release activities we do internally - it's going to be a little easier if we wrap up the release before making this change, so if this is not too urgent, I'd wait ~1 week or so.

[tvalentyn]

Run PythonDocker PreCommit

[tvalentyn]

Run Portable_Python PreCommit

[tvalentyn]

We can proceed with merging this now. Can you address conflicts & test failures?

[ryanthompson591]

Run Portable_Python PreCommit

[tvalentyn]

Run Python 3.8 PostCommit