Add cleanup implementation to filesystem backend

[eprigorodov]

Description

Filesystem back-end stores task results in files and currently has no cleanup() method. Without other ways of maintenance (e.g. logrotate or systemd-tmpfiles) that will eventually overflow file system. This PR adds cleanup() implementation, which we use in our production.

[codecov]

Codecov Report

Merging #6919 (ab9646b) into master (16959cd) will decrease coverage by 0.00%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master    #6919      +/-   ##
==========================================
- Coverage   89.21%   89.21%   -0.01%     
==========================================
  Files         138      138              
  Lines       16678    16692      +14     
  Branches     2107     2112       +5     
==========================================
+ Hits        14880    14892      +12     
  Misses       1577     1577              
- Partials      221      223       +2     

Flag	Coverage Δ
unittests	89.21% <85.71%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
celery/backends/filesystem.py	94.28% <85.71%> (-2.15%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 16959cd...ab9646b. Read the comment docs.

[lgtm-com]

This pull request introduces 2 alerts when merging 4660141 into 16959cd - view on LGTM.com

new alerts:

• 2 for Wrong name for an argument in a call

[auvipy]

if you can try to improve the code coverage that would be really great

[lgtm-com]

This pull request introduces 2 alerts when merging 9bdf8ae into 16959cd - view on LGTM.com

new alerts:

• 2 for Wrong name for an argument in a call

[eprigorodov]

Hi @auvipy, thank you for fast response and explanation. The pipelines in Celery project are great, but I need some advise on how to fix the coverage.

1. The test_cleanup() is marked to be skipped on Windows. That is done because test makes a small sleep to divide expired and non-expired sets of task result flies. To make that work reliably on Windows + FAT the sleep time (according to the docs) should be longer than two seconds, which is not a good thing for testing, I believe.

If no unit testing is run on Windows + FAT then the test can be enabled on Windows as is. Otherwise the only options are either add longer delay or ignore decreasing coverage. Please advise.

2. To fix new coverage finding I would need to add a third commit. Is that fine or should all commits better be squashed into one?

Thank you

[auvipy]

we can sqash commits while merge. i am ok with the current coverage. can you just check on pypy failures?

[eprigorodov]

From what I can understand:

  collecting ... ERROR: InvocationError for command 'D:\a\celery\celery\.tox\pypy3-unit\Scripts\pytest.EXE' --maxfail=10 -v --cov=celery --cov-report=xml --cov-report term (exited with code 3221225477)

exit code 3221225477 means segfault. And here is an issue that looks quite relevant: pytest-dev/pytest#6419.

I'll try to rewrite with os.listdir() and os.stat().

[eprigorodov]

hi @auvipy, thank you for reviewing and approval.

One last question: we have to maintain a number of boxes with Debian 9 -> Python 3.5 -> Celery 4.4. Can I somehow make a PR for backporting this feature to branch "4.x"?

[auvipy]

4.x is EOL. you should use pyenv to upgrade your python and should not rely on systems python.