Remove requirements files

[eapolinario]

TL;DR

Remove mentions to requirements.txt and dev-requirements.txt

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

Up to this point we operated under the assumption that maintaining a single set of requirements files would be enough. This would guarantee a semblance of reproducibility (since we would be able to reproduce the environment used for that change). That assumption broke down as we started seeing libraries not being available in all OS+architecture we support, for example, tensorflow does not install cleanly on Mac M1's.

In this change we trade reproducibility for the maintenance burden of having exhaustive requirements files by environment. jazzband/pip-tools#826 (comment) is the current advice coming from pip-tools, basically if we are to do this we need to run pip-tools once per-environment.

Instead, we're taking one step back in order to figure out what's the right next steps. The idea is that we're going to invest in a real multi-environment solution in the near future, but in the meanwhile, we're going to lose reproducibility in order to decrease the maintenance burden. Note that this change is going to subsume the need for #1424.

As a side-note, we removed the legacy all-spark extras in #807, but never removed the CI bits related to the spark-2.4 extra. Those are being removed in this PR.

Tracking Issue

flyteorg/flyte#3284

Follow-up issue

NA

[codecov]

Codecov Report

Merging #1511 (b2db3eb) into master (74cbe58) will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1511      +/-   ##
==========================================
+ Coverage   69.31%   69.34%   +0.02%     
==========================================
  Files         305      305              
  Lines       28657    28642      -15     
  Branches     2714     2310     -404     
==========================================
- Hits        19864    19861       -3     
+ Misses       8276     8266      -10     
+ Partials      517      515       -2     

Impacted Files	Coverage Δ
flytekit/core/promise.py	51.96% <0.00%> (-0.57%)	⬇️
flytekit/clis/flyte_cli/main.py	44.34% <0.00%> (ø)
flytekit/extras/persistence/http.py	24.48% <0.00%> (ø)
flytekit/types/directory/__init__.py	0.00% <0.00%> (ø)
flytekit/types/schema/types.py	38.30% <0.00%> (+0.15%)	⬆️
flytekit/models/security.py	13.15% <0.00%> (+0.33%)	⬆️
flytekit/types/file/file.py	61.11% <0.00%> (+0.42%)	⬆️
flytekit/types/directory/types.py	56.19% <0.00%> (+0.46%)	⬆️
flytekit/configuration/internal.py	18.46% <0.00%> (+2.02%)	⬆️
flytekit/interfaces/random.py	25.00% <0.00%> (+5.00%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.