[KED-2897] kedro pipeline pull doesn't work if requirements.txt contains -r or -c entries

[deepyaman]

Description

kedro pipeline pull doesn't work if requirements.txt contains -r or -c entries. requirements.txt gets copied without modification to requirements.in if it doesn't already exist, including the -r/-c flags.

Context

I have a constraints file, and I have to work around it if I want to be able to pull in packages.

Steps to Reproduce

1. Create a Kedro project
2. Add a line like -c constraints.txt or -r more-requirements.txt to src/requirements.txt
3. kedro pipeline pull something

Expected Result

It should process the linked constraints/requirements.

Actual Result

https://github.com/quantumblacklabs/kedro/blob/0.17.5/kedro/framework/cli/pipeline.py#L1107 results in an error:

Traceback (most recent call last):
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/bin/kedro", line 8, in <module>
    sys.exit(main())
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/kedro/framework/cli/cli.py", line 266, in main
    cli_collection()
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/kedro/framework/cli/cli.py", line 216, in main
    **extra,
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/kedro/framework/cli/pipeline.py", line 311, in pull_package
    _pull_package(package_path, metadata, env=env, alias=alias, fs_args=fs_args)
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/kedro/framework/cli/pipeline.py", line 349, in _pull_package
    _append_package_reqs(requirements_in, package_reqs, package_name)
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/kedro/framework/cli/pipeline.py", line 1107, in _append_package_reqs
    reqs_to_add = set(new_reqs) - set(existing_reqs)
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3084, in parse_requirements
    yield Requirement(line)
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3094, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/Users/deepyaman_datta/miniconda3/envs/customerone-retail/lib/python3.7/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 102, in __init__
    requirement_string[e.loc : e.loc + 8], e.msg
pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at "'-c const'": Expected W:(abcd...)

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro version used (pip show kedro or kedro -V): kedro, version 0.17.5
• Python version used (python -V): Python 3.7.11
• Operating system and version:

  ProductName:	macOS
  ProductVersion:	11.5.1
  BuildVersion:	20G80
  

[deepyaman]

It also doesn't clean up (i.e. you're left with the exploded pipeline as a side effect), even though the command failed.

[antonymilne]

Hi @deepyaman, I am confused by how you managed to get this error through the steps you describe and unable to reproduce it myself. It sounds to me like you are dealing with a src/project_name/pipelines/pipeline_name/requirements.txt file rather than src/requirements.txt file - is that right?

I would indeed expect there to be a problem with -r or -c, since it falls under 1b of my concerned comment 😅 However, I'd expect any error to be raised already at kedro pipeline package time so that it's not even possible to get to the kedro pipeline pull stage of the process for it. So I'm confused by how you would have generated a package that would have -r or -c in its requirements in the first place. Please can you explain exactly the steps you took here to get the error?

[deepyaman]

Hi @deepyaman, I am confused by how you managed to get this error through the steps you describe and unable to reproduce it myself. It sounds to me like you are dealing with a src/project_name/pipelines/pipeline_name/requirements.txt file rather than src/requirements.txt file - is that right?

I would indeed expect there to be a problem with -r or -c, since it falls under 1b of my concerned comment 😅 However, I'd expect any error to be raised already at kedro pipeline package time so that it's not even possible to get to the kedro pipeline pull stage of the process for it. So I'm confused by how you would have generated a package that would have -r or -c in its requirements in the first place. Please can you explain exactly the steps you took here to get the error?

Hi @AntonyMilneQB! This issue occurs with -r or -c in src/requirements.txt, and not a pipeline requirements.txt, which is why there's no issue at kedro pipeline package time. The issue occurs when trying to install and update the central requirements file (requirements.in file, IIRC? I need to double check).

[antonymilne]

Ahhhh yes I see. You are completely right, thank you for the bug report. The issue only occurs when you're pulling a pipeline that was packaged with requirements, which is why I didn't see it before. Here's full steps to reproduce:

yes spaceflights | kedro new -s spaceflights
cd spaceflights
echo -r blah.txt >> src/requirements.txt

# This will be fine
kedro pipeline package data_science
kedro pipeline pull src/dist/data_science-0.1-py3-none-any.whl

# This will not be fine
echo pandas > src/spaceflights/pipelines/data_science/requirements.txt
kedro pipeline package data_science
kedro pipeline pull src/dist/data_science-0.1-py3-none-any.whl

[antonymilne]

This will be fixed in 0.17.6. Thank you @deepyaman!