Attempt to fix transient nightly build errors: Remove poetry cache

[maxi297]

What

We have seen transient errors on the build phase of the nighty build. The errors are somewhat all over the places but always link to Poetry. The goal would be to scope the issue.

Here are a couple of examples (not sharing the links because they are not sharable as they expire):

source-s3 2024-02-29 nightly build

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/usr/local/bin/python', '-m', 'pip', 'uninstall', 'charset-normalizer', '-y'] errored with the following return code 1
  
  Output:
  Traceback (most recent call last):
    File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/usr/local/lib/python3.9/site-packages/pip/__main__.py", line 22, in <module>
      from pip._internal.cli.main import main as _main
    File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/main.py", line 10, in <module>
      from pip._internal.cli.autocompletion import autocomplete
    File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
      from pip._internal.cli.main_parser import create_main_parser
    File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/main_parser.py", line 9, in <module>
      from pip._internal.build_env import get_runnable_pip
    File "/usr/local/lib/python3.9/site-packages/pip/_internal/build_env.py", line 19, in <module>
      from pip._internal.cli.spinners import open_spinner
    File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/spinners.py", line 9, in <module>
      from pip._internal.utils.logging import get_indentation
    File "/usr/local/lib/python3.9/site-packages/pip/_internal/utils/logging.py", line 13, in <module>
      from pip._vendor.rich.console import (
    File "/usr/local/lib/python3.9/site-packages/pip/_vendor/rich/console.py", line 60, in <module>
      from .pretty import Pretty, is_expandable
    File "/usr/local/lib/python3.9/site-packages/pip/_vendor/rich/pretty.py", line 31, in <module>
      import attr as _attr_module
    File "/usr/local/lib/python3.9/site-packages/attr/__init__.py", line 10, in <module>
      from . import converters, exceptions, filters, setters, validators
    File "/usr/local/lib/python3.9/site-packages/attr/converters.py", line 11, in <module>
      from ._make import NOTHING, Factory, pipe
    File "/usr/local/lib/python3.9/site-packages/attr/_make.py", line 53, in <module>
      _ng_default_on_setattr = setters.pipe(setters.convert, setters.validate)
  AttributeError: module 'attr.setters' has no attribute 'pipe'

source-sentry 2024-02-28 nightly build

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/usr/local/bin/python', '-m', 'pip', 'uninstall', 'attrs', '-y'] errored with the following return code 2
  
  Output:
  Found existing installation: attrs 23.1.0
  Uninstalling attrs-23.1.0:
  ERROR: Exception:
  Traceback (most recent call last):
    File "/usr/local/lib/python3.9/shutil.py", line 825, in move
      os.rename(src, real_dst)
  OSError: [Errno 18] Invalid cross-device link: '/usr/local/lib/python3.9/site-packages/attr/' -> '/usr/local/lib/python3.9/site-packages/~ttr'
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
      status = run_func(*args)
    File "/usr/local/lib/python3.9/site-packages/pip/_internal/commands/uninstall.py", line 105, in run
      uninstall_pathset = req.uninstall(
    File "/usr/local/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 687, in uninstall
      uninstalled_pathset.remove(auto_confirm, verbose)
    File "/usr/local/lib/python3.9/site-packages/pip/_internal/req/req_uninstall.py", line 381, in remove
      moved.stash(path)
    File "/usr/local/lib/python3.9/site-packages/pip/_internal/req/req_uninstall.py", line 272, in stash
      renames(path, new_path)
    File "/usr/local/lib/python3.9/site-packages/pip/_internal/utils/misc.py", line 315, in renames
      shutil.move(old, new)
    File "/usr/local/lib/python3.9/shutil.py", line 843, in move
      rmtree(src)
    File "/usr/local/lib/python3.9/shutil.py", line 740, in rmtree
      onerror(os.rmdir, path, sys.exc_info())
    File "/usr/local/lib/python3.9/shutil.py", line 738, in rmtree
      os.rmdir(path)
  OSError: [Errno 39] Directory not empty: '/usr/local/lib/python3.9/site-packages/attr/'

How

As we don't seem to see this in CI checks, we assume this is something specific to nightly builds (even though, it could be because there is more volume on nightly build and statistically this is where we see the issue). Poetry cache is not used on CI checks but it is on nightly builds. Remove the poetry cache will help us scope the issue.

🚨 User Impact 🚨

We expect more network traffic and longer nightly builds but less flaky ones.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Mar 7, 2024 6:07pm

[maxi297]

@alafanechere Should there be tests to update here?

[alafanechere]

@alafanechere Should there be tests to update here?

I don't think so, but let see if the unit test pass. Feel free to add a test if you like.

[alafanechere]

Can we comment this out instead of deleting?

[maxi297]

Done and I added a comment to give more context

[ChristoGrab]

LGTM and pipeline is green, thanks for looking into this 👍 Non-blocking junior question: could there be any other potential downsides to the heavier traffic (ie connectivity issues), or are we confident the infrastructure can handle the increased load?

[maxi297]

could there be any other potential downsides to the heavier traffic (ie connectivity issues), or are we confident the infrastructure can handle the increased load?

@alafanechere would know more about this but I don't think it matters that much to our infra. Were there any point where we had no poetry cache? Else, we will learn about another issue and I'm fine with that.

[alafanechere]

LGTM and pipeline is green, thanks for looking into this 👍 Non-blocking junior question: could there be any other potential downsides to the heavier traffic (ie connectivity issues), or are we confident the infrastructure can handle the increased load?

@ChristoGrab the main downside is a loss of performance: we'll always redownload the same stuff from Pip which has a significant networking overhead. In other words: slower tests and publish pipelines.