Sporadic "resource deadlock avoided" errors for GCE credentials

[ihaque-freenome]

In the last few days I've been getting this error sporadically when trying to copy files from GCS onto local storage on GCE nodes:

Your "GCE" credentials are invalid. Please run
    $ gcloud auth login
  [Errno 35] Resource deadlock avoided
  CommandException: Some components of [redacted] were not downloaded successfully. Please retry this download.
  CommandException: 1 file/object could not be transferred.

It is unclear to me whether this is related to the metadata connection mentioned above, the locking bug fixed in oauth2client a couple years ago, or something else entirely.

[houglum]

What version of gsutil are you running? And are you able to get a stack trace for the related error if you run gsutil with the top-level -D option?

[ihaque-freenome]

This is gsutil 4.25. Stack trace looks like the following:

Encountered exception while copying:
Traceback (most recent call last):
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command.py", line 2096, in PerformTask
    results = task.func(cls, task.args, thread_state=self.thread_gsutil_api)
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 745, in _CopyFuncWrapper
    preserve_posix=cls.preserve_posix_attrs)
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 937, in CopyFunc
    preserve_posix=preserve_posix))
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/copy_helper.py", line 3265, in PerformCopy
    preserve_posix=preserve_posix)
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/copy_helper.py", line 2567, in _DownloadObjectToFile
    status_queue=gsutil_api.status_queue))
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/copy_helper.py", line 2266, in _DoSlicedDownload
    'Please retry this download.' % dst_url.object_name)
CommandException: CommandException: Some components of read2.fastq.gz were not downloaded successfully. Please retry this download.


total_bytes_transferred: 0
DEBUG: Exception stack trace:
    Traceback (most recent call last):
      File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 577, in _RunNamedCommandAndHandleExceptions
        collect_analytics=True)
      File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 299, in RunNamedCommand
        return_code = command_inst.RunCommand()
      File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1104, in RunCommand
        self.op_failure_count, plural_str, plural_str))
    CommandException: CommandException: 1 file/object could not be transferred.

[ihaque-freenome]

Some additional background: there are typically multiple copies of gsutil running simultaneously when this error occurs.

[houglum]

Ah, I see. That can lead to all sorts of issues. They're all trying to use the same state directory, and are probably stepping all over each other's files used to track state. If you're going to run multiple invocations of gsutil at the same time, set the state_dir option for overlapping invocations to different directories (by default, every invocation will use ~/.gsutil). E.g. if you have one scheduled to run periodically via a cron job, you might want the command to look something like:

gsutil -o "GSUtil:state_dir=/some/path/to/gsutil_cron_state/" <rest of command>