Investigate novel storage system test failures #1100

dhermes · 2015-08-27T00:47:42Z

Bad error response (non-JSON)

ERROR: test_copy_existing_file (storage.TestStorageWriteFiles)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/storage.py", line 181, in test_copy_existing_file
    new_blob = self.bucket.copy_blob(blob, self.bucket, 'CloudLogoCopy')
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/storage/bucket.py", line 460, in copy_blob
    method='POST', path=api_path, _target_object=new_blob)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/connection.py", line 337, in api_request
    error_info=method + ' ' + url)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/exceptions.py", line 185, in make_exception
    payload = json.loads(content)
  File "/opt/python/2.7.9/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/opt/python/2.7.9/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/python/2.7.9/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

From https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/84536058
2. Missing files

======================================================================
ERROR: tearDownModule (storage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/storage.py", line 47, in tearDownModule
    bucket.delete(force=True)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/storage/bucket.py", line 358, in delete
    _target_object=None)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/connection.py", line 419, in api_request
    error_info=method + ' ' + url)
gcloud.exceptions.Conflict: 409 The bucket you tried to delete was not empty. (DELETE https://www.googleapis.com/storage/v1/b/new1440625507341)
======================================================================
FAIL: test_second_level (storage.TestStoragePseudoHierarchy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/storage.py", line 290, in test_second_level
    'parent/child/file22.txt'])
AssertionError: Lists differ: [] != ['parent/child/file21.txt', 'parent/child/file22.txt']
Second list contains 2 additional elements.
First extra element 0:
parent/child/file21.txt
- []

From https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/77405118
3. Backend Error (500)

======================================================================
ERROR: test_list_buckets (storage.TestStorageBuckets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/storage.py", line 58, in tearDown
    CLIENT.bucket(bucket_name).delete()
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/storage/batch.py", line 263, in __exit__
    self.finish()
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/storage/batch.py", line 249, in finish
    self._finish_futures(responses)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/storage/batch.py", line 231, in _finish_futures
    raise make_exception(*exception_args)
gcloud.exceptions.InternalServerError: 500 Backend Error
----------------------------------------------------------------------

From:

https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/77600321
https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/82969179
https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/97488225
Delete blob -> 404 (probably eventual consistency error)

======================================================================
ERROR: setUpClass (storage.TestStoragePseudoHierarchy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/regression/storage.py", line 241, in setUpClass
    blob.delete()
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/regression3/lib/python3.4/site-packages/gcloud/storage/blob.py", line 264, in delete
    return self.bucket.delete_blob(self.name)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/regression3/lib/python3.4/site-packages/gcloud/storage/bucket.py", line 361, in delete_blob
    self.connection.api_request(method='DELETE', path=blob_path)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/regression3/lib/python3.4/site-packages/gcloud/connection.py", line 293, in api_request
    raise make_exception(response, content)
gcloud.exceptions.NotFound: 404 Not Found
----------------------------------------------------------------------

From:

https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/58811668
https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/97491709
Eventual consistency error

======================================================================
FAIL: test_paginate_files (storage.TestStorageListFiles)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/storage.py", line 231, in test_paginate_files
    self.assertEqual(len(last_blobs), truncation_size)
AssertionError: 0 != 1

From:

ERROR: test_list_buckets (storage.TestStorageBuckets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/storage.py", line 76, in test_list_buckets
    bucket = CLIENT.create_bucket(bucket_name)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/storage/client.py", line 215, in create_bucket
    bucket.create(client=self)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/storage/bucket.py", line 172, in create
    data={'name': self.name}, _target_object=self)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/connection.py", line 337, in api_request
    error_info=method + ' ' + url)
gcloud.exceptions.ServiceUnavailable: 503 Backend Error (POST https://www.googleapis.com/storage/v1/b?project=precise-truck-742)

From: https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/90171219

tseaver · 2016-08-08T18:04:58Z

Conflict deleting bucket in module teardown:

======================================================================
ERROR: tearDownModule (storage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/system_tests/storage.py", line 55, in tearDownModule
    Config.TEST_BUCKET.delete(force=True)
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/storage/bucket.py", line 358, in delete
    _target_object=None)
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/connection.py", line 347, in api_request
    error_info=method + ' ' + url)
Conflict: 409 The bucket you tried to delete was not empty. (DELETE https://www.googleapis.com/storage/v1/b/new_1470678826358)

----------------------------------------------------------------------

Found during a local run of tox -e system-tests.

dhermes · 2016-08-08T21:31:06Z

That's from the eventual consistency of listing objects. Using force=True

gets a list of all objects / blobs
deletes each object / blob
deletes the bucket

but if the list isn't truly comprehensive, the bucket deletion will fail with the 409 conflict you showed. Luckily just trying bucket.delete(force=True) again will eventually succeed.

tseaver · 2016-08-09T21:35:49Z

and 7) addressed via 00d42d8.

dhermes · 2016-08-09T21:45:47Z

@tseaver Since these reports are stale, let's just close out and then address as new failures occur

dhermes added api: storage Issues related to the Cloud Storage API. testing labels Aug 27, 2015

dhermes mentioned this issue Aug 28, 2015

Intermittent storage regression failure #836

Closed

tseaver mentioned this issue Mar 16, 2016

Adopt common timeout/retry/backoff strategy for system tests. #1619

Closed

dhermes mentioned this issue May 17, 2016

Flaky System Tests #1796

Closed

tseaver closed this as completed Aug 9, 2016

dhermes added the flaky label Aug 11, 2016

JustinBeckwith assigned tseaver Feb 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Investigate novel storage system test failures #1100

Investigate novel storage system test failures #1100

dhermes commented Aug 27, 2015

tseaver commented Aug 8, 2016 •

edited

Loading

dhermes commented Aug 8, 2016

tseaver commented Aug 9, 2016

dhermes commented Aug 9, 2016

Investigate novel storage system test failures #1100

Investigate novel storage system test failures #1100

Comments

dhermes commented Aug 27, 2015

tseaver commented Aug 8, 2016 • edited Loading

dhermes commented Aug 8, 2016

tseaver commented Aug 9, 2016

dhermes commented Aug 9, 2016

tseaver commented Aug 8, 2016 •

edited

Loading