Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storage - SocketException Socket closed (gcloud java 0.28) #1227

Closed
fondberg opened this issue Sep 7, 2016 · 3 comments · Fixed by #1233
Closed

Storage - SocketException Socket closed (gcloud java 0.28) #1227

fondberg opened this issue Sep 7, 2016 · 3 comments · Fixed by #1233
Assignees
Labels
api: storage Issues related to the Cloud Storage API.

Comments

@fondberg
Copy link

fondberg commented Sep 7, 2016

We still see StorageException caused by SocketExceptions even if they should have been retried.

java.net.SocketException: Socket is closed
    at sun.security.ssl.SSLSocketImpl.getInputStream
    at sun.net.www.http.HttpClient.parseHTTP
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream
    at java.net.HttpURLConnection.getResponseCode
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode
    at com.google.api.client.http.javanet.NetHttpResponse.<init>
    at com.google.api.client.http.javanet.NetHttpRequest.execute
    at com.google.api.client.http.HttpRequest.execute
    at com.google.cloud.storage.spi.DefaultStorageRpc.open
    at com.google.cloud.storage.BlobWriteChannel.<init>
    at com.google.cloud.storage.StorageImpl.writer
    at com.google.cloud.storage.StorageImpl.writer
    at com.google.cloud.storage.StorageImpl.writer
    at com.spotify.buildartifactarchiver.storage.GcsUploaderClient.storeInputStream
    at com.spotify.buildartifactarchiver.storage.ArtifactUploader.uploadZipArchive
    at com.spotify.buildartifactarchiver.storage.ArtifactUploader.upload
    at com.spotify.buildartifactarchiver.ArtifactArchiver.archiveArtifactsForBuild
    at com.spotify.buildartifactarchiver.ArtifactArchiver.archiveBuild
    at com.spotify.buildartifactarchiver.web.ArtifactArchiverPubSubResource.messageHandler
    at com.spotify.google.cloud.pubsub.client.Puller.lambda$pullBatch$11
    at java.util.concurrent.CompletableFuture.uniWhenComplete
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire
    at java.util.concurrent.CompletableFuture.postComplete
    at java.util.concurrent.CompletableFuture.complete
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.succeed
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.lambda$wrap$9
    at java.util.concurrent.CompletableFuture.uniWhenComplete
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire
    at java.util.concurrent.CompletableFuture.postComplete
    at java.util.concurrent.CompletableFuture.complete
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.succeed
    at com.spotify.google.cloud.pubsub.client.Pubsub.lambda$requestJavaNet$8
    at java.util.concurrent.ThreadPoolExecutor.runWorker
    at java.util.concurrent.ThreadPoolExecutor$Worker.run
    at java.lang.Thread.run
com.google.cloud.storage.StorageException: Socket is closed
    at com.google.cloud.storage.spi.DefaultStorageRpc.translate
    at com.google.cloud.storage.spi.DefaultStorageRpc.open
    at com.google.cloud.storage.BlobWriteChannel.<init>
    at com.google.cloud.storage.StorageImpl.writer
    at com.google.cloud.storage.StorageImpl.writer
    at com.google.cloud.storage.StorageImpl.writer
    at com.spotify.buildartifactarchiver.storage.GcsUploaderClient.storeInputStream
    at com.spotify.buildartifactarchiver.storage.ArtifactUploader.uploadZipArchive
    at com.spotify.buildartifactarchiver.storage.ArtifactUploader.upload
    at com.spotify.buildartifactarchiver.ArtifactArchiver.archiveArtifactsForBuild
    at com.spotify.buildartifactarchiver.ArtifactArchiver.archiveBuild
    at com.spotify.buildartifactarchiver.web.ArtifactArchiverPubSubResource.messageHandler
    at com.spotify.google.cloud.pubsub.client.Puller.lambda$pullBatch$11
    at java.util.concurrent.CompletableFuture.uniWhenComplete
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire
    at java.util.concurrent.CompletableFuture.postComplete
    at java.util.concurrent.CompletableFuture.complete
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.succeed
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.lambda$wrap$9
    at java.util.concurrent.CompletableFuture.uniWhenComplete
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire
    at java.util.concurrent.CompletableFuture.postComplete
    at java.util.concurrent.CompletableFuture.complete
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.succeed
    at com.spotify.google.cloud.pubsub.client.Pubsub.lambda$requestJavaNet$8
    at java.util.concurrent.ThreadPoolExecutor.runWorker
    at java.util.concurrent.ThreadPoolExecutor$Worker.run
    at java.lang.Thread.run

We are uploading files using the following code if there is something that we might be doing wrong...

  public void storeInputStream(InputStream input, String fileName, String contentType)
      throws IOException {

    Storage storage = getStorage();

    BlobId blobId = BlobId.of(this.bucketName, fileName);
    LOG.debug("blobId: {}", blobId);
    BlobInfo blobInfo = BlobInfo.builder(blobId).contentType(contentType).build();
    LOG.debug("blobInfo: {}", blobInfo);
    LOG.info("Starting upload to GCS for file: {} with content type:{}", fileName, contentType);
    //perform stream uploads of the InputStream
    try (WriteChannel writer = storage.writer(blobInfo)) {
      byte[] buffer = new byte[1024];
      int limit;
      while ((limit = input.read(buffer)) >= 0) {
        writer.write(ByteBuffer.wrap(buffer, 0, limit));
      }
    }
  }
@mziccard
Copy link
Contributor

mziccard commented Sep 7, 2016

Hi @fonzy2013,

this is odd but I might have an answer.

What is odd:

  • You are getting java.net.SocketException: Socket is closed when opening a resumable session, not when uploading data. This is odd as calls to open do not carry a lot of data with them (few bytes) and should therefore be less susceptible to timeouts/network disruptions (or whatever it is that is causing this weird exception) while in progress.

Why this is happening:

  • Calls to open in WriteChannel are not retried, not sure why

Good news:

  • I can fix this right away by adding our retry logic to open calls in WriteChannel

I am sorry you are experiencing these issues, and I thank you for reporting them.

@mziccard mziccard added the api: storage Issues related to the Cloud Storage API. label Sep 7, 2016
@fondberg
Copy link
Author

fondberg commented Sep 7, 2016

@mziccard Thanks for speedy and great advice/support!
Strange indeed. Might it be a network related socket closure?

Now we are in production and only after 3 hours I saw around 150 exceptions (around 1-2% failure rate) which isn't all that bad but still ;-)

@mziccard
Copy link
Contributor

mziccard commented Sep 7, 2016

around 1-2% failure rate

All these exceptions originate from a SocketException Socket closed exception with com.google.cloud.storage.spi.DefaultStorageRpc.open in the stacktrace? @Capstan what do you think about this failure rate?

Once again, adding retryals will mitigate this issue, I'll draft a PR ASAP.

github-actions bot pushed a commit to suztomo/google-cloud-java that referenced this issue Jul 1, 2022
🤖 I have created a release *beep* *boop*
---


## [3.4.0](googleapis/java-asset@v3.3.1...v3.4.0) (2022-06-30)


### Features

* Enable REST transport for most of Java and Go clients ([googleapis#1242](googleapis/java-asset#1242)) ([86eb248](googleapis/java-asset@86eb248))


### Bug Fixes

* update gapic-generator-java with mock service generation fixes ([googleapis#1249](googleapis/java-asset#1249)) ([7d124ad](googleapis/java-asset@7d124ad))


### Dependencies

* update dependency com.google.api.grpc:proto-google-cloud-orgpolicy-v1 to v2.2.1 ([googleapis#1245](googleapis/java-asset#1245)) ([9b0d4e9](googleapis/java-asset@9b0d4e9))
* update dependency com.google.api.grpc:proto-google-cloud-pubsub-v1 to v1.101.0 ([googleapis#1228](googleapis/java-asset#1228)) ([c5f1712](googleapis/java-asset@c5f1712))
* update dependency com.google.api.grpc:proto-google-cloud-pubsub-v1 to v1.101.1 ([googleapis#1233](googleapis/java-asset#1233)) ([42031ef](googleapis/java-asset@42031ef))
* update dependency com.google.api.grpc:proto-google-identity-accesscontextmanager-v1 to v1.3.1 ([googleapis#1246](googleapis/java-asset#1246)) ([d868803](googleapis/java-asset@d868803))
* update dependency com.google.cloud:google-cloud-bigquery to v2.13.1 ([googleapis#1227](googleapis/java-asset#1227)) ([113c52f](googleapis/java-asset@113c52f))
* update dependency com.google.cloud:google-cloud-bigquery to v2.13.3 ([googleapis#1234](googleapis/java-asset#1234)) ([7006c44](googleapis/java-asset@7006c44))
* update dependency com.google.cloud:google-cloud-bigquery to v2.13.4 ([googleapis#1239](googleapis/java-asset#1239)) ([9c59632](googleapis/java-asset@9c59632))
* update dependency com.google.cloud:google-cloud-bigquery to v2.13.6 ([googleapis#1244](googleapis/java-asset#1244)) ([a305745](googleapis/java-asset@a305745))
* update dependency com.google.cloud:google-cloud-core to v2.8.0 ([googleapis#1238](googleapis/java-asset#1238)) ([8224b02](googleapis/java-asset@8224b02))
* update dependency com.google.cloud:google-cloud-resourcemanager to v1.4.0 ([googleapis#1230](googleapis/java-asset#1230)) ([5655582](googleapis/java-asset@5655582))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v2.13.0 ([googleapis#1241](googleapis/java-asset#1241)) ([2ebe221](googleapis/java-asset@2ebe221))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
github-actions bot pushed a commit that referenced this issue Aug 25, 2022
…o v2.3.6 (#1227)

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [com.google.cloud:google-cloud-automl](https://togithub.com/googleapis/java-automl) | `2.3.5` -> `2.3.6` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-automl/2.3.6/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-automl/2.3.6/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-automl/2.3.6/compatibility-slim/2.3.5)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-automl/2.3.6/confidence-slim/2.3.5)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>googleapis/java-automl</summary>

### [`v2.3.6`](https://togithub.com/googleapis/java-automl/blob/HEAD/CHANGELOG.md#&#8203;236-httpsgithubcomgoogleapisjava-automlcomparev235v236-2022-08-25)

[Compare Source](https://togithub.com/googleapis/java-automl/compare/v2.3.5...v2.3.6)

##### Dependencies

-   update dependency com.google.cloud:google-cloud-core to v2.8.10 ([#&#8203;1224](https://togithub.com/googleapis/java-automl/issues/1224)) ([731ed9f](https://togithub.com/googleapis/java-automl/commit/731ed9f87487c85309956fb6387c093017dfb617))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-automl).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzMi4xNzYuMCIsInVwZGF0ZWRJblZlciI6IjMyLjE3Ni4wIn0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the Cloud Storage API.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants