Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BlobWriteOption to support MD5 and CRC32C checks on create/write #271

Merged
merged 3 commits into from
Oct 20, 2015

Conversation

mziccard
Copy link
Contributor

This PR fixes #267. Main changes are:

  • Add BlobWriteOption with support for md5Match and crc32cMatch
  • Add MD5 and CRC32C computation to create(blobInfo, content, options)
  • Change BlobTargetOption... to BlobWriteOption... in create from stream writer
  • Update unit tests
  • Update and add integration tests

To note that md5Match and crc32cMatch are boolean options always true. If not specified storageobject's crc32c and md5 fields are set to null (so no validation occurs).
Alternatives to boolean options were:

  • String options implicitly taken from BlobInfo: this would have required to modify storage.optionMap, that is now used to fetch implicit values only for (meta)generation matches. (meta)generation options are shared across all APIs, so I felt like md5Match and crc32cMatch did not fit here.
  • String options explicity set by the user (e.g. BlobWriteOption.md5Match("7uyg8yg8==")), still a valid option but a bit counter-intuitive: all values set in BlobInfo are transmitted, why not to use md5 and crc32c fields?

Comments are appreciated!

- Add MD5 and CRC32C computation to create and create from byte array
- Change BlobTargetOption... to BlobWriteOption... in create from stream and writer
- Change BlobTargetOption... to BlobWriteOption in Blob.writer
- Update unit tests
- Update and add integration tests
@mziccard mziccard added the api: storage Issues related to the Cloud Storage API. label Oct 19, 2015
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Oct 19, 2015
@@ -106,6 +108,15 @@ private static StorageException translate(GoogleJsonError exception) {
return new StorageException(exception.getCode(), exception.getMessage(), retryable);
}

private static void applyOptions(StorageObject storageObject, Map<Option, ?> options) {

This comment was marked as spam.

- BlobWriteOption extends Serializable instead of Option
- add BlobTargetObject.convert to convert BlobWriteOption and possibly mask md5 and crc32c fields
- add private Storage.create(blobInfo, inputStream, blobTargetOption)
- update Storage.create, Storage.writer and Blob.writer javadoc
@mziccard
Copy link
Contributor Author

Fixed the comments. I removed options from the RPC layer.
I don't like so much using Tuple for the PRC layer in the BlobTargetOption.convert method I added, what do you think?

BlobInfo updatedInfo = blobInfo.toBuilder()
.md5(BaseEncoding.base64().encode(Hashing.md5().hashBytes(content).asBytes()))
.crc32c(BaseEncoding.base64().encode(
Ints.toByteArray(Hashing.crc32c().hashBytes(content).asInt())))

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

@aozarov
Copy link
Contributor

aozarov commented Oct 20, 2015

I don't like so much using Tuple for the PRC layer in the BlobTargetOption.convert method I added, what do you think?

Honestly, I wished Tuple was part of the JDK (or at least guava) but I understand the opposition for it.
I am fine with it the way it and as this that is internal we can always be changed later.
If you feel better you can write a dedicate private inner class for it or pass a BlobInfo.Builder.

@mziccard
Copy link
Contributor Author

@aozarov I don't like the idea of doing side effects on a builder either, lets keep it as is, we can possibly change it later.

@mziccard
Copy link
Contributor Author

Removed the unnecessary final

aozarov added a commit that referenced this pull request Oct 20, 2015
Add BlobWriteOption to support MD5 and CRC32C checks on create/write
@aozarov aozarov merged commit 5595770 into googleapis:master Oct 20, 2015
github-actions bot pushed a commit that referenced this pull request Jun 23, 2022
🤖 I have created a release *beep* *boop*
---


## [1.2.1](googleapis/java-artifact-registry@v1.2.0...v1.2.1) (2022-06-23)


### Dependencies

* update dependency com.google.cloud:google-cloud-shared-dependencies to v2.13.0 ([#270](googleapis/java-artifact-registry#270)) ([43c1ae8](googleapis/java-artifact-registry@43c1ae8))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
github-actions bot pushed a commit to suztomo/google-cloud-java that referenced this pull request Jun 29, 2022
This PR was generated using Autosynth. 🌈


<details><summary>Log from Synthtool</summary>

```
2020-10-15 22:43:18,502 synthtool [DEBUG] > Executing /root/.cache/synthtool/java-resourcemanager/.github/readme/synth.py.
On branch autosynth-readme
nothing to commit, working directory clean
2020-10-15 22:43:19,459 synthtool [DEBUG] > Wrote metadata to .github/readme/synth.metadata/synth.metadata.

```
</details>

Full log will be available here:
https://source.cloud.google.com/results/invocations/128052fe-465f-41a7-88f9-a76876e9437e/targets

- [ ] To automatically regenerate this PR, check this box.
github-actions bot pushed a commit that referenced this pull request Jun 30, 2022
Source-Link: googleapis/synthtool@7a220e2
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-java:latest@sha256:6d4e3a15c62cfdcb823d60e16da7521e7c6fc00eba07c8ff12e4de9924a57d28
github-actions bot pushed a commit that referenced this pull request Jul 1, 2022
…s to v1 (#271)

[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [com.google.cloud:google-cloud-workflows](https://togithub.com/googleapis/java-workflows) | `0.4.2` -> `1.0.0` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-workflows/1.0.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-workflows/1.0.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-workflows/1.0.0/compatibility-slim/0.4.2)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-workflows/1.0.0/confidence-slim/0.4.2)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>googleapis/java-workflows</summary>

### [`v1.0.0`](https://togithub.com/googleapis/java-workflows/blob/master/CHANGELOG.md#&#8203;100-httpswwwgithubcomgoogleapisjava-workflowscomparev042v100-2021-05-26)

[Compare Source](https://togithub.com/googleapis/java-workflows/compare/v0.4.2...v1.0.0)

##### Features

-   promote to 1.0.0 ([#&#8203;267](https://www.github.com/googleapis/java-workflows/issues/267)) ([57f9b08](https://www.github.com/googleapis/java-workflows/commit/57f9b08ddda565f6d3209292df1a26a15daaebe8))

##### [0.4.2](https://www.github.com/googleapis/java-workflows/compare/v0.4.1...v0.4.2) (2021-05-19)

##### Dependencies

-   update dependency com.google.cloud:google-cloud-shared-dependencies to v1.2.0 ([#&#8203;257](https://www.github.com/googleapis/java-workflows/issues/257)) ([98c0d36](https://www.github.com/googleapis/java-workflows/commit/98c0d36b8c531224dfea76c85eb2a35bc55e45d8))

##### [0.4.1](https://www.github.com/googleapis/java-workflows/compare/v0.4.0...v0.4.1) (2021-05-14)

##### Dependencies

-   update dependency com.google.cloud:google-cloud-shared-dependencies to v1.1.0 ([#&#8203;249](https://www.github.com/googleapis/java-workflows/issues/249)) ([0326aa5](https://www.github.com/googleapis/java-workflows/commit/0326aa563bab44a93258fabe77e6d953a7c4e3c9))

</details>

---

### Configuration

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻️ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-workflows).
github-actions bot pushed a commit that referenced this pull request Jul 1, 2022
🤖 I have created a release *beep* *boop*
---


### Updating meta-information for bleeding-edge SNAPSHOT release.

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
github-actions bot pushed a commit that referenced this pull request Aug 9, 2022
🤖 I have created a release *beep* *boop*
---


### Updating meta-information for bleeding-edge SNAPSHOT release.

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
github-actions bot pushed a commit that referenced this pull request Aug 9, 2022
🤖 I have created a release *beep* *boop*
---


### Updating meta-information for bleeding-edge SNAPSHOT release.

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
github-actions bot pushed a commit that referenced this pull request Sep 15, 2022
🤖 I have created a release *beep* *boop*
---


### Updating meta-information for bleeding-edge SNAPSHOT release.

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
github-actions bot pushed a commit that referenced this pull request Sep 15, 2022
🤖 I have created a release *beep* *boop*
---


### Updating meta-information for bleeding-edge SNAPSHOT release.

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
github-actions bot pushed a commit that referenced this pull request Sep 15, 2022
…ator_java versions (#271)

- [ ] Regenerate this pull request now.

PiperOrigin-RevId: 472750037

Source-Link: googleapis/googleapis@88f2ea3

Source-Link: https://github.com/googleapis/googleapis-gen/commit/230a5588306aae18fe8f2a57f14d4039ad72c901
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMjMwYTU1ODgzMDZhYWUxOGZlOGYyYTU3ZjE0ZDQwMzlhZDcyYzkwMSJ9
github-actions bot pushed a commit that referenced this pull request Sep 15, 2022
…1575) (#271)

Source-Link: googleapis/synthtool@2e9ac19
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-java:latest@sha256:8175681a918181d306d9c370d3262f16b4c724cc73d74111b7d42fc985ca7f93
github-actions bot pushed a commit that referenced this pull request Sep 15, 2022
🤖 I have created a release *beep* *boop*
---


## [2.3.4](googleapis/java-service-usage@v2.3.3...v2.3.4) (2022-09-15)


### Dependencies

* Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.3 ([#270](googleapis/java-service-usage#270)) ([54ce129](googleapis/java-service-usage@54ce129))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
github-actions bot pushed a commit that referenced this pull request Sep 15, 2022
🤖 I have created a release *beep* *boop*
---


### Updating meta-information for bleeding-edge SNAPSHOT release.

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
github-actions bot pushed a commit that referenced this pull request Sep 30, 2022
🤖 I have created a release *beep* *boop*
---


## [2.2.6](googleapis/java-dms@v2.2.5...v2.2.6) (2022-09-29)


### Dependencies

* Update dependency cachetools to v5 ([#288](googleapis/java-dms#288)) ([9370173](googleapis/java-dms@9370173))
* Update dependency certifi to v2022.9.24 ([#268](googleapis/java-dms#268)) ([1745004](googleapis/java-dms@1745004))
* Update dependency charset-normalizer to v2.1.1 ([#272](googleapis/java-dms#272)) ([367ffd9](googleapis/java-dms@367ffd9))
* Update dependency click to v8.1.3 ([#273](googleapis/java-dms#273)) ([328f00b](googleapis/java-dms@328f00b))
* Update dependency gcp-releasetool to v1.8.8 ([#269](googleapis/java-dms#269)) ([7d68646](googleapis/java-dms@7d68646))
* Update dependency google-api-core to v2.10.1 ([#274](googleapis/java-dms#274)) ([03bc3c5](googleapis/java-dms@03bc3c5))
* Update dependency google-auth to v2.12.0 ([#275](googleapis/java-dms#275)) ([bd96264](googleapis/java-dms@bd96264))
* Update dependency google-cloud-core to v2.3.2 ([#270](googleapis/java-dms#270)) ([fcd6bbd](googleapis/java-dms@fcd6bbd))
* Update dependency google-cloud-storage to v2.5.0 ([#276](googleapis/java-dms#276)) ([80b39f1](googleapis/java-dms@80b39f1))
* Update dependency google-crc32c to v1.5.0 ([#277](googleapis/java-dms#277)) ([a561425](googleapis/java-dms@a561425))
* Update dependency googleapis-common-protos to v1.56.4 ([#271](googleapis/java-dms#271)) ([296d07c](googleapis/java-dms@296d07c))
* Update dependency importlib-metadata to v4.12.0 ([#278](googleapis/java-dms#278)) ([b4509fa](googleapis/java-dms@b4509fa))
* Update dependency jeepney to v0.8.0 ([#279](googleapis/java-dms#279)) ([b2b6e8f](googleapis/java-dms@b2b6e8f))
* Update dependency jinja2 to v3.1.2 ([#280](googleapis/java-dms#280)) ([c7c2a10](googleapis/java-dms@c7c2a10))
* Update dependency keyring to v23.9.3 ([#281](googleapis/java-dms#281)) ([981c58c](googleapis/java-dms@981c58c))
* Update dependency markupsafe to v2.1.1 ([#282](googleapis/java-dms#282)) ([5a78aa3](googleapis/java-dms@5a78aa3))
* Update dependency protobuf to v3.20.2 ([#283](googleapis/java-dms#283)) ([2444bcd](googleapis/java-dms@2444bcd))
* Update dependency protobuf to v4 ([#289](googleapis/java-dms#289)) ([b9e330a](googleapis/java-dms@b9e330a))
* Update dependency pyjwt to v2.5.0 ([#284](googleapis/java-dms#284)) ([9fc617d](googleapis/java-dms@9fc617d))
* Update dependency requests to v2.28.1 ([#285](googleapis/java-dms#285)) ([51574b4](googleapis/java-dms@51574b4))
* Update dependency typing-extensions to v4.3.0 ([#286](googleapis/java-dms#286)) ([c36d416](googleapis/java-dms@c36d416))
* Update dependency zipp to v3.8.1 ([#287](googleapis/java-dms#287)) ([34e4986](googleapis/java-dms@34e4986))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
github-actions bot pushed a commit that referenced this pull request Oct 4, 2022
github-actions bot pushed a commit that referenced this pull request Oct 4, 2022
🤖 I have created a release *beep* *boop*
---


## [1.2.5](https://togithub.com/googleapis/java-policy-troubleshooter/compare/v1.2.4...v1.2.5) (2022-10-03)


### Dependencies

* Update dependency cachetools to v5 ([#287](https://togithub.com/googleapis/java-policy-troubleshooter/issues/287)) ([2fea464](https://togithub.com/googleapis/java-policy-troubleshooter/commit/2fea464d4acf1cf78e01c58880053ff1d621e851))
* Update dependency certifi to v2022.9.24 ([#267](https://togithub.com/googleapis/java-policy-troubleshooter/issues/267)) ([9eeaf6c](https://togithub.com/googleapis/java-policy-troubleshooter/commit/9eeaf6c3d21e7c3eed8e50cd7bca445aba599313))
* Update dependency charset-normalizer to v2.1.1 ([#271](https://togithub.com/googleapis/java-policy-troubleshooter/issues/271)) ([9d3dfd8](https://togithub.com/googleapis/java-policy-troubleshooter/commit/9d3dfd8da1881a8d0c2dc3edd34ce21e4d8aa64b))
* Update dependency click to v8.1.3 ([#272](https://togithub.com/googleapis/java-policy-troubleshooter/issues/272)) ([5d92d7e](https://togithub.com/googleapis/java-policy-troubleshooter/commit/5d92d7e3551b2ca1b3ab6a91a8b6a589cb7c8028))
* Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.4 ([#292](https://togithub.com/googleapis/java-policy-troubleshooter/issues/292)) ([d47d642](https://togithub.com/googleapis/java-policy-troubleshooter/commit/d47d64222138b45ed5375bf3e9bf9e76cf8971ad))
* Update dependency gcp-releasetool to v1.8.8 ([#268](https://togithub.com/googleapis/java-policy-troubleshooter/issues/268)) ([a75ee9b](https://togithub.com/googleapis/java-policy-troubleshooter/commit/a75ee9be6a2ac6b9fe573d9d086a94d325ddd814))
* Update dependency google-api-core to v2.10.1 ([#273](https://togithub.com/googleapis/java-policy-troubleshooter/issues/273)) ([edf53b7](https://togithub.com/googleapis/java-policy-troubleshooter/commit/edf53b7785d559cb1f0b35a83982d8b36c7b1c6a))
* Update dependency google-auth to v2.12.0 ([#274](https://togithub.com/googleapis/java-policy-troubleshooter/issues/274)) ([1b4d43f](https://togithub.com/googleapis/java-policy-troubleshooter/commit/1b4d43facb6739b12cf4c037333237af206f6524))
* Update dependency google-cloud-core to v2.3.2 ([#269](https://togithub.com/googleapis/java-policy-troubleshooter/issues/269)) ([9e9681a](https://togithub.com/googleapis/java-policy-troubleshooter/commit/9e9681a4c0da7aa7e402584a1ac2219558b5352e))
* Update dependency google-cloud-storage to v2.5.0 ([#275](https://togithub.com/googleapis/java-policy-troubleshooter/issues/275)) ([7280eaf](https://togithub.com/googleapis/java-policy-troubleshooter/commit/7280eafabd013c704cc63a042fab3255bee55111))
* Update dependency google-crc32c to v1.5.0 ([#276](https://togithub.com/googleapis/java-policy-troubleshooter/issues/276)) ([b144928](https://togithub.com/googleapis/java-policy-troubleshooter/commit/b1449280a8c1e8caef134aae7b62a1c4f7b5b5b6))
* Update dependency googleapis-common-protos to v1.56.4 ([#270](https://togithub.com/googleapis/java-policy-troubleshooter/issues/270)) ([942ea39](https://togithub.com/googleapis/java-policy-troubleshooter/commit/942ea39d9b5f655736b0ce0c7c532803dffa4484))
* Update dependency importlib-metadata to v4.12.0 ([#277](https://togithub.com/googleapis/java-policy-troubleshooter/issues/277)) ([bf03ef8](https://togithub.com/googleapis/java-policy-troubleshooter/commit/bf03ef8a75846e74b0b509a518caf85dcea2b901))
* Update dependency jeepney to v0.8.0 ([#278](https://togithub.com/googleapis/java-policy-troubleshooter/issues/278)) ([f51e9e7](https://togithub.com/googleapis/java-policy-troubleshooter/commit/f51e9e724f088a864dd70deeef16edf94922c246))
* Update dependency jinja2 to v3.1.2 ([#279](https://togithub.com/googleapis/java-policy-troubleshooter/issues/279)) ([95f9152](https://togithub.com/googleapis/java-policy-troubleshooter/commit/95f91525eb3ff0d57a3639b0f66cf2be6b1f0f8b))
* Update dependency keyring to v23.9.3 ([#280](https://togithub.com/googleapis/java-policy-troubleshooter/issues/280)) ([33d8606](https://togithub.com/googleapis/java-policy-troubleshooter/commit/33d8606ea8da63f7cffb9c38ad5e52a52276e5cc))
* Update dependency protobuf to v4 ([#288](https://togithub.com/googleapis/java-policy-troubleshooter/issues/288)) ([8a836ed](https://togithub.com/googleapis/java-policy-troubleshooter/commit/8a836edb54c0cd182dfb4ee4e29bbb910d452150))
* Update dependency pyjwt to v2.5.0 ([#283](https://togithub.com/googleapis/java-policy-troubleshooter/issues/283)) ([a4483cf](https://togithub.com/googleapis/java-policy-troubleshooter/commit/a4483cf44a22ba571a6378587270aff46c76e544))
* Update dependency typing-extensions to v4.3.0 ([#285](https://togithub.com/googleapis/java-policy-troubleshooter/issues/285)) ([2ccd257](https://togithub.com/googleapis/java-policy-troubleshooter/commit/2ccd257da7680d3952f7dcfc29633448fbc91c74))
* Update dependency zipp to v3.8.1 ([#286](https://togithub.com/googleapis/java-policy-troubleshooter/issues/286)) ([0270359](https://togithub.com/googleapis/java-policy-troubleshooter/commit/0270359b0316fe14ddd700120485929acb97fb8a))

---
This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
github-actions bot pushed a commit that referenced this pull request Oct 5, 2022
🤖 I have created a release *beep* *boop*
---


### Updating meta-information for bleeding-edge SNAPSHOT release.

---
This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
github-actions bot pushed a commit to yoshi-code-bot/google-cloud-java that referenced this pull request Nov 8, 2022
…s#1687) (googleapis#271)

* chore(java): add a note in README for migrated split repos

Disable renovate bot and flaky bot for split repositories
that have moved to the Java monorepo.
The Java monorepo will pass the "monorepo=True" parameter
to java.common_templates method in its owlbot.py files so that
the migration note will not appear in the README in the monorepo.

Co-authored-by: Jeff Ching <chingor@google.com>
Source-Link: https://togithub.com/googleapis/synthtool/commit/d4b291604f148cde065838c498bc8aa79b8dc10e
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-java:latest@sha256:edae91ccdd2dded2f572ec341a768ad180305a3e8fbfd93064b28e237d35920a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the Cloud Storage API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BlobWriteChannel CRC32C and MD5 hash mismatch error
3 participants