Skip to content

Commit

Permalink
Stop doing actions.write UTF-8. This was causing a double encode.
Browse files Browse the repository at this point in the history
Since Bazel reads BUILD and .bzl files as (essentially) raw bytes, any UTF-8
encoded input is essentially pre-UTF-8 encoded on the way out.

Closes bazelbuild#10174

DO NOT SUBMIT: This is a breaking change, so it needs a migration path.
Current idea:
- introduce a new attribute to write 'encode_as_utf_8'. Default is gated by incompatible_write_action_encodes_as_utf_8.
- for bazel we do the multi-release cycle flip
- for google we agressively try to change the default
  - add encode_as_utf_8 to the few broken rules
  - flip the default
  - fix the user.
  • Loading branch information
aiuto committed May 26, 2020
1 parent 034b09e commit 096defc
Showing 1 changed file with 5 additions and 6 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@

package com.google.devtools.build.lib.analysis.actions;

import static java.nio.charset.StandardCharsets.UTF_8;

import com.google.common.annotations.VisibleForTesting;
import com.google.devtools.build.lib.actions.ActionExecutionContext;
import com.google.devtools.build.lib.actions.ActionKeyContext;
Expand All @@ -39,7 +37,8 @@
/**
* Action to write a file whose contents are known at analysis time.
*
* <p>The output is always UTF-8 encoded.
* <p>The output is not reencoded in any way. That is, the raw bytes from BUILD
* and .bzl files for string attributes will be emitted exactly as read.
*
* <p>TODO(bazel-team): Choose a better name to distinguish this class from {@link
* BinaryFileWriteAction}.
Expand Down Expand Up @@ -171,7 +170,7 @@ private static final class CompressedString extends LazyString {
final int uncompressedSize;

CompressedString(String chars) {
byte[] dataToCompress = chars.getBytes(UTF_8);
byte[] dataToCompress = chars.getBytes();
ByteArrayOutputStream byteStream = new ByteArrayOutputStream(dataToCompress.length);
try (GZIPOutputStream zipStream = new GZIPOutputStream(byteStream)) {
zipStream.write(dataToCompress);
Expand Down Expand Up @@ -202,7 +201,7 @@ public String toString() {
// This should be impossible since we're reading from a byte array.
throw new RuntimeException(e);
}
return new String(uncompressedBytes, UTF_8);
return new String(uncompressedBytes);
}
}

Expand Down Expand Up @@ -235,7 +234,7 @@ public DeterministicWriter newDeterministicWriter(ActionExecutionContext ctx) {
return new DeterministicWriter() {
@Override
public void writeOutputFile(OutputStream out) throws IOException {
out.write(getFileContents().getBytes(UTF_8));
out.write(getFileContents().getBytes());
}
};
}
Expand Down

0 comments on commit 096defc

Please sign in to comment.