Skip to content

Commit

Permalink
Merge pull request #672 from diffplug/feat/foreign
Browse files Browse the repository at this point in the history
Make it easy to integrate native formatters
  • Loading branch information
nedtwigg authored Aug 25, 2020
2 parents 6f1177f + 55d45e8 commit 5aa7385
Show file tree
Hide file tree
Showing 42 changed files with 1,259 additions and 48 deletions.
6 changes: 6 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@ This document is intended for Spotless developers.
We adhere to the [keepachangelog](https://keepachangelog.com/en/1.0.0/) format (starting after version `1.27.0`).

## [Unreleased]
### Added
* The ability to shell out to formatters with their own executables. ([#672](https://github.com/diffplug/spotless/pull/672))
* `ProcessRunner` makes it easy to efficiently and debuggably call foreign executables, and pipe their stdout and stderr to strings.
* `ForeignExe` finds executables on the path (or other strategies), and confirms that they have the correct version (to facilitate Spotless' caching). If the executable is not present or the wrong version, it points the user towards how to fix the problem.
* These classes were used to add support for [python black](https://github.com/psf/black) and [clang-format](https://clang.llvm.org/docs/ClangFormat.html).
* Incidental to this effort, `FormatterFunc.Closeable` now has new methods which make resource-handling safer. The old method is still available as `ofDangerous`, but it should not be used outside of a testing situation. There are some legacy usages of `ofDangerous` in the codebase, and it would be nice to fix them, but the existing usages are using it safely.

## [2.2.2] - 2020-08-21
### Fixed
Expand Down
9 changes: 9 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,15 @@ Here's a checklist for creating a new step for Spotless:

In order for Spotless' model to work, each step needs to look only at the `String` input, otherwise they cannot compose. However, there are some cases where the source `File` is useful, such as to look at the file extension. In this case, you can pass a `FormatterFunc.NeedsFile` instead of a `FormatterFunc`. This should only be used in [rare circumstances](https://github.com/diffplug/spotless/pull/637), be careful that you don't accidentally depend on the bytes inside of the `File`!

### Integrating outside the JVM

There are many great formatters (prettier, clang-format, black, etc.) which live entirely outside the JVM. We have two main strategies for these:

- shell out to an external command for every file (used by clang-format and black) // TODO: link
- open a headless server and make http calls to it from Spotless (used by prettier) // TODO: link

Because of Spotless' up-to-date checking and [git ratcheting](https://github.com/diffplug/spotless/tree/main/plugin-gradle#ratchet), Spotless actually doesn't have to call formatters very often, so even an expensive shell call for every single invocation isn't that bad. Anything that works is better than nothing, and we can always speed things up later if it feels too slow (but it probably won't).

## How to enable the _ext projects

The `_ext` projects are disabled per default, since:
Expand Down
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ output = [
[![License Apache](https://img.shields.io/badge/license-apache-brightgreen.svg)](https://tldrlegal.com/license/apache-license-2.0-(apache-2.0))
<!---freshmark /shields -->

Spotless can format &lt;java | kotlin | scala | sql | groovy | javascript | flow | typeScript | css | scss | less | jsx | vue | graphql | json | yaml | markdown | license headers | anything> using &lt;gradle | maven | anything>.
Spotless can format &lt;antlr | c | c# | c++ | css | flow | graphql | groovy | html | java | javascript | json | jsx | kotlin | less | license headers | markdown | objective-c | protobuf | python | scala | scss | sql | typeScript | vue | yaml | anything> using &lt;gradle | maven | anything>.

- [Spotless for Gradle](plugin-gradle)
- [VS Code extension](https://marketplace.visualstudio.com/items?itemName=richardwillis.vscode-spotless-gradle)
Expand Down Expand Up @@ -46,6 +46,7 @@ lib('generic.ReplaceRegexStep') +'{{yes}} | {{yes}}
lib('generic.ReplaceStep') +'{{yes}} | {{yes}} | {{no}} |',
lib('generic.TrimTrailingWhitespaceStep') +'{{yes}} | {{yes}} | {{no}} |',
lib('antlr4.Antlr4FormatterStep') +'{{yes}} | {{yes}} | {{no}} |',
lib('cpp.ClangFormatStep') +'{{yes}} | {{no}} | {{no}} |',
extra('cpp.EclipseFormatterStep') +'{{yes}} | {{yes}} | {{no}} |',
extra('groovy.GrEclipseFormatterStep') +'{{yes}} | {{no}} | {{no}} |',
lib('java.GoogleJavaFormatStep') +'{{yes}} | {{yes}} | {{no}} |',
Expand All @@ -57,6 +58,7 @@ lib('kotlin.KtfmtStep') +'{{yes}} | {{yes}}
lib('markdown.FreshMarkStep') +'{{yes}} | {{no}} | {{no}} |',
lib('npm.PrettierFormatterStep') +'{{yes}} | {{yes}} | {{no}} |',
lib('npm.TsFmtFormatterStep') +'{{yes}} | {{yes}} | {{no}} |',
lib('python.BlackStep') +'{{yes}} | {{no}} | {{no}} |',
lib('scala.ScalaFmtStep') +'{{yes}} | {{yes}} | {{no}} |',
lib('sql.DBeaverSQLFormatterStep') +'{{yes}} | {{no}} | {{no}} |',
extra('wtp.EclipseWtpFormatterStep') +'{{yes}} | {{yes}} | {{no}} |',
Expand All @@ -78,6 +80,7 @@ extra('wtp.EclipseWtpFormatterStep') +'{{yes}} | {{yes}}
| [`generic.ReplaceStep`](lib/src/main/java/com/diffplug/spotless/generic/ReplaceStep.java) | :+1: | :+1: | :white_large_square: |
| [`generic.TrimTrailingWhitespaceStep`](lib/src/main/java/com/diffplug/spotless/generic/TrimTrailingWhitespaceStep.java) | :+1: | :+1: | :white_large_square: |
| [`antlr4.Antlr4FormatterStep`](lib/src/main/java/com/diffplug/spotless/antlr4/Antlr4FormatterStep.java) | :+1: | :+1: | :white_large_square: |
| [`cpp.ClangFormatStep`](lib/src/main/java/com/diffplug/spotless/cpp/ClangFormatStep.java) | :+1: | :white_large_square: | :white_large_square: |
| [`cpp.EclipseFormatterStep`](lib-extra/src/main/java/com/diffplug/spotless/extra/cpp/EclipseFormatterStep.java) | :+1: | :+1: | :white_large_square: |
| [`groovy.GrEclipseFormatterStep`](lib-extra/src/main/java/com/diffplug/spotless/extra/groovy/GrEclipseFormatterStep.java) | :+1: | :white_large_square: | :white_large_square: |
| [`java.GoogleJavaFormatStep`](lib/src/main/java/com/diffplug/spotless/java/GoogleJavaFormatStep.java) | :+1: | :+1: | :white_large_square: |
Expand All @@ -89,6 +92,7 @@ extra('wtp.EclipseWtpFormatterStep') +'{{yes}} | {{yes}}
| [`markdown.FreshMarkStep`](lib/src/main/java/com/diffplug/spotless/markdown/FreshMarkStep.java) | :+1: | :white_large_square: | :white_large_square: |
| [`npm.PrettierFormatterStep`](lib/src/main/java/com/diffplug/spotless/npm/PrettierFormatterStep.java) | :+1: | :+1: | :white_large_square: |
| [`npm.TsFmtFormatterStep`](lib/src/main/java/com/diffplug/spotless/npm/TsFmtFormatterStep.java) | :+1: | :+1: | :white_large_square: |
| [`python.BlackStep`](lib/src/main/java/com/diffplug/spotless/python/BlackStep.java) | :+1: | :white_large_square: | :white_large_square: |
| [`scala.ScalaFmtStep`](lib/src/main/java/com/diffplug/spotless/scala/ScalaFmtStep.java) | :+1: | :+1: | :white_large_square: |
| [`sql.DBeaverSQLFormatterStep`](lib/src/main/java/com/diffplug/spotless/sql/DBeaverSQLFormatterStep.java) | :+1: | :white_large_square: | :white_large_square: |
| [`wtp.EclipseWtpFormatterStep`](lib-extra/src/main/java/com/diffplug/spotless/extra/wtp/EclipseWtpFormatterStep.java) | :+1: | :+1: | :white_large_square: |
Expand Down
16 changes: 16 additions & 0 deletions gradle/special-tests.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
def special = [
'Npm',
'Black',
'Clang'
]

tasks.named('test') {
useJUnit { excludeCategories special.collect { "com.diffplug.spotless.category.${it}Test" } as String[] }
}

special.forEach {
def category = "com.diffplug.spotless.category.${it}Test"
tasks.register("${it}Test", Test) {
useJUnit { includeCategories category }
}
}
135 changes: 135 additions & 0 deletions lib/src/main/java/com/diffplug/spotless/ForeignExe.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
/*
* Copyright 2020 DiffPlug
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.diffplug.spotless;

import java.io.IOException;
import java.nio.charset.Charset;
import java.util.Objects;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import javax.annotation.Nullable;

/**
* Finds a foreign executable and checks its version.
* If either part of that fails, it shows you why
* and helps you fix it.
*
* Usage: `ForeignExe.nameAndVersion("grep", "2.5.7").confirmVersionAndGetAbsolutePath()`
* will find grep, confirm that it is version 2.5.7, and then return.
*/
public class ForeignExe {
private @Nullable String pathToExe;
private String versionFlag = "--version";
private Pattern versionRegex = Pattern.compile("version (\\S*)");
private @Nullable String fixCantFind, fixWrongVersion;

// MANDATORY
private String name;
private String version;

/** The name of the executable, used by "where" (win) and "which" (unix). */
public static ForeignExe nameAndVersion(String exeName, String version) {
ForeignExe foreign = new ForeignExe();
foreign.name = Objects.requireNonNull(exeName);
foreign.version = Objects.requireNonNull(version);
return foreign;
}

/** The flag which causes the exe to print its version (defaults to --version). */
public ForeignExe versionFlag(String versionFlag) {
this.versionFlag = Objects.requireNonNull(versionFlag);
return this;
}

/** A regex which can parse the version out of the output of the {@link #versionFlag(String)} command (defaults to `version (\\S*)`) */
public ForeignExe versionRegex(Pattern versionRegex) {
this.versionRegex = Objects.requireNonNull(versionRegex);
return this;
}

/** Use {version} anywhere you would like to inject the actual version string. */
public ForeignExe fixCantFind(String msg) {
this.fixCantFind = msg;
return this;
}

/** Use {version} or {versionFound} anywhere you would like to inject the actual version strings. */
public ForeignExe fixWrongVersion(String msg) {
this.fixWrongVersion = msg;
return this;
}

/** Path to the executable. If null, will search for the executable on the system path. */
public ForeignExe pathToExe(@Nullable String pathToExe) {
this.pathToExe = pathToExe;
return this;
}

/**
* Searches for the executable and confirms that it has the expected version.
* If it can't find the executable, or if it doesn't have the correct version,
* throws an exception with a message describing how to fix.
*/
public String confirmVersionAndGetAbsolutePath() throws IOException, InterruptedException {
try (ProcessRunner runner = new ProcessRunner()) {
String exeAbsPath;
if (pathToExe != null) {
exeAbsPath = pathToExe;
} else {
ProcessRunner.Result cmdWhich = runner.shellWinUnix("where " + name, "which " + name);
if (cmdWhich.exitNotZero()) {
throw cantFind("Unable to find " + name + " on path", cmdWhich);
} else {
exeAbsPath = cmdWhich.assertExitZero(Charset.defaultCharset()).trim();
}
}
ProcessRunner.Result cmdVersion = runner.exec(exeAbsPath, versionFlag);
if (cmdVersion.exitNotZero()) {
throw cantFind("Unable to run " + exeAbsPath, cmdVersion);
}
Matcher versionMatcher = versionRegex.matcher(cmdVersion.assertExitZero(Charset.defaultCharset()));
if (!versionMatcher.find()) {
throw cantFind("Unable to parse version with /" + versionRegex + "/", cmdVersion);
}
String versionFound = versionMatcher.group(1);
if (!versionFound.equals(version)) {
throw wrongVersion("You specified version " + version + ", but Spotless found " + versionFound, cmdVersion, versionFound);
}
return exeAbsPath;
}
}

private RuntimeException cantFind(String message, ProcessRunner.Result cmd) {
return exceptionFmt(message, cmd, fixCantFind == null ? null : fixCantFind.replace("{version}", version));
}

private RuntimeException wrongVersion(String message, ProcessRunner.Result cmd, String versionFound) {
return exceptionFmt(message, cmd, fixWrongVersion == null ? null : fixWrongVersion.replace("{version}", version).replace("{versionFound}", versionFound));
}

private RuntimeException exceptionFmt(String msgPrimary, ProcessRunner.Result cmd, @Nullable String msgFix) {
StringBuilder errorMsg = new StringBuilder();
errorMsg.append(msgPrimary);
errorMsg.append('\n');
if (msgFix != null) {
errorMsg.append(msgFix);
errorMsg.append('\n');
}
errorMsg.append(cmd.toString());
return new RuntimeException(errorMsg.toString());
}
}
82 changes: 75 additions & 7 deletions lib/src/main/java/com/diffplug/spotless/FormatterFunc.java
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,17 @@ interface Closeable extends FormatterFunc, AutoCloseable {
@Override
void close();

/** Creates a {@link Closeable} from an AutoCloseable and a function. */
public static Closeable of(AutoCloseable closeable, FormatterFunc function) {
/**
* Dangerous way to create a {@link Closeable} from an AutoCloseable and a function.
*
* It's important for FormatterStep's to allocate their resources as lazily as possible.
* It's easy to create a resource inside the state, and not realize that it may not be
* released. It's far better to use one of the non-deprecated `of()` methods below.
*
* The bug (and its fix) which is easy to write using this method: https://github.com/diffplug/spotless/commit/7f16ecca031810b5e6e6f647e1f10a6d2152d9f4
* How the `of()` methods below make the correct thing easier to write and safer: https://github.com/diffplug/spotless/commit/18c10f9c93d6f18f753233d0b5f028d5f0961916
*/
public static Closeable ofDangerous(AutoCloseable closeable, FormatterFunc function) {
Objects.requireNonNull(closeable, "closeable");
Objects.requireNonNull(function, "function");
return new Closeable() {
Expand All @@ -52,12 +61,73 @@ public void close() {

@Override
public String apply(String unix, File file) throws Exception {
return function.apply(Objects.requireNonNull(unix), Objects.requireNonNull(file));
return function.apply(unix, file);
}

@Override
public String apply(String unix) throws Exception {
return function.apply(unix);
}
};
}

/** @deprecated synonym for {@link #ofDangerous(AutoCloseable, FormatterFunc)} */
@Deprecated
public static Closeable of(AutoCloseable closeable, FormatterFunc function) {
return ofDangerous(closeable, function);
}

@FunctionalInterface
interface ResourceFunc<T extends AutoCloseable> {
String apply(T resource, String unix) throws Exception;
}

/** Creates a {@link FormatterFunc.Closeable} which uses the given resource to execute the format function. */
public static <T extends AutoCloseable> Closeable of(T resource, ResourceFunc<T> function) {
Objects.requireNonNull(resource, "resource");
Objects.requireNonNull(function, "function");
return new Closeable() {
@Override
public void close() {
ThrowingEx.run(resource::close);
}

@Override
public String apply(String unix, File file) throws Exception {
return function.apply(resource, unix);
}

@Override
public String apply(String unix) throws Exception {
return function.apply(resource, unix);
}
};
}

@FunctionalInterface
interface ResourceFuncNeedsFile<T extends AutoCloseable> {
String apply(T resource, String unix, File file) throws Exception;
}

/** Creates a {@link FormatterFunc.Closeable} which uses the given resource to execute the file-dependent format function. */
public static <T extends AutoCloseable> Closeable of(T resource, ResourceFuncNeedsFile<T> function) {
Objects.requireNonNull(resource, "resource");
Objects.requireNonNull(function, "function");
return new Closeable() {
@Override
public void close() {
ThrowingEx.run(resource::close);
}

@Override
public String apply(String unix, File file) throws Exception {
FormatterStepImpl.checkNotSentinel(file);
return function.apply(resource, unix, file);
}

@Override
public String apply(String unix) throws Exception {
return function.apply(Objects.requireNonNull(unix));
return apply(unix, FormatterStepImpl.SENTINEL);
}
};
}
Expand All @@ -80,9 +150,7 @@ interface NeedsFile extends FormatterFunc {

@Override
default String apply(String unix, File file) throws Exception {
if (file == FormatterStepImpl.SENTINEL) {
throw new IllegalArgumentException("This step requires the underlying file. If this is a test, use StepHarnessWithFile");
}
FormatterStepImpl.checkNotSentinel(file);
return applyWithFile(unix, file);
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -114,4 +114,10 @@ protected String format(Integer state, String rawUnix, File file) throws Excepti

/** A dummy SENTINEL file. */
static final File SENTINEL = new File("");

static void checkNotSentinel(File file) {
if (file == SENTINEL) {
throw new IllegalArgumentException("This step requires the underlying file. If this is a test, use StepHarnessWithFile");
}
}
}
Loading

0 comments on commit 5aa7385

Please sign in to comment.