Merge pull request #672 from diffplug/feat/foreign

Make it easy to integrate native formatters

Author: nedtwigg

CHANGES.md

@@ -10,6 +10,12 @@ This document is intended for Spotless developers.
 We adhere to the [keepachangelog](https://keepachangelog.com/en/1.0.0/) format (starting after version `1.27.0`).
 
 ## [Unreleased]
+### Added
+* The ability to shell out to formatters with their own executables. ([#672](https://github.com/diffplug/spotless/pull/672))
+  * `ProcessRunner` makes it easy to efficiently and debuggably call foreign executables, and pipe their stdout and stderr to strings.
+  * `ForeignExe` finds executables on the path (or other strategies), and confirms that they have the correct version (to facilitate Spotless' caching). If the executable is not present or the wrong version, it points the user towards how to fix the problem.
+  * These classes were used to add support for [python black](https://github.com/psf/black) and [clang-format](https://clang.llvm.org/docs/ClangFormat.html).
+  * Incidental to this effort, `FormatterFunc.Closeable` now has new methods which make resource-handling safer.  The old method is still available as `ofDangerous`, but it should not be used outside of a testing situation. There are some legacy usages of `ofDangerous` in the codebase, and it would be nice to fix them, but the existing usages are using it safely.
 
 ## [2.2.2] - 2020-08-21
 ### Fixed

CONTRIBUTING.md

@@ -104,6 +104,15 @@ Here's a checklist for creating a new step for Spotless:
 
 In order for Spotless' model to work, each step needs to look only at the `String` input, otherwise they cannot compose.  However, there are some cases where the source `File` is useful, such as to look at the file extension.  In this case, you can pass a `FormatterFunc.NeedsFile` instead of a `FormatterFunc`.  This should only be used in [rare circumstances](https://github.com/diffplug/spotless/pull/637), be careful that you don't accidentally depend on the bytes inside of the `File`!
 
+### Integrating outside the JVM
+
+There are many great formatters (prettier, clang-format, black, etc.) which live entirely outside the JVM.  We have two main strategies for these:
+
+- shell out to an external command for every file (used by clang-format and black) // TODO: link
+- open a headless server and make http calls to it from Spotless (used by prettier) // TODO: link
+
+Because of Spotless' up-to-date checking and [git ratcheting](https://github.com/diffplug/spotless/tree/main/plugin-gradle#ratchet), Spotless actually doesn't have to call formatters very often, so even an expensive shell call for every single invocation isn't that bad.  Anything that works is better than nothing, and we can always speed things up later if it feels too slow (but it probably won't).
+
 ## How to enable the _ext projects
 
 The `_ext` projects are disabled per default, since:

README.md

@@ -12,7 +12,7 @@ output = [
 [![License Apache](https://img.shields.io/badge/license-apache-brightgreen.svg)](https://tldrlegal.com/license/apache-license-2.0-(apache-2.0))
 <!---freshmark /shields -->
 
-Spotless can format &lt;java | kotlin | scala | sql | groovy | javascript | flow | typeScript | css | scss | less | jsx | vue | graphql | json | yaml | markdown | license headers | anything> using &lt;gradle | maven | anything>.
+Spotless can format &lt;antlr | c | c# | c++ | css | flow | graphql | groovy | html | java | javascript | json | jsx | kotlin | less | license headers | markdown | objective-c | protobuf | python | scala | scss | sql | typeScript | vue | yaml | anything> using &lt;gradle | maven | anything>.
 
 - [Spotless for Gradle](plugin-gradle)
   - [VS Code extension](https://marketplace.visualstudio.com/items?itemName=richardwillis.vscode-spotless-gradle)
@@ -46,6 +46,7 @@ lib('generic.ReplaceRegexStep')                  +'{{yes}}       | {{yes}}
 lib('generic.ReplaceStep')                       +'{{yes}}       | {{yes}}      | {{no}}  |',
 lib('generic.TrimTrailingWhitespaceStep')        +'{{yes}}       | {{yes}}      | {{no}}  |',
 lib('antlr4.Antlr4FormatterStep')                +'{{yes}}       | {{yes}}      | {{no}}  |',
+lib('cpp.ClangFormatStep')                       +'{{yes}}       | {{no}}       | {{no}}  |',
 extra('cpp.EclipseFormatterStep')                +'{{yes}}       | {{yes}}      | {{no}}  |',
 extra('groovy.GrEclipseFormatterStep')           +'{{yes}}       | {{no}}       | {{no}}  |',
 lib('java.GoogleJavaFormatStep')                 +'{{yes}}       | {{yes}}      | {{no}}  |',
@@ -57,6 +58,7 @@ lib('kotlin.KtfmtStep')                          +'{{yes}}       | {{yes}}
 lib('markdown.FreshMarkStep')                    +'{{yes}}       | {{no}}       | {{no}}  |',
 lib('npm.PrettierFormatterStep')                 +'{{yes}}       | {{yes}}      | {{no}}  |',
 lib('npm.TsFmtFormatterStep')                    +'{{yes}}       | {{yes}}      | {{no}}  |',
+lib('python.BlackStep')                          +'{{yes}}       | {{no}}       | {{no}}  |',
 lib('scala.ScalaFmtStep')                        +'{{yes}}       | {{yes}}      | {{no}}  |',
 lib('sql.DBeaverSQLFormatterStep')               +'{{yes}}       | {{no}}       | {{no}}  |',
 extra('wtp.EclipseWtpFormatterStep')             +'{{yes}}       | {{yes}}      | {{no}}  |',
@@ -78,6 +80,7 @@ extra('wtp.EclipseWtpFormatterStep')             +'{{yes}}       | {{yes}}
 | [`generic.ReplaceStep`](lib/src/main/java/com/diffplug/spotless/generic/ReplaceStep.java) | :+1:       | :+1:      | :white_large_square:  |
 | [`generic.TrimTrailingWhitespaceStep`](lib/src/main/java/com/diffplug/spotless/generic/TrimTrailingWhitespaceStep.java) | :+1:       | :+1:      | :white_large_square:  |
 | [`antlr4.Antlr4FormatterStep`](lib/src/main/java/com/diffplug/spotless/antlr4/Antlr4FormatterStep.java) | :+1:       | :+1:      | :white_large_square:  |
+| [`cpp.ClangFormatStep`](lib/src/main/java/com/diffplug/spotless/cpp/ClangFormatStep.java) | :+1:       | :white_large_square:       | :white_large_square:  |
 | [`cpp.EclipseFormatterStep`](lib-extra/src/main/java/com/diffplug/spotless/extra/cpp/EclipseFormatterStep.java) | :+1:       | :+1:      | :white_large_square:  |
 | [`groovy.GrEclipseFormatterStep`](lib-extra/src/main/java/com/diffplug/spotless/extra/groovy/GrEclipseFormatterStep.java) | :+1:       | :white_large_square:       | :white_large_square:  |
 | [`java.GoogleJavaFormatStep`](lib/src/main/java/com/diffplug/spotless/java/GoogleJavaFormatStep.java) | :+1:       | :+1:      | :white_large_square:  |
@@ -89,6 +92,7 @@ extra('wtp.EclipseWtpFormatterStep')             +'{{yes}}       | {{yes}}
 | [`markdown.FreshMarkStep`](lib/src/main/java/com/diffplug/spotless/markdown/FreshMarkStep.java) | :+1:       | :white_large_square:       | :white_large_square:  |
 | [`npm.PrettierFormatterStep`](lib/src/main/java/com/diffplug/spotless/npm/PrettierFormatterStep.java) | :+1:       | :+1:      | :white_large_square:  |
 | [`npm.TsFmtFormatterStep`](lib/src/main/java/com/diffplug/spotless/npm/TsFmtFormatterStep.java) | :+1:       | :+1:      | :white_large_square:  |
+| [`python.BlackStep`](lib/src/main/java/com/diffplug/spotless/python/BlackStep.java) | :+1:       | :white_large_square:       | :white_large_square:  |
 | [`scala.ScalaFmtStep`](lib/src/main/java/com/diffplug/spotless/scala/ScalaFmtStep.java) | :+1:       | :+1:      | :white_large_square:  |
 | [`sql.DBeaverSQLFormatterStep`](lib/src/main/java/com/diffplug/spotless/sql/DBeaverSQLFormatterStep.java) | :+1:       | :white_large_square:       | :white_large_square:  |
 | [`wtp.EclipseWtpFormatterStep`](lib-extra/src/main/java/com/diffplug/spotless/extra/wtp/EclipseWtpFormatterStep.java) | :+1:       | :+1:      | :white_large_square:  |

gradle/special-tests.gradle

@@ -0,0 +1,16 @@
+def special = [
+	'Npm',
+	'Black',
+	'Clang'
+]
+
+tasks.named('test') {
+	useJUnit { excludeCategories special.collect { "com.diffplug.spotless.category.${it}Test" } as String[] }
+}
+
+special.forEach {
+	def category = "com.diffplug.spotless.category.${it}Test"
+	tasks.register("${it}Test", Test) {
+		useJUnit { includeCategories category }
+	}
+}

lib/src/main/java/com/diffplug/spotless/ForeignExe.java

@@ -0,0 +1,135 @@
+/*
+ * Copyright 2020 DiffPlug
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.diffplug.spotless;
+
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import javax.annotation.Nullable;
+
+/**
+ * Finds a foreign executable and checks its version.
+ * If either part of that fails, it shows you why
+ * and helps you fix it.
+ *
+ * Usage: `ForeignExe.nameAndVersion("grep", "2.5.7").confirmVersionAndGetAbsolutePath()`
+ * will find grep, confirm that it is version 2.5.7, and then return.
+ */
+public class ForeignExe {
+	private @Nullable String pathToExe;
+	private String versionFlag = "--version";
+	private Pattern versionRegex = Pattern.compile("version (\\S*)");
+	private @Nullable String fixCantFind, fixWrongVersion;
+
+	// MANDATORY
+	private String name;
+	private String version;
+
+	/** The name of the executable, used by "where" (win) and "which" (unix). */
+	public static ForeignExe nameAndVersion(String exeName, String version) {
+		ForeignExe foreign = new ForeignExe();
+		foreign.name = Objects.requireNonNull(exeName);
+		foreign.version = Objects.requireNonNull(version);
+		return foreign;
+	}
+
+	/** The flag which causes the exe to print its version (defaults to --version). */
+	public ForeignExe versionFlag(String versionFlag) {
+		this.versionFlag = Objects.requireNonNull(versionFlag);
+		return this;
+	}
+
+	/** A regex which can parse the version out of the output of the {@link #versionFlag(String)} command (defaults to `version (\\S*)`) */
+	public ForeignExe versionRegex(Pattern versionRegex) {
+		this.versionRegex = Objects.requireNonNull(versionRegex);
+		return this;
+	}
+
+	/** Use {version} anywhere you would like to inject the actual version string. */
+	public ForeignExe fixCantFind(String msg) {
+		this.fixCantFind = msg;
+		return this;
+	}
+
+	/** Use {version} or {versionFound} anywhere you would like to inject the actual version strings. */
+	public ForeignExe fixWrongVersion(String msg) {
+		this.fixWrongVersion = msg;
+		return this;
+	}
+
+	/** Path to the executable. If null, will search for the executable on the system path. */
+	public ForeignExe pathToExe(@Nullable String pathToExe) {
+		this.pathToExe = pathToExe;
+		return this;
+	}
+
+	/**
+	 * Searches for the executable and confirms that it has the expected version.
+	 * If it can't find the executable, or if it doesn't have the correct version,
+	 * throws an exception with a message describing how to fix.
+	 */
+	public String confirmVersionAndGetAbsolutePath() throws IOException, InterruptedException {
+		try (ProcessRunner runner = new ProcessRunner()) {
+			String exeAbsPath;
+			if (pathToExe != null) {
+				exeAbsPath = pathToExe;
+			} else {
+				ProcessRunner.Result cmdWhich = runner.shellWinUnix("where " + name, "which " + name);
+				if (cmdWhich.exitNotZero()) {
+					throw cantFind("Unable to find " + name + " on path", cmdWhich);
+				} else {
+					exeAbsPath = cmdWhich.assertExitZero(Charset.defaultCharset()).trim();
+				}
+			}
+			ProcessRunner.Result cmdVersion = runner.exec(exeAbsPath, versionFlag);
+			if (cmdVersion.exitNotZero()) {
+				throw cantFind("Unable to run " + exeAbsPath, cmdVersion);
+			}
+			Matcher versionMatcher = versionRegex.matcher(cmdVersion.assertExitZero(Charset.defaultCharset()));
+			if (!versionMatcher.find()) {
+				throw cantFind("Unable to parse version with /" + versionRegex + "/", cmdVersion);
+			}
+			String versionFound = versionMatcher.group(1);
+			if (!versionFound.equals(version)) {
+				throw wrongVersion("You specified version " + version + ", but Spotless found " + versionFound, cmdVersion, versionFound);
+			}
+			return exeAbsPath;
+		}
+	}
+
+	private RuntimeException cantFind(String message, ProcessRunner.Result cmd) {
+		return exceptionFmt(message, cmd, fixCantFind == null ? null : fixCantFind.replace("{version}", version));
+	}
+
+	private RuntimeException wrongVersion(String message, ProcessRunner.Result cmd, String versionFound) {
+		return exceptionFmt(message, cmd, fixWrongVersion == null ? null : fixWrongVersion.replace("{version}", version).replace("{versionFound}", versionFound));
+	}
+
+	private RuntimeException exceptionFmt(String msgPrimary, ProcessRunner.Result cmd, @Nullable String msgFix) {
+		StringBuilder errorMsg = new StringBuilder();
+		errorMsg.append(msgPrimary);
+		errorMsg.append('\n');
+		if (msgFix != null) {
+			errorMsg.append(msgFix);
+			errorMsg.append('\n');
+		}
+		errorMsg.append(cmd.toString());
+		return new RuntimeException(errorMsg.toString());
+	}
+}

lib/src/main/java/com/diffplug/spotless/FormatterFunc.java

@@ -40,8 +40,17 @@ interface Closeable extends FormatterFunc, AutoCloseable {
 		@Override
 		void close();
 
-		/** Creates a {@link Closeable} from an AutoCloseable and a function. */
-		public static Closeable of(AutoCloseable closeable, FormatterFunc function) {
+		/**
+		 * Dangerous way to create a {@link Closeable} from an AutoCloseable and a function.
+		 *
+		 * It's important for FormatterStep's to allocate their resources as lazily as possible.
+		 * It's easy to create a resource inside the state, and not realize that it may not be
+		 * released.  It's far better to use one of the non-deprecated `of()` methods below.
+		 *
+		 * The bug (and its fix) which is easy to write using this method: https://github.com/diffplug/spotless/commit/7f16ecca031810b5e6e6f647e1f10a6d2152d9f4
+		 * How the `of()` methods below make the correct thing easier to write and safer: https://github.com/diffplug/spotless/commit/18c10f9c93d6f18f753233d0b5f028d5f0961916
+		 */
+		public static Closeable ofDangerous(AutoCloseable closeable, FormatterFunc function) {
 			Objects.requireNonNull(closeable, "closeable");
 			Objects.requireNonNull(function, "function");
 			return new Closeable() {
@@ -52,12 +61,73 @@ public void close() {
 
 				@Override
 				public String apply(String unix, File file) throws Exception {
-					return function.apply(Objects.requireNonNull(unix), Objects.requireNonNull(file));
+					return function.apply(unix, file);
+				}
+
+				@Override
+				public String apply(String unix) throws Exception {
+					return function.apply(unix);
+				}
+			};
+		}
+
+		/** @deprecated synonym for {@link #ofDangerous(AutoCloseable, FormatterFunc)} */
+		@Deprecated
+		public static Closeable of(AutoCloseable closeable, FormatterFunc function) {
+			return ofDangerous(closeable, function);
+		}
+
+		@FunctionalInterface
+		interface ResourceFunc<T extends AutoCloseable> {
+			String apply(T resource, String unix) throws Exception;
+		}
+
+		/** Creates a {@link FormatterFunc.Closeable} which uses the given resource to execute the format function. */
+		public static <T extends AutoCloseable> Closeable of(T resource, ResourceFunc<T> function) {
+			Objects.requireNonNull(resource, "resource");
+			Objects.requireNonNull(function, "function");
+			return new Closeable() {
+				@Override
+				public void close() {
+					ThrowingEx.run(resource::close);
+				}
+
+				@Override
+				public String apply(String unix, File file) throws Exception {
+					return function.apply(resource, unix);
+				}
+
+				@Override
+				public String apply(String unix) throws Exception {
+					return function.apply(resource, unix);
+				}
+			};
+		}
+
+		@FunctionalInterface
+		interface ResourceFuncNeedsFile<T extends AutoCloseable> {
+			String apply(T resource, String unix, File file) throws Exception;
+		}
+
+		/** Creates a {@link FormatterFunc.Closeable} which uses the given resource to execute the file-dependent format function. */
+		public static <T extends AutoCloseable> Closeable of(T resource, ResourceFuncNeedsFile<T> function) {
+			Objects.requireNonNull(resource, "resource");
+			Objects.requireNonNull(function, "function");
+			return new Closeable() {
+				@Override
+				public void close() {
+					ThrowingEx.run(resource::close);
+				}
+
+				@Override
+				public String apply(String unix, File file) throws Exception {
+					FormatterStepImpl.checkNotSentinel(file);
+					return function.apply(resource, unix, file);
 				}
 
 				@Override
 				public String apply(String unix) throws Exception {
-					return function.apply(Objects.requireNonNull(unix));
+					return apply(unix, FormatterStepImpl.SENTINEL);
 				}
 			};
 		}
@@ -80,9 +150,7 @@ interface NeedsFile extends FormatterFunc {
 
 		@Override
 		default String apply(String unix, File file) throws Exception {
-			if (file == FormatterStepImpl.SENTINEL) {
-				throw new IllegalArgumentException("This step requires the underlying file. If this is a test, use StepHarnessWithFile");
-			}
+			FormatterStepImpl.checkNotSentinel(file);
 			return applyWithFile(unix, file);
 		}
 

lib/src/main/java/com/diffplug/spotless/FormatterStepImpl.java

@@ -114,4 +114,10 @@ protected String format(Integer state, String rawUnix, File file) throws Excepti
 
 	/** A dummy SENTINEL file. */
 	static final File SENTINEL = new File("");
+
+	static void checkNotSentinel(File file) {
+		if (file == SENTINEL) {
+			throw new IllegalArgumentException("This step requires the underlying file. If this is a test, use StepHarnessWithFile");
+		}
+	}
 }