Option to run Smithy CLI without isolated classpath?

[jeskew]

Thanks for this plugin! I've been using it without issue on OS X and Linux environments, but a colleague on Windows is getting classpath errors from the smithyBuildJar task. A typical error looks like this:

> Task :model:smithyBuildJar FAILED
Running smithy build
CLI classpath JAR does not exist: C:\Users\tooth\Workspaces\psf-api\model\C
CLI classpath JAR does not exist: C:\Users\tooth\Workspaces\psf-api\model\Users\tooth\.gradle\caches\modules-2\files-2.1\software.amazon.smithy\smithy-openapi\1.0.8\cbebc91325ac824ccc71a0738aeeab6a3b828b12\smithy-openapi-1.0.8.jar;C
CLI classpath JAR does not exist: C:\Users\tooth\Workspaces\psf-api\model\Users\tooth\.gradle\caches\modules-2\files-2.1\software.amazon.smithy\smithy-build\1.0.8\1943056b6e6d61e51b16396d4343b9d6fcf3c9a5\smithy-build-1.0.8.jar;C
CLI classpath JAR does not exist: C:\Users\tooth\Workspaces\psf-api\model\Users\tooth\.gradle\caches\modules-2\files-2.1\software.amazon.smithy\smithy-jsonschema\1.0.8\969868b1cc2c693001f33093f46d07df597ed362\smithy-jsonschema-1.0.8
.jar;C
CLI classpath JAR does not exist: C:\Users\tooth\Workspaces\psf-api\model\Users\tooth\.gradle\caches\modules-2\files-2.1\software.amazon.smithy\smithy-aws-traits\1.0.8\4c75784ac723e64b88f7f8ce39b9633582a48444\smithy-aws-traits-1.0.8
.jar;C
CLI classpath JAR does not exist: C:\Users\tooth\Workspaces\psf-api\model\Users\tooth\.gradle\caches\modules-2\files-2.1\software.amazon.smithy\smithy-model\1.0.8\97ec736517441a7ea61e505b6fea5c555e3c4685\smithy-model-1.0.8.jar;C
CLI classpath JAR does not exist: C:\Users\tooth\Workspaces\psf-api\model\Users\tooth\.gradle\caches\modules-2\files-2.1\software.amazon.smithy\smithy-utils\1.0.8\27d4c21862631385c91749c228cff090e8bafffd\smithy-utils-1.0.8.jar
CLI classpath JAR does not exist: C:\Users\tooth\Workspaces\psf-api\model\Users\tooth\.gradle\caches\modules-2\files-2.1\software.amazon.smithy\smithy-gradle-plugin\0.5.1\4ed8279d17a81f9f9fbbc30789d8dd23ea85694c\smithy-gradle-plugin
-0.5.1.jar;C
CLI classpath JAR does not exist: C:\Users\tooth\Workspaces\psf-api\model\Users\tooth\.gradle\caches\modules-2\files-2.1\software.amazon.smithy\smithy-cli\1.0.8\de64615e551d89ce77649d37d2d72cbc73a5791c\smithy-cli-1.0.8.jar;C
CLI classpath JAR does not exist: C:\Users\tooth\Workspaces\psf-api\model\Users\tooth\.gradle\caches\modules-2\files-2.1\software.amazon.smithy\smithy-linters\1.0.8\57410fcb69eeb9e9fdae26dbf5219262d0bdde1\smithy-linters-1.0.8.jar;C
CLI classpath JAR does not exist: C:\Users\tooth\Workspaces\psf-api\model\Users\tooth\.gradle\caches\modules-2\files-2.1\software.amazon.smithy\smithy-diff\1.0.8\5fc218b017f3737695d4ac724f61257afe02e414\smithy-diff-1.0.8.jar;C
software.amazon.smithy.cli.Cli

FAILURE: Build failed with an exception.

Gradle seems to be prepending the module root (C:\Users\tooth\Workspaces\psf-api\model) to the enumerated classpath when those entries start with a backslash. I ran the plugin in a debugger and saw that the fileset stored by the collection passed to executeCliThread is correct (e.g., the entries look like \Users\tooth\.gradle\caches\modules-2\files-2.1\software.amazon.smithy\smithy-aws-traits\1.0.8\4c75784ac723e64b88f7f8ce39b9633582a48444\smithy-aws-traits-1.0.8 .jar;C), but when getFiles is called, the classpath entries get the unwanted prefix of the module root path, and when the CLI is invoked with that classpath, it of course can't find any files. I don't think this is due to anything in the plugin itself (at least not as far as I could tell) but may be a more general bug in how classpath entries get translated into a FileCollection in Gradle's plugin support classes.

My ask here is, could you add a configuration flag to the plugin to not run the Smithy CLI in an isolated manner? I see I can choose between a forked process or a Java thread, but both of those options use an incorrect classpath on Windows.

[vinoddin]

@jeskew ,

Happened to be playing around with Smithy over the weekend, and ran into this too. I've managed to get past the specific issue by commenting out the inclusion of protocol tests in the main settings file because all i wanted was a built smithy binary. That might not work for your friend temporarily, but if it does, there you go.

@JordonPhillips

I wonder if the issue itself is because of smithy-gradle-plugin's does this in SmithyBuildJar:

        builder.libClasspath(SmithyUtils.getClasspath(getProject(), RUNTIME_CLASSPATH).getAsPath());
        builder.buildClasspath(SmithyUtils.getBuildscriptClasspath(getProject()).getAsPath());

...which is passed to smithy-cli's BuildParameterBuilder that does this:
`

    this.buildClasspath.addAll(splitAndFilterString(":", buildClasspath));

`
getAsPath returns a platform-specific string, and splitting by ":" doesnt work on Windows, likely? This might need fixing even if it's not the cause of this issue, btw, since it's reading in platform-specific strings and splitting them by a unix-only delimiter.

Just a suggestion, I couldnt get all the repos building to local maven and gradle repos/caches to actually repro/confirm.

[vinoddin]

Update: I did get the local repos/caches working and after changing all use of ":" with System.getProperty("path.separator") in the two classes mentioned above, I'm past the CLI classpath JAR does not exist problem.

However, re-enabling protocol tests still fails with tons of errors like pasted below, dont know enough of Smithy's model resolution logic in smithy-build to understand if my fix caused it, or if its unrelated.

ERROR: aws.protocoltests.json10#NoInputAndNoOutput (Model)
     @ C:\Users\vinoddin\Documents\Official\projects\smithy\smithy-aws-protocol-tests\model\awsJson1_0\empty-input-output.smithy
     |
  17 | apply NoInputAndNoOutput @httpRequestTests([
     |                                            ^
     = Unable to resolve trait `smithy.test#httpRequestTests`. If this is a custom trait, then it must be defined before it can be used in a model.


DANGER: - (SyntacticShapeIdTarget)
     @ C:\Users\vinoddin\Documents\Official\projects\smithy\smithy-aws-protocol-tests\model\awsJson1_0\empty-input-output.smithy
     |
  21 |         protocol: awsJson1_0,
     |                   ^
     = Syntactic shape ID `awsJson1_0` does not resolve to a valid shape ID: `aws.protocols#awsJson1_0`. Did you mean to quote this string? Are you missing a model file?

<snip>
Validation result: 228 ERROR(s), 444 DANGER(s), 0 WARNING(s), 0 NOTE(s)
Validated 1090 shapes in model
The model is invalid: 228 ERROR(s), 444 DANGER(s)
Error executing Smithy CLI (ReflectiveOperationException)
java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at software.amazon.smithy.gradle.SmithyUtils.lambda$executeCliThread$2(SmithyUtils.java:279)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: software.amazon.smithy.cli.CliError: The model is invalid: 228 ERROR(s), 444 DANGER(s)
	at software.amazon.smithy.cli.commands.Validator.validate(Validator.java:86)
	at software.amazon.smithy.cli.commands.CommandUtils.buildModel(CommandUtils.java:47)
	at software.amazon.smithy.cli.commands.BuildCommand.execute(BuildCommand.java:109)
	at software.amazon.smithy.cli.Cli.run(Cli.java:144)
	at software.amazon.smithy.cli.SmithyCli.run(SmithyCli.java:93)
	at software.amazon.smithy.cli.SmithyCli.run(SmithyCli.java:78)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at software.amazon.smithy.gradle.SmithyUtils.lambda$executeCliThread$2(SmithyUtils.java:279)

[jeskew]

Update: I did get the local repos/caches working and after changing all use of ":" with System.getProperty("path.separator") in the two classes mentioned above, I'm past the CLI classpath JAR does not exist problem.

That would explain the trailing ;C on the classpath entries.

In the meantime, I’ve been using the Smithy CLI directly as a Gradle task so that the project will build on Windows:

configurations {
    smithy
}

dependencies {
    smithy("software.amazon.smithy:smithy-cli:1.0.8")
    smithy("software.amazon.smithy:smithy-model:1.0.8")
    smithy("software.amazon.smithy:smithy-openapi:1.0.8")
}


task build(type: JavaExec) {
    classpath configurations.smithy
    main = "software.amazon.smithy.cli.SmithyCli"
    args("build",
        "--config",
        file("$project.projectDir/smithy-build.json").toString(),
        "--output",
        file("$project.buildDir/smithyprojections/model").toString(),
        "--discover",
        file("$project.projectDir/src/main/smithy").toString())
}

The main drawback of this approach is that the model will be recompiled every time the build target is invoked (as opposed to only when the model files change as happens with the smithy Gradle plugin).

[mtdowling]

Thanks for the help on this, @vinoddin.

I've got two PRs that address this. After merging them, I'll fire it up in Windows to see if I can fix any remaining issues:

• Use System's path separator for classpaths smithy#575
• Use System's path separator for classpaths #28

After that, we'll see if we can set our tests up to automatically run on Windows too.

Update: With the changes in https://github.com/awslabs/smithy/pull/576/files and the above two PRs, I was able to get the Gradle plugin to build and pass all tests on Windows. Smithy's Gradle tests are pretty in-depth integration tests, so this should be working once those are all merged and we cut a release. There are still some failing tests in smithy-build and smithy-cli, but those seem to just be issues we need to fix specifically in the tests.

[JordonPhillips]

Both the main smithy repo and this repo now have windows ci runnning. As mentioned above, there were a few more things that needed touching up on the main smithy repo. If you have any other problems, don't hesitate to open up another issue!