Feature request: make the KarateRuntimeOptions more flexible

[celcius112]

Hello Peter,

I have a Spring Boot application in which I launch my Karate tests at runtime, meaning that the cucumber features will be bundled in a jar (or a bootJar to be more precise). I'm using the Cucumber's parallel runner (CucumberRunner).

Since Spring Boot manages by itself its jar, the MultiLoader used by KarateRuntimeOptions will not be able to discover the features on the classpath, as described in this issue.

I was able to use my own ResourceLoader, and everything works fine, but I had to extend KarateRuntimeOptions and pretty much override everything and modify the class loader, which is not very lenient in case of any modification in the current API.

Sorry if I don't have a very precise request, since I'm not sure what's the best way to resolve that, and for having a pretty farfetched use-case, but I can say for sure that on the long term it might be a good idea to make KarateRuntimeOptions a bit more flexible.

If you want a demo project with my use-case and my solution, I'm more than willing to provide it.

Thank you very much

[ptrthomas]

@celcius112 can you please do me a favor, check out the 'cukexit' branch and try.

the file loading has been completely re-written and as far as I can tell, it should support loading from a JAR file as well as the normal classpath.

would appreciate your help here :)

https://github.com/intuit/karate/blob/cukexit/karate-core/src/main/java/com/intuit/karate/FileUtils.java#L341

[celcius112]

Oh yeah, saw your tweet on removing Cucumber :)
I'll try your branch with pleasure. Thank you very much and I'll come back to you soon

[celcius112]

So I tried the most recent version of you branch. There seems to be some issues at the moment:

1. When launching locally, on IntelliJ and with Gradle, Karate is not able to find the .feature files in the current classpath. It will try to find the files in the /out/production/classes directory, but will only find .classes files, since .features files are moved to a special /out/production/resources directory that is not taken into account now. I'm not sure whether it is a Gradle or Intellij-specific error, but previously the Cucumber's ClasspathResourceIterable that was wrapping a URLClassLoader was able to resolve the resources.
2. In a packaged application, when using CucumberRunner#parallel(Class<?>, int, String) and resolving the class relative to classpath (using FileUtils#toRelativeClassPath(Class)), I've got a odd ProviderMismatchException in FileUtils#toRelativeClassPath(File) when trying to relativize the rootPath (the rootPath being resolved to / by getClassPathRoot()).
3. Again in a packaged application, when using CucumberRunner#parallel(List<String>, List<String>, int, String), with the second parameter (paths) being a singleton list of classpath: + package of the class launching the tests, the method FileUtils#getPathFor(String) will be called twice in FileUtils#scanForFeatureFiles(String), and will trigger twice FileSystems.newFileSystem with the second time logically failing with a FileSystemAlreadyExistsException.

Would you like a sample project in which we can easily reproduce these steps ?

[ptrthomas]

@celcius112 awesome, thanks ! So by "packaged" application, I assume you mean it is a JAR ?

ideally I'd like the sample project to be included in the gradle project side-by-side with the karate-demo maven project: https://github.com/intuit/karate/blob/master/karate-demo/build.gradle

I was resisting running this as part of CI until now but maybe we should.

Hopefully even the creation of a JAR can be stuffed into the same project and we can run some tests to test all scenarios, would you be able to take a look at those ?

By the way I'm right now working on getting the Intelli-J "right-click-and-run" working with the new Cucumber-free situation. As of now the IDE results (green bar etc) is broken, but do see if you spot any other issues !

[ptrthomas]

@celcius112 nevermind, tried to work on all 3 issues and I think fixed.

checked in a tiny jar file with some feature-files bundled, including relative-path call complications. here is the test: https://github.com/intuit/karate/blob/ef22ef37fb4e2511c4bb2e9a6eb182d2dd1b8daf/karate-junit4/src/test/java/com/intuit/karate/junit4/files/JarLoadingTest.java

[celcius112]

Thank you @ptrthomas !

So by "packaged" application, I assume you mean it is a JAR ?

Yep exactly :).

So if we come back to my 3 points:

1. Works well for me ! But I still had to add a small but weird modification in FileLogAppender#collect. If I run locally with your branch as it is, I've got a java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip in the bug.flip() call in the aforementioned method. I've found this Github comment that proposes to use the strange ((Buffer)byteBuffer).flip(), which works for me. I say strange because the variable buf is of type HeapByteBuffer at runtime, which inherits from ByteBuffer and Buffer and should thus have access to flip(). For your interest I'm running this program on JDK 8.
2. The small bugs in points 2 and 3 seem to be fixed :) However we are still coming back to my initial point, which is to be able to detect the feature files in a Spring Boot jar. Unfortunately it has the same behavior as before. But with your code you were still able to provide more information for my research, and I found this Spring Boot issue that seems to explain that ZipFileSystem has few limitations regarding fully-executable jar. Andy Wilkinson proposes this Spring-specific solution, which is also my current solution for resolving feature files and works when trying it on your branch by replacing Files.walk(searchPath) in collectFeatureFilesFromJar(). Again it seems to be very Spring-Boot-specific, and might be difficult to implement as it is in Karate, but maybe we can find a way to plug a lenient solution for those in need 😢 ?
   Here's another related issue: Support java.nio.file Paths and FileSystems with nested jars spring-projects/spring-boot#7161

[ptrthomas]

@celcius112 great ! I think it is time you submitted a PR. I've honestly reached the limit of my patience struggling with the weirdness of the nio Path API all weekend :|

[celcius112]

@ptrthomas ok, hold my 🍺 (might take some time though)

[jlee70]

whoops, I should have read this thread...

[ptrthomas]

@celcius112 @jlee70 - I took another look at this and see the last commit above, I think I have been able to solve scanning within JAR files. we were missing adding anything in JAR files to classpath scanning earlier.

would be great if you can build fro source and test.

[celcius112]

Hello @ptrthomas,
I couldn't test you classpath modification thoroughly for diverse reasons, but I did find a possible issue.

I have a feature file being called in my karate-config.js. This file is correctly called, but the Scenario step seems to be blocked indefinitely without any exception thrown. The Background step seems to work correctly though.
Further information:

• I call this feature file with karate.call()
• I'm passing some parameters, which are usable in the Background step (so I know they are correctly passed)
• I'm calling the feature file using the classpath: classpath:path/file.feature. I know it is correctly called since the Background step is launched

[ptrthomas]

@celcius112 sorry I won't be able to look at your issue unless you submit a full working example as per the instructions here: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue

[jlee70]

Hello @ptrthomas.
Loading from jar worked fine for me. Now, I am trying to run feature with Report.
This procedure for me seem to work to generate report.
Please correct me if I am doing wrong thing.

I was following JarLoadingTest in the com.intuit.karate.junit4.files package.

 Feature feature = FeatureParser.parse(resource);
 CallContext callContext = new CallContext(null, true);
//I ended up using Engine because I need FeatureResult to generate report.
 FeatureResult result = Engine.executeFeatureSync(feature, (String)null, callContext);
 File resultFile = Engine.saveResultHtml("target/surefire-reports/", result);

In my prototype with above procedure, I am able to serve generated html in boot app and seems to work for my need and nice.

It will be nice if I can specify file name in Engine.saveResultHtml call. So I can bind dynamic test run with report.

Also, not sure what is a proper way to get karate-config.js in java. Is there some converter util method between karate-config.js and java map. config.js is not pure map and value I assume.
It can have javascript function...etc.. Maybe ability to specify config file in CallContext Constructor would be nice. So I can specify config file dynamically in known location and config.

thank you very much.

[ptrthomas]

@jlee70 I made the changes so you can override the filename for the report artifacts.

I don't understand your question on karate-config.js

[jlee70]

oh thank you @ptrthomas.

For karate-config.js, according to my debug tracing, karate-configuration gets loaded through

public ScenarioContext(FeatureContext featureContext, CallContext call) {
...
if (call.parentContext == null && call.evalKarateConfig) {
            // base config is only looked for in the classpath
            try {
                Script.callAndUpdateConfigAndAlsoVarsIfMapReturned(false, ScriptBindings.READ_KARATE_CONFIG_BASE, null, this);
            } catch (Exception e) {
                if (e instanceof KarateFileNotFoundException) {
                    logger.trace("skipping 'classpath:karate-base.js': {}", e.getMessage());
                } else {
                    throw new RuntimeException("evaluation of 'classpath:karate-base.js' failed", e);
                }
            String configDir = System.getProperty(ScriptBindings.KARATE_CONFIG_DIR);
            String configScript = ScriptBindings.readKarateConfigForEnv(true, configDir, null);
            try {
                Script.callAndUpdateConfigAndAlsoVarsIfMapReturned(false, configScript, null, this);
            } catch (Exception e) {
                if (e instanceof KarateFileNotFoundException) {
                    logger.warn("skipping bootstrap configuration: {}", e.getMessage());
                } else {
                    throw new RuntimeException("evaluation of '" + ScriptBindings.KARATE_CONFIG_JS + "' failed", e);
                }
            }
            //maybe third configuration loading here if callcontext has custom config file location.
            //then load that custom config with specified file location?
            
...

and whoops I did not read code completely... there is a way to specify configuration folder and loaded config from the folder... I will try that approach first before asking new feature.
However, this should enable me to do dynamic configuration generation through web-ui and able to run test with dynamic configuration file.

thanks

[ptrthomas]

0.9.0 released