[JENKINS-57928] Allow configuring JMH benchmarks using CASC #921

AbhyudayaSharma · 2019-06-11T10:55:20Z

Your checklist for this pull request

🚨 Please review the guidelines for contributing to this repository.

Make sure you are requesting to pull a topic/feature/bugfix branch (right side) and not your master branch!
Please describe what you did
Link to relevant GitHub issues or pull requests
Link to relevant Jenkins JIRA issues
Did you provide a test-case? That demonstrates feature works or fixes the issue.

Description

Adds support for running JMH benchmarks with temporary instances configured using Configuration as Code. This is a direct follow-up to jenkinsci/jenkins-test-harness#135.

JIRA issue: https://issues.jenkins-ci.org/browse/JENKINS-57928

@jenkinsci/gsoc2019-role-strategy

AbhyudayaSharma · 2019-06-11T12:33:59Z

I'm not sure about the Codacy review. The test method will throw an error if the code doesn't behave properly so I don't think there is any need for an assert or a fail().

jetersen

LGTM

plugin/src/test/java/io/jenkins/plugins/casc/CascJmhBenchmarkStateTest.java

plugin/src/test/java/io/jenkins/plugins/casc/SampleBenchmark.java

jetersen · 2019-06-14T06:26:29Z

plugin/src/test/java/io/jenkins/plugins/casc/misc/jmh/CascJmhBenchmarkState.java

+     * @throws ConfiguratorException when unable to configure
+     */
+    @Override
+    public void setup() throws Exception {


for setup, I suggest passing this so the resource loader will work for the current benchmark 😓
I don't know if there is a way to do it inside JUnit / benchmark framework

This how it is used here:

configuration-as-code-plugin/plugin/src/test/java/io/jenkins/plugins/casc/misc/Util.java

Lines 138 to 160 in 08b1a48

* Reads a resource from the classpath to use in asserting expected export content.

* <p>

* The resource is expected to be UTF-8 encoded.

* <p>

* Example usage:

* <pre>{@code

* toStringFromYamlFile(this, "expected-output.yml");}</pre>

*

* @param clazz pass in {@code this}

* @param resourcePath the file name to read, should be in the same package as your test class in resources

* @return the string content of the file

* @throws URISyntaxException invalid path

* @throws IOException invalid path or file not found in general

*/

public static String toStringFromYamlFile(Object clazz, String resourcePath) throws URISyntaxException, IOException {

URL resource = clazz.getClass().getResource(resourcePath);

if (resource == null) {

throw new FileNotFoundException("Couldn't find file: " + resourcePath);

}

byte[] bytes = Files.readAllBytes(Paths.get(resource.toURI()));

return new String(bytes, StandardCharsets.UTF_8).replaceAll("\r\n?", "\n");

}

plugin/src/test/java/io/jenkins/plugins/casc/SampleBenchmark.java

AbhyudayaSharma · 2019-06-14T07:47:56Z

@Casz I've made it work though it's not too pretty. The problem is that we cannot pass an object to a benchmark as it is run in a different JVM. Also, the class that will extend CascJmhBenchmarkState would completely transformed when the benchmarks are run, so we cannot use getClass().getEnclosingClass().

Also, I don't know why the Travis builds are failing. Reflections should be present as it is a dependency of Jenkins Test Harness now.

jetersen · 2019-06-14T07:56:36Z

plugin/src/test/java/io/jenkins/plugins/casc/misc/jmh/CascJmhBenchmarkState.java

-        String config = Objects.requireNonNull(getClass().getClassLoader().getResource(getResourcePath())).toExternalForm();
+        Class<?> enclosingClass = getEnclosingClass();
+
+        if (!enclosingClass.isAnnotationPresent(JmhBenchmark.class)) {


jetersen · 2019-06-14T07:59:20Z

plugin/src/test/java/io/jenkins/plugins/casc/SampleBenchmark.java

+        @Nonnull
+        @Override
+        protected Class<?> getEnclosingClass() {
+            return SampleBenchmark.class;


Might as well return this; or does that not work?

That doesn't work. Like I said above io.jenkins.plugins.casc.SampleBenchmark becomes io.jenkins.plugins.casc.generated.SampleBenchmark_MyState_jmhType

AbhyudayaSharma · 2019-06-14T16:43:59Z

@Casz Is it ready for merging into master?

jetersen · 2019-06-15T12:18:55Z

No CI and Codacy checks failed 😓

AbhyudayaSharma · 2019-06-15T13:15:31Z

I need help here. The Jenkins build seems to fail just because the Docker container does not have enough space.

[2019-06-15T13:04:55.209Z] [ERROR] io.jenkins.plugins.casc.vault.VaultSecretSourceTest  Time elapsed: 2.348 s  <<< FAILURE!

[2019-06-15T13:04:55.209Z] java.lang.AssertionError: Docker environment should have more than 2GB free disk space

The Travis build on the other hand seems to have it's build cache corrupted. Reflections comes from Jenkins Test Harness now and should be present on the classpath. Any ideas?

jetersen · 2019-06-15T13:20:42Z

Regarding Jenkins: https://issues.jenkins-ci.org/browse/INFRA-2062

Regarding Travis seems to resolve to a different version of Guava than that of Jenkins

AbhyudayaSharma · 2019-06-15T13:34:37Z

Thanks @Casz ! Now what should I do about Travis?

jetersen · 2019-06-15T13:48:32Z

ronmamo/reflections#219

This should be fixed in Jenkins test harness?

jetersen · 2019-06-15T13:50:03Z

Or just say no to guava CC @oleg-nenashev

oleg-nenashev · 2019-06-16T15:51:17Z

Yes, likely a JTH fix needed. Is there a suitable reflections version which is compatible with guava inside the core?

AbhyudayaSharma · 2019-06-17T05:20:15Z

I have created a pull request to Jenkins Test Harness using ~~ClassGraph~~ SezPoz.

jenkinsci/jenkins-test-harness#141

jetersen · 2019-06-17T07:55:54Z

@AbhyudayaSharma the incremental is ready: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/jenkins-test-harness/2.51-rc1188.061ed497968c/

AbhyudayaSharma · 2019-06-17T07:58:06Z

@Casz Yes, thanks! I'm just waiting for tests on my machine to finish.

plugin/src/test/java/io/jenkins/plugins/casc/CascJmhBenchmarkStateTest.java

jetersen · 2019-06-17T08:15:46Z

all green except for codacy 😆

…tateTest.java Co-Authored-By: Joseph Petersen <josephp90@gmail.com>

jetersen · 2019-06-17T09:09:18Z

Now just waiting on a release of JTH 😓

AbhyudayaSharma · 2019-06-17T09:13:44Z

Oleg said in the Role Strategy Plugin Gitter chat that he will try to release a new JTH version (and plugin-pom) today.

AbhyudayaSharma · 2019-06-18T07:54:28Z

Newer JTH was released: jenkinsci/plugin-pom#213

AbhyudayaSharma · 2019-06-18T08:01:33Z

Thanks a lot @Casz ! Would we be getting a release soon?

jetersen · 2019-06-18T08:03:44Z

Incremental should be available in the meantime 😅

Allow configuring JMH benchmarks using CASC

28888a2

jetersen approved these changes Jun 11, 2019

View reviewed changes

jetersen reviewed Jun 11, 2019

View reviewed changes

plugin/src/test/java/io/jenkins/plugins/casc/CascJmhBenchmarkStateTest.java Show resolved Hide resolved

timja reviewed Jun 11, 2019

View reviewed changes

plugin/src/test/java/io/jenkins/plugins/casc/SampleBenchmark.java Outdated Show resolved Hide resolved

Add documentation for benchmarks

1cfa252

oleg-nenashev mentioned this pull request Jun 13, 2019

[JENKINS-57653] Introduce JMH benchmarks to Jenkins Test Harness jenkinsci/jenkins-test-harness#135

Merged

Use released JTH and Plugin-POM

4183953

AbhyudayaSharma marked this pull request as ready for review June 14, 2019 05:31

AbhyudayaSharma changed the title ~~[JENKINS-57928] Allow configuring JMH benchmarks using CASC 🚧~~ [JENKINS-57928] Allow configuring JMH benchmarks using CASC Jun 14, 2019

AbhyudayaSharma changed the title ~~[JENKINS-57928] Allow configuring JMH benchmarks using CASC~~ [JENKINS-57928] Allow configuring JMH benchmarks using CASC 🚧 Jun 14, 2019

AbhyudayaSharma added 2 commits June 14, 2019 11:25

Fix package

5377498

Fix import

c17ba6b

jetersen requested changes Jun 14, 2019

View reviewed changes

Use the enclosing class's getResource()

54eacf9

AbhyudayaSharma changed the title ~~[JENKINS-57928] Allow configuring JMH benchmarks using CASC 🚧~~ [JENKINS-57928] Allow configuring JMH benchmarks using CASC Jun 14, 2019

jetersen reviewed Jun 14, 2019

View reviewed changes

Add assert to make Codacy happy

eae7ecc

AbhyudayaSharma closed this Jun 15, 2019

AbhyudayaSharma reopened this Jun 15, 2019

Use incrementals from jenkins-test-harness#141

828d441

jetersen reviewed Jun 17, 2019

View reviewed changes

plugin/src/test/java/io/jenkins/plugins/casc/CascJmhBenchmarkStateTest.java Outdated Show resolved Hide resolved

Update plugin/src/test/java/io/jenkins/plugins/casc/CascJmhBenchmarkS…

1527ea0

…tateTest.java Co-Authored-By: Joseph Petersen <josephp90@gmail.com>

jetersen approved these changes Jun 17, 2019

View reviewed changes

jetersen mentioned this pull request Jun 17, 2019

[JENKINS-58038] Use Annotation-Indexer instead of Reflections jenkinsci/jenkins-test-harness#141

Merged

AbhyudayaSharma added 3 commits June 17, 2019 19:46

Test with latest incremental

74ed51f

Update jmh-benchmarks.md

0314b7a

Use Plugin POM 3.46

4916512

jetersen approved these changes Jun 18, 2019

View reviewed changes

jetersen merged commit 6311ebf into jenkinsci:master Jun 18, 2019

AbhyudayaSharma deleted the jmh branch June 18, 2019 08:01

jetersen added the test A PR that adds to testing - used by Release Drafter label Jun 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JENKINS-57928] Allow configuring JMH benchmarks using CASC #921

[JENKINS-57928] Allow configuring JMH benchmarks using CASC #921

AbhyudayaSharma commented Jun 11, 2019

AbhyudayaSharma commented Jun 11, 2019

jetersen left a comment

jetersen Jun 14, 2019

AbhyudayaSharma commented Jun 14, 2019 •

edited

Loading

jetersen Jun 14, 2019

jetersen Jun 14, 2019

AbhyudayaSharma Jun 14, 2019

AbhyudayaSharma commented Jun 14, 2019

jetersen commented Jun 15, 2019

AbhyudayaSharma commented Jun 15, 2019 •

edited

Loading

jetersen commented Jun 15, 2019

AbhyudayaSharma commented Jun 15, 2019

jetersen commented Jun 15, 2019

jetersen commented Jun 15, 2019

oleg-nenashev commented Jun 16, 2019

AbhyudayaSharma commented Jun 17, 2019 •

edited

Loading

jetersen commented Jun 17, 2019

AbhyudayaSharma commented Jun 17, 2019 •

edited

Loading

jetersen commented Jun 17, 2019

jetersen commented Jun 17, 2019

AbhyudayaSharma commented Jun 17, 2019 •

edited

Loading

AbhyudayaSharma commented Jun 18, 2019

AbhyudayaSharma commented Jun 18, 2019

jetersen commented Jun 18, 2019

	* Reads a resource from the classpath to use in asserting expected export content.
	* <p>
	* The resource is expected to be UTF-8 encoded.
	* <p>
	* Example usage:
	* <pre>{@code
	* toStringFromYamlFile(this, "expected-output.yml");}</pre>
	*
	* @param clazz pass in {@code this}
	* @param resourcePath the file name to read, should be in the same package as your test class in resources
	* @return the string content of the file
	* @throws URISyntaxException invalid path
	* @throws IOException invalid path or file not found in general
	*/
	public static String toStringFromYamlFile(Object clazz, String resourcePath) throws URISyntaxException, IOException {
	URL resource = clazz.getClass().getResource(resourcePath);
	if (resource == null) {
	throw new FileNotFoundException("Couldn't find file: " + resourcePath);
	}

	byte[] bytes = Files.readAllBytes(Paths.get(resource.toURI()));
	return new String(bytes, StandardCharsets.UTF_8).replaceAll("\r\n?", "\n");
	}

[JENKINS-57928] Allow configuring JMH benchmarks using CASC #921

[JENKINS-57928] Allow configuring JMH benchmarks using CASC #921

Conversation

AbhyudayaSharma commented Jun 11, 2019

Your checklist for this pull request

Description

AbhyudayaSharma commented Jun 11, 2019

jetersen left a comment

Choose a reason for hiding this comment

jetersen Jun 14, 2019

Choose a reason for hiding this comment

AbhyudayaSharma commented Jun 14, 2019 • edited Loading

jetersen Jun 14, 2019

Choose a reason for hiding this comment

jetersen Jun 14, 2019

Choose a reason for hiding this comment

AbhyudayaSharma Jun 14, 2019

Choose a reason for hiding this comment

AbhyudayaSharma commented Jun 14, 2019

jetersen commented Jun 15, 2019

AbhyudayaSharma commented Jun 15, 2019 • edited Loading

jetersen commented Jun 15, 2019

AbhyudayaSharma commented Jun 15, 2019

jetersen commented Jun 15, 2019

jetersen commented Jun 15, 2019

oleg-nenashev commented Jun 16, 2019

AbhyudayaSharma commented Jun 17, 2019 • edited Loading

jetersen commented Jun 17, 2019

AbhyudayaSharma commented Jun 17, 2019 • edited Loading

jetersen commented Jun 17, 2019

jetersen commented Jun 17, 2019

AbhyudayaSharma commented Jun 17, 2019 • edited Loading

AbhyudayaSharma commented Jun 18, 2019

AbhyudayaSharma commented Jun 18, 2019

jetersen commented Jun 18, 2019

AbhyudayaSharma commented Jun 14, 2019 •

edited

Loading

AbhyudayaSharma commented Jun 15, 2019 •

edited

Loading

AbhyudayaSharma commented Jun 17, 2019 •

edited

Loading

AbhyudayaSharma commented Jun 17, 2019 •

edited

Loading

AbhyudayaSharma commented Jun 17, 2019 •

edited

Loading