Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build config: Use buildSrc convention plugins to configure the Dokka subprojects #2704

Merged
merged 65 commits into from
Mar 7, 2023

Conversation

aSemy
Copy link
Contributor

@aSemy aSemy commented Oct 11, 2022

This is a large part of the implementation for #2703

Status: ready for review

  • set up buildSrc convention plugins
  • move shared configuration into the convention plugins
  • tidy up the subproject build files to remove duplicated configuration
  • set up centralized repo definitions

Important to note:

  • This should have no impact on the actual source code or published artifacts, this is only updating the build config
  • All Gradle plugins must be defined in ./buildSrc/build.gradle.kts, so they are aligned across all subprojects

Related

@aSemy aSemy marked this pull request as draft October 11, 2022 17:58
@aSemy
Copy link
Contributor Author

aSemy commented Oct 11, 2022

I made some good progress but then I got stuck when I reached the integration test config, because it's a bit of a beast.

https://github.com/Kotlin/dokka/blob/83174361becb2af227159834cdf6e14db9300c53/integration-tests/build.gradle.kts

I think most of the config in here should be converted to a specific 'integration test' buildSrc plugin.

@IgnatBeresnev IgnatBeresnev added the runner: Gradle plugin An issue/PR related to Dokka's Gradle plugin label Oct 12, 2022
@aSemy
Copy link
Contributor Author

aSemy commented Oct 12, 2022

@IgnatBeresnev I see you've added the 'Gradle Plugin' label, and while this PR would help with that, it's more about the improving the general Gradle build config of the entire project :)

@IgnatBeresnev
Copy link
Member

IgnatBeresnev commented Oct 12, 2022

Yeah, true. I don't want to create too many labels, but I definitely want to mark all Gradle-related PRs in case we ask for anyone's help with review and whatnot

If it makes anyone uneasy, we can certainly create a separate one or rename the existing. We'll discuss it with the team this week, just in case

@aSemy
Copy link
Contributor Author

aSemy commented Oct 13, 2022

With regards to labels, what about a 'build config' or 'ci/cd' label that would cover the build scripts, and also GitHub Actions, Qodana, TeamCity etc?

@aSemy
Copy link
Contributor Author

aSemy commented Oct 13, 2022

I've made some good progress! All shared build-config now comes from buildSrc convention plugins, not from subprojects{} or allprojects{} blocks. This makes the build much more modular, because it's not restricted by inheritance, and easier to configure because a build script is more self contained - it doesn't have config applied from some arbitrary parent.

@aSemy aSemy marked this pull request as ready for review October 18, 2022 10:08
@aSemy aSemy marked this pull request as draft October 18, 2022 10:42
@aSemy aSemy marked this pull request as ready for review October 18, 2022 11:15
@IgnatBeresnev
Copy link
Member

IgnatBeresnev commented Mar 7, 2023

Hi! Not sure if you noticed that the integration tests for coroutines are unable to be run because of the toolchain being set to 1.8 globally. This refactoring has uncovered quite a number of problems already, which is good, I guess :)

A problem occurred evaluating root project 'kotlinx.coroutines'.
> Failed to apply plugin 'jdk-convention'.
> Project required JDK 11+, but found 1.8

Can you please fix it when you have the time, so that the checks pass? I believe that's the last thing holding up the merge

Not sure what the best solution for the fix is, but I'd say any that builds the same bytecode as before and makes the tests pass. We can revisit our requirements for the used java/kotlin versions later and separately

@aSemy
Copy link
Contributor Author

aSemy commented Mar 7, 2023

thanks for checking @IgnatBeresnev, I didn't realise that was the problem.

I set the toolchain for Dokka to use Java 8 because that matches the existing jvmTarget

jvmTarget = "1.8"

If I set the toolchain to 11, then the :runners:maven-plugin build fails:

> Task :runners:maven-plugin:helpMojo
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building dokka-maven-plugin 1.8.20-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-plugin-plugin:3.5.2:helpmojo (default-cli) @ dokka-maven-plugin ---
> Task :plugins:base:frontend:nodeSetup
> Task :plugins:base:frontend:npmSetup SKIPPED
> Task :runners:maven-plugin:helpMojo
[WARNING] Using platform encoding (UTF-8 actually) to read mojo source files, i.e. build is platform dependent!
[INFO] java-javadoc mojo extractor found 0 mojo descriptor.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.425 s
[INFO] Finished at: 2023-03-07T16:44:34+01:00
[INFO] Final Memory: 12M/68M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-plugin-plugin:3.5.2:helpmojo (default-cli) on project dokka-maven-plugin: Execution default-cli of goal org.apache.maven.plugins:maven-plugin-plugin:3.5.2:helpmojo failed: Unsupported class file major version 55 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-plugin-plugin:3.5.2:helpmojo (default-cli) on project dokka-maven-plugin: Execution default-cli of goal org.apache.maven.plugins:maven-plugin-plugin:3.5.2:helpmojo failed: Unsupported class file major version 55

And there's a warning for :runners:gradle-plugin

> Task :runners:gradle-plugin:compileTestKotlin
'compileTestJava' task (current target is 11) and 'compileTestKotlin' task (current target is 1.8) jvm target compatibility should be set to the same Java version.
By default will become an error since Gradle 8.0+! Read more: https://kotl.in/gradle/jvm/target-validation
Consider using JVM toolchain: https://kotl.in/gradle/jvm/toolchain

It looks like kotlinx.coroutines has a custom check for the JDK version...

https://github.com/Kotlin/kotlinx.coroutines/blob/4116d4a178f32b1241db8247a38cd2823fb8b03e/buildSrc/src/main/kotlin/jdk-convention.gradle.kts#L7-L14

I'll look into it a little more. Perhaps the kotlinx.coroutines int. test requires a little more config.

aSemy added 5 commits March 7, 2023 16:46
# Conflicts:
#	build.gradle.kts
Merge pull request Kotlin#3582 from isaialvarado/patch-1

Fix example code
Converted Serialization formats list to table

PR Kotlin#2010
Updating docs after 7.0.0 release
@IgnatBeresnev
Copy link
Member

IgnatBeresnev commented Mar 7, 2023

I set the toolchain for Dokka to use Java 8 because that matches the existing jvmTarget

Yeah, that's the most important part - we cannot break compatibility with Java 8 for now.

We can set jvmTarget to 8 in the kotlin convention + the gradle plugin, but run the whole project with java 11 - I don't mind that. Adding the toolchain is definitely a good idea, even if jvmTarget stays at 8

But whatever works is fine :) We can revisit the used versions later

Tell me once it's ready to be looked at, I'll trigger the teamcity build

@aSemy
Copy link
Contributor Author

aSemy commented Mar 7, 2023

@IgnatBeresnev Do you know why the jvm-version is set to 12 in the example workflows?

@aSemy
Copy link
Contributor Author

aSemy commented Mar 7, 2023

We can set jvmTarget to 8 in the kotlin convention + the gradle plugin, but run the whole project with java 11 - I don't mind that. Adding the toolchain is definitely a good idea, even if jvmTarget stays at 8

Toolchains don't let you separately define a language version; the JVM target is set to that of the Java version of the Toolchain. So if the project needs to target Java 8, the Toolchain needs to be Java 8. Don't ask me why jvmTarget can't be set separately...

Fortunately Toolchains are very flexible, so I set the integrationTest task launcher to be Java 11.

1bbbd9b

@IgnatBeresnev Tests have passed, it's ready for the TeamCity build!

@IgnatBeresnev
Copy link
Member

IgnatBeresnev commented Mar 7, 2023

@IgnatBeresnev Do you know why the jvm-version is set to 12 in the example workflows?

It was copied from s3-snapshots.yml, but why it's set there to 12 - zero clue :( I think it's fine to set it to 11 like you did

@IgnatBeresnev Tests have passed, it's ready for the TeamCity build!

Thank you once again! Triggered the builds, will merge once they pass :)

@aSemy
Copy link
Contributor Author

aSemy commented Mar 8, 2023

@IgnatBeresnev By the way, good job on the GitHub labels! They look really smart, and they're helpful.

@IgnatBeresnev
Copy link
Member

@aSemy yeah, going through all of the issues :) Part of the reason is actually for the dokkatoo DSL. In particular, the configuration and the Gradle plugin tags could be of interest to go along with the re-write since we're breaking compatibility anyway

Comment on lines +14 to +18
java {
toolchain {
languageVersion.set(JavaLanguageVersion.of(8))
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we apply toolchain for all modules, they'll run tests on Java 8 instead of the JDK matrix on CI, we need to apply more toolchains in Test which will slow down the build, I believe we should not use toolchain for now. Check

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

Toolchains can still help with this though, because it's possible to set them per-task. I did something similar for MockK - I'll put together a PR.

Copy link
Contributor

@Goooler Goooler Mar 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right, yes that would be slower. I agree, using GitHub parallel workflows is best, not multiple Gradle tasks.

@aSemy aSemy deleted the feat/buildsrc_convention_plugins branch March 13, 2023 10:31
@aSemy aSemy mentioned this pull request Mar 16, 2023
4 tasks
@ilya-g
Copy link
Member

ilya-g commented Apr 6, 2023

Hi @aSemy.
I've noticed that after this change the artifact dokka-analysis started publishing an additional shadowRuntimeElements variant with the following attributes:

    {
      "name": "shadowRuntimeElements",
      "attributes": {
        "org.gradle.category": "library",
        "org.gradle.dependency.bundling": "shadowed",
        "org.gradle.libraryelements": "jar",
        "org.gradle.usage": "java-runtime"
      },
      "files": [
        {
          "name": "dokka-analysis-1.8.20-local-1-all.jar",
          "url": "dokka-analysis-1.8.20-local-1-all.jar",
          "size": 100899229,
          ...
        }
      ]
    }

This is unfortunate, because it causes variant resolution ambiguity with the runtimeElements variant in ...Plugin configurations of manually created DokkaTaskPartial tasks in the stdlib docs build.

Is this an expected consequence of this change?

  • if no, could you investigate why it has happened and how to stop publishing this variant?
  • if yes, could you provide a guidance how to overcome this variant ambiguity in a project?

@aSemy
Copy link
Contributor Author

aSemy commented Apr 6, 2023

I will take a look! I'm not very familiar with the Shadow plugin, but it looks like there are a few issues relating to shadowRuntimeElements https://github.com/johnrengelman/shadow/issues?q=shadowRuntimeElements

What might be a quick fix is disabling publication of this specific configuration, with a similar way to how Test Fixtures publication can be disabled.

val javaComponent = components["java"] as AdhocComponentWithVariants
javaComponent.withVariantsFromConfiguration(configurations["testFixturesApiElements"]) { skip() }
javaComponent.withVariantsFromConfiguration(configurations["testFixturesRuntimeElements"]) { skip() }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Everything related to builds tools, CI configurations and project tooling
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants