SOLR-17406: Introduce Version Catalogs #2706
Conversation
The implementation of the eclipse plugin throws "Value is null" error when trying to sync project without the palantir consistent versions plugin. Either fix the error or remove the configuration.
Move buildSrc into a composite included build (build-infra). Expose a plugin with buildinfra extension. See apache/lucene@4afae6a
github-actions
bot
added
documentation
|
I think the next step is just some dependency version alignment? |
|
I think it would be a good idea to first resolve the current solrbot PRs before aligning the versions. I am already looking into some of the PRs to see how they can be resolved. |
…sion-catalogs # Conflicts: # build-tools/scriptDepVersions.gradle # build.gradle # gradle/validation/check-environment.gradle # solr/core/build.gradle # solr/modules/hadoop-auth/build.gradle # solr/modules/hdfs/build.gradle # solr/server/build.gradle # solr/solrj-zookeeper/build.gradle # versions.lock # versions.props
malliaridis
force-pushed
the
SOLR-17406/rebase-version-catalogs
branch
from
788d305
to
6caff99
Compare
…sion-catalogs # Conflicts: # solr/modules/opentelemetry/build.gradle # versions.lock # versions.props
|
The current state should differ now only in hamcrest-core, which is being merged with hamcrest in the |
There was a problem hiding this comment.
What prompted you to update the NOTICE files of existing dependencies? -- very thorough of you but it would not have occurred to me. I found a couple misattributed LGPL references to correct but no other issue.
I ran the bats/integration tests and docker tests, and build the ref guide, just to exercise some parts of our build that might not otherwise in the PR validation. I think we should merge this very soon. A shout-out of the change arriving to our dev list would be helpful for general awareness and its impacts (e.g. dependency upgrades are now different and won't backport).
solr/licenses/netty-NOTICE.txt
Outdated
| * license/LICENSE.jboss-marshalling.txt (GNU LGPL 2.1) | ||
| * license/LICENSE.jboss-marshalling.txt (Apache License 2.0) |
There was a problem hiding this comment.
I checked this -- looks accurate. Then I got curious and searched out entire licenses folder for "LGPL". netty-transport-native-epoll-NOTICE.txt (and unix-common-NOTICE.txt) also makes a reference to LGPL but it's via the same jboss-marshalling line you are correcting here. So I think they can also be updated.
There was a problem hiding this comment.
The updates are originally coming from #2702. I looked up the repository and noticed a few changes there. It's probably best to make it a required step of the dependency update procedure. It should also be quite simple to find and copy-paste the files if they are on GitHub (or part of the jar, which is often not the case).
Apparently I missed to update the license and notice file of the rest of the netty dependencies. Since they come all from the same repository, it is probably best to merge the files into netty-LICENSE.txt and netty-NOTICE-txt by simply removing the other files. Should be part of SOLR-15929 though.
solr/core/src/java/org/apache/solr/update/processor/ClassificationUpdateProcessorFactory.java
Outdated
Show resolved
Hide resolved
solr/core/src/test/org/apache/solr/handler/admin/IndexSizeEstimatorTest.java
Outdated
Show resolved
Hide resolved
solr/core/src/test/org/apache/solr/handler/admin/api/ReloadCoreAPITest.java
Outdated
Show resolved
Hide resolved
solr/core/src/test/org/apache/solr/handler/admin/api/UnloadCoreAPITest.java
Outdated
Show resolved
Hide resolved
…sion-catalogs # Conflicts: # solr/core/src/test/org/apache/solr/handler/admin/IndexSizeEstimatorTest.java # versions.lock # versions.props
| 4. Run `./gradlew updateLicenses` to re-generate SHA1 checksums of the new jar files. | ||
| 5. Once in a while, a new version of a dependency will transitively bring in brand-new dependencies. | ||
| 4. In case of a conflict, resolve the conflict according to `help/dependencies.txt` | ||
| 5. Check if there are any constraints that are obsolete after the dependency update |
There was a problem hiding this comment.
Check where? How does one identify an obsolete constraint?
There was a problem hiding this comment.
The constraints are defined in gradle/validation/dependencies.gradle. There, if the updated dependency is listed, the constraint can be reviewed, updated or removed.
The constraints fall into two "groups". The first group are dependency constraints from dependencies that our project directly includes and require version alignment to sync the versions across all transitive dependencies.
The second group are dependencies that are only present as transitive dependencies. There, I followed the convention to provide additional information with "which dependencies use what version", so that the next person reviewing the constraint does not have to look it up. However, this is quite time-consuming to analyze the dependencies and therefore subject to change.
Now, in order to review a constraint, you have to check if the updated dependency is mentioned in any of the constraints, either as a reason for another dependency constraint or as the constraint's dependency. Removing temporarily a constraint, the task writeLocks will fail if the constraint is required.
This process and the constraints of dependencies.gradle are not optimal, as it is quite time-consuming and not obvious by just looking at it. We just haven't found yet a more efficient way to maintain these constraints.
There was a problem hiding this comment.
I meant to imply that the documentation should answer these questions for the folks attempting to follow it :)
| @@ -100,12 +166,12 @@ If you want to upgrade Lucene to a newer build proceed like the following: | |||
| queued) | |||
| - remember the build number of Jenkins (left side, first build in list, | |||
| prefixed by '#') | |||
| - Edit ./versions.props and change Lucene's version to '9.0.0-prereleaseX', | |||
| - Edit gradle/libs.versions.toml and change Lucene's version to '9.0.0-prereleaseX', | |||
There was a problem hiding this comment.
When I look at versions.props we don't seem to be doing this. Is this a legacy from pre-split? Maybe this section should go away since Lucene definitely marches ahead on it's own now? (Main is not using 10.0 let alone a pre-release)
There was a problem hiding this comment.
When I started these changes I wasn't sure how Lucene is updated. We still have some custom logic in some gradle files (see gradle/lucene-dev/*), and I believe now that Lucene is "just another dependency", we may be able to simplify and remove any custom logic related to Lucene versions. But this should be handled in a separate PR.
| 5. Once in a while, a new version of a dependency will transitively bring in brand-new dependencies. | ||
| 4. In case of a conflict, resolve the conflict according to `help/dependencies.txt` | ||
| 5. Check if there are any constraints that are obsolete after the dependency update | ||
| 6. Update the license and notice files of the changed dependencies if necessary. See `help/dependencies.txt` for |
There was a problem hiding this comment.
Generally said, any difference in the LICENSE or NOTICE file of the affected dependencies is a change we have to include.
Updating LICENSE and NOTICE files is also subject to change, see SOLR-15929.
There was a problem hiding this comment.
This is detail that should be provided in the doc so that there's no variation in what people think is "necessary", even if it's just a reference to the doc mentioned in that ticket.
|
Thanks @gus-asf for your review and input. Your questions are pointing out some of our upcoming TODOs. I'll leave some comments about the current state on each of your comments, hopefully clarifying and answering your questions. FYI the documentation will be updated periodically in the near future as we work to clean up and improve specific processes. |
https://issues.apache.org/jira/browse/SOLR-17406
Description
With the version catalogs all versions and references to them are managed in a single file and updated from there.
Solution
This PR introduces version catalogs and removes the palantir consistent versions plugin, since it is no longer needed.
For syncing versions a new gradle file (dependencies.gradle) is introduced, where the version of a dependency (either transitive or non-transitive) is set via constraints. This part of the changes should be improved in the future if possible to reduce the maintenance overhead.
This PR is split from #2646.
This PR is following the practices from apache/lucene#13484.
Note that this PR is not tested for backwards compatibility (9x).
Tests
No tests should be affected by this.
Checklist
Please review the following and check all that apply:
mainbranch../gradlew check.