-
Notifications
You must be signed in to change notification settings - Fork 73
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
resolve race condition with jar manifest manipulation #1150
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
type: fix | ||
fix: | ||
description: resolve race condition with jar manifest manipulation | ||
links: | ||
- https://github.com/palantir/sls-packaging/pull/1150 |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -185,58 +185,6 @@ class CreateManifestIntegrationSpec extends GradleIntegrationSpec { | |
file('bar-server/product-dependencies.lock').text.contains 'com.palantir.group:foo-service ($projectVersion, 1.x.x)' | ||
} | ||
|
||
def "createManifest does not force compilation of sibling projects"() { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agree with Dan. Is there another solution? Maybe using variants? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I had suggested using variants and artifactTransformers offline. I think a solution similar to what we do for diagnostics could work here. The only issue I see is that it is more challenging to get access to a manifest than to a file in the resource directory. I think we could work around this though by producing both an entry in the JARs manifest (for backwards compatability) and a file in the resources directory that contained the Pdeps of a given JAR. We could then cleanly re-use the existing transforms (https://github.com/palantir/sls-packaging/blob/develop/gradle-sls-packaging/src/main/java/com/palantir/gradle/dist/service/DiagnosticsManifestPlugin.java#L165-L233) and leverage regular gradle cache validation There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I believe that writing the information to resource files is the way to go (it was the original design for the additional conjure information which is now being written), though that won't necessarily clear all of this up, especially if we are also writing to the meta-inf file (because we still have the existing problem). However, there may be an optimization (a.k.a. hack) we can appy for in-project dependencies for lock files. We do not need to actually compute the minimum version for such when writing lock files - we always write the template variable '$projectVersion'. So if we think of updating locks separately, then we may be able to avoid compiling and writing jars. I'll ponder a bit more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I like Felipe's idea of also producing a pdeps artifact in addition to writing it in the JAR manifest as this would potentially simplify configuration.
Wouldn't project dependencies use the same version of sls-packaging though so we could rely on them using the new code path and producing an artifact containing the pdeps that does not require compiling the source?
Maybe I misunderstand but I think we need to know the concrete minimum version for writing the "product-dependencies.lock" file (ref)? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. in-project dependencies are always changed to the template (ref), unless there is some comparison I am missing? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we need to revert this. After this PR, excavator checks will not open PRs if the resulting code does not compile. Here's an example of a dependency upgrade excavator run on our internal server repo that failed because |
||
setup: | ||
buildFile << """ | ||
allprojects { | ||
project.version = '1.0.0' | ||
group "com.palantir.group" | ||
} | ||
""" | ||
helper.addSubproject("foo-api", """ | ||
apply plugin: 'java' | ||
apply plugin: 'com.palantir.sls-recommended-dependencies' | ||
|
||
recommendedProductDependencies { | ||
productDependency { | ||
productGroup = 'com.palantir.group' | ||
productName = 'foo-service' | ||
minimumVersion = '0.0.0' | ||
maximumVersion = '1.x.x' | ||
recommendedVersion = rootProject.version | ||
} | ||
} | ||
""".stripIndent()) | ||
helper.addSubproject("foo-server", """ | ||
apply plugin: 'com.palantir.sls-java-service-distribution' | ||
|
||
dependencies { | ||
compile project(':foo-api') | ||
} | ||
|
||
distribution { | ||
serviceGroup 'com.palantir.group' | ||
serviceName 'foo-service' | ||
mainClass 'com.palantir.foo.bar.MyServiceMainClass' | ||
args 'server', 'var/conf/my-service.yml' | ||
} | ||
""".stripIndent()) | ||
|
||
when: | ||
def result = runTasks('foo-server:createManifest') | ||
|
||
then: | ||
result.task(":foo-server:createManifest").outcome == TaskOutcome.SUCCESS | ||
result.task(":foo-api:configureProductDependencies").outcome == TaskOutcome.SUCCESS | ||
result.task(':foo-api:jar') == null | ||
result.tasks.collect({ it.path }).toSet() == ImmutableSet.of( | ||
":foo-api:configureProductDependencies", | ||
":foo-api:processResources", | ||
":foo-server:mergeDiagnosticsJson", | ||
":foo-server:resolveProductDependencies", | ||
":foo-server:createManifest") | ||
} | ||
|
||
def "check depends on createManifest"() { | ||
when: | ||
def result = runTasks(':check') | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Based on reviews from @CRogers I had thought that the Provider would automatically set task dependencies. Is that not the case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good question. A property/provider that is the output of task A will create a dependency if it is set to an input property of task B. However, because this provider is not associated with a task, I do not know if the same logic applies. I can say the behavior was different once I added the explicit dependsOn statement - the upstream jar tasks were run.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That might be a stupid question, but what does it actually mean to depend on a provider? The provider will be evaluated before the task runs? How is that different from evaluating the provider passed through as a task input?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
actually, the right answer to this may be that the Configuration is not listed as an Input property to the task. See -
sls-packaging/gradle-sls-packaging/src/main/java/com/palantir/gradle/dist/tasks/ResolveProductDependenciesTask.java
Lines 153 to 164 in 856fba2
While I didn't set things up this way here (predated me), I have encountered this issue before with in similar situations. I do not know if it is still a problem of if newer versions of gradle have some way to get around it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More data - The task has this method to try to enforce the dependencies between the tasks as well -
sls-packaging/gradle-sls-packaging/src/main/java/com/palantir/gradle/dist/tasks/ResolveProductDependenciesTask.java
Lines 106 to 130 in 856fba2
This was added with this commit -
3649f1b
I didn't change the functionality of this method when I extracted the task, but it clearly is not doing what we hoped. I have a gradle build scan where configureProductDependencies is starting after resolveProductDependencies
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just writing notes to myself - this commit was to try to avoid building jars - b855a77#diff-ffb538d33fb6230122fe54f923c3b269363d3fd26e7f1f38c58b9750aacf9e82R136-R143
It was about a month before the one that added the otherProjectProductDependenciesTasks method above. The problem with all of these is that they assume they have complete knowledge of what could write dependency info into the jar manifest (i.e. the ConfigureProductDependenciesTask). Though I still don't know why I have seen the task ordering difference noted above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Discussed offline and we'll move forward with this for now. If the overhead by compiling jars becomes too bad, we can look into other options.