Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: issue-5850 - Settle override conflicts between edges and propagate changes #7025

Open
wants to merge 33 commits into
base: latest
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
865070c
Settle overrides conflicts between edges, and propagate changes to th…
AlonNavon Nov 23, 2023
cd20196
cleanup
AlonNavon Nov 26, 2023
0cad4e0
Optimization
AlonNavon Nov 26, 2023
6b276f3
Fix override sets comparisons
AlonNavon Nov 28, 2023
fff3072
Add comparator
AlonNavon Nov 28, 2023
e97176e
Check for undefined
AlonNavon Nov 28, 2023
19155ed
Another place
AlonNavon Nov 28, 2023
dedbccf
Fix
AlonNavon Nov 28, 2023
4a83786
Lint fixes
AlonNavon Dec 6, 2023
7ee2f2e
Added tests
AlonNavon Dec 6, 2023
0917421
Overrides aren't inherited from parent, but from to nodes
AlonNavon Oct 27, 2024
6d0ad42
Always use raw spec when analyzing overrides
AlonNavon Oct 28, 2024
67d7d5c
Replaceability
AlonNavon Oct 28, 2024
643cbea
Overridden status
AlonNavon Oct 28, 2024
c5f6e4a
Fix undefined bugs
AlonNavon Oct 28, 2024
6c9bbf9
Parent doesn't matter to overrides
AlonNavon Oct 28, 2024
63760d9
Update override set in reload
AlonNavon Oct 28, 2024
6111630
Erroneous node
AlonNavon Oct 28, 2024
b42d4c8
Add comments
AlonNavon Oct 29, 2024
7d97726
Code review fixes
AlonNavon Nov 4, 2024
35e5790
Fix edge satisfaction logic
AlonNavon Nov 4, 2024
35ce2f7
Fix dedupe logic
AlonNavon Nov 4, 2024
5630954
Code review fixes
AlonNavon Nov 4, 2024
f859e92
Oops
AlonNavon Nov 4, 2024
29f0750
Static function
AlonNavon Nov 5, 2024
17ec457
Typo
AlonNavon Nov 5, 2024
4cde831
Shouldn't throw
AlonNavon Nov 7, 2024
701b125
CR fixes
AlonNavon Nov 7, 2024
b119f67
CR fixes
AlonNavon Nov 7, 2024
3136b64
Move functions
AlonNavon Nov 10, 2024
80126c7
Comments
AlonNavon Nov 11, 2024
c4b5338
Override sets unit tests
AlonNavon Nov 11, 2024
9db1232
Clarify comment
AlonNavon Nov 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions workspaces/arborist/lib/edge.js
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,7 @@ class Edge {
const newTo = this.#from.resolve(this.#name)
if (newTo !== this.#to) {
if (this.#to) {
this.#to.edgesIn.delete(this)
this.#to.deleteEdgeIn(this)
}
this.#to = newTo
this.#error = null
Expand All @@ -273,7 +273,7 @@ class Edge {
detach () {
this.#explanation = null
if (this.#to) {
this.#to.edgesIn.delete(this)
this.#to.deleteEdgeIn(this)
}
this.#from.edgesOut.delete(this.#name)
this.#to = null
Expand Down
95 changes: 93 additions & 2 deletions workspaces/arborist/lib/node.js
Original file line number Diff line number Diff line change
Expand Up @@ -1344,9 +1344,100 @@ class Node {
this.edgesOut.set(edge.name, edge)
}

addEdgeIn (edge) {
recalculateOutEdgesOverrides () {
// For each edge out propogate the new overrides through.
for (const [, edge] of this.edgesOut) {

This comment was marked as resolved.

edge.reload(true)
if (edge.to) {
edge.to.updateNodeOverrideSet(edge.overrides)
}
}
}

findSpecificOverrideSet (first, second) {

This comment was marked as resolved.

for (let overrideSet = second; overrideSet; overrideSet = overrideSet.parent) {
if (overrideSet.isEqual(first)) {
return second
}
}
for (let overrideSet = first; overrideSet; overrideSet = overrideSet.parent) {
if (overrideSet.isEqual(second)) {
return first
}
}
console.log('Conflicting override sets')
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wraithgar This case means that the same node has two edges with completely different override sets pointing to it. This log is useful for debugging, not sure if just to remove the line or there's some other convention used in the arborist.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

proc-log is what we use to log in npm.

log.silly is probably what we want here, defined as

Extremely noisy excessive logging messages that are typically only useful for debugging.

}

updateNodeOverrideSetDueToEdgeRemoval (otherOverrideSet) {
// If this edge's overrides isn't equal to this node's overrides, then removing it won't change newOverrideSet later.
if (!this.overrides || !this.overrides.isEqual(otherOverrideSet)) {
return false
}
let newOverrideSet
for (const edge of this.edgesIn) {
if (newOverrideSet) {
newOverrideSet = this.findSpecificOverrideSet(edge.overrides, newOverrideSet)
} else {
newOverrideSet = edge.overrides
}
}
if (this.overrides.isEqual(newOverrideSet)) {
return false
}
this.overrides = newOverrideSet
if (this.overrides) {
// Optimization: if there's any override set at all, then no non-extraneous node has an empty override set. So if we temporarily have no
// override set (for example, we removed all the edges in), there's no use updating all the edges out right now. Let's just wait until
// we have an actual override set later.
this.recalculateOutEdgesOverrides()
}
return true
}

// This logic isn't perfect either. When we have two edges in that have different override sets, then we have to decide which set is correct.
// This function assumes the more specific override set is applicable, so if we have dependencies A->B->C and A->C
// and an override set that specifies what happens for C under A->B, this will work even if the new A->C edge comes along and tries to change
// the override set.
// The strictly correct logic is not to allow two edges with different overrides to point to the same node, because even if this node can satisfy
// both, one of its dependencies might need to be different depending on the edge leading to it.
// However, this might cause a lot of duplication, because the conflict in the dependencies might never actually happen.
updateNodeOverrideSet (otherOverrideSet) {
if (!this.overrides) {
// Assuming there are any overrides at all, the overrides field is never undefined for any node at the end state of the tree.
// So if the new edge's overrides is undefined it will be updated later. So we can wait with updating the node's overrides field.
if (!otherOverrideSet) {
return false
}
this.overrides = otherOverrideSet
this.recalculateOutEdgesOverrides()
return true
}
if (this.overrides.isEqual(otherOverrideSet)) {
return false
}
const newOverrideSet = this.findSpecificOverrideSet(this.overrides, otherOverrideSet)
if (newOverrideSet) {
if (!this.overrides.isEqual(newOverrideSet)) {
this.overrides = newOverrideSet
this.recalculateOutEdgesOverrides()
return true
}
return false
}
// This is an error condition. We can only get here if the new override set is in conflict with the existing.
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wraithgar This case means that the same node has two edges with completely different override sets pointing to it, which means that this node can't really decide which one is correct.
Our options are:

  1. Go with the flow - in the current logic the out edges continue with the first overrides set they got, so this isn't breaking comparing to the existing logic. But it's obviously the wrong behavior.
  2. Raise an error - just fail the command. Not sure how you do that in the arborist.
  3. Duplicate the node - this is the same package with different override requirements. The proper logic is to duplicate the node, and then dedup it later if there's no conflict.

Currently I chose (1), but maybe it should at least show some kind of warning.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A hard-won lesson we learned with peer dependencies is that 1 is not the right choice here. npm 6 can install completely invalid trees, and this was fixed in npm 7. --legacy-peer-deps can turn the behavior back on, but again it will install trees that have incorrect peer dependencies if there was a conflict. npm should never guess the user's intent.

2 is the easiest solution, especially given that 3 may eventually fail if we end up with an already duplicated node with conflicting override requirements.

So, unless there's a really elegant way to do 3 that does't require an extra mountain of effort, 2 is best here. We'll throw with as much context as possible to help the user build a working override set, but we won't build an invalid tree.

This also leaves the door open later to implementing 3, since we can gradually add smaller, specific cases where 3 will work, and the fallback to 2 will already be in place.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how you do that in the arborist.

Just throw an Error object. If we want npm to format the message in a special way based on any other attributes you assign to the object we can do that. A good example of that is here, with the cli's custom error output switch here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @wraithgar, I've encountered a problem with throwing an exception here.
This function is reached both by npm install and npm ls.
If I throw an exception here, npm ls fails without showing the tree, and running npm install again on a badly installed project fails in the initial mapping phase. What do you suggest?

Copy link
Contributor

@jdalton jdalton Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlonNavon

What do you suggest?

Until clarity on the issue you can always return false or return false and log.silly it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I checked the behavior of the algorithms here. @jdalton @wraithgar
We shouldn't throw here, we should just log the problem.
The fix to satisfiedBy function in edge.js causes that to return false, and so the edge is marked as invalid.
In the ls case it is marked as erroneous, and in the install case it is detached from this node and connected to a proper node.
So without throwing, we get the correct behavior.

}

deleteEdgeIn (edge) {
this.edgesIn.delete(edge)
if (edge.overrides) {
this.overrides = edge.overrides
this.updateNodeOverrideSetDueToEdgeRemoval(edge.overrides)
}
}

addEdgeIn (edge) {
// We need to handle the case where the new edge in has an overrides field which is different from the current value.
if (!this.overrides || !this.overrides.isEqual(edge.overrides)) {
this.updateNodeOverrideSet(edge.overrides)
}

this.edgesIn.add(edge)
Expand Down
37 changes: 37 additions & 0 deletions workspaces/arborist/lib/override-set.js
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,43 @@ class OverrideSet {
}
}

childrenAreEqual (other) {
AlonNavon marked this conversation as resolved.
Show resolved Hide resolved
if (this.children.size !== other.children.size) {
return false
}
for (const [key] of this.children) {
if (!other.children.has(key)) {
return false
}
if (this.children.get(key).value !== other.children.get(key).value) {
return false
}
if (!this.children.get(key).childrenAreEqual(other.children.get(key))) {
return false
}
}
return true
}
AlonNavon marked this conversation as resolved.
Show resolved Hide resolved

isEqual (other) {
if (this === other) {
return true
}
if (!other) {
return false
}
if (this.key !== other.key || this.value !== other.value) {
return false
}
if (!this.childrenAreEqual(other)) {
return false
}
if (!this.parent) {
return !other.parent
}
return this.parent.isEqual(other.parent)
}

getEdgeRule (edge) {
for (const rule of this.ruleset.values()) {
if (rule.name !== edge.name) {
Expand Down
Loading
Loading