feat: Declarative Shadow Variables

[Christopher-Chianelli]

VariableListeners are notoriously difficult to write, especially when you have multiple variable depending on each others.
Declarative shadow variables fixes this by providing a declarative way to define the dependencies of your shadow variables and how to calculate them.

First, annotate your variables as @ProvidedShadowVariable:

@PlanningEntity
public class Visit {
    String id;

    @InvalidityMarker
    boolean isInvalid;   // If true, the visit is in a "loop" and some of its provided shadow variables will be null

    @ProvidedShadowVariable(MyShadowVariableProvider.class)
    LocalDateTime serviceReadyTime;
    
    // ...
}

Next, create a ShadowVariableProvider to define your variables:

public class MyShadowVariableProvider implements ShadowVariableProvider {
    @Override
    public void defineVariables(ShadowVariableFactory variableFactory) {
        // The previous visit; you don't need to use @PreviousElementShadowVariable
        var previousVisit = variableFactory.entity(Visit.class).previous();
        // The vehicle the visit is assignedf to; you don't need to use @InverseRelationShadowVariable
        var vehicle = variableFactory.entity(Visit.class).inverse(Vehicle.class);
        
        // The service ready time; when the technician will arrive
        var serviceReadyTime = variableFactory.newShadow(Visit.class)
               // If a visit has a previous visit, the ready time 
                .compute(previousVisit.fact(Location.class, Visit::getLocation),
                         previousVisit.variable(LocalDateTime.class, "serviceFinishTime"),
                         (visit, previousLocation, previousEndTime) -> previousEndTime.plus(visit.getLocation().travelTimeFrom(previousLocation))
                )
                // If a visit has no previous visit, use the vehicle's start time
                .orCompute(vehicle.fact(Location.class, Vehicle::getLocation),
                        vehicle.fact(LocalDateTime.class, Vehicle::getStartTime),
                        (visit, startLocation, startTime) -> startTime.plus(visit.getLocation().travelTimeFrom(startLocation))
                )
                // If a visit is unassigned, its serviceReadyTime is null
                .as("serviceReadyTime");
        
         // The service start time; when service can start
         // If the visit is a part of a group, service cannot start until all technicians arrive
         // Otherwise, it the same as the service start time
         var visitGroup = variableFactory
                        .entity(Visit.class)
                        .group(Visit.class, Visit::getVisitGroup);
         var serviceStartTime = variableFactory.newShadow(Visit.class)
                // If the visit has a visit group, its start time is the max of the ready times of the visit in its visit group
                .compute(visitGroup.variables(LocalDateTime.class, "serviceReadyTime"),
                        (visit, groupReadyTimes) -> groupReadyTimes.isEmpty() ? null : Collections.max(groupReadyTimes))
                // Otherwise, it the service ready time of this visit
                .orCompute(serviceReadyTime, (visit, readyTime) -> readyTime)
                .as("serviceStartTime");

         // Each service ends 30 minutes after it started
         var serviceFinishTime = variableFactory.newShadow(Visit.class)
                .compute(serviceStartTime, (visit, startTime) -> startTime.plusMinutes(30))
                .as("serviceFinishTime");
    }
}

and finally, write tests to ensure your shadow variables work correctly:

@Test
public void shadowVariables() {
      var sessionFactory = ShadowVariableSessionFactory.create(
              SolutionDescriptor.buildSolutionDescriptor(RoutePlan.class,
                      Vehicle.class, Visit.class),
              new MyShadowVariableProvider());

      var vehicle = Vehicle("v1");
      var visit1 = Visit("c1");
      var visit2 = Visit("c2");
      var visit3 = Visit("c3");

      var session = sessionFactory.forEntities(vehicle, visit1, visit2, visit3);
      session.setInverse(visit1, vehicle);
      session.setPrevious(visit2, visit1);
      session.setPrevious(visit3, visit2);
      session.updateVariables();

      assertThat(visit1.getServiceReadyTime()).isEqualTo(TestShadowVariableProvider.BASE_START_TIME);
      assertThat(visit1.getServiceStartTime()).isEqualTo(TestShadowVariableProvider.BASE_START_TIME);
      assertThat(visit1.getServiceFinishTime()).isEqualTo(TestShadowVariableProvider.BASE_START_TIME.plusMinutes(30L));
      assertThat(visit1.isInvalid()).isFalse();

      assertThat(visit2.getServiceReadyTime()).isEqualTo(TestShadowVariableProvider.BASE_START_TIME.plusMinutes(60L));
      assertThat(visit2.getServiceStartTime()).isEqualTo(TestShadowVariableProvider.BASE_START_TIME.plusMinutes(60L));
      assertThat(visit2.getServiceFinishTime()).isEqualTo(TestShadowVariableProvider.BASE_START_TIME.plusMinutes(90L));
      assertThat(visit2.isInvalid()).isFalse();

      assertThat(visit3.getServiceReadyTime()).isEqualTo(TestShadowVariableProvider.BASE_START_TIME.plusMinutes(120L));
      assertThat(visit3.getServiceStartTime()).isEqualTo(TestShadowVariableProvider.BASE_START_TIME.plusMinutes(120L));
      assertThat(visit3.getServiceFinishTime()).isEqualTo(TestShadowVariableProvider.BASE_START_TIME.plusMinutes(150L));
      assertThat(visit3.isInvalid()).isFalse();
}

Behind the scenes, Timefold Solver calculates a valid topological order for each of your shadow variables.
This allows Timefold Solver to:

1. Only recalculate changed shadow variables. A shadow variable is recalculated if any of its declared inputs has changed. Using an undeclared variable input will lead to score corruptions.
2. Determine what variables are in a "loop". Variables are in a "loop" if:

• They directly or indirectly depend on each other.
• They depend on looped variables.

Currently a WIP:

• Write javadoc
• Write documentation
• Write additional tests
• Write additional use cases
• Finalize API

[triceo]

Before I do a proper review, I suggest we do some house-keeping first. (Otherwise comments would get lost when files are being renamed and moved around.)

• Are we happy with the package? Variables generally go into api.domain.variable, so in this case preview.api.domain.variable.declarative?
• Preview features need to be declared in PreviewFeature; features not enabled through this enum need to fail fast when used. (In this case, probably when the new annotations are being processed.)

[triceo]

We need to be careful here. This feature should have no performance impact when disabled. I'll run some numbers to confirm that.

[triceo]

And this can be stored on a score director, meaning there would be a 1:1 mapping between these instances and score directors.

[Christopher-Chianelli]

The issue with storing it on the ScoreDirector is that the entire ScoreDirector would then need to be mocked to be mocked to implement tests. The class exists to abstract away the before...Change and after...Changed method calls (tests do not care, but they are required when solving).

[triceo]

My understanding of this API is still limited, but do we actually need this public API anywhere in the solver? IMO this is internal in core, and only exposed in the tests; therefore I'd make this public in the tests, not in the solver itself.

[Christopher-Chianelli]

It can be moved; it is here for convenience when testing the prototype. (core internally depends on the default implementation which has before/after methods).

[triceo]

Dtto.

[triceo]

Hypothetical scenario:

• I have two different shadow providers.
• But they end up declaring/depending on the same shadow variables.

Do they all become part of the same graph? In other words - is the graph limited to one shadow provider, or is it built from all shadow providers combined, and then split into individual independent components?

[Christopher-Chianelli]

Hypothetical scenario:

* I have two different shadow providers.

* But they end up declaring/depending on the same shadow variables.

Do they all become part of the same graph? In other words - is the graph limited to one shadow provider, or is it built from all shadow providers combined, and then split into individual independent components?

Although untested, it is built from all shadow providers combined, and then split into individual independent components. Defining the same shadow variable twice is an error though; (you can use it multiple time, but cannot declare it multiple times).

[triceo]

Please add this to the list of things that need to be tested.

[ge0ffrey]

Will this mark any type of invalidity or only loops?

[Christopher-Chianelli]

Any invalidity caused by loops (so a variable in a cycle OR a variable that depends on a variable in a cycle). Any type of invalidity is too vague; some people will consider null to be a valid values, others would not.

[ge0ffrey]

This seems similar to ConstraintVerifier, so the naming should be similar.
Does the ConstraintVerifier API have anything that is a Session?

[ge0ffrey]

Feels similar to ConstraintVerifier.given(), so maybe rename it to "given"?

[ge0ffrey]

Maybe rename to something like ShadowVariableVerifier, as it serves a a simular purpose?

[ge0ffrey]

Is FSR being copy pasted into the solver test sources?
Or is it a subset?

[Christopher-Chianelli]

A heavily simplified subset.

[sonarqubecloud]

Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE