Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix high memory usage with DC Sensitivity Analysis and DC Woodbury Security Analysis #1109

Merged
merged 8 commits into from
Nov 4, 2024
+36 −12
Merged

Fix high memory usage with DC Sensitivity Analysis and DC Woodbury Security Analysis #1109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1500061
Modified new dense matrices to shallow copied matrices
SylvestreSakti Oct 29, 2024
5e64fc7
Merge branch 'main' into fix_memory_dc_sensitivity
SylvestreSakti Oct 29, 2024
7ee6677
Modifciation to keep previous implementation of woodbury engine
SylvestreSakti Oct 29, 2024
0b91b0b
Rename variable
SylvestreSakti Oct 29, 2024
e045aea
flow and factor states do not change their sizes
Hadrien-Godard Oct 29, 2024
8cfc443
Added same modifications to DC SA
SylvestreSakti Oct 29, 2024
1ce884a
Small typo
p-arvy Oct 31, 2024
3f0bb37
Merge branch 'main' into fix_memory_dc_sensitivity
geofjamg Nov 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/main/java/com/powsybl/openloadflow/sa/WoodburyDcSecurityAnalysis.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,7 @@ protected SecurityAnalysisResult runSimulations(LfNetwork lfNetwork, List<Propag
cleanContingencies(lfNetwork, propagatedContingencies);

double[] preContingencyStates = DcLoadFlowEngine.run(context, new DisabledNetwork(), reportNode);
double[] baseContingencyStates = preContingencyStates.clone();

// set pre contingency angle states as state vector of equation system
context.getEquationSystem().getStateVector().set(preContingencyStates);
Expand Down Expand Up @@ -238,7 +239,7 @@ protected SecurityAnalysisResult runSimulations(LfNetwork lfNetwork, List<Propag

logPostContingencyStart(lfNetwork, lfContingency);
Stopwatch stopwatch = Stopwatch.createStarted();

System.arraycopy(baseContingencyStates, 0, preContingencyStates, 0, baseContingencyStates.length);
double[] postContingencyStates = calculatePostContingencyStatesForAContingency(context, connectivityBreakAnalysisResults.contingenciesStates(), preContingencyStates, propagatedContingency,
connectivityBreakAnalysisResults.contingencyElementByBranch(), Collections.emptySet(), Collections.emptySet(), reportNode, Collections.emptySet());
// compute post contingency result with post contingency states
Expand Down
37 changes: 30 additions & 7 deletions src/main/java/com/powsybl/openloadflow/sensi/DcSensitivityAnalysis.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
import com.powsybl.iidm.network.Network;
import com.powsybl.loadflow.LoadFlowParameters;
import com.powsybl.math.matrix.DenseMatrix;
import com.powsybl.math.matrix.DenseMatrixFactory;
import com.powsybl.math.matrix.MatrixFactory;
import com.powsybl.openloadflow.OpenLoadFlowParameters;
import com.powsybl.openloadflow.dc.DcLoadFlowContext;
Expand Down Expand Up @@ -231,7 +232,6 @@ private void calculateSensitivityValuesForAContingency(DcLoadFlowContext loadFlo
DenseMatrix postContingencyFlowStates = engine.run(newFlowStates);
DenseMatrix postContingencyFactorStates = engine.run(newFactorStates);
calculateSensitivityValues(factors, postContingencyFactorStates, postContingencyFlowStates, contingency, resultWriter, disabledNetwork);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since nexFlowStates and newFactorSates are modified by run, and since they can be the input values factorSate and flowState the javadoc of calculateSensitivityValuesForAContingency should explicit that the matrices may be modified.


// write contingency status
if (contingency.hasNoImpact()) {
resultWriter.writeContingencyStatus(contingency.getIndex(), SensitivityAnalysisResult.Status.NO_IMPACT);
Expand Down Expand Up @@ -438,13 +438,14 @@ public void analyse(Network network, List<PropagatedContingency> contingencies,
: Collections.emptyList();

// run DC load on pre-contingency network
DenseMatrix flowStates = calculateFlowStates(loadFlowContext, participatingElements, new DisabledNetwork(), reportNode);

DenseMatrix baseFlowStates = calculateFlowStates(loadFlowContext, participatingElements, new DisabledNetwork(), reportNode);
DenseMatrix flowStates = (DenseMatrix) baseFlowStates.copy(new DenseMatrixFactory());
// compute the pre-contingency factor states
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems useless to copu the values here since they will be restored to the base value at each iteration. You could just create it and avoid here the iteration on all non null values (can be important on large matrixes).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would also be clearer to rename the variable flowStatesWorkingMatrix to make it clear that is is a reused matrix that is modified at each iteration.

DenseMatrix factorsStates = calculateFactorStates(loadFlowContext, factorGroups, participatingElements);
DenseMatrix baseFactorStates = calculateFactorStates(loadFlowContext, factorGroups, participatingElements);
DenseMatrix factorStates = (DenseMatrix) baseFactorStates.copy(new DenseMatrixFactory());

// calculate sensitivity values for pre-contingency network
calculateSensitivityValues(validFactorHolder.getFactorsForBaseNetwork(), factorsStates, flowStates, null, resultWriter, new DisabledNetwork());
calculateSensitivityValues(validFactorHolder.getFactorsForBaseNetwork(), factorStates, flowStates, null, resultWriter, new DisabledNetwork());

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't you use baseFactorStates here ? Isn't the function read-only ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes calculateSensisitivityValues (as well as createBranchSensitivityValue) is read-only. At this line, it can take baseFactorSates and baseFlowStates indeed.

// filter contingencies without factors
List<PropagatedContingency> contingenciesWithFactors = new ArrayList<>();
Expand All @@ -464,23 +465,45 @@ public void analyse(Network network, List<PropagatedContingency> contingencies,

// process contingencies with no connectivity break
for (PropagatedContingency contingency : connectivityBreakAnalysisResults.nonBreakingConnectivityContingencies()) {
matrixShallowCopy(baseFlowStates, flowStates);
matrixShallowCopy(baseFactorStates, factorStates);

calculateSensitivityValuesForAContingency(loadFlowContext, lfParametersExt, validFactorHolder, factorGroups,
factorsStates, connectivityBreakAnalysisResults.contingenciesStates(), flowStates, contingency,
factorStates, connectivityBreakAnalysisResults.contingenciesStates(), flowStates, contingency,
connectivityBreakAnalysisResults.contingencyElementByBranch(), Collections.emptySet(), participatingElements, Collections.emptySet(), resultWriter, reportNode, Collections.emptySet(), false);
}

LOGGER.info("Processing contingencies with connectivity break");

// process contingencies with connectivity break
for (ConnectivityBreakAnalysis.ConnectivityAnalysisResult connectivityAnalysisResult : connectivityBreakAnalysisResults.connectivityAnalysisResults()) {
matrixShallowCopy(baseFlowStates, flowStates);
matrixShallowCopy(baseFactorStates, factorStates);

processContingenciesBreakingConnectivity(connectivityAnalysisResult, loadFlowContext, lfParameters, lfParametersExt,
validFactorHolder, factorGroups, participatingElements, connectivityBreakAnalysisResults.contingencyElementByBranch(),
flowStates, factorsStates, connectivityBreakAnalysisResults.contingenciesStates(), resultWriter, reportNode);
flowStates, factorStates, connectivityBreakAnalysisResults.contingenciesStates(), resultWriter, reportNode);
}
}

stopwatch.stop();
LOGGER.info("DC sensitivity analysis done in {} ms", stopwatch.elapsed(TimeUnit.MILLISECONDS));
}
}

/**
* Copy all the values that are in an originalMatrix and paste it in the copyMatrix (without allocating new memory spaces)
* The dimensions of both matrices must be the same
*/
// TODO : implement this method for DenseMatrix in powsybl-core ?
private static void matrixShallowCopy(DenseMatrix originalMatrix, DenseMatrix copyMatrix) {
if (originalMatrix.getRowCount() == copyMatrix.getRowCount() && originalMatrix.getColumnCount() == copyMatrix.getColumnCount()) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The term shallow copy is incorrect. See definition here https://developer.mozilla.org/en-US/docs/Glossary/Shallow_copy
In a shallow copy in a shallow copy when you change either the source or the copy, you may also cause the other object to change too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we speak about "deep copy" ? (but without creating a new container)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copyValues light be a bettre term and make clear you expect the same matrix dimensions.

for (int columnIndex = 0; columnIndex < originalMatrix.getColumnCount(); columnIndex++) {
for (int rowIndex = 0; rowIndex < originalMatrix.getRowCount(); rowIndex++) {
copyMatrix.set(rowIndex, columnIndex, originalMatrix.get(rowIndex, columnIndex));
}
}
}
}
Copy link
Collaborator

@vidaldid-rte vidaldid-rte Nov 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should throw an exception (IllegalArgumentException) if the matrix sizes do not match instead of silently ignoring the error.


}
8 changes: 4 additions & 4 deletions src/main/java/com/powsybl/openloadflow/sensi/WoodburyEngine.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,12 +84,12 @@ private void setAlphas(DenseMatrix states, int columnState) {

/**
* Calculate post-contingency states values using pre-contingency states values and some flow transfer factors (alphas).
* @return a matrix of post-contingency voltage angle states.
* @return a matrix of post-contingency voltage angle states (which is the same modified pre-contingency states java object).
*/
public DenseMatrix run(DenseMatrix preContingencyStates) {
Objects.requireNonNull(preContingencyStates);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The API xould be clearer with something like
public void toPostContingencyStates(DenseMatrix preContingencyStates);

It would clarify that the input matrix is modified.

// fill the post contingency matrices
DenseMatrix postContingencyStates = new DenseMatrix(preContingencyStates.getRowCount(), preContingencyStates.getColumnCount());
DenseMatrix postContingencyStates = preContingencyStates;
for (int columnIndex = 0; columnIndex < preContingencyStates.getColumnCount(); columnIndex++) {
setAlphas(preContingencyStates, columnIndex);
for (int rowIndex = 0; rowIndex < preContingencyStates.getRowCount(); rowIndex++) {
Expand All @@ -106,11 +106,11 @@ public DenseMatrix run(DenseMatrix preContingencyStates) {

/**
* Calculate post-contingency states values using pre-contingency states values and some flow transfer factors (alphas).
* @return an array of post-contingency voltage angle states.
* @return an array of post-contingency voltage angle states (which is the same modified pre-contingency java object).
*/
public double[] run(double[] preContingencyStates) {
Objects.requireNonNull(preContingencyStates);
double[] postContingencyStates = new double[preContingencyStates.length];
double[] postContingencyStates = preContingencyStates;
setAlphas(new DenseMatrix(preContingencyStates.length, 1, preContingencyStates), 0);
for (int rowIndex = 0; rowIndex < preContingencyStates.length; rowIndex++) {
double postContingencyValue = preContingencyStates[rowIndex];
Expand Down