Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix CGM export dependencies issues #3128

Merged
merged 6 commits into from
Sep 16, 2024
Merged

Fix CGM export dependencies issues #3128

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b8fa338
Add dependentOn relation of SV model to all TP_BD models in case of a…
rcourtier Sep 3, 2024
fef831b
Update documentation
rcourtier Sep 3, 2024
0793f9a
Merge branch 'main' into cgmes_cgm_export_fix_dependencies_issues
zamarrenolm Sep 4, 2024
013750e
Add dependency (dependentOn) from updated SSHs to original EQs
rcourtier Sep 4, 2024
6e8586a
Check that the BOUNDARY_TP_ID parameter is considered in the export d…
rcourtier Sep 4, 2024
a01d65e
Merge branch 'main' into cgmes_cgm_export_fix_dependencies_issues
zamarrenolm Sep 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 29 additions & 15 deletions cgmes/cgmes-conversion/src/main/java/com/powsybl/cgmes/conversion/CgmesExport.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,22 +107,24 @@ public void export(Network network, Properties parameters, DataSource dataSource
private void exportCGM(Network network, DataSource dataSource, CgmesExportContext context) {
checkCgmConsistency(network, context);

// Initialize models for export. The original IGM TP and SSH don't get exported,
// Initialize models for export. The original IGM EQ, SSH, TP and TP_BD don't get exported,
// but we need to init their models to retrieve their IDs when building the dependencies.
Map<Network, IgmModelsForCgm> igmModels = new HashMap<>();
for (Network subnetwork : network.getSubnetworks()) {
IgmModelsForCgm igmModelsForCgm = new IgmModelsForCgm(
initializeModelForExport(subnetwork, CgmesSubset.STEADY_STATE_HYPOTHESIS, context, false, false),
initializeModelForExport(subnetwork, CgmesSubset.STEADY_STATE_HYPOTHESIS, context, false, true),
initializeModelForExport(subnetwork, CgmesSubset.TOPOLOGY, context, false, false)
initializeModelForExport(subnetwork, CgmesSubset.EQUIPMENT, context, false, false),
initializeModelForExport(subnetwork, CgmesSubset.STEADY_STATE_HYPOTHESIS, context, false, false),
initializeModelForExport(subnetwork, CgmesSubset.TOPOLOGY, context, false, false),
initializeModelForExport(subnetwork, CgmesSubset.TOPOLOGY_BOUNDARY, context, false, false)
);
igmModels.put(subnetwork, igmModelsForCgm);
}
CgmesMetadataModel updatedCgmSvModel = initializeModelForExport(network, CgmesSubset.STATE_VARIABLES, context, true, true);

// Update dependencies
if (context.updateDependencies()) {
updateDependenciesCGM(igmModels.values(), updatedCgmSvModel);
updateDependenciesCGM(igmModels.values(), updatedCgmSvModel, context.getBoundaryTpId());
}

// Export the SSH for the IGMs and the SV for the CGM
Expand Down Expand Up @@ -227,22 +229,29 @@ public static CgmesMetadataModel initializeModelForExport(
}

/**
* Update cross dependencies between the subset models through the dependentOn relationship.
* The IGMs updated SSH supersede the original ones.
* The CGM updated SV depends on the IGMs updated SSH and on the IGMs original TP.
* @param igmModels For each IGM: the original SSH model, the updated SSH model and the original TP model.
* Update cross dependencies between the subset models (including boundaries) through the dependentOn relationship.
* The IGMs updated SSH supersede the original ones and depend on the original EQ. Other dependencies are kept.
* The CGM updated SV depends on the IGMs updated SSH and on the IGMs original TP and TP_BD.
* @param igmModels For each IGM: the updated SSH model and the original SSH, TP and TP_BD models.
* @param updatedCgmSvModel The SV model for the CGM.
* @param boundaryTpId The model id for the TP_BD subset.
*/
private void updateDependenciesCGM(Collection<IgmModelsForCgm> igmModels, CgmesMetadataModel updatedCgmSvModel) {
private void updateDependenciesCGM(Collection<IgmModelsForCgm> igmModels, CgmesMetadataModel updatedCgmSvModel, String boundaryTpId) {
// Each updated SSH model depends on the original EQ model
igmModels.forEach(m -> m.updatedSsh.addDependentOn(m.originalEq.getId()));

// Each updated SSH model supersedes the original one
// Clear previous dependencies
igmModels.forEach(m -> m.updatedSsh.clearDependencies());
igmModels.forEach(m -> m.updatedSsh.clearSupersedes());
igmModels.forEach(m -> m.updatedSsh.addSupersedes(m.originalSsh.getId()));

// Updated SV model depends on updated SSH models and original TP models
// Updated SV model depends on updated SSH models and original TP and TP_BD models
updatedCgmSvModel.addDependentOn(igmModels.stream().map(m -> m.updatedSsh.getId()).collect(Collectors.toSet()));
updatedCgmSvModel.addDependentOn(igmModels.stream().map(m -> m.originalTp.getId()).collect(Collectors.toSet()));
if (boundaryTpId != null) {
updatedCgmSvModel.addDependentOn(boundaryTpId);
} else {
updatedCgmSvModel.addDependentOn(igmModels.stream().map(m -> m.originalTpBd.getId()).collect(Collectors.toSet()));
}
}

/**
Expand Down Expand Up @@ -492,14 +501,19 @@ private String getBaseName(CgmesExportContext context, DataSource dataSource, Ne
* when setting the relationships (dependOn, supersedes) between them in a CGM export.
*/
private static class IgmModelsForCgm {
CgmesMetadataModel originalSsh;
CgmesMetadataModel updatedSsh;
CgmesMetadataModel originalEq;
CgmesMetadataModel originalSsh;
CgmesMetadataModel originalTp;
CgmesMetadataModel originalTpBd;

public IgmModelsForCgm(CgmesMetadataModel originalSsh, CgmesMetadataModel updatedSsh, CgmesMetadataModel originalTp) {
this.originalSsh = originalSsh;
public IgmModelsForCgm(CgmesMetadataModel updatedSsh, CgmesMetadataModel originalEq, CgmesMetadataModel originalSsh,
CgmesMetadataModel originalTp, CgmesMetadataModel originalTpBd) {
this.updatedSsh = updatedSsh;
this.originalEq = originalEq;
this.originalSsh = originalSsh;
this.originalTp = originalTp;
this.originalTpBd = originalTpBd;
}
}

Expand Down
72 changes: 62 additions & 10 deletions ...ion/src/test/java/com/powsybl/cgmes/conversion/test/export/CommonGridModelExportTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,14 +140,20 @@ void testCgmExportNoModelsNoProperties() throws IOException {
String updatedNlSshId = "urn:uuid:Network_NL_N_STEADY_STATE_HYPOTHESIS_2021-02-03T04:30:00Z_2_1D__FM";
String originalBeTpId = "urn:uuid:Network_BE_N_TOPOLOGY_2021-02-03T04:30:00Z_1_1D__FM";
String originalNlTpId = "urn:uuid:Network_NL_N_TOPOLOGY_2021-02-03T04:30:00Z_1_1D__FM";
Set<String> expectedDependencies = Set.of(updatedBeSshId, updatedNlSshId, originalBeTpId, originalNlTpId);
String originalBeTpBdId = "urn:uuid:Network_BE_N_TOPOLOGY_BOUNDARY_2021-02-03T04:30:00Z_1_1D__FM";
String originalNlTpBdId = "urn:uuid:Network_NL_N_TOPOLOGY_BOUNDARY_2021-02-03T04:30:00Z_1_1D__FM";
Set<String> expectedDependencies = Set.of(updatedBeSshId, updatedNlSshId, originalBeTpId, originalNlTpId, originalBeTpBdId, originalNlTpBdId);
assertEquals(expectedDependencies, getOccurrences(updatedCgmSvXml, REGEX_DEPENDENT_ON));

// Each updated IGM SSH should supersede the original one
// Each updated IGM SSH should supersede the original one and depend on the original EQ
String originalBeSshId = "urn:uuid:Network_BE_N_STEADY_STATE_HYPOTHESIS_2021-02-03T04:30:00Z_1_1D__FM";
String originalBeEqId = "urn:uuid:Network_BE_N_EQUIPMENT_2021-02-03T04:30:00Z_1_1D__FM";
String originalNlSshId = "urn:uuid:Network_NL_N_STEADY_STATE_HYPOTHESIS_2021-02-03T04:30:00Z_1_1D__FM";
String originalNlEqId = "urn:uuid:Network_NL_N_EQUIPMENT_2021-02-03T04:30:00Z_1_1D__FM";
assertEquals(originalBeSshId, getFirstOccurrence(updatedBeSshXml, REGEX_SUPERSEDES));
assertEquals(originalBeEqId, getFirstOccurrence(updatedBeSshXml, REGEX_DEPENDENT_ON));
assertEquals(originalNlSshId, getFirstOccurrence(updatedNlSshXml, REGEX_SUPERSEDES));
assertEquals(originalNlEqId, getFirstOccurrence(updatedNlSshXml, REGEX_DEPENDENT_ON));

// Profiles should be consistent with the instance files
assertEquals("http://entsoe.eu/CIM/SteadyStateHypothesis/1/1", getFirstOccurrence(updatedBeSshXml, REGEX_PROFILE));
Expand Down Expand Up @@ -195,14 +201,17 @@ void testCgmExportWithModelsForSubnetworks() throws IOException {
String updatedNlSshId = "urn:uuid:Network_NL_N_STEADY_STATE_HYPOTHESIS_2021-02-03T04:30:00Z_2_1D__FM";
String originalBeTpId = "urn:uuid:Network_BE_N_TOPOLOGY_2021-02-03T04:30:00Z_1_1D__FM";
String originalNlTpId = "urn:uuid:Network_NL_N_TOPOLOGY_2021-02-03T04:30:00Z_1_1D__FM";
Set<String> expectedDependencies = Set.of(updatedBeSshId, updatedNlSshId, originalBeTpId, originalNlTpId);
String originalTpBdId = "Common TP_BD model ID";
Set<String> expectedDependencies = Set.of(updatedBeSshId, updatedNlSshId, originalBeTpId, originalNlTpId, originalTpBdId);
assertEquals(expectedDependencies, getOccurrences(updatedCgmSvXml, REGEX_DEPENDENT_ON));

// Each updated IGM SSH should supersede the original one
// Each updated IGM SSH should supersede the original one and depend on the original EQ
String originalBeSshId = "urn:uuid:Network_BE_N_STEADY_STATE_HYPOTHESIS_2021-02-03T04:30:00Z_1_1D__FM";
String originalNlSshId = "urn:uuid:Network_NL_N_STEADY_STATE_HYPOTHESIS_2021-02-03T04:30:00Z_1_1D__FM";
assertEquals(originalBeSshId, getFirstOccurrence(updatedBeSshXml, REGEX_SUPERSEDES));
assertEquals(originalNlSshId, getFirstOccurrence(updatedNlSshXml, REGEX_SUPERSEDES));
assertEquals(Set.of("BE EQ model ID"), getOccurrences(updatedBeSshXml, REGEX_DEPENDENT_ON));
assertEquals(Set.of("NL EQ model ID"), getOccurrences(updatedNlSshXml, REGEX_DEPENDENT_ON));

// Profiles should be consistent with the instance files
assertEquals("http://entsoe.eu/CIM/SteadyStateHypothesis/1/1", getFirstOccurrence(updatedBeSshXml, REGEX_PROFILE));
Expand Down Expand Up @@ -254,15 +263,19 @@ void testCgmExportWithModelsForAllNetworks() throws IOException {
String updatedNlSshId = "urn:uuid:Network_NL_N_STEADY_STATE_HYPOTHESIS_2022-03-04T05:30:00Z_4_1D__FM";
String originalBeTpId = "urn:uuid:Network_BE_N_TOPOLOGY_2022-03-04T05:30:00Z_1_1D__FM";
String originalNlTpId = "urn:uuid:Network_NL_N_TOPOLOGY_2022-03-04T05:30:00Z_1_1D__FM";
String originalTpBdId = "Common TP_BD model ID";
String additionalDependency = "Additional dependency";
Set<String> expectedDependencies = Set.of(updatedBeSshId, updatedNlSshId, originalBeTpId, originalNlTpId, additionalDependency);
Set<String> expectedDependencies = Set.of(updatedBeSshId, updatedNlSshId, originalBeTpId,
originalNlTpId, originalTpBdId, additionalDependency);
assertEquals(expectedDependencies, getOccurrences(updatedCgmSvXml, REGEX_DEPENDENT_ON));

// Each updated IGM SSH should supersede the original one
// Each updated IGM SSH should supersede the original one and depend on the original EQ
String originalBeSshId = "urn:uuid:Network_BE_N_STEADY_STATE_HYPOTHESIS_2022-03-04T05:30:00Z_1_1D__FM";
String originalNlSshId = "urn:uuid:Network_NL_N_STEADY_STATE_HYPOTHESIS_2022-03-04T05:30:00Z_1_1D__FM";
assertEquals(originalBeSshId, getFirstOccurrence(updatedBeSshXml, REGEX_SUPERSEDES));
assertEquals(originalNlSshId, getFirstOccurrence(updatedNlSshXml, REGEX_SUPERSEDES));
assertEquals(Set.of("BE EQ model ID"), getOccurrences(updatedBeSshXml, REGEX_DEPENDENT_ON));
assertEquals(Set.of("NL EQ model ID"), getOccurrences(updatedNlSshXml, REGEX_DEPENDENT_ON));

// Profiles should be consistent with the instance files
// The model of the main network brings an additional profile
Expand All @@ -288,6 +301,7 @@ void testCgmExportWithProperties() throws IOException {
exportParams.put(CgmesExport.MODELING_AUTHORITY_SET, "Regional Coordination Center");
exportParams.put(CgmesExport.MODEL_DESCRIPTION, "Common Grid Model export");
exportParams.put(CgmesExport.MODEL_VERSION, "4");
exportParams.put(CgmesExport.BOUNDARY_TP_ID, "ENTSOE TP_BD model ID");
String basename = "test_bare+properties";
network.write("CGMES", exportParams, tmpDir.resolve(basename));
String updatedBeSshXml = Files.readString(tmpDir.resolve(basename + "_BE_SSH.xml"));
Expand All @@ -314,14 +328,19 @@ void testCgmExportWithProperties() throws IOException {
String updatedNlSshId = "urn:uuid:Network_NL_N_STEADY_STATE_HYPOTHESIS_2021-02-03T04:30:00Z_4_1D__FM";
String originalBeTpId = "urn:uuid:Network_BE_N_TOPOLOGY_2021-02-03T04:30:00Z_1_1D__FM";
String originalNlTpId = "urn:uuid:Network_NL_N_TOPOLOGY_2021-02-03T04:30:00Z_1_1D__FM";
Set<String> expectedDependencies = Set.of(updatedBeSshId, updatedNlSshId, originalBeTpId, originalNlTpId);
String originalTpBdId = "ENTSOE TP_BD model ID";
Set<String> expectedDependencies = Set.of(updatedBeSshId, updatedNlSshId, originalBeTpId, originalNlTpId, originalTpBdId);
assertEquals(expectedDependencies, getOccurrences(updatedCgmSvXml, REGEX_DEPENDENT_ON));

// Each updated IGM SSH should supersede the original one
// Each updated IGM SSH should supersede the original one and depend on the original EQ
String originalBeSshId = "urn:uuid:Network_BE_N_STEADY_STATE_HYPOTHESIS_2021-02-03T04:30:00Z_1_1D__FM";
String originalBeEqId = "urn:uuid:Network_BE_N_EQUIPMENT_2021-02-03T04:30:00Z_1_1D__FM";
String originalNlSshId = "urn:uuid:Network_NL_N_STEADY_STATE_HYPOTHESIS_2021-02-03T04:30:00Z_1_1D__FM";
String originalNlEqId = "urn:uuid:Network_NL_N_EQUIPMENT_2021-02-03T04:30:00Z_1_1D__FM";
assertEquals(originalBeSshId, getFirstOccurrence(updatedBeSshXml, REGEX_SUPERSEDES));
assertEquals(originalBeEqId, getFirstOccurrence(updatedBeSshXml, REGEX_DEPENDENT_ON));
assertEquals(originalNlSshId, getFirstOccurrence(updatedNlSshXml, REGEX_SUPERSEDES));
assertEquals(originalNlEqId, getFirstOccurrence(updatedNlSshXml, REGEX_DEPENDENT_ON));

// Profiles should be consistent with the instance files
assertEquals("http://entsoe.eu/CIM/SteadyStateHypothesis/1/1", getFirstOccurrence(updatedBeSshXml, REGEX_PROFILE));
Expand All @@ -347,6 +366,7 @@ void testCgmExportWithModelsAndProperties() throws IOException {
exportParams.put(CgmesExport.MODELING_AUTHORITY_SET, "Regional Coordination Center");
exportParams.put(CgmesExport.MODEL_DESCRIPTION, "Common Grid Model export");
exportParams.put(CgmesExport.MODEL_VERSION, "4");
exportParams.put(CgmesExport.BOUNDARY_TP_ID, "ENTSOE TP_BD model ID");
String basename = "test_bare+models+properties";
network.write("CGMES", exportParams, tmpDir.resolve(basename));
String updatedBeSshXml = Files.readString(tmpDir.resolve(basename + "_BE_SSH.xml"));
Expand Down Expand Up @@ -375,15 +395,19 @@ void testCgmExportWithModelsAndProperties() throws IOException {
String updatedNlSshId = "urn:uuid:Network_NL_N_STEADY_STATE_HYPOTHESIS_2022-03-04T05:30:00Z_4_1D__FM";
String originalBeTpId = "urn:uuid:Network_BE_N_TOPOLOGY_2022-03-04T05:30:00Z_1_1D__FM";
String originalNlTpId = "urn:uuid:Network_NL_N_TOPOLOGY_2022-03-04T05:30:00Z_1_1D__FM";
String originalTpBdId = "ENTSOE TP_BD model ID"; // the parameter prevails on the extension
String additionalDependency = "Additional dependency";
Set<String> expectedDependencies = Set.of(updatedBeSshId, updatedNlSshId, originalBeTpId, originalNlTpId, additionalDependency);
Set<String> expectedDependencies = Set.of(updatedBeSshId, updatedNlSshId, originalBeTpId,
zamarrenolm marked this conversation as resolved.
Show resolved Hide resolved
originalNlTpId, originalTpBdId, additionalDependency);
assertEquals(expectedDependencies, getOccurrences(updatedCgmSvXml, REGEX_DEPENDENT_ON));

// Each updated IGM SSH should supersede the original one
// Each updated IGM SSH should supersede the original one and depend on the original EQ
String originalBeSshId = "urn:uuid:Network_BE_N_STEADY_STATE_HYPOTHESIS_2022-03-04T05:30:00Z_1_1D__FM";
String originalNlSshId = "urn:uuid:Network_NL_N_STEADY_STATE_HYPOTHESIS_2022-03-04T05:30:00Z_1_1D__FM";
assertEquals(originalBeSshId, getFirstOccurrence(updatedBeSshXml, REGEX_SUPERSEDES));
assertEquals(originalNlSshId, getFirstOccurrence(updatedNlSshXml, REGEX_SUPERSEDES));
assertEquals(Set.of("BE EQ model ID"), getOccurrences(updatedBeSshXml, REGEX_DEPENDENT_ON));
assertEquals(Set.of("NL EQ model ID"), getOccurrences(updatedNlSshXml, REGEX_DEPENDENT_ON));

// Profiles should be consistent with the instance files, CGM SV has an additional profile
assertEquals("http://entsoe.eu/CIM/SteadyStateHypothesis/1/1", getFirstOccurrence(updatedBeSshXml, REGEX_PROFILE));
Expand Down Expand Up @@ -647,6 +671,20 @@ private void addModelsForSubnetworks(Network network, int version) {
.addSupersedes("BE SSH previous ID")
.addProfile("http://entsoe.eu/CIM/SteadyStateHypothesis/1/1")
.add()
.newModel()
.setId("BE EQ model ID")
.setSubset(CgmesSubset.EQUIPMENT)
.setVersion(1)
.setModelingAuthoritySet("http://elia.be/CGMES/2.4.15")
.addProfile("http://entsoe.eu/CIM/EquipmentCore/3/1")
.add()
.newModel()
.setId("Common TP_BD model ID")
.setSubset(CgmesSubset.TOPOLOGY_BOUNDARY)
.setVersion(1)
.setModelingAuthoritySet("http://www.entsoe.eu/OperationalPlanning")
.addProfile("http://entsoe.eu/CIM/TopologyBoundary/3/1")
.add()
.add();
network.getSubnetwork("Network_NL")
.newExtension(CgmesMetadataModelsAdder.class)
Expand All @@ -659,6 +697,20 @@ private void addModelsForSubnetworks(Network network, int version) {
.addSupersedes("NL SSH previous ID")
.addProfile("http://entsoe.eu/CIM/SteadyStateHypothesis/1/1")
.add()
.newModel()
.setId("NL EQ model ID")
.setSubset(CgmesSubset.EQUIPMENT)
.setVersion(1)
.setModelingAuthoritySet("http://tennet.nl/CGMES/2.4.15")
.addProfile("http://entsoe.eu/CIM/EquipmentCore/3/1")
.add()
.newModel()
.setId("Common TP_BD model ID")
.setSubset(CgmesSubset.TOPOLOGY_BOUNDARY)
.setVersion(1)
.setModelingAuthoritySet("http://www.entsoe.eu/OperationalPlanning")
.addProfile("http://entsoe.eu/CIM/TopologyBoundary/3/1")
.add()
.add();
}

Expand Down
6 changes: 3 additions & 3 deletions docs/grid_exchange_formats/cgmes/export.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ If a version number is given as a parameter, it is used for the exported files.
The quick CGM export will always write updated SSH files for IGMs and a single SV for the CGM. The parameter for selecting which profiles to export is ignored in this kind of export.

If the dependencies have to be updated automatically (see parameter **iidm.export.cgmes.update-dependencies** below), the exported instance files will contain metadata models where:
* Updated SSH for IGMs supersede the original ones.
* Updated SV for the CGM depends on the updated SSH from IGMs and on the original TP from IGMs.
* Updated SSH for IGMs supersede the original ones, and depend on the original EQ from IGMs.
* Updated SV for the CGM depends on the updated SSH from IGMs and on the original TP and TP_BD from IGMs.

The filenames of the exported instance files will follow the pattern:
* For the CGM SV: `<basename>_SV.xml`.
Expand Down Expand Up @@ -71,7 +71,7 @@ exampleBase_NL_SSH.xml
exampleBase_SV.xml
```

where the updated SSH files will supersede the original ones, and the SV will contain the correct dependencies of new SSH and original TPs.
where the updated SSH files will supersede the original ones and depend on the original EQs, and the SV will contain the correct dependencies of new SSH and original TPs and TP_BD.

(cgmes-cgm-manual-export)=
## CGM (Common Grid Model) manual export
Expand Down