Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Website: show total number of (filtered) benchmarks #80

Merged
merged 6 commits into from
Dec 12, 2024

Conversation

jacek-oet
Copy link
Collaborator

Test result:
New result CSV with the latest HiGHS run time
2024-12-11T08-31_export_with_highs_run_time.csv

FIELD1 Benchmark Size Solver Solver Version Solver Release Year Status Termination Condition Runtime (s) Memory Usage (MB) Objective Value Max Integrality Violation Duality Gap highs_run_time
101 Sienna_modified_RTS_GMLC_DA_sys_NetCopperPlate_Horizon12_Day29 1-1h glpk 5.0 2020 warning unknown 60 8192 60
102 Sienna_modified_RTS_GMLC_DA_sys_NetCopperPlate_Horizon12_Day314 1-1h glpk 5.0 2020 warning unknown 60 8192 60
103 Sienna_modified_RTS_GMLC_DA_sys_NetCopperPlate_Horizon12_Day332 1-1h glpk 5.0 2020 warning unknown 60 8192 21.268036365509033
.........
880 tulipa-1_EU_investment_simple 28-672h scip 9.1.1 2024 TO Timeout 60 8192 60
882 tulipa-1_EU_investment_simple 28-2016h scip 9.1.1 2024 TO Timeout 60 8192 60
884 tulipa-1_EU_investment_simple 28-4032h scip 9.1.1 2024 TO Timeout 60 8192 60
886 tulipa-1_EU_investment_simple 28-8760h scip 9.1.1 2024 TO Timeout 60 8192 60

When we add a filter for size, such as showing only XXS, it will become...

FIELD1 Benchmark Size Solver Solver Version Solver Release Year Status Termination Condition Runtime (s) Memory Usage (MB) Objective Value Max Integrality Violation Duality Gap highs_run_time
86 genx-7_three_zones_w_colocated_VRE_storage 3-24h glpk 5.0 2020 warning unknown 60 8192 0.4413185119628906
36 pypsa-eur-elec-op 2-24h glpk 5.0 2020 ok optimal 6.36882758140564 137.076 7269027498 1.224797248840332
37 pypsa-eur-elec-op 3-24h glpk 5.0 2020 ok optimal 17.552338123321533 154.268 7346414429 3.970861196517944
38 pypsa-eur-elec-op 2-12h glpk 5.0 2020 ok optimal 30.239359378814697 174.064 7389777942 5.883429288864136
14 pypsa-eur-elec-trex 2-24h glpk 5.0 2020 ok optimal 6.441955327987671 136.976 7269143813 1.790940284729004
15 pypsa-eur-elec-trex 3-24h glpk 5.0 2020 ok optimal 19.342355012893677 154.192 7250311849 6.746302843093872
78 pypsa-gas+wind+sol+ely 1-1h glpk 5.0 2020 TO Timeout 60 8192 6.2434680461883545
95 tulipa-1_EU_investment_simple 28-24h glpk 5.0 2020 warning unknown 60 8192 8.176160335540771
857 genx-7_three_zones_w_colocated_VRE_storage 3-24h highs 1.8.1 2024 ok optimal 0.4413185119628906 138.312 443334.430587612 0.499681269698442 0.0000056893869610606685 0.4413185119628906
757 pypsa-eur-elec-op 2-24h highs 1.8.1 2024 ok optimal 1.224797248840332 196.712 7269027497.86813 1.224797248840332
759 pypsa-eur-elec-op 3-24h highs 1.8.1 2024 ok optimal 3.970861196517944 248.24 7346414429.227344 3.970861196517944
761 pypsa-eur-elec-op 2-12h highs 1.8.1 2024 ok optimal 5.883429288864136 279.06 7389777941.591801 5.883429288864136
713 pypsa-eur-elec-trex 2-24h highs 1.8.1 2024 ok optimal 1.790940284729004 199.572 7269143813.066846 1.790940284729004
715 pypsa-eur-elec-trex 3-24h highs 1.8.1 2024 ok optimal 6.746302843093872 254.488 7250311849.251861 6.746302843093872
841 pypsa-gas+wind+sol+ely 1-1h highs 1.8.1 2024 ok optimal 6.2434680461883545 422.72 31714138111.86105 6.2434680461883545
875 tulipa-1_EU_investment_simple 28-24h highs 1.8.1 2024 ok optimal 8.176160335540771 206.452 223314143.52914777 0.5 0.00009676663361641652 8.176160335540771
858 genx-7_three_zones_w_colocated_VRE_storage 3-24h scip 9.1.1 2024 ok optimal 3.213060140609741 232.316 443334.4305876122 0.499943913150048 0 0.4413185119628906
758 pypsa-eur-elec-op 2-24h scip 9.1.1 2024 ok optimal 2.8956098556518555 270.608 7269027497.868132 1.224797248840332
760 pypsa-eur-elec-op 3-24h scip 9.1.1 2024 ok optimal 15.086172819137571 355.332 7346414429.227432 3.970861196517944
762 pypsa-eur-elec-op 2-12h scip 9.1.1 2024 ok optimal 15.212609767913818 423.084 7389777941.591944 5.883429288864136
714 pypsa-eur-elec-trex 2-24h scip 9.1.1 2024 ok optimal 4.4813501834869385 276.036 7269143813.066574 1.790940284729004
716 pypsa-eur-elec-trex 3-24h scip 9.1.1 2024 ok optimal 12.085017681121826 354.912 7250311849.251635 6.746302843093872
842 pypsa-gas+wind+sol+ely 1-1h scip 9.1.1 2024 TO Timeout 60 8192 6.2434680461883545
876 tulipa-1_EU_investment_simple 28-24h scip 9.1.1 2024 ok optimal 8.024469375610352 209.644 223310171.76019543 0.5 0 8.176160335540771

We will have 5 GLPK, 8 HiGHS, and 7 SCIP instances with an OK status.
image

@siddharth-krishna
Copy link
Contributor

Thanks Jacek. What you did makes sense, but I'm worried it might be confusing, because when I remove the higher size categories, the main "Details" table also changes, including the timeout amount. This isn't true, because as you can see from the max runtime for scip, the timeout is more than 1 minute.
image

Would it be possible to use the number of instances after the filter in the following line, but not in the Details table (that always shows the total number of benchmarks and the overall timeout).

Solved benchmarks is the number of benchmarks where the solver returns an ok status, out of 6 benchmarks

If possible, could we also change the line to say out of 6 **filtered** benchmarks when filters are active, to remind the user?

Copy link
Contributor

@siddharth-krishna siddharth-krishna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor changes, please 🙏

@siddharth-krishna siddharth-krishna changed the title Feat/add min and max runtime Website: show total number of (filtered) benchmarks Dec 12, 2024
@siddharth-krishna siddharth-krishna merged commit 857f954 into main Dec 12, 2024
1 check passed
@siddharth-krishna siddharth-krishna deleted the feat/add-min-and-max-runtime branch December 12, 2024 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants