Fix `CSVLogger` trying to append to file from previous run in same version folder #19446

awaelchli · 2024-02-11T14:13:02Z

What does this PR do?

Fixes #19432

This fixes an edge case when you try to log to a version folder from a previous run. We already had a warning that says

Experiment logs directory {self.log_dir} exists and is not empty.
Previous log files in this directory will be deleted when the new ones are saved!

But we should delete the previous file, otherwise the logger will try to append to it, and run into an issue when the metric keys are different.

cc @Borda @carmocca @justusschock @awaelchli

github-actions · 2024-02-11T14:23:13Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.13, oldest)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.1)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.2)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.13, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.1)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.2)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.13, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.1)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.2)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅
pl-cpu (macOS-12, pytorch, 3.11, 2.0)	success	✅
pl-cpu (macOS-12, pytorch, 3.11, 2.1)	success	✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.0)	success	✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.1)	success	✅
pl-cpu (windows-2022, pytorch, 3.11, 2.0)	success	✅
pl-cpu (windows-2022, pytorch, 3.11, 2.1)	success	✅

These checks are required after the changes to .github/workflows/ci-tests-pytorch.yml, src/lightning/fabric/loggers/csv_logs.py, tests/tests_pytorch/loggers/test_csv.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs) (testing Lightning \| latest)	success	✅
pytorch-lightning (GPUs) (testing PyTorch \| latest)	success	✅

These checks are required after the changes to tests/tests_pytorch/loggers/test_csv.py, src/lightning/fabric/loggers/csv_logs.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to src/lightning/fabric/loggers/csv_logs.py.

🟢 fabric: Docs

Check ID	Status
docs-make (fabric, doctest)	success	✅
docs-make (fabric, html)	success	✅

These checks are required after the changes to src/lightning/fabric/loggers/csv_logs.py.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, lightning, 3.8, 1.13, oldest)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
fabric-cpu (macOS-11, lightning, 3.11, 2.1)	success	✅
fabric-cpu (macOS-11, lightning, 3.11, 2.2)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.13, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.1)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.2)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.13, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
fabric-cpu (windows-2022, lightning, 3.11, 2.1)	success	✅
fabric-cpu (windows-2022, lightning, 3.11, 2.2)	success	✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13)	success	✅
fabric-cpu (macOS-12, fabric, 3.11, 2.0)	success	✅
fabric-cpu (macOS-12, fabric, 3.11, 2.1)	success	✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.0)	success	✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.1)	success	✅
fabric-cpu (windows-2022, fabric, 3.11, 2.0)	success	✅
fabric-cpu (windows-2022, fabric, 3.11, 2.1)	success	✅

These checks are required after the changes to src/lightning/fabric/loggers/csv_logs.py, tests/tests_fabric/loggers/test_csv.py, .github/workflows/ci-tests-fabric.yml.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs) (testing Fabric \| latest)	success	✅
lightning-fabric (GPUs) (testing Lightning \| latest)	success	✅

These checks are required after the changes to src/lightning/fabric/loggers/csv_logs.py, tests/tests_fabric/loggers/test_csv.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/fabric/loggers/csv_logs.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.11)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.11)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.11)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.11)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.11)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.11)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.11)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.11)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.11)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.11)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.11)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.11)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.11)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.11)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.11)	success	✅

These checks are required after the changes to src/lightning/fabric/loggers/csv_logs.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

codecov · 2024-02-11T14:36:53Z

Codecov Report

Merging #19446 (bf41052) into master (b097a4d) will decrease coverage by 30%.
The diff coverage is 100%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #19446      +/-   ##
==========================================
- Coverage      84%      53%     -30%     
==========================================
  Files         450      445       -5     
  Lines       38087    37993      -94     
==========================================
- Hits        31810    20272   -11538     
- Misses       6277    17721   +11444

…rsion folder (#19446)

Fix

57f265c

awaelchli added bug Something isn't working logger: csv labels Feb 11, 2024

awaelchli added this to the 2.2.x milestone Feb 11, 2024

github-actions bot added the fabric lightning.fabric.Fabric label Feb 11, 2024

awaelchli added fun Staff contributions outside working hours - to differentiate from the "community" label and removed fabric lightning.fabric.Fabric labels Feb 11, 2024

awaelchli added 2 commits February 11, 2024 15:19

update

ac8923d

test

bc818d1

awaelchli marked this pull request as ready for review February 11, 2024 14:22

awaelchli requested review from carmocca, justusschock, Borda and williamFalcon as code owners February 11, 2024 14:22

github-actions bot added fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package labels Feb 11, 2024

awaelchli changed the title ~~Fix CSVLogger trying to append to file from previous run in save version folder~~ Fix CSVLogger trying to append to file from previous run in same version folder Feb 11, 2024

carmocca approved these changes Feb 12, 2024

View reviewed changes

Merge branch 'master' into bugfix/csv-file-exists

bca94c7

Borda changed the title ~~Fix CSVLogger trying to append to file from previous run in same version folder~~ Fix CSVLogger trying to append to file from previous run in same version folder Feb 13, 2024

Borda approved these changes Feb 13, 2024

View reviewed changes

Merge branch 'master' into bugfix/csv-file-exists

ed7abef

mergify bot added the ready PRs ready to be merged label Feb 13, 2024

awaelchli added 2 commits February 13, 2024 15:05

Merge branch 'master' into bugfix/csv-file-exists

65042bd

I messed up the checkgroup again

bf41052

awaelchli requested a review from ethanwharris as a code owner February 13, 2024 15:13

github-actions bot added the ci Continuous Integration label Feb 13, 2024

awaelchli merged commit 3fbc29b into master Feb 13, 2024
111 checks passed

awaelchli deleted the bugfix/csv-file-exists branch February 13, 2024 18:59

awaelchli added a commit that referenced this pull request Mar 1, 2024

Fix CSVLogger trying to append to file from previous run in same ve…

836319e

…rsion folder (#19446)

lantiga pushed a commit that referenced this pull request Mar 4, 2024

Fix CSVLogger trying to append to file from previous run in same ve…

e6a763b

…rsion folder (#19446)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `CSVLogger` trying to append to file from previous run in same version folder #19446

Fix `CSVLogger` trying to append to file from previous run in same version folder #19446

awaelchli commented Feb 11, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Feb 11, 2024 •

edited

Loading

codecov bot commented Feb 11, 2024 •

edited

Loading

Fix CSVLogger trying to append to file from previous run in same version folder #19446

Fix CSVLogger trying to append to file from previous run in same version folder #19446

Conversation

awaelchli commented Feb 11, 2024 • edited by github-actions bot Loading

What does this PR do?

github-actions bot commented Feb 11, 2024 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

codecov bot commented Feb 11, 2024 • edited Loading

Codecov Report

Fix `CSVLogger` trying to append to file from previous run in same version folder #19446

Fix `CSVLogger` trying to append to file from previous run in same version folder #19446

awaelchli commented Feb 11, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Feb 11, 2024 •

edited

Loading

codecov bot commented Feb 11, 2024 •

edited

Loading