Skip to content

only generate checkpoint file when the job does not complete #452

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
weideng1 wants to merge 1 commit into
base: 1.x
Choose a base branch
from
Open

only generate checkpoint file when the job does not complete #452

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion workflow/commons/src/main/java/com/datastax/oss/dsbulk/workflow/commons/log/LogManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,7 @@ public void reportAvailableFiles() throws IOException {
}
if (checkpointEnabled) {
CheckpointManager manager = mergeCheckpointManagers();
if (!manager.isEmpty()) {
if (!manager.isEmpty() && !manager.isComplete(replayStrategy)) {
Copy link
Contributor

@adutra adutra Aug 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will indeed fix the issue but will trigger a bunch of integration test failures because we have a few tests expecting such a file to be generated. Especially the tests in CSVEndToEndSimulacronIT.

Copy link
Collaborator Author

@weideng1 weideng1 Aug 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adutra I see what you're talking about. Basically we need to address the following 10 integration test failures, right?

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   CSVEndToEndSimulacronIT.full_load:202
Expecting path:
  /var/folders/kt/wswnnkcn1mxg8bkdxwm_6k1h0000gn/T/logs8486694306328228675/LOAD_20220816-160910-818666/checkpoint.csv
to exist (symbolic links were followed).
[ERROR]   CSVEndToEndSimulacronIT.full_unload:1209
Expecting path:
  /var/folders/kt/wswnnkcn1mxg8bkdxwm_6k1h0000gn/T/logs2302077949864412390/UNLOAD_20220816-160916-095270/checkpoint.csv
to exist (symbolic links were followed).
[ERROR]   CSVEndToEndSimulacronIT.full_unload_csv_default_modification:1288
Expecting path:
  /var/folders/kt/wswnnkcn1mxg8bkdxwm_6k1h0000gn/T/logs5564278254796049513/UNLOAD_20220816-160910-627139/checkpoint.csv
to exist (symbolic links were followed).
[ERROR]   CSVEndToEndSimulacronIT.full_unload_large_result_set:1320
Expecting path:
  /var/folders/kt/wswnnkcn1mxg8bkdxwm_6k1h0000gn/T/logs1208374229285479517/UNLOAD_20220816-160917-095643/checkpoint.csv
to exist (symbolic links were followed).
[ERROR]   CSVEndToEndSimulacronIT.massive_load_with_errors:896
Expecting path:
  /var/folders/kt/wswnnkcn1mxg8bkdxwm_6k1h0000gn/T/logs7228255075026913737/LOAD_20220816-160914-338597/checkpoint.csv
to exist (symbolic links were followed).
[ERROR]   CSVEndToEndSimulacronIT.massive_unload_with_errors:1128
Expecting path:
  /var/folders/kt/wswnnkcn1mxg8bkdxwm_6k1h0000gn/T/logs5864708909355599806/UNLOAD_20220816-160919-971570/checkpoint.csv
to exist (symbolic links were followed).
[ERROR]   CSVEndToEndSimulacronIT.massive_unload_with_errors:1128
Expecting path:
  /var/folders/kt/wswnnkcn1mxg8bkdxwm_6k1h0000gn/T/logs7558778109783596575/UNLOAD_20220816-160922-513469/checkpoint.csv
to exist (symbolic links were followed).
[ERROR]   CSVEndToEndSimulacronIT.unload_binary:1531
Expecting path:
  /var/folders/kt/wswnnkcn1mxg8bkdxwm_6k1h0000gn/T/logs4874744943006336612/UNLOAD_20220816-160916-867395/checkpoint.csv
to exist (symbolic links were followed).
[ERROR]   CSVEndToEndSimulacronIT.unload_binary:1531
Expecting path:
  /var/folders/kt/wswnnkcn1mxg8bkdxwm_6k1h0000gn/T/logs7539088680158356009/UNLOAD_20220816-160916-899891/checkpoint.csv
to exist (symbolic links were followed).
[ERROR]   CSVEndToEndSimulacronIT.validate_stdout:1488
Expecting path:
  /var/folders/kt/wswnnkcn1mxg8bkdxwm_6k1h0000gn/T/logs667372273589927031/UNLOAD_20220816-160916-807864/checkpoint.csv
to exist (symbolic links were followed).
[INFO]
[ERROR] Tests run: 75, Failures: 10, Errors: 0, Skipped: 0

Copy link
Contributor

@adutra adutra Aug 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are many more I'm afraid. Our internal CI shows a lot of failures in CSVConnectorEndToEndCCMIT and JsonConnectorEndToEndCCMIT.

You can run CCM tests locally by enabling the medium profile:

mvn clean install -Pmedium

Side note: we should reserve the branches ending with .x for releases only. Next time, please fork the repo and create a branch targeting 1.10.x and base your PR on it.

writeCheckpointFile(manager);
LOGGER.info("Checkpoints for the current operation were written to {}.", CHECKPOINT_CSV);
LOGGER.info(
Expand Down