Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make sure we don't write duplicated indicators #1058

Merged
merged 2 commits into from
Aug 27, 2024

Conversation

ZakarFin
Copy link
Member

@ZakarFin ZakarFin commented Mar 18, 2024

When processing indicators from a statistical datasource. It breaks the UI completely when the user scrolls to such indicator:

aineistohaku

This might happen when we have multiple servers that are processing the indicator list at the same time. One server might save the indicator as processed while another still has it's on its work queue. When the other server saves it gets the processed one from redis and adds a duplicate from its work queue.

ant-design/ant-design#33864

@ZakarFin
Copy link
Member Author

Maybe we should just enable restricting the update to be done on a single node on clustered env. Nothing here really addresses the race condition issue where two nodes get a common list of saved indicators from Redis, but might add different sets when saving the changes at the same time (overwriting the other nodes processed data). This might lead to some indicators not being listed after the processing has been done.

@ZakarFin
Copy link
Member Author

Disabling this might work with just adding this empty value prop: oskari.scheduler.job.StatisticsDatasources=. The scheduler will log a warning with "Available scheduled job StatisticsDatasources needs the cronLine configuration parameter" but it should not run the update when the line is empty.

@ZakarFin
Copy link
Member Author

ZakarFin commented Mar 28, 2024

Documenting here that configuring oskari.scheduler.job.StatisticsDatasources= did not disable the scheduler to run the update. But taking a better look it needs .cronLine added to the key:

So trying again with oskari.scheduler.job.StatisticsDatasources.cronLine=

@ZakarFin
Copy link
Member Author

oskari.scheduler.job.StatisticsDatasources.cronLine= with the empty value works for disabling the processing on multi-server setup.

@ZakarFin ZakarFin added this to the 2.14.0 milestone Aug 27, 2024
@ZakarFin ZakarFin merged commit 07d4725 into oskariorg:develop Aug 27, 2024
2 checks passed
@ZakarFin ZakarFin deleted the duplicate-indicators branch August 27, 2024 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant