Deal with deleted lics in rtn ver import & clean #1045
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
https://eaflood.atlassian.net/browse/WATER-4654 > Part of the work to migrate management of return versions from NALD to WRLS As part of migrating management of return versions to WRLS we have been making a series of changes to [water-abstraction-import](https://github.com/DEFRA/water-abstraction-import) to ensure our data is as 'clean' as possible. Users in NALD are able to delete records but until we started this work, they were never deleted on the WRLS side. We moved everything related to return versions into its own job, then updated the import side to ensure the NALD data was transformed to a state we can work with, and parts that were previously not included such as points are now imported. After that we then added a clean step to ensure return information that no longer existed in NALD was also deleted in WRLS. This work has fed the next data clean up task: cleaning up the licence information. One of the things that will block us deleting a licence from WRLS (that has been deleted in NALD) is if it has a 'completed' return log. This is external user submitted data so we can't delete it, nor can we delete the licence record it is linked to, even if it has been deleted in NALD. We made an assumption that when a licence is deleted in NALD _all_ child records are also deleted. So, when it comes to deleting the licence, we can assume if a `water.return_version` record still exists in WRLS it is because we blocked deletion due to a 'completed' return log. It turns out that core assumption is wrong. There are examples of licences that have been deleted from NALD, but their child records still exist, including return versions. This means our import is importing data for deleted licences, and our clean is never removing it because the record _is in_ NALD! This change updates our return versions job to cater for what we now know. - **import** - exclude return information where the licence record no longer exists - **clean** - include return information where the child record might exist, but the parent doesn't
Cruikshanks
added
the
housekeeping
|
This was not the issue. What was actually going on was that the return versions were linked to the wrong WRLS licence record. WATER-4794 covers it in more detail, and Move return versions to correct licence fixes it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://eaflood.atlassian.net/browse/WATER-4654
As part of migrating management of return versions to WRLS, we have been making a series of changes to water-abstraction-import to ensure our data is as 'clean' as possible. Users in NALD can delete records, but until we started this work, they were never deleted on the WRLS side.
We moved everything related to return versions into its own job, then updated the import side to ensure the NALD data was transformed to a state we can work with, and parts previously not included, such as points, are now imported. After that, we added a clean step to ensure return information that no longer existed in NALD was also deleted in WRLS.
This work has fed the subsequent data clean-up task: cleaning up the licence information. One of the things that will block us from deleting a licence from WRLS (that has been deleted in NALD) is if it has a 'completed' return log. This is external user-submitted data, so we can't delete it or the licence record it is linked to, even if it has been deleted in NALD.
We assumed that when a licence is deleted in NALD all child records are also deleted. So, when it comes to deleting the licence, we can assume if a
water.return_versionrecord still exists in WRLS, it is because we blocked deletion due to a 'completed' return log.It turns out that core assumption is wrong. There are examples of licences that have been deleted from NALD, but their child records still exist, including return versions. This means our import imports data for deleted licences, and our clean never remove it because the record is in NALD!
This change updates our return versions job to cater to what we now know.