Move return versions to correct licence #2660
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
https://eaflood.atlassian.net/browse/WATER-4794 > TL;DR; Never amend a licence reference in NALD! Whilst working on WATER-4734 we were pulling together a list of all licences deleted from NALD that still exist in WRLS. Because we have the latest version of the WATER-4654 (return version data clean-up) changes in our non-prod environments (excluding pre), we were expecting no deleted licences to be flagged as having return logs. However, six came back in the results. We know the changes to clean the return versions handle return versions linked to 'completed' return logs. If this scenario occurs, we won't delete it until it can be looked at. So, when cleaning the licences, we can do a bit of a shortcut: any WRLS licence due to be deleted which still has a `water.return_version` record must do so because it is linked to a 'completed' return log. Taking **MD/0280003/012** as an example, we confirmed there are no return logs. Huh. If this licence is deleted in NALD, then so must be the return versions and all child records, so why have we not deleted it in our clean job? What we found is the NALD return version has not been deleted. It and all the child records remain. However, when we look at the NALD licence record it is linked to, it has a `LIC_NO` of **MD/028/0003/012**. Can you spot the difference?! 😂 We've deduced that when first creating the licence in NALD, a typo was made with the licence reference. But WRLS would have imported the record and created a `water.licences` record (plus others in the other schemas). When the typo was spotted, rather than creating a new NALD licence record, the typo was corrected. If the previous team had used the NALD table ID, as they do with all other tables, then the import would have just corrected the WRLS licence record. However, they use the licence reference as the unique identifier for reasons known only to them! This means the import thinks **MD/028/0003/012** is an entirely new licence, so it creates a second WRLS record. During the nightly import, the NALD return version records update the existing information, all pointing to the wrong WRLS licence record! This change adds a migration to update the licence the affected return versions are linked to. The end result is when searching for the licence ref as now set in NALD, users will see the return versions. In the future. we'll be able to delete the abandoned WRLS record.
Cruikshanks
added a commit
to DEFRA/water-abstraction-import
that referenced
this pull request
Nov 26, 2024
https://eaflood.atlassian.net/browse/WATER-4794 While working on WATER-4734, we pulled together a list of all licences deleted from NALD that still exist in WRLS. Because we have the latest version of the WATER-4654 (return version data clean-up) changes in our non-prod environments (excluding pre), we were expecting no deleted licences to be flagged as having return logs. However, six came back in the results. We know our previous changes to clean the return versions handle return versions linked to 'completed' return logs. If this scenario occurs, we won't delete it until it can be looked at. So, when it comes to cleaning the licences, we can do a bit of a shortcut: any WRLS licence due to be deleted which still has a `water.return_version` record must do so because it is linked to a 'completed' return log. Taking **MD/0280003/012** as an example, we confirmed there are no return logs but it was one of the six returned in the results. Huh? If this licence is deleted in NALD, then so must be the return versions and all child records, so why have we not deleted it in our clean job? What we found is the NALD return version has not been deleted. It and all the child records remain in NALD. However, when we look at the NALD licence record it is linked to, it has a `LIC_NO` of **MD/028/0003/012**. Can you spot the difference?! 😂 We've deduced that when first creating the licence in NALD, a typo was made with the licence reference. But WRLS would have imported the record and created a **MD/0280003/012** `water.licences` record. When the typo was spotted, rather than creating a new NALD licence record, the typo was corrected. If the previous team had used the NALD table ID, as they do with all other tables, then the import would have just corrected the WRLS licence record. However, they use the licence reference as the unique identifier for reasons known only to them! This means the import thinks **MD/028/0003/012** is an entirely new licence, so it creates a second WRLS record. During the nightly import, the NALD return version records update the existing information, but they point to the wrong WRLS licence record! Initially, we thought this would require a manual intervention so added [a fix migration to water-abstraction-service](DEFRA/water-abstraction-service#2660). However, after double checking the import query for the return versions we realised if we include `licence_id` in the `ON CONFLICT DO UPDATE` statement, this would also fix the issue. Plus, this would fix any future licence reference amendments.
Cruikshanks
added a commit
to DEFRA/water-abstraction-import
that referenced
this pull request
Nov 26, 2024
https://eaflood.atlassian.net/browse/WATER-4794 While working on WATER-4734, we pulled together a list of all licences deleted from NALD that still exist in WRLS. Because we have the latest version of the WATER-4654 (return version data clean-up) changes in our non-prod environments (excluding pre), we were expecting no deleted licences to be flagged as having return logs. However, six came back in the results. We know our previous changes to clean the return versions handle return versions linked to 'completed' return logs. If this scenario occurs, we won't delete it until it can be looked at. So, when it comes to cleaning the licences, we can do a bit of a shortcut: any WRLS licence due to be deleted which still has a `water.return_version` record must do so because it is linked to a 'completed' return log. Taking **MD/0280003/012** as an example, we confirmed there are no return logs but it was one of the six returned in the results. Huh? If this licence is deleted in NALD, then so must be the return versions and all child records, so why have we not deleted it in our clean job? What we found is the NALD return version has not been deleted. It and all the child records remain in NALD. However, when we look at the NALD licence record it is linked to, it has a `LIC_NO` of **MD/028/0003/012**. Can you spot the difference?! 😂 We've deduced that when first creating the licence in NALD, a typo was made with the licence reference. But WRLS would have imported the record and created a **MD/0280003/012** `water.licences` record. When the typo was spotted, rather than creating a new NALD licence record, the typo was corrected. If the previous team had used the NALD table ID, as they do with all other tables, then the import would have just corrected the WRLS licence record. However, they use the licence reference as the unique identifier for reasons known only to them! This means the import thinks **MD/028/0003/012** is an entirely new licence, so it creates a second WRLS record. During the nightly import, the NALD return version records update the existing information, but they point to the wrong WRLS licence record! Initially, we thought this would require manual intervention, so we added [a fix migration to water-abstraction-service](DEFRA/water-abstraction-service#2660). However, after double-checking the import query for the return versions, we realised if we included `licence_id` in the `ON CONFLICT DO UPDATE` statement, this would also fix the issue. Plus, this would fix any future licence reference amendments.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://eaflood.atlassian.net/browse/WATER-4794
While working on WATER-4734, we pulled together a list of all licences deleted from NALD that still exist in WRLS.
Because we have the latest version of the WATER-4654 (return version data clean-up) changes in our non-prod environments (excluding pre), we were expecting no deleted licences to be flagged as having return logs. However, six came back in the results.
We know the changes to clean the return versions handle return versions linked to 'completed' return logs. If this scenario occurs, we won't delete it until it can be looked at. So, when cleaning the licences, we can do a bit of a shortcut: any WRLS licence due to be deleted which still has a
water.return_versionrecord must do so because it is linked to a 'completed' return log.Taking MD/0280003/012 as an example, we confirmed there are no return logs. Huh. If this licence is deleted in NALD, then so must be the return versions and all child records, so why have we not deleted it in our clean job?
What we found is the NALD return version has not been deleted. It and all the child records remain. However, when we look at the NALD licence record it is linked to, it has a
LIC_NOof MD/028/0003/012. Can you spot the difference?! 😂We've deduced that when first creating the licence in NALD, a typo was made with the licence reference. But WRLS would have imported the record and created a
water.licencesrecord (plus others in the other schemas). When the typo was spotted, rather than creating a new NALD licence record, the typo was corrected.If the previous team had used the NALD table ID, as they do with all other tables, then the import would have just corrected the WRLS licence record. However, they use the licence reference as the unique identifier for reasons known only to them! This means the import thinks MD/028/0003/012 is an entirely new licence, so it creates a second WRLS record.
During the nightly import, the NALD return version records update the existing information, all pointing to the wrong WRLS licence record!
This change adds a migration to update the licence to which the affected return versions are linked. The result is that users will see the return versions when searching for the licence reference, as it is now set in NALD. In the future, we can delete the abandoned WRLS record.