Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add new licence changes table #2666

Merged
merged 5 commits into from
Jan 7, 2025
Merged

Add new licence changes table #2666

merged 5 commits into from
Jan 7, 2025

Conversation

Cruikshanks
Copy link
Member

@Cruikshanks Cruikshanks commented Jan 5, 2025 •
edited
Loading

https://eaflood.atlassian.net/browse/WATER-4546

Part of the work to migrate management of return versions to WRLS from NALD

During the import from NALD, if a licence end date is changed, for example, if it has been revoked in NALD, our return version functionality needs to know about it. This is so we can reissue the return logs for the licence to match the changed end date.

We are actively trying to move away from the legacy code base, so this work was always going to be done in water-abstraction-system. Initially, it would be triggered by our own import process due to replace the legacy one. But that was when we thought ReSP would be taking over from NALD.

Now, the plan is for WRLS to encompass the final abstraction leg and take over from NALD. There is little point in replacing a complex import we intend to switch off in the next year.

But, knowing we'd need something in the interim, we created the /jobs/licence-changes job. The intent was to schedule this after the first NALD import job had completed (the one that downloads and extracts the NALD data) but before the main licence import
job. That way, our job could compare the NALD and WRLS licence records to determine if an end date has changed.

But then we had our 'doh!' moment.

Our reissue return logs engine expects to be given a licence ID and the date the change applies. However, the WRLS licence record needs to have been updated for it to determine the start and end dates of the new return logs.

If we schedule /jobs/licence-changes before the licence import job, it will see the change but won't reissue anything because the WRLS record won't have been updated. If we schedule it after, it won't see any difference and won't trigger the reissue.

Doh!

So, we've needed to change tact on this completely. Now, we intend to

  • trigger /jobs/licence-changes (though we'll move it to a different endpoint as it will no longer be a job) from the NALD import job rather than schedule it
  • update /jobs/licence-changes to store the details of changed licences in a new table
  • add a new endpoint that processes these 'licence change' records, and which can be triggered from the licence import job

This change is needed to create the new table in the water schema.

@Cruikshanks
Add new licence changes table
489bf84 
https://eaflood.atlassian.net/browse/WATER-4546

> Part of the work to migrate management of return versions to WRLS from NALD

During the import from NALD, if a licence end date is changed, for example, it has been revoked in NALD our return versions functionality needs to know about it. This is so we can reissue the return logs for the licence in order to match the changed end date.

We are actively trying to move away from the legacy code base, so this work was always going to be done in [water-abstraction-system](https://github.com/DEFRA/water-abstraction-system). Initially, it was going to be triggered by our [own import process that was due to replace the legacy one](https://eaflood.atlassian.net/browse/WATER-4535). But that was when we thought ReSP would be taking over from NALD.

Now the plan is WRLS will encompass the final abstraction leg and takeover from NALD. There is little point in replacing a complex import we intend to switch off in the next year.

But, knowing we'd need something in the interim we created the `/jobs/licence-changes` job in **water-abstraction-system**. The intent was to schedule this after the first NALD import job had completed (that one that downloads and extracts the NALD data), but before the main licence import job. That way our job could compare the NALD and WRLS licence records in order to determine if an end date has changed.

But then we had our 'doh!' moment.

Our reissue return logs engine expects to be given a licence ID and date the change applies. But it needs the WRLS licence record to have been updated in order to determine what the new return logs start and end dates should be.

If we schedule `/jobs/licence-changes` before the licence import job, it will see the change but won't reissue anything because the WRLS record won't have been updated. If we schedule it after, it won't see any different so won't trigger the reissue.

Doh!

So, we've needed to completely change tact on this. Now we intend to

- _trigger_ `/jobs/licence-changes` (though we'll move the it to a different endpoint as it will no longer be a job) from the [water-abstraction-import](https://github.com/DEFRA/water-abstraction-import) NALD import job rather than schedule it
- update `/jobs/licence-changes` to store the details of changed licences in a new table
- add a new endpoint that processes these 'licence change' records, and which can be triggered from the **water-abstraction-import** main licence import job

This change is needed to create the new table in the `water` schema.
@Cruikshanks Cruikshanks added the enhancement New feature or request label Jan 5, 2025
@Cruikshanks Cruikshanks self-assigned this Jan 5, 2025
@Cruikshanks
Add migration to create new table
6e982dc
@Cruikshanks
Allow date fields to be nullable
d402cc1 
Doh! The different might be that a date has now been set in NALD, but WRLS is null, or vice versa. Therefore we need to be able to store null for the dates in the table.
Cruikshanks added a commit to DEFRA/water-abstraction-system that referenced this pull request Jan 6, 2025
@Cruikshanks
Move licence-changes to endpoint and split work
8f64ca3 
https://eaflood.atlassian.net/browse/WATER-4546

> Part of the work to migrate management of return versions to WRLS from NALD

During the import from NALD, if a licence end date is changed, for example, if it has been revoked in NALD, our return version functionality needs to know about it. This is so we can reissue the return logs for the licence to match the changed end date.

We are actively trying to move away from the legacy code base, so this work was always going to be done in here. Initially, it would be triggered by our [own import process](https://eaflood.atlassian.net/browse/WATER-4535), which was due to replace the legacy one. But that was when we thought ReSP would be taking over from NALD.

Now, the plan is for WRLS to encompass the final abstraction leg and take over from NALD. There is little point in replacing a complex import we intend to switch off in the next year.

But, knowing we'd need something in the interim, we created the [/jobs/licence-changes job](#1593). The intent was to schedule this after the first [NALD import job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#nald-import) had completed (the one that downloads and extracts the NALD data) but before the main [licence import
  job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#licence-import). That way, our job could compare the NALD and WRLS licence records to determine if an end date has changed.

But then we had our 'doh!' moment.

Our reissue return logs engine expects to be given a licence ID and the date the change applies. However, the WRLS licence record needs to have been updated for it to determine the start and end dates of the new return logs.

If we schedule `/jobs/licence-changes` before the licence import job, it will see the change but won't reissue anything because the WRLS record won't have been updated. If we schedule it after, it won't see any difference and won't trigger the reissue.

Doh!

So, we've needed to change tact on this completely. Now, we intend to

- _trigger_ `/licences/end-dates/check` from the [NALD import job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#nald-import) rather than schedule it
- refactor `/jobs/licence-changes` as `/licences/end-dates`, but instead of processing changed licences it [stores the details of the change in a table](DEFRA/water-abstraction-service#2666)
- add a new `/licences/end-dates/process` endpoint that processes these 'licence change' records, and which will be triggered from the [licence import job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#licence-import)

This change handles the refactoring and creating of the endpoints, plus addition of a new model for for the new table.
@Cruikshanks Cruikshanks mentioned this pull request Jan 6, 2025
Merged
@Cruikshanks
Add a constraint to the table
730f0fe 
This will support the event that the check is performed a second time before the WRLS record has been updated, or the end date change process has had a chance to run.

If the `/licences/end-date/check` process flags the same licence again, for the same date type, we can use the constraint to apply an Upsert query and update the dates to the latest information.

This avoids us erroring if we attempt to insert the record for a second time.
@Cruikshanks
Doh! Add change_date field
d3ff4c7 
Forgot to add this when creating the table. Our code in water-abstraction-system, having identified what date change needs to be captured, also determines what change date to use, for example, if 2 dates have been updated at the same time.

We need this to be able to process the change in our later process.
Cruikshanks added a commit to DEFRA/water-abstraction-import that referenced this pull request Jan 7, 2025
@Cruikshanks
Trigger new licence end date change endpoints
d506e88 
https://eaflood.atlassian.net/browse/WATER-4546

> Part of the work to migrate management of return versions to WRLS from NALD

During the import from NALD, if a licence end date is changed, for example, if it has been revoked in NALD, our return version functionality needs to know about it. This is so we can reissue the return logs for the licence to match the changed end date.

We are actively trying to move away from the legacy code base, so this work was always going to be done in [water-abstraction-system](https://github.com/DEFRA/water-abstraction-system). Initially, it would be triggered by our [own import process](https://eaflood.atlassian.net/browse/WATER-4535), which was due to replace the legacy one. But that was when we thought ReSP would be taking over from NALD.

Now, the plan is for WRLS to encompass the final abstraction leg and take over from NALD. So, there is little point in replacing a complex import we intend to switch off in the next year.

But, knowing we'd need something in the interim, we created the [/jobs/licence-changes job](DEFRA/water-abstraction-system#1593). The intent was to schedule this after the first [NALD import job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#nald-import) had completed (the one that downloads and extracts the NALD data) but before the main [licence import
  job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#licence-import). That way, our job could compare the NALD and WRLS licence records to determine if an end date has changed.

But then we had our 'doh!' moment.

Our reissue return logs engine expects to be given a licence ID and the date the change applies. However, the WRLS licence record needs to have been updated for it to determine the start and end dates of the new return logs.

If we schedule `/jobs/licence-changes` before the licence import job, it will see the change but won't reissue anything because the WRLS record won't have been updated. If we schedule it after, it won't see any difference and won't trigger the reissue.

Doh!

So, we've needed to change tact on this completely. Now, we intend to

- _trigger_ `/licences/end-dates/check` from the [NALD import job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#nald-import) rather than schedule it
- refactor `/jobs/licence-changes` as `/licences/end-dates`, but instead of processing changed licences, it [stores the details of the change in a table](DEFRA/water-abstraction-service#2666)
- add a new `/licences/end-dates/process` endpoint that processes these 'licence change' records, which will be triggered from the [licence import job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#licence-import)

This change adds new steps to the NALD and Licence import jobs to trigger these new endpoints.
@Cruikshanks Cruikshanks mentioned this pull request Jan 7, 2025
Merged
@Cruikshanks Cruikshanks marked this pull request as ready for review January 7, 2025 13:30
@Cruikshanks Cruikshanks merged commit 4eb7674 into main Jan 7, 2025
5 checks passed
@Cruikshanks Cruikshanks deleted the add-new-licence-changes-table branch January 7, 2025 13:30
Cruikshanks added a commit to DEFRA/water-abstraction-import that referenced this pull request Jan 7, 2025
@Cruikshanks
Trigger new licence end date change endpoints (#1049)
626d6c8 
https://eaflood.atlassian.net/browse/WATER-4546

> Part of the work to migrate management of return versions to WRLS from NALD

During the import from NALD, if a licence end date is changed, for example, if it has been revoked in NALD, our return version functionality needs to know about it. This is so we can reissue the return logs for the licence to match the changed end date.

We are actively trying to move away from the legacy code base, so this work was always going to be done in [water-abstraction-system](https://github.com/DEFRA/water-abstraction-system). Initially, it would be triggered by our [own import process](https://eaflood.atlassian.net/browse/WATER-4535), which was due to replace the legacy one. But that was when we thought ReSP would be taking over from NALD.

Now, the plan is for WRLS to encompass the final abstraction leg and take over from NALD. So, there is little point in replacing a complex import we intend to switch off in the next year.

But, knowing we'd need something in the interim, we created the [/jobs/licence-changes job](DEFRA/water-abstraction-system#1593). The intent was to schedule this after the first [NALD import job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#nald-import) had completed (the one that downloads and extracts the NALD data) but before the main [licence import
  job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#licence-import). That way, our job could compare the NALD and WRLS licence records to determine if an end date has changed.

But then we had our 'doh!' moment.

Our reissue return logs engine expects to be given a licence ID and the date the change applies. However, the WRLS licence record needs to have been updated for it to determine the start and end dates of the new return logs.

If we schedule `/jobs/licence-changes` before the licence import job, it will see the change but won't reissue anything because the WRLS record won't have been updated. If we schedule it after, it won't see any difference and won't trigger the reissue.

Doh!

So, we've needed to change tact on this completely. Now, we intend to

- _trigger_ `/licences/end-dates/check` from the [NALD import job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#nald-import) rather than schedule it
- refactor `/jobs/licence-changes` as `/licences/end-dates`, but instead of processing changed licences, it [stores the details of the change in a table](DEFRA/water-abstraction-service#2666)
- add a new `/licences/end-dates/process` endpoint that processes these 'licence change' records, which will be triggered from the [licence import job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#licence-import)

This change adds new steps to the NALD and Licence import jobs to trigger these new endpoints.
Cruikshanks added a commit to DEFRA/water-abstraction-system that referenced this pull request Jan 7, 2025
@Cruikshanks
Move licence-changes to endpoint and split work (#1600)
9018907 
https://eaflood.atlassian.net/browse/WATER-4546

> Part of the work to migrate management of return versions to WRLS from NALD

During the import from NALD, if a licence end date is changed, for example, if it has been revoked in NALD, our return version functionality needs to know about it. This is so we can reissue the return logs for the licence to match the changed end date.

We are actively trying to move away from the legacy code base, so this work was always going to be done in here. Initially, it would be triggered by our [own import process](https://eaflood.atlassian.net/browse/WATER-4535), which was due to replace the legacy one. But that was when we thought ReSP would be taking over from NALD.

Now, the plan is for WRLS to encompass the final abstraction leg and take over from NALD. There is little point in replacing a complex import we intend to switch off in the next year.

But, knowing we'd need something in the interim, we created the [/jobs/licence-changes job](#1593). The intent was to schedule this after the first [NALD import job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#nald-import) had completed (the one that downloads and extracts the NALD data) but before the main [licence import
  job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#licence-import). That way, our job could compare the NALD and WRLS licence records to determine if an end date has changed.

But then we had our 'doh!' moment.

Our reissue return logs engine expects to be given a licence ID and the date the change applies. However, the WRLS licence record needs to have been updated for it to determine the start and end dates of the new return logs.

If we schedule `/jobs/licence-changes` before the licence import job, it will see the change but won't reissue anything because the WRLS record won't have been updated. If we schedule it after, it won't see any difference and won't trigger the reissue.

Doh!

So, we've needed to change tact on this completely. Now, we intend to

- _trigger_ `/licences/end-dates/check` from the [NALD import job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#nald-import) rather than schedule it
- refactor `/jobs/licence-changes` as `/licences/end-dates`, but instead of processing changed licences, it [stores the details of the change in a table](DEFRA/water-abstraction-service#2666)
- add a new `/licences/end-dates/process` endpoint that processes these 'licence change' records, which will be triggered from the [licence import job](https://github.com/DEFRA/water-abstraction-team/blob/main/jobs/import.md#licence-import)

This change handles the refactoring and creation of the endpoints, plus the addition of a new model for the new table.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@Cruikshanks Cruikshanks

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@Cruikshanks