Speed up first reservable time calculation

[matti-lamppu]

🛠️ Changelog

Speeds up first reservable time calculation in a two different ways:

1. Pre-calculates affecting reservations with a materialized view. The view should be updated with a scheduled task.

2. Run calculation for fewer units based on pagination arguments. Use a custom QuerySet.iterator() to fetch chunks of reservation units and their affecting reservations

The above optimizations achieve <1s fetching with our test data when calculating FRT, regardless of showOnlyReservable or any pagination args.

🧪 Test plan

• Automated tests

🚧 Dependencies

• Setup a periodic job in Django to refresh AffectingTimeSpans in platta envs.

🎫 Tickets

• TILA-3436

[github-actions]

Triggered from #1252 by @​matti-lamppu.

Checking if we can fast forward main (1ee0c87) to speed-up-first-reservable-time-calculation (8884c0a).

Target branch (main):

commit 1ee0c8758afa804960d9424e0b7fd56124405d77 (HEAD -> main, origin/main, origin/HEAD)
Author: Matti Lamppu <matti.lamppu@vincit.fi>
Date:   Fri Aug 2 11:21:06 2024 +0300

    Switch to ECR for pulling docker images
    
    This is done due to hitting pull rate limits for Docker Hub
    (for unauthenticated users) in Azure Pipelines. At the time,
    this is 100 pulls per 6 hours, which is shared with all platta
    developers. ECR offers 1 pull per second (for unauthenticated users),
    which we shouldn't hit too often, since CI also has retries for
    pulling images.

Pull request (speed-up-first-reservable-time-calculation):

commit 8884c0a3bbf0c962c4de276b1885ca78ca22430b (pull_request/speed-up-first-reservable-time-calculation)
Author: Matti Lamppu <matti.lamppu@vincit.fi>
Date:   Fri Aug 2 16:14:47 2024 +0300

    Optimize affecting time spans fetching for FRT calc
    
    Instead of fetching affecting time span elements with a prefetch,
    create a custom QuerySet iterator method that can fetch the
    time spans in chunks when a new iterator chunk needs to be fetched.
    This way we also don't create so many duplicates for in memory.
    The method seems much faster (up to 4x), executing in less than a
    second with our current test dataset.

Can't fast forward main (1ee0c87) to speed-up-first-reservable-time-calculation (8884c0a). main (1ee0c87) is not a direct ancestor of speed-up-first-reservable-time-calculation (8884c0a). Branches appear to have diverged at 09c30da:

* 8884c0a3bbf0c962c4de276b1885ca78ca22430b Optimize affecting time spans fetching for FRT calc
* d51377d640b907248fbbadcb0b51b31de40199bb Use affecting tims spans for first reservable time calculation
* be9ba0a66b5b394b0c6d82f79e2cbf495cabf491 Use affecting time spans in recurring reservation generation
* 80b6a0bb9df63f67e6bacf92a37e63ca51f96b63 Add affecting time spans materialized view
| * 1ee0c8758afa804960d9424e0b7fd56124405d77 Switch to ECR for pulling docker images
| * ed7b8cfda646e6cf20de4d0016668ed4310f46a6 Add a warning for using test data creation in production environment
| * c26a4c0a2937ec6e5198672dd9d1ff6bfa2e48b5 Add `allow_cascade=True` to test data creation flush
|/  
* 09c30daeba5cdc89c9a8f97e5f0f9c0bc6ab7b15 Add another option for fast-forward merge comment

commit 09c30daeba5cdc89c9a8f97e5f0f9c0bc6ab7b15 (tag: v0.53.1)
Author: Matti Lamppu <matti.lamppu@vincit.fi>
Date:   Wed Jul 31 12:31:29 2024 +0300

    Add another option for fast-forward merge comment

Rebase locally, and then force push to speed-up-first-reservable-time-calculation.

[matti-lamppu]

Idea: Maybe there should be a flag set in cache which indicates that the materialized view refresh is valid, in case the job fails for some reason. This would then force a refresh in the endpoint if the value is missing.

[vergama]

Idea: Maybe there should be a flag set in cache which indicates that the materialized view refresh is valid, in case the job fails for some reason. This would then force a refresh in the endpoint if the value is missing.

Or value of when the view was last updated; force refresh if too old?
Some complications may be caused by parallel endpoint hits encountering invalid mat.view, in which case the refresh should be synchronized...

[vergama]

"The dude abides."

[github-actions]

Triggered from #1252 by @​matti-lamppu.

Checking if we can fast forward main (30dea12) to speed-up-first-reservable-time-calculation (35f3900).

Target branch (main):

commit 30dea125e7440ea3a5fa9c4d14fcb2e84a5df9f9 (HEAD -> main, origin/main, origin/HEAD)
Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date:   Mon Aug 5 21:16:47 2024 +0000

    [pre-commit.ci] pre-commit autoupdate
    
    updates:
    - [github.com/astral-sh/ruff-pre-commit: v0.5.5 → v0.5.6](https://github.com/astral-sh/ruff-pre-commit/compare/v0.5.5...v0.5.6)

Pull request (speed-up-first-reservable-time-calculation):

commit 35f3900f1f0294548376a51975e0fa760ac222f1 (pull_request/speed-up-first-reservable-time-calculation)
Author: Matti Lamppu <matti.lamppu@vincit.fi>
Date:   Tue Aug 6 07:44:57 2024 +0300

    Use a value in cache to check if AffectingTimeSpans are valid
    
    Set the value whenever the view is refreshed. Force a refresh
    in the FRT calculation if the view is invalid.

Can't fast forward main (30dea12) to speed-up-first-reservable-time-calculation (35f3900). main (30dea12) is not a direct ancestor of speed-up-first-reservable-time-calculation (35f3900). Branches appear to have diverged at 1ee0c87:

* 35f3900f1f0294548376a51975e0fa760ac222f1 Use a value in cache to check if AffectingTimeSpans are valid
* 941d31112ccba5b8f17f24814cc866182cb6cbae Optimize affecting time spans fetching for FRT calc
* 52bfdb202bade8ef33d00d2ef8da3e3d86579d98 Use affecting tims spans for first reservable time calculation
* 188152ebac64569f0560a6f47c8a573458908478 Use affecting time spans in recurring reservation generation
* 86dfb00172a8f7f51427cffcd799c5cd347740ea Add affecting time spans materialized view
| * 30dea125e7440ea3a5fa9c4d14fcb2e84a5df9f9 [pre-commit.ci] pre-commit autoupdate
| * 92cf2304d603ed0e93876db4030f836193811b79 build(deps): bump the python-dependencies group with 7 updates
|/  
* 1ee0c8758afa804960d9424e0b7fd56124405d77 Switch to ECR for pulling docker images

commit 1ee0c8758afa804960d9424e0b7fd56124405d77
Author: Matti Lamppu <matti.lamppu@vincit.fi>
Date:   Fri Aug 2 11:21:06 2024 +0300

    Switch to ECR for pulling docker images
    
    This is done due to hitting pull rate limits for Docker Hub
    (for unauthenticated users) in Azure Pipelines. At the time,
    this is 100 pulls per 6 hours, which is shared with all platta
    developers. ECR offers 1 pull per second (for unauthenticated users),
    which we shouldn't hit too often, since CI also has retries for
    pulling images.

Rebase locally, and then force push to speed-up-first-reservable-time-calculation.

[matti-lamppu]

Added a value to cache for when the view was last updated. The FRT calculation will check this and refresh the view if it is too old. Refreshing the view with AffectingTimeSpan.refresh() updates the cached value.

Also, added error handling logic to the .refresh() methods on both materialized views to raise an error if the refresh fails during local development, but log to sentry in other environments.

[matti-lamppu]

Merged #1234, #1274, #1261 to this PR so that they can be merged to main together.

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
84.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[matti-lamppu]

/ff

[github-actions]

Triggered from #1252 (comment) by @​matti-lamppu.

Trying to fast forward main (30dea12) to speed-up-first-reservable-time-calculation (60e7771).

Target branch (main):

commit 30dea125e7440ea3a5fa9c4d14fcb2e84a5df9f9 (HEAD -> main, origin/main, origin/HEAD)
Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date:   Mon Aug 5 21:16:47 2024 +0000

    [pre-commit.ci] pre-commit autoupdate
    
    updates:
    - [github.com/astral-sh/ruff-pre-commit: v0.5.5 → v0.5.6](https://github.com/astral-sh/ruff-pre-commit/compare/v0.5.5...v0.5.6)

Pull request (speed-up-first-reservable-time-calculation):

commit 60e7771f302587389f8774ecfb7988478f2189cf (pull_request/speed-up-first-reservable-time-calculation)
Author: Eemeli Ranta <eemeli.ranta@vincit.fi>
Date:   Wed Jul 3 20:58:21 2024 +0300

    Update email feedback URL

Fast forwarding main (30dea12) to speed-up-first-reservable-time-calculation (60e7771).

$ git push origin 60e7771f302587389f8774ecfb7988478f2189cf:main
To https://github.com/City-of-Helsinki/tilavarauspalvelu-core.git
   30dea125..60e7771f  60e7771f302587389f8774ecfb7988478f2189cf -> main