Add integration tests for bit shift functionality

[patrick-austin]

This adds "integration" tests for the bit depth changes in #151 . It will require a single hdf5 file, generated with v1.1 of the simulated data tool for the timestamp 20230606120000 to be ingested into the bucket the tests run against. This has not yet been added to the CI to avoid failing other branches, so as a result the tests on this branch will fail. I haven't attached this file as it's fairly large, but can circulate if others want to test locally or put it into the CI bucket when we're ready.

New tests:

• Two new test cases for test_valid_get_records:

  • That bit_depth does not appear in the metadata for old data (v1.0)
  • That bit_depth does appear in the metadata for new data (v1.1)

• Three new test cases for test_valid_get_image:

  • 12 bit original image:
    

  • 12 bit image with 8 bit limits applied:
    

  • 12 bit image with 12 bit limits applied (this should result in the same image as in both cases these values are scaled up to the "storage" bit depth, which is 16 bits, before being applied):
    

• Unit test of _bit_shift_to_storage as not all lines were covered by the above test

Existing tests updated:

• most_recent_date and recent_sample include values for the new timestamp
• Count records incremented by 1 for total, and greater than range
• Range converter includes the new most recent timestamp/shotnumber conversion

[moonraker595]

Hi, Just picking this up as well as I thought it'd be a good idea to pull down the bit shift code and try and run some tests against it.
Just some general questions:

• PR 151 & 153: Just wondering if these can be merged into one branch so I can get both code and tests. I can merge them myself locally if not.

• "This has not yet been added to the CI to avoid failing other branches"
  Was I the hold up here? If so, could this be added to the CI bucket now?

• Side note: do you want to merge the changes to main into 151 or worry about that later? Also, I'll try and grab @kevinphippsstfc and yourself after stand up tomorrow to see if we can clean out any of the older/dangling branches.

[patrick-austin]

Hi, Just picking this up as well as I thought it'd be a good idea to pull down the bit shift code and try and run some tests against it. Just some general questions:

* PR 151 & 153: Just wondering if these can be merged into one branch so I can get both code and tests. I can merge them myself locally if not.

* "This has not yet been added to the CI to avoid failing other branches"
  Was I the hold up here? If so, could this be added to the CI bucket now?

* Side note: do you want to merge the changes to main into 151 or worry about that later? Also, I'll try and grab @kevinphippsstfc and yourself after stand up tomorrow to see if we can clean out any of the older/dangling branches.

Yeah sorry I'm only looking at this comment now. I know we spoke about this earlier but to reiterate:

• https://github.com/ral-facilities/operationsgateway-api/tree/DSEGOG-325-Integration-tests contains all the commits from https://github.com/ral-facilities/operationsgateway-api/tree/DSEGOG-325-Update-ingestion-to-support-epac_ops_data_version-1.1 in addition to 1 more commit which adds "integration" tests. If you prefer only having one PR, then we can change this PR to target main instead and the diff will show all commits. The only reason to open this separately is to (ideally) make review easier by splitting into smaller parts and keeping the "integration" tests separate. If you want to run everything locally, just use this branch.
• I can push the h5 file to the CI bucket, as long as Louise and Josh are happy to (as I understand it, it will impact both front and back end repos) - I didn't quite catch if they were at the sprint meeting.
• I can merge main in if needed, but it isn't showing me any conflicts so it should be fine? Other than the safety job failing, which is a known entity.

Happy to discuss and clarify more if needed.

[moonraker595]

It's all good. I should've realised that it was taken from the other branch. I'm happy to wait until everything is approved for merging/adding the hdf file to the bucket. If this happens again, I think we should check the tests and maybe refactor anything that checks for an exact number of test files.

I've spun up the API, ingested the hdf file and am just running the nox stages now 👍

[moonraker595]

Tests look good. The only thing i can't see is if the code coverage has changed. I'm happy to approve so we can get the CI working with the new test data, and then create a separate PR for any code coverage concerns if needed.