First version implementation of preload_data and the rest of CLMS data store

[b-yogesh]

This PR introduces the initial mechanism of preloading data, including cache management, downloading, and file processing.

The following classes (components) are responsible for the mechanism:

CLMS

• Serves as the main interface to interact with the CLMS API. This class coordinates with the PreloadData class to preload the data into a local filestore.

CacheManager

• Manages the local cache of preloaded data.
• Maintains a dictionary (cache) that maps data_ids to their respective file paths.
• Handles file store from the xcube data store in a local directory and refreshes the cache when necessary.

DownloadTaskManager

• Handles the download process, including managing download requests and checking their statuses.
• Retrieves task statuses based on dataset and file IDs or task IDs, determining whether the download is pending, completed, or cancelled.
• Initiates data downloads in chunks and manages zip file extraction, looking specifically for geo data. Definition of geo data is defined in the function docstring in the notes.

CLMSAPITokenHandler

• Handles the creation and refreshing of the CLMS API token given the credentials which can be obtained following the steps

FileProcessor

• Handles the postprocessing of downloaded data, extracting, stacking and storing geo files from downloaded zip files.

PreloadData

• The main class responsible for orchestrating the preloading of datasets.
• It coordinates with CacheManager, DownloadTaskManager, CLMSAPITokenHandler and FileProcessor classes to handle the complete process of caching, data downloading, making sure token is valid and post-processing of downloaded data.
• Utilizes threading for handling multiple data preloading tasks concurrently.
• Uses notebook.tqdm for displaying progress bars

Currently, only logs are shown in the jupyter cells that let the user know of the request status which is updated every 60 seconds to avoid sending several requests to the CLMS API. This is configurable, so it can be changed if we need a lower or higher waiting time between request status. tqdm is used here that shows the progress bar for the actual download, unzipping, and postprocessing.

A tutorial notebook has also been added to the examples folder to show how to use this data store.

There are some missing tests for a few classes, but they will be added in the next PRs

Closes #3, #4, #5

[forman]

Store Implementation

• The preload mechanism lacks the preload handle concept (but that's ok for time being)
• Do not limit the caching data store to file. Allow for any MutableDataStore instead.

Coding

• Using camel-case names, do not concatenate upper-case abbreviations, use camel-case instead: CLMSAPITokenHandler is not user friendly, use ClmsApiTokenHandler instead
• Don't use os, use fsspec instead. Actually, don't use specific local filesystem calls at all, if not urgently required.

Testing

• Test code is often hard to read because of the bloating mock setup
• Use mocking more carefully! It can cause architectural code changes to be very expensive in the end!
• Test with respect to correct results or behavious, not implementation.
• Build test class names like so: ${class_or_func}Test, not Test${class_or_func}
• Using plain assert doesn't distinguish between expected and actual values - which makes it harder to understand what went wrong if tests fail. Prefer test classes that use assert methods from unitest.TestCase.

Docstrings

• Should start with a short, one-line sentence that follows immediately the """ and ends with a dot .. Use active speech in this sentence.
• Void methods are supposed to return None. Don't document this.

FYI @konstntokas

[forman]

If you import a package in non-test code, add it as a true project dependency.
Do not rely on transitive dependencies.

[b-yogesh]

Currently, I only use numpy in the tests, so I have not added it in the project dependency.

[forman]

Move into dependencies list and potentially also others.

[b-yogesh]

I only use numpy in the tests, so I have not added it in the dependencies list. By others, do you mean the pytest, black, flake8 ... etc.?

[konstntokas]

Approved (since the preload api will change) with some comments which you may address or note down for a later PR.

[konstntokas]

Why do you want to keep them? There was a comment in Normans review last time that this is redundant.

[b-yogesh]

Based on what I understand on Norman's comment was to let the constants be here if they are used in multiple files, which is the case for these.

[konstntokas]

in the credentials json which I got from the CLMS store, there are more fields. Are the others not needed for the store? Please double check if required is correct here.

[konstntokas]

Are the needed?

[b-yogesh]

Yes. I would like all the hardcoded strings to be defined as constants for easier manipulation later (if required).

[konstntokas]

cannot you do something like this?

        response = requests.request()
        if response.ok:
            return response # or do something else
        else:
            raise DataStoreError(response.raise_for_status())

[b-yogesh]

That can be done but if I dont do this, the actual error message is lost leaving the user confused about the reason for the exception.
I have added a comment as well explaining the reason I do this.

[b-yogesh]

Closing this PR as #12 refactors this PR.