Make metadata two-way synced

[thclark]

Feature request

Use Case

Database migrations are a powerful way of adding new metadata to objects already in a store (or of updating that metadata).

Additionally, it should be possible to change metadata on an existing file via django admin and have that propagate to the store correctly.

Current state

Some objects have been found in staging store whose metadata doesn't contain values which have been added via a migration.

We need to check that bulk_updates and bulk_creates of database entries, as well as savin existing objects, does update metadata in the store. This should be part of reinstating the store test suites.

[thclark]

@nvn_nil raised this as an issue and worked around it with the following signal code; we should do it elegantly so he doesn't have to:

import logging
from django.db.models.signals import post_save
from django.dispatch import receiver
from octue.resources import Datafile

from datalake.models import ElevationMap, MastTimeseries, WindMap


logger = logging.getLogger(__name__)


@receiver(post_save, sender=MyDataLakeModel)
def update_cloud_metadata(sender, instance, **kwargs):
    """Update the cloud metadata from the instance values on save
    Should be handled in django_twined but it's not (or it's not working) at the moment.
    """

    # We should get the datafile using the 'to_datafile' method and with 'update_from_store=False' it is expected to use the instance metadata.
    # However, all the datafile returning methods fetch the metadata from the store so we have to create the datafile ourselves with 'ignore_stored_metadata=True'
    df = Datafile(
        instance.gs_path,
        id=str(instance.id),
        ignore_stored_metadata=True,
        tags=instance.get_tags_from_instance(),
        labels=instance.get_labels_from_instance(),
    )
    df.update_metadata()