Storage Service doesn't handle deleted containers

[cristiancalugaru]

Affected module
Backend

Describe the bug
When deleting an S3 bucket (container) and running S3 Storage Service ingestion, the deleted container is not removed from OM.

To Reproduce
Delete an S3 bucket which was already ingested in OM, then re-ingest S3 containers in OM. The deleted S3 bucket is still present as a container in OM.

Expected behavior
OM should reflect the fact that the S3 bucket was deleted and also delete it.

Additional context
The fix most likely involves something similar with how tables are handled, in the database_service, as a post_process step:
https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/database_service.py#L139

[harshach]

@cristiancalugaru will you be able to contribute this?

[cristiancalugaru]

@harshach yes, please assign it to me.