feat: adding the operation delete_all_documents to the OpenSearchDocumentStore

[davidsbatista]

Related Issues

• fixes #2315

Proposed Changes:

• adds a delete_all_documents() method to the OpenSearchDocumentStore class to both Synchronous and Asynchronous versions
• recreate_index=False (default): Uses delete_by_query API for faster deletion of large datasets
• recreate_index=True: Recreates the entire index, preserving original mappings and settings
• other small improvments:
  • _deserialize_search_hits() → @staticmethod
  • _process_bulk_write_errors() → @staticmethod
  • _deserialize_document() → @staticmethod
  • _postprocess_bm25_search_results() -> @staticmethod
• minor typo corrections in docstrings
• added proper SPDX license headers to test files

📝 Usage Example

# Delete all documents (fast, keeps index structure)
document_store.delete_all_documents()

# Delete all documents and recreate index (slower, resets index state)
document_store.delete_all_documents(recreate_index=True)

---

How did you test it?

• unit tests, integration tests + CI tests
• test_document_store.py - Added test_delete_all_documents() with and without index recreation
• test_document_store_async.py - Added test_delete_all_documents_async() with and without index recreation

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

[anakin87]

Just a quick thought... I am not sure deleting and recreating the index is a good idea.

As a user, I'd expect delete_all_documents to only remove the data, not touch the index itself. And even though we recommend managing the DB via Haystack, some users might still have changed index attributes manually. In that case, recreating the index could cause surprises.

A bulk delete might be a safer option. Maybe it's also worth checking how this was done in Haystack 1.x.

[davidsbatista]

I was investigating into it and for both Elastic and Open, specially for large volumes of data, this is the most efficient way to do it. In both cases the indexes are recreated with the same mappings/settings.

The only issue is if a user changes the index after creation/population.

[davidsbatista]

What about something like:

def delete_all_documents(self, recreate_index=False)

By default we apply a usual delete operation, and if recreate_index=True, we just recreate the index using the mappings/settings defined by the user or the default ones.

[anakin87]

What about something like:

def delete_all_documents(self, recreate_index=False)

I would prefer this idea!

[julian-risch]

Looks quite good to me already. My main suggestion is that if we have a try except in delete_all_documents_async, it makes sense to add the same to the sync implementation of delete_all_documents to ensures consistent behavior.
Other than that, I have a suggestion for the tests to make sure they only test one thing. You could create a separate test, for example test_index_functionality_after_delete_all_documents for testing writing and retrieving of documents after deleting all documents. Something like:

def test_index_functionality_after_delete_all_documents(self, document_store):
    """Test that documents can be written and retrieved after delete_all_documents"""
    document_store.write_documents([Document(id="1", content="Test")])  # or add more documents
    document_store.delete_all_documents()
    
    new_doc = Document(id="2", content="New test")
    document_store.write_documents([new_doc])  # or add more documents
    assert document_store.count_documents() == 1
    
    results = document_store.filter_documents()
    assert len(results) == 1
    assert results[0].content == "New test"

[julian-risch]

It's consistent with the other methods to have this assert here. Therefore, I agree it's good to have it here.

However, I believe the only case it covers is if self._client was set at some point and then for some reason is set to None again. We have the same assert in _ensure_index_exists, which get's called as part of _ensure_initialized when it is run for the first time.