Improve support for subcollections by propagating dynamic vars and backfilling collectionGroup

[jsasitorn]

Change Summary

Currently users can provide dynamic collection paths in typesense config, "users/{userId}/books" even though this isn't properly supported. This improves Firebase plugin support for subcollections in indexOnWrite and backfill by the following:

For onWrite, pass context.params so that placeholders in a dynamic collection path, e.g. users/{userId}/books/{bookId}/chapters, are also indexed in typesense, thus allowing querying not only for a chapter with title="Intro" or id=789, but also userId=123 and bookId=456.

For backfill, query using .collectionGroup(collectionGroup), to backfill following a dynamic path: "users/{userId}/books/{bookId}/title".

This draws inspiration from #93 to improve backfills. However, this version adds guards to prevent group backfills for static paths: e.g.,"users/123/books/456/chapters" and unrelated paths, e.g. using .collectionGroup("books") could end up backfilling both "books" and "users/{userId}/books".

Only change to the typesense collection is the inclusion of ids in the dynamic path.

PR Checklist

• I have read and signed the Contributor License Agreement.

[tharropoulos]

Just some small nitpicks I noticed

[jsasitorn]

updated. let me know any other comments or feedback

[tharropoulos]

lgtm!

[jasonbosco]

Thank you for the PR @jsasitorn.

This is now available in this RC version: https://github.com/typesense/firestore-typesense-search/releases/tag/v2.0.0-rc.0

Could you install the RC version of the extension using the installation link above and let me know how it goes?

[jsasitorn]

@jasonbosco tested both indexOnWrite and backfill on my data set (~600 entries), data appears to be all there. Noticed some innocuous errors addressed in #100.

Also, couple of things I noticed testing:

1. Might be useful to note which permissions are required: indexOnWrite uses "documents:create","documents:update"," while backfill will also need "documents:import". Error from typesense is the generic invalid key, so wasn't clear.

2. This debug is pretty useful for debugging, but think its dumping more than it should in a steady production setup:

   firestore-typesense-search/functions/src/utils.js

   Line 148 in eb4c7a4

   console.log("typesenseDocument", typesenseDocument);

[jasonbosco]

Thanks @jsasitorn. Published PR #100 in v2.0.0-rc.1

Might be useful to note which permissions are required: indexOnWrite uses "documents:create","documents:update"," while backfill will also need "documents:import". Error from typesense is the generic invalid key, so wasn't clear.

@tharropoulos could you add this to the README?

This debug is pretty useful for debugging, but think its dumping more than it should in a steady production setup:

Yeah, may be we should add a debug: true key to the typesense_sync collection and only log that particular line when it's set to true. Do you want to do a PR for this?

[jsasitorn]

@jasonbosco adding a firestore call to typesense_sync is a little heavy since typesenseDocumentFromSnapshot is also called in .onWrite for every object update. What do you think of adding debug:true/false to extension.yaml and piping it in through config.js? Takes a couple min to update after toggling, but would this action is infrequent.

[vazome]

Thank you all for this!

[jasonbosco]

@jsasitorn Good point. Adding a debug mode to extensions.yml sounds good to me.