Semantic Search for Events #8998

hunterjm · 2023-12-18T07:16:22Z

Implementation for #8980

Still a work in progress, but far enough along to play around with if interested. I still need to implement a way to keep the Events database and the vector database in sync and add missing historical events to the vector database on startup.

…shear#8673) * Generate low res low fps previews for recordings viewer * Make sure previews end on the hour * Fix durations and decrase keyframe interval to ensure smooth scrubbing * Ensure minimized resolution is compatible with yuv * Add ability to configure preview quality * Fix * Clean up previews more efficiently * Use iterator

* Add external sub label to timeline * Include icon * Update timeline.py Co-authored-by: Sergey Krashevich <svk@svk.su> * Formatting --------- Co-authored-by: Sergey Krashevich <svk@svk.su>

) * Write a low resolution low fps stream from decoded frames (blakeblackshear#8673) * Generate low res low fps previews for recordings viewer * Make sure previews end on the hour * Fix durations and decrase keyframe interval to ensure smooth scrubbing * Ensure minimized resolution is compatible with yuv * Add ability to configure preview quality * Fix * Clean up previews more efficiently * Use iterator * Ensure final frame in preview is not duplicated * initial react/ts framework * fix gitignore glob excluding ts files * ignore folders in web-new * SWRConfig changes for swr 2.x * use frigateConfig type in websocket handlers --------- Co-authored-by: Nicolas Mowen <nickmowen213@gmail.com>

* remove rtmp from python * remove rtmp from nginx * remove rtmp from docs * fix test for missing role

* Add config editor to new webUI and fix double scroll * Address review comments and enable light theme for vscode

…akeblackshear#8895) * Add support for review grid * Cleanup reloading on focus * Adjust timeline api to include metadata and before * Be more efficient about getting info * Adjust to new data format * Cleanup types * Cleanup text * Transition to history * Cleanup * remove old web implementations * Cleanup

* Add table and oveview components * Add toooltips for cards * Add storage * Undo

* Use constant aspect ratio for review grid * Add infinite scrolling * Don't have horizontal scrolling * Handle autoplay for mobile * Load more efficiently

* Add support for live views * Lint fixes * Fix width * Fix lint * Cleanup hooks

* Playback recording when clicking on review item * Add timeline overlay selector * Lint fixes * Lint fixes * Set video plays inline * Stop autoplay on open * Reverse order * Improve autoplay performance * Remove chromecast icon by default * Improve margin and remove lazy image loading to reduce jumping * Fix root margin for mobile * Fix scrolling behavior * Fix width

* Show camera dashboard * Cleanup * Cleanup * Cleanup * Cleanup * fix * Fix input color * Show recent events on dashboard

* fixup build * swap frontends

hunterjm · 2023-12-18T07:18:50Z

And yes, I know that I added the frontend to the old web - but the new frontend only uses timeline for now, and it does not have enough detail to perform this type of search, or filters implemented at this point in time.

…arch

hunterjm · 2023-12-19T06:33:54Z

Implementation is mostly done at this point. I implemented the frontend in web-old because it was further along (and less work) than trying something in the new UI.

For the new UI, what do you think about a search bar at the top that is available on every page? When searched, it shows a styled dropdown of matching events allowing for additional filtering, downloading, or opening in the recordings/timeline viewer.

Examples

Search for "man with brown hair and beard"

Most results from description

Search for "golfers"

Most results from description

Search for "black and white dog"

Clicked "Find Similar" and returned image search results

…abled

…/usr/bin/env /usr/bin/python3 /home/vscode/.vscode-server/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 40321 -- -m frigate

…g wrong results

docker/main/Dockerfile

hunterjm · 2023-12-19T22:48:57Z

I just installed this on my parent's instance without configuring Gemini. Just using the CLIP model gets some pretty decent results:

Example 1

Search Term: "chevy bolt"

Example 2:

Search Term: "package delivery"
Filtered to the day of 12/15
Requested similar images to the first result shown in the screenshot

hunterjm · 2023-12-20T05:44:17Z

I've updated the docs and have been running this in production for a couple of days. I think it's ready for review now.

docker/main/rootfs/etc/s6-overlay/s6-rc.d/chroma/run

docs/docs/integrations/api.md

NickM-27 · 2023-12-20T14:32:44Z

frigate/app.py

@@ -337,6 +342,13 @@ def cleanup_timeline_db(db: SqliteExtDatabase) -> None:

 migrate_db.close()

+ def init_embeddings(self) -> None:
+ self.embeddings: Optional[Embeddings] = None


For all of these new services I think it would be good to only run them if this has been enabled in the config.

That is the case with the client. The init method will return a ValueError if semantic_search is not enabled and self.embeddings will remain None. See: https://github.com/blakeblackshear/frigate/pull/8998/files#diff-a87fb799a0f5537bd6fb9049628a14131fb21e8bc2f0fc21363d664bb6aff7feR50-R51

NickM-27 · 2023-12-20T14:32:56Z

frigate/app.py

@@ -584,19 +597,26 @@ def start_timeline_processor(self) -> None:
 )
 self.timeline_processor.start()

+ def start_embeddings_processor(self) -> None:
+ self.embeddings_processor = EmbeddingProcessor(


Same as above, only start the service if enabled in the config

This class will return and kill the thread immediately if semantic_search is not enabled. See: https://github.com/blakeblackshear/frigate/pull/8998/files#diff-bb5fcf4020ca2bf73e181a7b467529ec9f574c7292f720b742014f795b82054aR47-R48

Makes sense, I figured that was the case but think it might be better to have it explicit in this file vs the implicit behavior. Not a big deal though

frigate/events/cleanup.py

skrashevich · 2023-12-22T05:17:06Z

you seen this project?

hunterjm · 2023-12-22T05:57:33Z

you seen this project?

I had not until now. GPT-4 Vision can be integrated this way relatively easy as well, or that project can be updated to submit the summary as a description directly to Frigate and have it indexed and searchable.

I like the use of multiple image frames representing a fuller context, but wanted to keep it relatively simple for the first implementation.

NickM-27 · 2024-01-31T13:17:06Z

@blakeblackshear need to reopen this with new base branch just fyi

hunterjm · 2024-06-03T21:32:59Z

Closing this for now. New PR will be forthcoming targeting the new UI for 0.14 after discussing with @NickM-27 and @blakeblackshear.

NickM-27 and others added 15 commits December 13, 2023 06:13

Ensure final frame in preview is not duplicated

8fb8c38

Add external sub label as timeline entry (blakeblackshear#8855)

e666fc5

* Add external sub label to timeline * Include icon * Update timeline.py Co-authored-by: Sergey Krashevich <svk@svk.su> * Formatting --------- Co-authored-by: Sergey Krashevich <svk@svk.su>

Remove rtmp (blakeblackshear#8941)

e41f40d

* remove rtmp from python * remove rtmp from nginx * remove rtmp from docs * fix test for missing role

Add export page in to new web UI (blakeblackshear#8929)

992aca7

Support config editor in new webUI (blakeblackshear#8928)

02a082e

* Add config editor to new webUI and fix double scroll * Address review comments and enable light theme for vscode

Support storage page in new ui (blakeblackshear#8948)

9bc15f7

* Add table and oveview components * Add toooltips for cards * Add storage * Undo

Improve review grid (blakeblackshear#8953)

a4ae15e

* Use constant aspect ratio for review grid * Add infinite scrolling * Don't have horizontal scrolling * Handle autoplay for mobile * Load more efficiently

Add live views to new webui (blakeblackshear#8970)

6a400a5

* Add support for live views * Lint fixes * Fix width * Fix lint * Cleanup hooks

Camera dashboard (blakeblackshear#8981)

8206497

* Show camera dashboard * Cleanup * Cleanup * Cleanup * Cleanup * fix * Fix input color * Show recent events on dashboard

Use new UI (blakeblackshear#8983)

2160963

* fixup build * swap frontends

Initial vector similarity implementation

3b529ed

hunterjm added 3 commits December 18, 2023 02:21

ignore ruff errors with reason

59586e0

fix test run and chroma connection in final docker image

3431363

delete old vectors as events removed from DB, add image similarity se…

dbf9f0c

…arch

hunterjm added 8 commits December 19, 2023 01:48

hide search field and find similar button when semantic search is dis…

45d1c51

…abled

cleanup pysqlite imports and chroma errors - fix tests

3da86df

restore __main__.py

7066688

fix mypy

f3a8eef

fix typing

99cdf10

add ability to update description in frontend

f220e4d

show textarea if no description but selected cd /workspace/frigate ; …

4e6495f

…/usr/bin/env /usr/bin/python3 /home/vscode/.vscode-server/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 40321 -- -m frigate

fix issue where similarity would limit events before sorting returnin…

5ac1baa

…g wrong results

hunterjm commented Dec 19, 2023

View reviewed changes

docker/main/Dockerfile Outdated Show resolved Hide resolved

hunterjm added 4 commits December 19, 2023 23:01

add the ability to choose searching text by thumbnail or description

a029f13

update documentation

1d3dd5b

add tip about .reindex_events to docs

2e3cc12

make Docker build web again

4ff0c3a

hunterjm marked this pull request as ready for review December 20, 2023 05:44

NickM-27 reviewed Dec 20, 2023

View reviewed changes

hunterjm added 2 commits December 21, 2023 21:37

address PR comment on thumb search param name

8de27ab

move delete to own method to reduce duplication

bf8b612

blakeblackshear force-pushed the 0.14 branch from 530b0f9 to 944afd7 Compare December 31, 2023 13:50

blakeblackshear added the pinned label Jan 26, 2024

blakeblackshear force-pushed the 0.14 branch from 5c5a6ba to e8fd4ca Compare January 31, 2024 12:46

blakeblackshear deleted the branch blakeblackshear:dev January 31, 2024 12:56

blakeblackshear closed this Jan 31, 2024

blakeblackshear reopened this Jan 31, 2024

blakeblackshear changed the base branch from 0.14 to dev January 31, 2024 13:25

hunterjm closed this Jun 3, 2024

hunterjm mentioned this pull request Jun 12, 2024

Semantic Search for Detections #11899

Merged

19 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Semantic Search for Events #8998

Semantic Search for Events #8998

hunterjm commented Dec 18, 2023

hunterjm commented Dec 18, 2023

hunterjm commented Dec 19, 2023 •

edited

Loading

hunterjm commented Dec 19, 2023

hunterjm commented Dec 20, 2023

NickM-27 Dec 20, 2023

hunterjm Dec 22, 2023

NickM-27 Dec 20, 2023

hunterjm Dec 22, 2023

NickM-27 Dec 22, 2023

skrashevich commented Dec 22, 2023

hunterjm commented Dec 22, 2023

NickM-27 commented Jan 31, 2024

hunterjm commented Jun 3, 2024

Semantic Search for Events #8998

Semantic Search for Events #8998

Conversation

hunterjm commented Dec 18, 2023

hunterjm commented Dec 18, 2023

hunterjm commented Dec 19, 2023 • edited Loading

Examples

Search for "man with brown hair and beard"

Search for "golfers"

Search for "black and white dog"

hunterjm commented Dec 19, 2023

Example 1

Example 2:

hunterjm commented Dec 20, 2023

NickM-27 Dec 20, 2023

Choose a reason for hiding this comment

hunterjm Dec 22, 2023

Choose a reason for hiding this comment

NickM-27 Dec 20, 2023

Choose a reason for hiding this comment

hunterjm Dec 22, 2023

Choose a reason for hiding this comment

NickM-27 Dec 22, 2023

Choose a reason for hiding this comment

skrashevich commented Dec 22, 2023

hunterjm commented Dec 22, 2023

NickM-27 commented Jan 31, 2024

hunterjm commented Jun 3, 2024

hunterjm commented Dec 19, 2023 •

edited

Loading