Separate cluster and matches endpoints. #183
Comments
stepan-anokhin
added a commit
to stepan-anokhin/VideoDeduplication
that referenced
this issue
Nov 11, 2020
johnhbenetech
pushed a commit
that referenced
this issue
Nov 11, 2020
* Separate cluster and matches enpoints (#183) * Update server tests * Test matches endpoint * Update REST client * Refacto API client * Creat generic hook for entity loading * Consume matches and cluster API separately * Manage fields inclusion * Update immediate match loading * Update comparison page * Remove trash * Remove trash
johnhbenetech
added a commit
that referenced
this issue
Nov 20, 2020
* File cluster proof of concept (#162) * Implement files data-access object * Migrate to files-dao * Update server tests * Fix regression * Make module naming more ideomatic * Resolve linting issues * Support expunging by session scope * Draft matches extraction * Filter matches by distance * Test query matches with cycles * Test match loading * Emulate matches pagination * Update MatchesDAO tests * Migrate server.api to MatchesDAO * Update api/matches tests * Test multiple hops with cycles * Update file matches page * Update cluster page to consume new matches format * Improve graph container responsiveness * Fix linting issues * Improve graph responsiveness * Implement generic loading trigger * Support matches filtering * Implement dynamic matches loading * Fix loading trigger message * Fix match reducers * Improve dynamic match loading * Implement neighbor loading * Adjust graph style * Enable zooming * Enable cluster navigation * Fix tooltips * Make edge opacity dynamic * Make color scheme static * Handle playback issues (#165) * Handle missing file in thumbnail endpoint * Handle missing file in watch endpoint * Display playback/loading errors * Handle file missing in database case * Use bundled flv.js By default react-player tries to lazy-load playback SDK from CDN. But the application must be able play video files when Internet connection is not available. To solve that we bundle flv.js and initialize global variable consumed by react-player's FilePlayer. See https://www.npmjs.com/package/react-player#sdk-overrides See cookpete/react-player#605 (comment) * Suppress excessive flv.js error logs (#149) * coding style improvements and small refactors * Fix matches loading (#166) (#168) * Support config tags (#167) * Support tags by PathReprStorage * Support tags by SQLiteReprStorage * Update extract_features.py to support tags * Add missing dependency * Make config tag opaque * Update extract_features.py sript * Delete obsolete code * Update repr storage tests * Update generate_matches.py script * Update template_matching.py script * Update general_tests * Add missing unit-test dependency * Optimize module dependencies * Implement side-by-side match comparison view (#155) (#169) * Refactor file cluster page * Refacto VideoInformationPane * Make video details elements collapsible * Move distance element to common package * Add comparable file component * Implement reusable file summary * Implement match file list * Implement mother file view * Fetch matched files scenes * Setup comparison page routing * Reset video player on file change * Fix matched file header * Improve distance style * Hook up compare button * Fix match duplication * Fix linting issues * Make compare button primary-colored * added missing dependency * removed unnecessary files * additional refactor and linting fixes * Improve cluster view (#171) * Refactor FileSummaryHeader * Refactor linear list item * Extract match file id from URL * Navigate to comparison page on edge click * Navigate to compare from single match preview * Improve cluster tooltips * Implement node highlighting * Improve link popover * Fix linting issues * Add navigation from comparison page (#172) (#177) * Handle partial processing results (#178) * Handle missing data in backend (#135) * Test missing data processing * Fix go back navigation (#179) * Fix go-back navigation * Preserve filters and search results Previous file filters and fetched files should not be updated when we navigate from video-details page to the collection page. * Handle missing video length * Fix match category selector * Ensure initial file loading * Add match file enumeration & sorting (#181) * Add match enumeration (#175) * Sort files by match score * Handle uninitialized ref * Add image version option (#180) * Ensure shell is bash * Refactor setup script * Select image version during setup (#97) * Add make-goal for forced setup update * Improve scripts output * Add docker update and purge goals (#114) * Use arrow-select for pre-built option * Add various UI improvements (#173, #174, #176) (#182) * Handle dense layout * Add active filters indication (#176) * Preserve file view type during session (#174) * Enlarge links hitbox (#173) * Fix link click handler * augmented dataset evaluation pipeline * Modifications to support benchmarking script * Add cluster API endpoint (#183) (#184) * Separate cluster and matches enpoints (#183) * Update server tests * Test matches endpoint * Update REST client * Refacto API client * Creat generic hook for entity loading * Consume matches and cluster API separately * Manage fields inclusion * Update immediate match loading * Update comparison page * Remove trash * Remove trash * Refactor ui state management (#185, #189) (#190) * Move helpers to separate modules * Move file cache to separate package * Move file matches state to separate package * Move cluster state to separate package * Move file-list state to separate package * Collect reusable prop-types in a public package * Extract entity fetching logic * Refactor file-cluster state Use generic approach to manage file-cluster state. * Refactor file-matches state Use generic approach to manage file-matches state. * Update server client * Update matches params (#189) * Fix file cluster update * Disable false-positive linting issue * Create post_push * Simplify docker-compose workflow (#187, #188) (#191) * Simplify docker-compose workflow Use `build` and `image` simultaneously in compose-file. As a result no shared configuration is needed and all the configuration is managed by environment variables (located in the .env file). To use prebuild images user will need to run `make pull` To use local images (if production mode is disabled) user will need to run `make build`. That's it. * Commit missing script * Add revision information to docker images (#188) * Benchmarking improvements * Update and rename post_push to build * Delete build * Parse exif general encoded date (#137) (#193) * Update db schema * Fix media-info output parsing * Parse exif date-time * Use date-time on backend * Update tests * Use numeric timestamps on frontend * Handle NaN-representation of missing values The exif metadata is pre-processed by the pandas.json_normalize method which replaces any missing value with NaNs. As a result the float value may appear in place of string. * Handle pandas missing data representation pandas.Series heuristically determines type of the underlying data and tries to represent a missing values according to that data time. In case of datetime the missing values are represented by pandas.NaT which is not compatible with SQLAlchemy framework. To fix that we have to explicitly transform NaTs to Nones. See https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#datetimes * Fix deprecated import * Fix database URI * Handle more date columns * Update video metadata model (#139) (#196) * Update db schema * Update db access logic * Update frontend * Update server tests * Update storage tests * Additional error handling for dealing with missing files * Added Missing frame level features check * Quick merge fix Co-authored-by: Stepan Anokhin <38860530+stepan-anokhin@users.noreply.github.com> Co-authored-by: Felipe Batista <fsbatista1@gmail.com> Co-authored-by: Stepan Anokhin <stepan.anokhin@gmail.com>
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently there is a single API endpoint (namely
/api/v1/files/:id/matches) for the following use cases:Retrieving cluster fragments is unnecessarily complex and slow for use case 1. We need to create a separate endpoint (
/api/v1/files/:id/cluster) for cluster querying.The text was updated successfully, but these errors were encountered: