[BIG Refactoring] Add abstraction for search backend client #464

fgravin · 2023-04-24T19:26:03Z

Work in progress for OGC sprint.
Refactor the search client, which is now a full elasticsearch centric service, to be backend agnostic.
Add an abstraction, with clean architecture in mind and first add the gn4 (elasticsearch) implementation. This will prepare the work for adding OGC Records API later on.

We should work with DDD and try to focus on the domain and describe the models and usecases.

github-actions · 2023-04-24T19:27:37Z

Affected libs: api-metadata-converter, feature-editor, api-repository, feature-catalog, feature-record, feature-router, feature-search, feature-map, feature-dataviz, ui-search, common-domain, common-fixtures, util-shared, ui-elements, ui-catalog, ui-widgets, ui-inputs, ui-map, feature-auth, ui-dataviz, data-access-datafeeder, data-access-gn4, util-app-config, util-i18n, util-data-fetcher, ui-layout,
Affected apps: data-platform, datahub, metadata-converter, metadata-editor, webcomponents, demo, search, map-viewer, datafeeder,

🚀 Build and deploy storybook and demo on GitHub Pages
📦 Build and push affected docker images

libs/feature/search/src/lib/domain/search.client.ts

fgravin

Thanks,

Few remarks

we should move out gn4 models from the domain
atomic.operation should not depend on gn4 model
we should only manipulate the pivot format, and remove the old model
where will you put the mappers ?
we should spread files in folders for the domain
shared/links should be moved

It's on a good way though, congratz

fgravin · 2023-07-27T14:08:13Z

libs/feature/search/src/lib/constants.ts

+
+export const DEFAULT_PAGE_SIZE = 10
+
+export const FIELDS_SUMMARY: FieldName[] = [


This should go in the domain with domain fields

fgravin · 2023-07-27T14:12:04Z

apps/datahub/src/app/home/news-page/last-created/last-created.component.ts

-      includes: [...ES_SOURCE_BRIEF, 'createDate', 'changeDate'],
-    })
+    this.searchFacade.setConfigRequestFields([
+      ...FIELDS_BRIEF,


should be pivot format fields

This needs a stronger abstraction and will be done later on

fgravin

Testing

clicking on header badges does not update the sortby input

jahow · 2023-08-22T16:59:12Z

This is rebased on main and generally finished, we need to do more tests in the various apps to make sure nothing is broken.

I will rework the commit history to make the various steps of the refactoring clearer.

Also note that I've left on the side the handling of favorites and filter geometry in the search state; I think this can be done in another PR because it was just too many changes at once.

coveralls · 2023-08-23T17:58:12Z

coverage: 83.902% (-2.0%) from 85.859% when pulling 88f0225 on refactor-search-client into 9811dd5 on main.

fgravin

Massive work !
Looks good, I noticed a breaking change with the wc elastic configuration. I added a note in the release description.

We can merge and slowly test and fix the side effects.

fgravin · 2023-08-25T13:16:21Z

libs/feature/search/src/lib/state/reducer.spec.ts

      })
      const state = reducerSearch(initialStateSearch, action)
      expect(state.config.filters).toEqual({
-        custom: { any: 'blah', other: 'Some value' },
-        elastic: {},


Will break web components

filter='{"elastic": [{ "term": { "OrgForResource": "Géo2France" } }]}'

Also make some simplifications/improvements to it and adapt the iso19139 converter

All types related to the gn4 repository are moved as well

These will have to be reworked

The sort by combos have been slightly changed to avoid using JSON as values. This was breaking the e2e tests and doing a join(',') is also much simpler. The values for the orgs filter dropdown have been updated to show correct values. also fixed a bug when clicking on the org name in the dataset view

filters are SearchFilters like others and are merged with state filters before building the ES payload

f-necas · 2023-08-25T16:38:19Z

🥳

jahow · 2023-08-25T18:47:38Z

🎉

jahow reviewed Apr 26, 2023

View reviewed changes

libs/feature/search/src/lib/domain/search.client.ts Outdated Show resolved Hide resolved

jahow force-pushed the refactor-search-client branch 2 times, most recently from dfe001a to d6cd5a1 Compare May 2, 2023 06:01

fgravin changed the title ~~Add abstraction for search backend client~~ [BIG Refactoring] Add abstraction for search backend client Jul 5, 2023

fgravin added the refactoring label Jul 5, 2023

fgravin assigned jahow Jul 13, 2023

jahow force-pushed the refactor-search-client branch 3 times, most recently from b025a8b to 493ed7c Compare July 25, 2023 15:25

fgravin commented Jul 27, 2023

View reviewed changes

jahow added this to the 2.0.0 milestone Aug 2, 2023

jahow force-pushed the refactor-search-client branch 3 times, most recently from b6d9654 to b2e96f8 Compare August 22, 2023 14:14

fgravin commented Aug 22, 2023

View reviewed changes

jahow force-pushed the refactor-search-client branch from b2e96f8 to 62d2916 Compare August 22, 2023 16:58

jahow marked this pull request as ready for review August 22, 2023 16:58

jahow force-pushed the refactor-search-client branch 7 times, most recently from fd97210 to 73a786b Compare August 23, 2023 17:47

jahow force-pushed the refactor-search-client branch from db3fb7c to 88f0225 Compare August 24, 2023 10:17

jahow mentioned this pull request Aug 24, 2023

Reimplement favorites & geometry filtering following the internal data format change #590

Closed

fgravin commented Aug 25, 2023

View reviewed changes

jahow added 17 commits August 25, 2023 16:11

chore: remove build target from md-converter lib

c7f3887

ci: upgrade node to stable

32ff780

fix: fix metadata-converter project name

535e53b

feat: move pivot format to common module

4dee454

Also make some simplifications/improvements to it and adapt the iso19139 converter

feat: introduce gn4 records repository with associated mapper

a8790c3

All types related to the gn4 repository are moved as well

refactor: remove old types

505666a

refactor: move fixtures to common

b6dd663

refactor: adjust fileds service to use repository

734d762

refactor: adjust search state & effects to use repository

454ee42

refactor: adjust record view state & effects to use repository

d7a2a29

refactor: adjust links & data service to use pivot format

009e6dd

refactor: adjust sort by component to use pivot format

4dc681a

refactor: adjust all files to use pivot format

384b499

test: deactivate two tests

c7418a6

These will have to be reworked

feat(search): better handle missing orgs/thumbnails

98ccc40

fix(data-service): fix wrong protocol & improve error handling

ade09fb

jahow force-pushed the refactor-search-client branch from 88f0225 to 6d70763 Compare August 25, 2023 14:12

fgravin added 4 commits August 25, 2023 16:25

fix(downloads): display link description first

d24311e

fix(wc): remove ConfigFilters specificity

4ebbe7c

filters are SearchFilters like others and are merged with state filters before building the ES payload

refactor(es): clean useless interfaces

a09a3fb

fix(sortby): avoid null value from sortBy$ observable

07c78fe

fgravin merged commit ba8b141 into main Aug 25, 2023
5 checks passed

fgravin deleted the refactor-search-client branch August 25, 2023 16:35

This was referenced Aug 25, 2023

[Sponsor by MEL] Add feature Metadata Quality Widget #497

Merged

Pivot format as internal format #551

Closed

This was referenced Sep 19, 2023

Reactivate filtering by favorites & geometry #623

Merged

ElasticSearch filters processing sometime fails #628

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BIG Refactoring] Add abstraction for search backend client #464

[BIG Refactoring] Add abstraction for search backend client #464

fgravin commented Apr 24, 2023

github-actions bot commented Apr 24, 2023 •

edited

Loading

fgravin left a comment

fgravin Jul 27, 2023

fgravin Jul 27, 2023

jahow Aug 22, 2023

fgravin left a comment

jahow commented Aug 22, 2023 •

edited

Loading

coveralls commented Aug 23, 2023 •

edited

Loading

fgravin left a comment

fgravin Aug 25, 2023

f-necas commented Aug 25, 2023

jahow commented Aug 25, 2023


		export const DEFAULT_PAGE_SIZE = 10

		export const FIELDS_SUMMARY: FieldName[] = [

[BIG Refactoring] Add abstraction for search backend client #464

[BIG Refactoring] Add abstraction for search backend client #464

Conversation

fgravin commented Apr 24, 2023

github-actions bot commented Apr 24, 2023 • edited Loading

fgravin left a comment

Choose a reason for hiding this comment

fgravin Jul 27, 2023

Choose a reason for hiding this comment

fgravin Jul 27, 2023

Choose a reason for hiding this comment

jahow Aug 22, 2023

Choose a reason for hiding this comment

fgravin left a comment

Choose a reason for hiding this comment

jahow commented Aug 22, 2023 • edited Loading

coveralls commented Aug 23, 2023 • edited Loading

fgravin left a comment

Choose a reason for hiding this comment

fgravin Aug 25, 2023

Choose a reason for hiding this comment

f-necas commented Aug 25, 2023

jahow commented Aug 25, 2023

github-actions bot commented Apr 24, 2023 •

edited

Loading

jahow commented Aug 22, 2023 •

edited

Loading

coveralls commented Aug 23, 2023 •

edited

Loading