Feat/fed search cleanup #920

ciaranschutte · 2025-03-14T15:39:48Z

remove unused code
clean up env config with correct parse functions, default and alphanumeric order
cleanup schema, mapping and startup config types and functions

…t files, improve usage'

justincorrigible · 2025-03-14T20:29:21Z

modules/server/src/gqlServer.ts

+import { GraphQLResolveInfo } from 'graphql';
+
+export type Context = {
+	esClient: Client;


TODO: we will need to abstract this into an ESClient provider, so users can switch seamlessly between ES v7 and v8

justincorrigible · 2025-03-14T20:29:56Z

modules/server/src/graphqlRoutes.js

 };

-const createSchemasFromConfigs = async ({
+export const createSchemasFromConfigs = async ({


🙏 I knew we'd need this export at some point, just couldn't remember which was the one function needed after a few months

justincorrigible · 2025-03-14T20:31:54Z

modules/server/src/mapping/resolveAggregations.ts

 		const response = await esSearch(esClient)({
 			index: type.index,
 			size: 0,
+			// @ts-expect-error - valid search query parameter in ES 7.17, not in types


* add placeholder * Adjust config template example and file * remove unused arranger constructor param * reorder constants file, parse env var * update arranger network config type * type, add gql server types * graphqlRoutes, remove unused import, export func * resolver creation, improve TS usage, split functionality into distinct files, improve usage' * add surrounding brackets for func * cleanup schema resolvers --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* Network aggregation search - Initial Config (#876) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * Update modules/server/README.md Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update types.ts * Update README.md * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Feat/#873 retrieve remote schemas (#877) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * use axios * axios network check, doesn't throw like fetch * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add logs to thrown errors * small feedback fixes * add typings + additional info in comments * add better errors * remove unused func. renaming * add return type * remove assertion. add TS filter * change nulls to undefined * change explicit undefined to optional ts * add tsdoc comment * change var type to loose record instead of uknown * if else TS * if else TS * change to nested promise return --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Pr feedback - doc updates (#879) * update tsdoc * update tsdoc --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/#874 merge schemas (#878) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * basic merge schema * cleanup comment * exclusuions * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * use axios * axios network check, doesn't throw like fetch * update packages for typedefs * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add logs to thrown errors * fix merge conflict * add test watch command with basic test * tests * Fix type. narrow scope. hoist filtering * remove debug code. add comment * update comments + add type assertion * add comments * remove redundant options arg for schema merginging * seperate test utils and fixture from main file * seperate test utils and fixture from main file * tests, cleanup, extra tests, split responsibility --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Add core functionality for remote connection resolvers (#880) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/agg resolver (#881) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check * fetchGQL helper func working with axios * rework dynamic network search types based on remote connections types * use __type query instead of full introspectionQuery * remove old code * add single typeDef export, responsible for merging remote connection and aggregation types * correct structure for merging typedefs with other typedefs, merge on Query * get all types from remote connections once * agg resolver structuring * add aggregations resolvers * add agg resolver network calls * rename and narrow NetworkType type across files * add common file * add network queries file * rename var * query remote connections * move query * add comments * add supported and unsupported aggregations with test * ts-jest disable diagnostics to run tests without all types passing * rework connection status property to do ping healthcehck * types cleanup across files * improved typing for reducer in field types, covering supported and unsupported aggregates * tighten and rename types * type cleanup and comments * add explicit name mapping * rename type correctly * Cleanup query lookup for aggregations for remote connections * move TSdoc comment * move types * add util func to convert gql AST into string * move aggregations resolver code into a module * simplify query to be composable, better naming for agg query map * aggregation logic placeholder * cleanup * rework logic to use single request and mapping fields, instead of slicing central query --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/resolve stats (#883) * add resolve aggregation function with tests * add basic aggregation TS types * cleanup * improve comments --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/resolver responses (#884) * add documentName to config and use in remote connection gql queries * move fulfilled promise type guard to util and make it a generic * resolve aggregations from network query results' * return resolved response object * PR feedback --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Fix network agg resolution (#885) * add Bucket gql object type * text change, move func outside of nested func * fix resolved aggregations overriding * Update modules/server/src/network/aggregations/index.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Response data & batching requests (#887) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * remove log --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Fed Search request timeouts (#889) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * add GQL request timeout with config option and default value --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Reorg connection (#891) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Accumulator pipeline (#893) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/fed total hits (#895) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * fix logging --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * feat/agg acc tests (#897) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/fed numeric agg resolution (#898) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test * add test + expand types * rework resolution iteration and add numericAggregations resolver * add typename to types * cleanup --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * use Math methods (#900) Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Individual node config query fields (#899) * use more detailed NodeConfig object to store available agg types * filter requested aggs with available aggs * add available aggs to node connection resolver type defs * cleanup --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/aggregate not available (#903) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/filters (#904) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests * add args to Aggregations gql typedef * add args to network field * pass gql resolvers args to query creation * remove console log * add include missing: * add overture sqon builder dep * move up type * add SQON check and args type * rename file * check for SQON correctness but pass ordinary object through on success * add comment * Update modules/server/src/network/utils/sqon.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Minor comment updates (#905) * improve comment * improve comment --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Fed search cleanup typings (#906) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * address feedback --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Fix broken build (#907) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * fix gql query creation test * query back to normal test * disables tests, adds comment * fix merged file --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * add agg mode env config (#908) * add agg mode env config * env config to strip out section of gql schema, typedef + resolver * prettier * prettier * Update modules/server/.env.schema Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/createConnectionResolvers.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * alphabetize * move up import * change AGG_ONLY_MODE to enabled DOCUMENT_HITS * add empty line at EOF * add comment * fix typo * fix inverted logic --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Feature/threshold calc (#909) * add conditional prop to schema * conditional schema field * conditional add dataMasking schema fields * add data masking logic * add agg only conditional to resolvers * clarify todo * get optional aggregations field and values for hits resolution * add mask func stub * fix missing prop from spread * masking logic * conditional resolve hits * move thresholding higher on resolver chain * add falsey belowThreshold value instead of null * data mask threshold env var * update wrong process.env path * seperate resolver creation from routing * check for undefiend value in lookup * clarify comment * add threshold value explanation * move types to common file * typing --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * multiple node with data thresholding (#912) * fix build error * single endpoint * fix GQL type name collisions * move data mask threshold env. ensure data masking applies even if hits.total is not queried * change belowThreshold to relation object * surface api errors to axios responses * add relation logic and agg merging for network search * read data mask threshold min from config * check for valid env val * remove unused func * fix TS build errors * fix schema merge error --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/fed search cleanup (#920) * add placeholder * Adjust config template example and file * remove unused arranger constructor param * reorder constants file, parse env var * update arranger network config type * type, add gql server types * graphqlRoutes, remove unused import, export func * resolver creation, improve TS usage, split functionality into distinct files, improve usage' * add surrounding brackets for func * cleanup schema resolvers --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/fed search networked cleanup (#921) * cleanup utils * reorg types, decompose file, improve comments * clean up types * cleanup types, redundant code and fix a couple of bad merge files * fix enum * cleanup test types * type fix * explicit type import, use path alias --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Update modules/server/src/app.js Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * make horrible ajax.ts console logging somewhat useful * improve release process * fix paths and imports for new build config * update tests to todos * fix some imports * fix build issues with lodash and missing config flag * fix paths * comment out test files, broken with tsx vs jest * fix path * fix path * improve release process * allow toolbar options customization * fix setsIndex conflict resolutioon misses * default for ENABLE_DOCUMENT_HITS env flag is true * default enable document hits to true * add DATA_MASK_MIN_THRESHOLD env var as pass through var * debug * debug * debug * add default param value * debug * string util important fix * fix for tests * Add comment re data masking * remove debug log * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/masking.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/resolveHitsFromAggs.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * remove specific NetworkAggError class * fix error response type narrow * missing param * add explanation comment for aggs resolver, gql info lookup --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> Co-authored-by: Anders Richardsson <2107110+caravinci@users.noreply.github.com>

* Feature/federated (#926) * Network aggregation search - Initial Config (#876) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * Update modules/server/README.md Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update types.ts * Update README.md * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Feat/#873 retrieve remote schemas (#877) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * use axios * axios network check, doesn't throw like fetch * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add logs to thrown errors * small feedback fixes * add typings + additional info in comments * add better errors * remove unused func. renaming * add return type * remove assertion. add TS filter * change nulls to undefined * change explicit undefined to optional ts * add tsdoc comment * change var type to loose record instead of uknown * if else TS * if else TS * change to nested promise return --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Pr feedback - doc updates (#879) * update tsdoc * update tsdoc --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/#874 merge schemas (#878) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * basic merge schema * cleanup comment * exclusuions * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * use axios * axios network check, doesn't throw like fetch * update packages for typedefs * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add logs to thrown errors * fix merge conflict * add test watch command with basic test * tests * Fix type. narrow scope. hoist filtering * remove debug code. add comment * update comments + add type assertion * add comments * remove redundant options arg for schema merginging * seperate test utils and fixture from main file * seperate test utils and fixture from main file * tests, cleanup, extra tests, split responsibility --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Add core functionality for remote connection resolvers (#880) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/agg resolver (#881) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check * fetchGQL helper func working with axios * rework dynamic network search types based on remote connections types * use __type query instead of full introspectionQuery * remove old code * add single typeDef export, responsible for merging remote connection and aggregation types * correct structure for merging typedefs with other typedefs, merge on Query * get all types from remote connections once * agg resolver structuring * add aggregations resolvers * add agg resolver network calls * rename and narrow NetworkType type across files * add common file * add network queries file * rename var * query remote connections * move query * add comments * add supported and unsupported aggregations with test * ts-jest disable diagnostics to run tests without all types passing * rework connection status property to do ping healthcehck * types cleanup across files * improved typing for reducer in field types, covering supported and unsupported aggregates * tighten and rename types * type cleanup and comments * add explicit name mapping * rename type correctly * Cleanup query lookup for aggregations for remote connections * move TSdoc comment * move types * add util func to convert gql AST into string * move aggregations resolver code into a module * simplify query to be composable, better naming for agg query map * aggregation logic placeholder * cleanup * rework logic to use single request and mapping fields, instead of slicing central query --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/resolve stats (#883) * add resolve aggregation function with tests * add basic aggregation TS types * cleanup * improve comments --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/resolver responses (#884) * add documentName to config and use in remote connection gql queries * move fulfilled promise type guard to util and make it a generic * resolve aggregations from network query results' * return resolved response object * PR feedback --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Fix network agg resolution (#885) * add Bucket gql object type * text change, move func outside of nested func * fix resolved aggregations overriding * Update modules/server/src/network/aggregations/index.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Response data & batching requests (#887) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * remove log --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Fed Search request timeouts (#889) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * add GQL request timeout with config option and default value --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Reorg connection (#891) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Accumulator pipeline (#893) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/fed total hits (#895) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * fix logging --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * feat/agg acc tests (#897) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/fed numeric agg resolution (#898) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test * add test + expand types * rework resolution iteration and add numericAggregations resolver * add typename to types * cleanup --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * use Math methods (#900) Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Individual node config query fields (#899) * use more detailed NodeConfig object to store available agg types * filter requested aggs with available aggs * add available aggs to node connection resolver type defs * cleanup --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/aggregate not available (#903) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/filters (#904) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests * add args to Aggregations gql typedef * add args to network field * pass gql resolvers args to query creation * remove console log * add include missing: * add overture sqon builder dep * move up type * add SQON check and args type * rename file * check for SQON correctness but pass ordinary object through on success * add comment * Update modules/server/src/network/utils/sqon.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Minor comment updates (#905) * improve comment * improve comment --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Fed search cleanup typings (#906) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * address feedback --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Fix broken build (#907) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * fix gql query creation test * query back to normal test * disables tests, adds comment * fix merged file --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * add agg mode env config (#908) * add agg mode env config * env config to strip out section of gql schema, typedef + resolver * prettier * prettier * Update modules/server/.env.schema Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/createConnectionResolvers.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * alphabetize * move up import * change AGG_ONLY_MODE to enabled DOCUMENT_HITS * add empty line at EOF * add comment * fix typo * fix inverted logic --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Feature/threshold calc (#909) * add conditional prop to schema * conditional schema field * conditional add dataMasking schema fields * add data masking logic * add agg only conditional to resolvers * clarify todo * get optional aggregations field and values for hits resolution * add mask func stub * fix missing prop from spread * masking logic * conditional resolve hits * move thresholding higher on resolver chain * add falsey belowThreshold value instead of null * data mask threshold env var * update wrong process.env path * seperate resolver creation from routing * check for undefiend value in lookup * clarify comment * add threshold value explanation * move types to common file * typing --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * multiple node with data thresholding (#912) * fix build error * single endpoint * fix GQL type name collisions * move data mask threshold env. ensure data masking applies even if hits.total is not queried * change belowThreshold to relation object * surface api errors to axios responses * add relation logic and agg merging for network search * read data mask threshold min from config * check for valid env val * remove unused func * fix TS build errors * fix schema merge error --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/fed search cleanup (#920) * add placeholder * Adjust config template example and file * remove unused arranger constructor param * reorder constants file, parse env var * update arranger network config type * type, add gql server types * graphqlRoutes, remove unused import, export func * resolver creation, improve TS usage, split functionality into distinct files, improve usage' * add surrounding brackets for func * cleanup schema resolvers --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/fed search networked cleanup (#921) * cleanup utils * reorg types, decompose file, improve comments * clean up types * cleanup types, redundant code and fix a couple of bad merge files * fix enum * cleanup test types * type fix * explicit type import, use path alias --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Update modules/server/src/app.js Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * make horrible ajax.ts console logging somewhat useful * improve release process * fix paths and imports for new build config * update tests to todos * fix some imports * fix build issues with lodash and missing config flag * fix paths * comment out test files, broken with tsx vs jest * fix path * fix path * improve release process * allow toolbar options customization * fix setsIndex conflict resolutioon misses * default for ENABLE_DOCUMENT_HITS env flag is true * default enable document hits to true * add DATA_MASK_MIN_THRESHOLD env var as pass through var * debug * debug * debug * add default param value * debug * string util important fix * fix for tests * Add comment re data masking * remove debug log * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/masking.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/resolveHitsFromAggs.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * remove specific NetworkAggError class * fix error response type narrow * missing param * add explanation comment for aggs resolver, gql info lookup --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> Co-authored-by: Anders Richardsson <2107110+caravinci@users.noreply.github.com> * initial cleanup * remove remaining data masking config + calculations * redo remove data masking relation ref * remove leftover env vars --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> Co-authored-by: Anders Richardsson <2107110+caravinci@users.noreply.github.com>

* Feature/federated (#926) * Network aggregation search - Initial Config (#876) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * Update modules/server/README.md Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update types.ts * Update README.md * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Feat/#873 retrieve remote schemas (#877) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * use axios * axios network check, doesn't throw like fetch * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add logs to thrown errors * small feedback fixes * add typings + additional info in comments * add better errors * remove unused func. renaming * add return type * remove assertion. add TS filter * change nulls to undefined * change explicit undefined to optional ts * add tsdoc comment * change var type to loose record instead of uknown * if else TS * if else TS * change to nested promise return --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Pr feedback - doc updates (#879) * update tsdoc * update tsdoc --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/#874 merge schemas (#878) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * basic merge schema * cleanup comment * exclusuions * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * use axios * axios network check, doesn't throw like fetch * update packages for typedefs * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add logs to thrown errors * fix merge conflict * add test watch command with basic test * tests * Fix type. narrow scope. hoist filtering * remove debug code. add comment * update comments + add type assertion * add comments * remove redundant options arg for schema merginging * seperate test utils and fixture from main file * seperate test utils and fixture from main file * tests, cleanup, extra tests, split responsibility --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Add core functionality for remote connection resolvers (#880) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/agg resolver (#881) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check * fetchGQL helper func working with axios * rework dynamic network search types based on remote connections types * use __type query instead of full introspectionQuery * remove old code * add single typeDef export, responsible for merging remote connection and aggregation types * correct structure for merging typedefs with other typedefs, merge on Query * get all types from remote connections once * agg resolver structuring * add aggregations resolvers * add agg resolver network calls * rename and narrow NetworkType type across files * add common file * add network queries file * rename var * query remote connections * move query * add comments * add supported and unsupported aggregations with test * ts-jest disable diagnostics to run tests without all types passing * rework connection status property to do ping healthcehck * types cleanup across files * improved typing for reducer in field types, covering supported and unsupported aggregates * tighten and rename types * type cleanup and comments * add explicit name mapping * rename type correctly * Cleanup query lookup for aggregations for remote connections * move TSdoc comment * move types * add util func to convert gql AST into string * move aggregations resolver code into a module * simplify query to be composable, better naming for agg query map * aggregation logic placeholder * cleanup * rework logic to use single request and mapping fields, instead of slicing central query --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/resolve stats (#883) * add resolve aggregation function with tests * add basic aggregation TS types * cleanup * improve comments --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/resolver responses (#884) * add documentName to config and use in remote connection gql queries * move fulfilled promise type guard to util and make it a generic * resolve aggregations from network query results' * return resolved response object * PR feedback --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Fix network agg resolution (#885) * add Bucket gql object type * text change, move func outside of nested func * fix resolved aggregations overriding * Update modules/server/src/network/aggregations/index.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Response data & batching requests (#887) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * remove log --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Fed Search request timeouts (#889) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * add GQL request timeout with config option and default value --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Reorg connection (#891) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Accumulator pipeline (#893) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/fed total hits (#895) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * fix logging --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * feat/agg acc tests (#897) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/fed numeric agg resolution (#898) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test * add test + expand types * rework resolution iteration and add numericAggregations resolver * add typename to types * cleanup --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * use Math methods (#900) Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Individual node config query fields (#899) * use more detailed NodeConfig object to store available agg types * filter requested aggs with available aggs * add available aggs to node connection resolver type defs * cleanup --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/aggregate not available (#903) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/filters (#904) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests * add args to Aggregations gql typedef * add args to network field * pass gql resolvers args to query creation * remove console log * add include missing: * add overture sqon builder dep * move up type * add SQON check and args type * rename file * check for SQON correctness but pass ordinary object through on success * add comment * Update modules/server/src/network/utils/sqon.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Minor comment updates (#905) * improve comment * improve comment --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Fed search cleanup typings (#906) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * address feedback --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Fix broken build (#907) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * fix gql query creation test * query back to normal test * disables tests, adds comment * fix merged file --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * add agg mode env config (#908) * add agg mode env config * env config to strip out section of gql schema, typedef + resolver * prettier * prettier * Update modules/server/.env.schema Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/createConnectionResolvers.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * alphabetize * move up import * change AGG_ONLY_MODE to enabled DOCUMENT_HITS * add empty line at EOF * add comment * fix typo * fix inverted logic --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Feature/threshold calc (#909) * add conditional prop to schema * conditional schema field * conditional add dataMasking schema fields * add data masking logic * add agg only conditional to resolvers * clarify todo * get optional aggregations field and values for hits resolution * add mask func stub * fix missing prop from spread * masking logic * conditional resolve hits * move thresholding higher on resolver chain * add falsey belowThreshold value instead of null * data mask threshold env var * update wrong process.env path * seperate resolver creation from routing * check for undefiend value in lookup * clarify comment * add threshold value explanation * move types to common file * typing --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * multiple node with data thresholding (#912) * fix build error * single endpoint * fix GQL type name collisions * move data mask threshold env. ensure data masking applies even if hits.total is not queried * change belowThreshold to relation object * surface api errors to axios responses * add relation logic and agg merging for network search * read data mask threshold min from config * check for valid env val * remove unused func * fix TS build errors * fix schema merge error --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/fed search cleanup (#920) * add placeholder * Adjust config template example and file * remove unused arranger constructor param * reorder constants file, parse env var * update arranger network config type * type, add gql server types * graphqlRoutes, remove unused import, export func * resolver creation, improve TS usage, split functionality into distinct files, improve usage' * add surrounding brackets for func * cleanup schema resolvers --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Feat/fed search networked cleanup (#921) * cleanup utils * reorg types, decompose file, improve comments * clean up types * cleanup types, redundant code and fix a couple of bad merge files * fix enum * cleanup test types * type fix * explicit type import, use path alias --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> * Update modules/server/src/app.js Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * make horrible ajax.ts console logging somewhat useful * improve release process * fix paths and imports for new build config * update tests to todos * fix some imports * fix build issues with lodash and missing config flag * fix paths * comment out test files, broken with tsx vs jest * fix path * fix path * improve release process * allow toolbar options customization * fix setsIndex conflict resolutioon misses * default for ENABLE_DOCUMENT_HITS env flag is true * default enable document hits to true * add DATA_MASK_MIN_THRESHOLD env var as pass through var * debug * debug * debug * add default param value * debug * string util important fix * fix for tests * Add comment re data masking * remove debug log * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/masking.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/resolveHitsFromAggs.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * remove specific NetworkAggError class * fix error response type narrow * missing param * add explanation comment for aggs resolver, gql info lookup --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> Co-authored-by: Anders Richardsson <2107110+caravinci@users.noreply.github.com> * initial cleanup * remove remaining data masking config + calculations * add network agg property from env * main conflicts * read directly from config --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> Co-authored-by: Anders Richardsson <2107110+caravinci@users.noreply.github.com>

add placeholder

c3d1c25

ciaranschutte changed the base branch from main to feature/federated_search March 14, 2025 15:41

Ciaran Schutte added 9 commits March 14, 2025 11:45

Adjust config template example and file

9d6fe98

remove unused arranger constructor param

f1e4666

reorder constants file, parse env var

d36c021

update arranger network config type

4d00dde

type, add gql server types

0d22b37

graphqlRoutes, remove unused import, export func

b00e5b2

resolver creation, improve TS usage, split functionality into distinc…

2268f85

…t files, improve usage'

add surrounding brackets for func

77d4e59

cleanup schema resolvers

2eb1e5c

ciaranschutte marked this pull request as ready for review March 14, 2025 20:24

justincorrigible reviewed Mar 14, 2025

View reviewed changes

justincorrigible approved these changes Mar 14, 2025

View reviewed changes

ciaranschutte merged commit 5ffe5e9 into feature/federated_search Mar 15, 2025
0 of 2 checks passed

ciaranschutte deleted the feat/fed_search_cleanup branch March 15, 2025 18:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feat/fed search cleanup #920

Feat/fed search cleanup #920

Uh oh!

ciaranschutte commented Mar 14, 2025 •

edited

Loading

Uh oh!

justincorrigible Mar 14, 2025

Uh oh!

justincorrigible Mar 14, 2025 •

edited

Loading

Uh oh!

justincorrigible Mar 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Feat/fed search cleanup #920

Feat/fed search cleanup #920

Uh oh!

Conversation

ciaranschutte commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

justincorrigible Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

justincorrigible Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justincorrigible Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ciaranschutte commented Mar 14, 2025 •

edited

Loading

justincorrigible Mar 14, 2025 •

edited

Loading