Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
1f743da
change resolver code based on new gql type
Sep 16, 2024
a088522
comments and cleanup
Sep 18, 2024
e7bc1c9
break up main entry point into two modules, one for querying and one …
Sep 19, 2024
b4c7230
rename network config type, remove supported aggs from type
Sep 19, 2024
9412fed
type cleanup
Sep 19, 2024
f764bd0
only send network queries with original query fields
Sep 19, 2024
7d1a46b
move gql health check into module
Sep 19, 2024
5046131
cleanup, renaming types, removing redundant code
Sep 19, 2024
c11dd43
add type
Sep 19, 2024
68ab587
fix merged conflict
Sep 20, 2024
f0ddc0c
adds full stop. fixes comment typo
Sep 20, 2024
e264b12
fix pipeline code, move accumulator into class
Sep 24, 2024
da646e3
handle unavailable node
Sep 24, 2024
8160e19
check undefined in Success
Sep 24, 2024
bba55c2
add note
Sep 24, 2024
5105765
remove unused code
Sep 24, 2024
347b9e5
rename poorly named field
Sep 24, 2024
711b693
add comment:
Sep 24, 2024
2388add
rename file
Sep 26, 2024
6d665b7
http response success status as const
Sep 26, 2024
36b7fc6
renamed AggAccumulator file
Sep 26, 2024
762beeb
clean up types, change loose object to use record, fix any typings
Sep 27, 2024
d046aea
Merge branch 'accumulator_pipeline' into feat/fed_total_hits
Sep 27, 2024
e6f8fee
add total hits to query string
Sep 29, 2024
0d8a3ab
Merge branch 'feature/federated_search' into feat/fed_total_hits
Sep 29, 2024
aaf9278
cleanup field, param order
Sep 29, 2024
22405de
add test
Sep 29, 2024
6876ba9
rename RemoteAggregations type
Sep 30, 2024
9bfaeeb
add hits TS type
Sep 30, 2024
bdedb05
fix env toggle for network fed search
Sep 30, 2024
8ddf011
renaming, update types
Oct 1, 2024
6daf0bc
documentName and documentType are the same for functionality
Oct 1, 2024
10590f7
return unique fields
Oct 1, 2024
958fc6d
fix accumulator resolve for new input shape
Oct 1, 2024
301fdd4
tighten types. add Hits resolution
Oct 1, 2024
ac5b5f7
rename count to hits
Oct 1, 2024
112fa1a
add first test for requested fields
Oct 1, 2024
dc876e0
tighten types and error checking
Oct 2, 2024
2836cf8
fix up types and comments, resolvers code fix
Oct 2, 2024
7f486b0
update aggregation test
Oct 2, 2024
2dc800a
Merge remote-tracking branch 'origin/feature/federated_search' into f…
Oct 2, 2024
c108bc2
add test + expand types
Oct 2, 2024
9727fec
Merge branch 'feature/federated_search' into feat/fed_numeric_agg_res…
Oct 2, 2024
e7d5896
rework resolution iteration and add numericAggregations resolver
Oct 3, 2024
e36b064
add typename to types
Oct 3, 2024
cd1b150
cleanup
Oct 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 36 additions & 44 deletions modules/server/src/network/aggregations/AggregationAccumulator.ts
Original file line number Diff line number Diff line change
@@ -1,41 +1,33 @@
import { SUPPORTED_AGGREGATIONS } from '../common';
import { Aggregations, Bucket, NumericAggregations } from '../types/aggregations';
import { AllAggregations } from '../types/types';
import { RequestedFieldsMap } from '../util';

type ResolveAggregationInput = {
aggregationsMap: AllAggregations;
requestedAggregationFields: string[];
accumulator: AllAggregations;
};

type AggregationsTuple = [Aggregations, Aggregations];
type NumericAggregationsTuple = [NumericAggregations, NumericAggregations];

Comment on lines +10 to +12
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to tuple instead of loose array. resolving logic takes previous aggregation and current aggregation and returns a single aggregation

/**
* Resolves returned aggregations from network queries into single accumulated aggregation
*
* @param
*/
const resolveAggregations = ({
aggregationsMap,
requestedAggregationFields,
Copy link
Contributor

@demariadaniel demariadaniel Oct 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No more requested fields, just aggregate what's there -- definitely simplifies things

I'm just curious why there was a need to 'request specific fields' before and why that is no longer needed

Copy link
Contributor Author

@ciaranschutte ciaranschutte Oct 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is still a need to request specific fields, it just happens before the aggregation resolving.
POST GQL Query to request fields => resolve response

Example:
query to central server: {donors_gender: {...}, donors_age: {...}}
NodeA has [donors_gender, donors_age]
NodeB only has [donors_age]

Before code change if we are iterating on original query, NodeB.response[donors_gender] will be undefined

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it so it's all just stream lined handling of the same scenario, just wanted to make sure

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Other relevant comment for reference too #898 (comment)

For sure - thanks for checking!

accumulator,
}: ResolveAggregationInput) => {
requestedAggregationFields.forEach((fieldName) => {
const resolveAggregations = ({ aggregationsMap, accumulator }: ResolveAggregationInput) => {
Object.keys(aggregationsMap).forEach((fieldName) => {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iterate over response instead of request. possibility of undefined fields when using response object

const aggregation = aggregationsMap[fieldName];
const aggregationType = aggregation?.__typename || '';
const accumulatedFieldAggregation = accumulator[fieldName];

if (aggregation && accumulatedFieldAggregation) {
const resolvedAggregation = resolveToNetworkAggregation(aggregationType, [
aggregation,
accumulatedFieldAggregation,
]);
const accumulatedFieldAggregation = accumulator[fieldName];

// mutation - update a single aggregations field in the accumulator
accumulator[fieldName] = resolvedAggregation;
}
// mutation - update a single aggregations field in the accumulator
// if first aggregation, nothing to resolve yet
accumulator[fieldName] = !accumulatedFieldAggregation
? aggregation
: resolveToNetworkAggregation(aggregationType, [aggregation, accumulatedFieldAggregation]);
});
Comment on lines +27 to 30
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clean up resolution calling. if there's no existing aggregation, nothing to resolve with


return accumulator;
};

/**
Expand All @@ -46,12 +38,12 @@ const resolveAggregations = ({
*/
const resolveToNetworkAggregation = (
type: string,
aggregations: Aggregations[] | NumericAggregations[],
aggregations: AggregationsTuple | NumericAggregationsTuple,
): Aggregations | NumericAggregations => {
if (type === SUPPORTED_AGGREGATIONS.Aggregations) {
return resolveAggregation(aggregations as Aggregations[]);
return resolveAggregation(aggregations as AggregationsTuple);
} else if (type === SUPPORTED_AGGREGATIONS.NumericAggregations) {
return resolveNumericAggregation(aggregations as NumericAggregations);
return resolveNumericAggregation(aggregations as NumericAggregationsTuple);
} else {
// no types match
throw Error('No matching aggregation type');
Expand Down Expand Up @@ -153,7 +145,7 @@ const updateComputedBuckets = (bucket: Bucket, computedBuckets: Bucket[]) => {
* }
* ```
*/
export const resolveAggregation = (aggregations: Aggregations[]): Aggregations => {
export const resolveAggregation = (aggregations: AggregationsTuple): Aggregations => {
const resolvedAggregation = aggregations.reduce((resolvedAggregation, agg) => {
const computedBuckets = resolvedAggregation.buckets;
agg.buckets.forEach((bucket) => updateComputedBuckets(bucket, computedBuckets));
Expand All @@ -163,34 +155,34 @@ export const resolveAggregation = (aggregations: Aggregations[]): Aggregations =
return resolvedAggregation;
};

const resolveNumericAggregation = (aggregations: NumericAggregations) => {
// TODO: implement
throw Error('Not implemented');
};
const resolveNumericAggregation = (aggregations: NumericAggregationsTuple): NumericAggregations => {
return aggregations.reduce((resolvedAggregation, agg) => {
// max
if (agg.stats.max > resolvedAggregation.stats.max) {
resolvedAggregation.stats.max = agg.stats.max;
}
Comment on lines +160 to +163
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolvedAggregation.stats.max = Math.max(agg.stats.max, resolvedAggregation.stats.max);
resolvedAggregation.stats.min = Math.min(agg.stats.min, resolvedAggregation.stats.min);

// min
if (agg.stats.min < resolvedAggregation.stats.min) {
resolvedAggregation.stats.min = agg.stats.min;
}
// count
resolvedAggregation.stats.count += agg.stats.count;
// sum
resolvedAggregation.stats.sum += agg.stats.sum;
// avg
resolvedAggregation.stats.avg = resolvedAggregation.stats.sum / resolvedAggregation.stats.count;

Comment on lines +160 to 174
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NumericAggregations - Stats resolving logic

const emptyAggregation: Aggregations = { bucket_count: 0, buckets: [] };
return resolvedAggregation;
});
};

export class AggregationAccumulator {
requestedFields: string[];
totalAgg: AllAggregations;

constructor(requestedFieldsMap: RequestedFieldsMap) {
const requestedFields = Object.keys(requestedFieldsMap);
this.requestedFields = requestedFields;
/*
* seed accumulator with the requested field keys
* this will make it easier to add to using key lookup instead of Array.find
*/
this.totalAgg = requestedFields.reduce<AllAggregations>((accumulator, field) => {
return { ...accumulator, [field]: emptyAggregation };
}, {});
}
Comment on lines -178 to -187
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not needed anymore, saves an iteration by checking if aggregation exists while resolving

totalAgg: AllAggregations = {};

resolve(data: AllAggregations) {
resolveAggregations({
accumulator: this.totalAgg,
aggregationsMap: data,
requestedAggregationFields: this.requestedFields,
aggregationsMap: structuredClone(data),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ensure no input data is mutated. shouldn't be an issue due to life time of a response data in real time application but definitely causes issues with test fixtures.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL!

});
}

Expand Down
62 changes: 48 additions & 14 deletions modules/server/src/network/aggregations/tests/aggregation.test.ts
Original file line number Diff line number Diff line change
@@ -1,29 +1,24 @@
import { AggregationAccumulator } from '../AggregationAccumulator';
import { aggregation as fixture } from './fixture';

describe('Network aggregation resolution', () => {
it('should compute requested keys from info map constructor param', () => {
const requestedFields = { donors_age: {}, donors_gender: {} };
const totalAggs = new AggregationAccumulator(requestedFields);
expect(totalAggs.requestedFields.sort()).toEqual(Object.keys(requestedFields).sort());
});
// expected values
const maleCount = 835;
const femaleCount = 812;
const unknownCount = 2;
const bucketCount = 3;

const expectedStats = { max: 100, min: 1, count: 15, avg: 56, sum: 840 };

describe('Network aggregation resolution', () => {
describe('resolves multiple aggregations into a single aggregation:', () => {
it('should resolve multiple Aggregations type fields', () => {
const requestedFields = { donors_gender: {} };
const totalAggs = new AggregationAccumulator(requestedFields);
const totalAggs = new AggregationAccumulator();
const aggregationsToResolve = [
{ donors_gender: fixture.inputA },
{ donors_gender: fixture.inputC },
];
aggregationsToResolve.forEach((agg) => totalAggs.resolve(agg));

// expected values
const maleCount = 835;
const femaleCount = 812;
const unknownCount = 2;
const bucketCount = 3;

const result = totalAggs.result();
const aggregation = result['donors_gender'];

Expand All @@ -38,5 +33,44 @@ describe('Network aggregation resolution', () => {
unknownCount,
);
});
it('should resolve multiple NumericAggregations type fields', () => {
const totalAggs = new AggregationAccumulator();
const aggregationsToResolve = [
{ donors_weight: fixture.inputD },
{ donors_weight: fixture.inputE },
];
aggregationsToResolve.forEach((agg) => totalAggs.resolve(agg));

const result = totalAggs.result();
const aggregation = result['donors_weight'];
expect(aggregation.stats).toEqual(expectedStats);
});
it('should resolve a combination of Aggregations and NumericAggregations type fields', () => {
const aggregationsToResolve = [
{ donors_gender: fixture.inputA, donors_weight: fixture.inputE },
{ donors_gender: fixture.inputC, donors_weight: fixture.inputD },
];

const totalAggs = new AggregationAccumulator();
aggregationsToResolve.forEach((agg) => {
totalAggs.resolve(agg);
});

const result = totalAggs.result();
const donorsGenderAgg = result['donors_gender'];
const donorsWeightAgg = result['donors_weight'];

expect(donorsGenderAgg.bucket_count).toEqual(bucketCount);
expect(donorsGenderAgg.buckets.find((bucket) => bucket.key === 'Male')?.doc_count).toEqual(
maleCount,
);
expect(donorsGenderAgg.buckets.find((bucket) => bucket.key === 'Female')?.doc_count).toEqual(
femaleCount,
);
expect(donorsGenderAgg.buckets.find((bucket) => bucket.key === 'Unknown')?.doc_count).toEqual(
unknownCount,
);
expect(donorsWeightAgg.stats).toEqual(expectedStats);
});
});
});
15 changes: 14 additions & 1 deletion modules/server/src/network/aggregations/tests/fixture.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { Aggregations } from '@/network/types/aggregations';
import { Aggregations, NumericAggregations } from '@/network/types/aggregations';

const inputA: Aggregations = {
__typename: 'Aggregations',
Expand Down Expand Up @@ -49,8 +49,21 @@ const inputC: Aggregations = {
],
};

// NumericAggregations
const inputD: NumericAggregations = {
__typename: 'NumericAggregations',
stats: { max: 100, min: 12, count: 10, avg: 42, sum: 420 },
};

const inputE: NumericAggregations = {
__typename: 'NumericAggregations',
stats: { max: 75, min: 1, count: 5, avg: 84, sum: 420 },
};

export const aggregation = {
inputA,
inputB,
inputC,
inputD,
inputE,
};
12 changes: 10 additions & 2 deletions modules/server/src/network/types/aggregations.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,17 @@ export type Bucket = {
};

export type Aggregations = {
__typename?: string;
__typename: 'Aggregations';
bucket_count: number;
buckets: Bucket[];
};

export type NumericAggregations = { __typename?: string };
type Stats = {
max: number;
min: number;
count: number;
avg: number;
sum: number;
};

export type NumericAggregations = { __typename: 'NumericAggregations'; stats: Stats };