Feat/filters

[ciaranschutte]

• add graphql args to network field, this is a Query top level field and the args can be passed on through to node query creation
• uses already established gql query functions to pass gql args as seperate variable (JSON type appears to only work in this manner anyway)
• API always has to query aggregations as a subfield of network

[joneubank]

I think this looks fine but have a question about making sure our variables in the fetch request match the variables we list in the gql query. Perhaps important thing to test.

[joneubank]

🙌

[joneubank]

we are declaring 3 variables here... if they are not provided in the args object will GQL throw an error?

[ciaranschutte]

it's ok for these. default gql args are nullable. there would be an error if args were required using '!' eg. (filter: JSON!)

[joneubank]

Type declared after it is used.

[joneubank]

We declare some specific variables in the gql query, should we be requiring these in the args?

[ciaranschutte]

We could. I'd love your typescript insight here.

Queries passing args that aren't defined will get an error straight from the gql server. Something like:

 "message": "Unknown argument \"random_arg\" on field \"Query.network\".",
      "extensions": {
        "code": "GRAPHQL_VALIDATION_FAILED",

So once we trust gql (and we should, strongly typed queries is a benefit it provides), args: Record<string, unknown> could always be args: Record<string, { ...known gql args types... }>

I think it's safe to pass on through the args "as is" because:

• We don't do anything with the args right now, they're just passed through as variables.
  The query creation is only string manipulation so TS wouldn't provide any benefit there by typing the args.
• GQL server will error itself because a type not matching the schema will throw an error

Bigger question is something that impacts a lot of the project - we have an intersection of GQL types and TS types. There's no quick and easy way to do this. The most straightforward we've had so far requires a build step (that's the thing Sam added for platform - I can't recall the library name - some kind of gql-codegen)

[joneubank]

Luckily for us, this module only exposes a single function whose interface needs to be typed. The function aggregationPipeline is exported and shouldn't allow any args, it should be specifying what we expect to be set. To that effect, consider introducing the type NetworkAggregationArgs and then update the function declaration to:

type NetworkAggregationArgs = {
	filters: object;
	aggregations_filter_themselves: boolean;
	include_missing: boolean;
};

/**
 * Query each remote connection
 *
 * @param queries - Query for each remote connection
 * @param requestedAggregationFields
 * @returns
 */
export const aggregationPipeline = async (
	configs: NodeConfig[],
	requestedAggregationFields: RequestedFieldsMap,
	args: NetworkAggregationArgs,
) => {

You'll then need to put corresponding updates into the GQL network resolvers to make sure the args provided from the client are properly validated and typed to be passed into the aggregation pipeline.

[joneubank]

Shouldn't this return boolean to check the data type? If you want an error thrown at a specific location then throw from there, but this won't be very reusable if it either does nothing or throws.

Also, lets add a TSDocs comment for this exported utility.

[ciaranschutte]

the return value is a SQON object, which we don't want, we want to keep it as a string.
So the only purpose this has is to throw an error if it's not a SQON

[joneubank]

Does GQL want us to throw strings instead of an error like new Error(message)?

It seems weird that you created isSQONFilter to throw an error and then immediately caught and threw a different error. Would be more reusable to have the sqon type check function return a Result with success containing the SQON and the failure containing the error message you want to include in your new thrown error.

import SQONBuilder, {SQON} from '@overture-stack/sqon-builder';

/**
 * Attempts to convert a variable to a SQON. A Result is returned, where if successful
 * the type checked SQON object is returned, and if a failure occurs then the error data
 * from the SQONBuilder is returned as the failure message.
 * 
 * @param filter 
 * @returns 
 */
export const convertToSqon = (filter: unknown): Result<SQON, 'INVALID_SQON'> => {
	try {
		const output = SQONBuilder.from(filter);
		return success(output.toValue());
	} catch (error: unknown) { 
		let message = `${error}`; // Will convert the error to a string, works fine with the expected ZodError without reuiring importing Zod to arranger
		return failure('INVALID_SQON', message)
	}
};

With this, you can also update the NetworkAggregationArgs to have filters as a SQON instead of object.

[joneubank]

Bonus, I'd rename your file httpResponse.ts to be result.ts.

[justincorrigible]

ToDo: consider logger instance/level, so this doesn't come up in regular Arranger instances (unless that may actually be desirable?)

[joneubank]

Looks functionally good. Some comments here for small updates, mostly regarding commenting the code.

[joneubank]

This is not a useful description. Consider expanding to describe what the content of the query string will be and how that string can be used. Also useful, consider: what are the arguments and how are they used?

[joneubank]

I think a comment block for this would be good. You have a comment below that says "type should match gql typedefs", but it would be good to directly mention which gql typedef, maybe including the path to that file or at least its name. As a first time reader of this file, I should know that this type is describing the inputs to a specific gql query.

[ciaranschutte]

Good idea! I'll include some kind of name link. I will avoid using path because if files are moved there's a string in a comment that needs to be updated.

[joneubank]

This is working fine, consider using the SQON schema exported with SQONBuilder:

import SQONBuilder, {SQON} from '@overture-stack/sqon-builder';

const isSqon = (input: unknown): input is SQON => SQON.safeParse(input).success;