Missing support for arbitrary query parameter names?

[dcode]

Description

I'm trying to support arbitrary query parameters in my API. The API is a published specification where the user may offer query parameters for an object that is permitted to have additionalProperties per the jsonschema. The idea is the backend will return all objects subject to a query filter presented.

For example:

http://127.0.0.1:8000/api/images/?architecture=x86_64&os-distro=debian

In my specification, I have the following, pulled verbatim from the linked comment above.

openapi: 3.0.0

paths:
  /images:
    get:
      operationId: "images.read_all"
      tags:
        - "Images"
      summary: "Read the list of images, with optional filter"
      parameters:
        - in: query
          name: tags  # Arbitrary name
          schema:
            type: object
            additionalProperties:  # Schema for parameter values
              type: string
              pattern: '^[A-Za-z][A-Za-z0-9]*$'
            example:
              tagName1: tagValue1
              tagName2: tagValue2
      responses:
        "200":
          description: "Successfully read images list"

The function images.read_all is:

@FlaskDecorator()
def read_all(**kwargs):
    logger.info(f"Received query: {kwargs}")
    images = Image.query.all()
    return images_schema.dump(images)

related logs:

DEBUG - Query Parameter 'tags' (sanitized: 'tags') in function arguments
DEBUG - tags is a {'in': 'query', 'name': 'tags', 'schema': {'type': 'object', 'additionalProperties': {'type': 'string', 'pattern': '^[A-Za-z][A-Za-z0-9]*$'}, 'example': {'tagName1': 'tagValue1', 'tagName2': 'tagValue2'}}}
DEBUG - Query Parameter 'tagName1' (sanitized: 'tagName1') in function arguments
ERROR - Function argument 'tagName1' (non-sanitized: tagName1) not defined in specification
DEBUG - Query Parameter 'tagName2' (sanitized: 'tagName2') in function arguments
ERROR - Function argument 'tagName2' (non-sanitized: tagName2) not defined in specification

In the UI, I can submit the default example and given the specification above, it fails seemingly due to the query parameter deserialization issue mentioned in #1291. I try the workaround of using style: deepObject and explode: true and this now works in the UI, but the URL looks like this (url-encoded):

http://127.0.0.1:8000/api/images?tags[tagName1]=tagValue1&tags[tagName2]=tagValue2

Specification

        - in: query
          name: tags  # Arbitrary name
          style: deepObject
          explode: true
          schema:
            type: object
            additionalProperties:  # Schema for parameter values
              type: string
              pattern: '^[A-Za-z][A-Za-z0-9]*$'
            example:
              tagName1: tagValue1
              tagName2: tagValue2

Logs:

DEBUG - Query Parameter 'tags' (sanitized: 'tags') in function arguments
DEBUG - tags is a {'in': 'query', 'name': 'tags', 'style': 'deepObject', 'explode': True, 'schema': {'type': 'object', 'additionalProperties': {'type': 'string', 'pattern': '^[A-Za-z][A-Za-z0-9]*$'}, 'example': {'tagName1': 'tagValue1', 'tagName2': 'tagValue2'}}}
DEBUG - Query Parameter 'tags' (sanitized: 'tags') in function arguments
DEBUG - tags is a {'in': 'query', 'name': 'tags', 'style': 'deepObject', 'explode': True, 'schema': {'type': 'object', 'additionalProperties': {'type': 'string', 'pattern': '^[A-Za-z][A-Za-z0-9]*$'}, 'example': {'tagName1': 'tagValue1', 'tagName2': 'tagValue2'}}}
INFO - Received query: {'tags': {'tagName2': 'tagValue2', 'tagName1': 'tagValue1'}}
INFO:     127.0.0.1:58654 - "GET /images?tags%5BtagName1%5D=tagValue1&tags%5BtagName2%5D=tagValue2 HTTP/1.1" 200 OK

All the other options I've tried so far have also come up short, including using the full jsonschema specification as a reference and setting additionalProperties: True.

Expected behaviour

I expected to be able to pass additionalProperties via arbitrary key-values in the query string and they would be passed into the function kwargs dictionary and validated according to the additionalProperties schema, if given.

Desired query:

curl 'http://127.0.0.1:8000/api/images?foo=bar&architecture=x86_64'

Actual behaviour

The parameters are scrubbed after being mistakenly not in the specification.

Related logs:

DEBUG - /api/images validating parameters...
DEBUG - Query Parameter 'tags' (sanitized: 'tags') in function arguments
DEBUG - tags is a {'in': 'query', 'name': 'tags', 'style': 'deepObject', 'explode': True, 'schema': {'type': 'object', 'additionalProperties': {'type': 'string', 'pattern': '^[A-Za-z][A-Za-z0-9]*$'}, 'example': {'tagName1': 'tagValue1', 'tagName2': 'tagValue2'}}}
DEBUG - Query Parameter 'foo' (sanitized: 'foo') in function arguments
ERROR - Function argument 'foo' (non-sanitized: foo) not defined in specification
DEBUG - Query Parameter 'architecture' (sanitized: 'architecture') in function arguments
ERROR - Function argument 'architecture' (non-sanitized: architecture) not defined in specification
DEBUG - Query Parameter 'tags' (sanitized: 'tags') in function arguments
DEBUG - tags is a {'in': 'query', 'name': 'tags', 'style': 'deepObject', 'explode': True, 'schema': {'type': 'object', 'additionalProperties': {'type': 'string', 'pattern': '^[A-Za-z][A-Za-z0-9]*$'}, 'example': {'tagName1': 'tagValue1', 'tagName2': 'tagValue2'}}}
DEBUG - Query Parameter 'foo' (sanitized: 'foo') in function arguments
ERROR - Function argument 'foo' (non-sanitized: foo) not defined in specification
DEBUG - Query Parameter 'architecture' (sanitized: 'architecture') in function arguments
ERROR - Function argument 'architecture' (non-sanitized: architecture) not defined in specification
INFO - Received query: {'tags': {}}
INFO:     127.0.0.1:50460 - "GET /images?foo=bar&architecture=x86_64 HTTP/1.1" 200 OK

Steps to reproduce

Use the documented specification and queries noted above.

Additional info:

Per the comment on the OpenAPI specification here, it seems like this is officially supported in OAS 3.0, and with a bit more flexibility in OAS 3.1.

After reading through the code, I think the issue lies in this segment. When **kwargs is used, the function signature is ignored, however the query_definitions lookup ignores additionalProperties and blindly attempts to access the parameter by key (which doesn't exist). This is mistakenly interpreted as "not being defined in specification".

connexion/connexion/decorators/parameter.py

Lines 349 to 364 in 0c0c517

	logger.debug(
	"Query Parameter '%s' (sanitized: '%s') in function arguments",
	key,
	sanitized_key,
	)
	try:
	query_defn = query_definitions[key]
	except KeyError: # pragma: no cover
	logger.error(
	"Function argument '%s' (non-sanitized: %s) not defined in specification",
	sanitized_key,
	key,
	)
	else:
	logger.debug("%s is a %s", key, query_defn)
	result.update({sanitized_key: _get_val_from_param(value, query_defn)})

Workaround: As a workaround, it seems I can pull this information out of the Flask request object, but I'd rather not do that unless I'm desperate. I note it here in case someone else is hitting the same thing and needs it working right away.

RELATED WEIRDNESS: In the "Actual behavior" logs above, it seems like the query args are processed twice. This doesn't happen with deepObject style.

Output of the commands:

• python --version : Python 3.11.3
• pip show connexion | grep "^Version\:": Version: 3.0.0a6

I initially tried this with the current stable, "2.14.2", and had the same issue. I reworked it all hoping the new 3.x alpha might work.

[RobbeSneyders]

Thanks for the detailed description @dcode

I think there's a couple of issues at play here:

• Connexion doesn't pass in unspecified top level arguments to the view function.
• query params of type object with style: form are not being deserialized #1291

I think there's two options to work around this, which you seem to have figured out already:

• Don't specify the arguments in your specification, make sure strict_validation is disabled, and access the arguments via the request object exposed by connexion:

  curl 'http://127.0.0.1:8000/api/images?foo=bar&architecture=x86_64'
  

  from connexion import request
  request.query_params
  

  This means you cannot specify additionalProperties though, which means connexion cannot validate against this.

• Use style: deepObject and explode: true and use the following query:

  curl http://127.0.0.1:8000/api/images?tags[foo]=bar&tags[architecture]=x86_64
  

  This way you can specify the additionalProperties and Connexion should validate them. But this is only possible if you have control of the client and can change the query this way.

[RobbeSneyders]

Closing this in favor of #1291