Low-code CDK: request_parameters do not accept options

[isaacharrisholt]

Environment

• Airbyte version: 0.40.13
• OS Version / Instance: Kubuntu 5.18.5
• Deployment: Docker
• Source Connector and version: Alpha Vantage (WIP)
• Step where error happened: Stream check

Current Behavior

When trying to use {{ options['function'] }} in the request_parameters, the parameter is not included in the request, despite the option being present for the stream.

Expected Behavior

The function request parameter should be in the API call made.

Logs

Logs for py main.py check --config secrets/config.json --debug

{"type": "DEBUG", "message": "Debug logs enabled", "data": {}}
{"type": "DEBUG", "message": "parsed YAML into declarative source", "data": {"source_name": "SourceAlphaVantage", "path_to_yaml_file": "alpha_vantage.yaml", "parsed_config": "{\"version\": \"0.1.0\", \"definitions\": {\"schema_loader\": {\"type\": \"JsonSchema\", \"file_path\": \"./source_alpha_vantage/schemas/{{ options['name'] }}.json\"}, \"selector\": {\"type\": \"RecordSelector\", \"extractor\": {\"type\": \"DpathExtractor\", \"field_pointer\": [\"{{ options['dpath'] }}\"]}}, \"requester\": {\"url_base\": \"https://www.alphavantage.co\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"NoAuth\"}, \"request_options_provider\": {\"request_parameters\": {\"apikey\": \"{{ config['api_key'] }}\", \"symbol\": \"{{ config['symbol'] }}\", \"function\": \"{{ options['function'] }}\", \"datatype\": \"json\", \"outputsize\": \"{{ config['outputsize'] }}\"}}}, \"retriever\": {\"record_selector\": {\"type\": \"RecordSelector\", \"extractor\": {\"type\": \"DpathExtractor\", \"field_pointer\": [\"{{ options['dpath'] }}\"]}}, \"paginator\": {\"type\": \"NoPagination\"}, \"requester\": {\"url_base\": \"https://www.alphavantage.co\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"NoAuth\"}, \"request_options_provider\": {\"request_parameters\": {\"apikey\": \"{{ config['api_key'] }}\", \"symbol\": \"{{ config['symbol'] }}\", \"function\": \"{{ options['function'] }}\", \"datatype\": \"json\", \"outputsize\": \"{{ config['outputsize'] }}\"}}}}, \"base_stream\": {\"retriever\": {\"record_selector\": {\"type\": \"RecordSelector\", \"extractor\": {\"type\": \"DpathExtractor\", \"field_pointer\": [\"{{ options['dpath'] }}\"]}}, \"paginator\": {\"type\": \"NoPagination\"}, \"requester\": {\"url_base\": \"https://www.alphavantage.co\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"NoAuth\"}, \"request_options_provider\": {\"request_parameters\": {\"apikey\": \"{{ config['api_key'] }}\", \"symbol\": \"{{ config['symbol'] }}\", \"function\": \"{{ options['function'] }}\", \"datatype\": \"json\", \"outputsize\": \"{{ config['outputsize'] }}\"}}}}}, \"time_series_daily_stream\": {\"retriever\": {\"record_selector\": {\"type\": \"RecordSelector\", \"extractor\": {\"type\": \"DpathExtractor\", \"field_pointer\": [\"{{ options['dpath'] }}\"]}}, \"paginator\": {\"type\": \"NoPagination\"}, \"requester\": {\"url_base\": \"https://www.alphavantage.co\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"NoAuth\"}, \"request_options_provider\": {\"request_parameters\": {\"apikey\": \"{{ config['api_key'] }}\", \"symbol\": \"{{ config['symbol'] }}\", \"function\": \"{{ options['function'] }}\", \"datatype\": \"json\", \"outputsize\": \"{{ config['outputsize'] }}\"}}}}, \"$options\": {\"name\": \"time_series_daily\", \"dpath\": \"Time Series (Daily)\", \"function\": \"TIME_SERIES_DAILY\", \"path\": \"/query\"}, \"class_name\": \"airbyte_cdk.sources.declarative.declarative_stream.DeclarativeStream\"}}, \"streams\": [{\"retriever\": {\"record_selector\": {\"type\": \"RecordSelector\", \"extractor\": {\"type\": \"DpathExtractor\", \"field_pointer\": [\"{{ options['dpath'] }}\"]}}, \"paginator\": {\"type\": \"NoPagination\"}, \"requester\": {\"url_base\": \"https://www.alphavantage.co\", \"http_method\": \"GET\", \"authenticator\": {\"type\": \"NoAuth\"}, \"request_options_provider\": {\"request_parameters\": {\"apikey\": \"{{ config['api_key'] }}\", \"symbol\": \"{{ config['symbol'] }}\", \"function\": \"{{ options['function'] }}\", \"datatype\": \"json\", \"outputsize\": \"{{ config['outputsize'] }}\"}}}}, \"$options\": {\"name\": \"time_series_daily\", \"dpath\": \"Time Series (Daily)\", \"function\": \"TIME_SERIES_DAILY\", \"path\": \"/query\"}, \"class_name\": \"airbyte_cdk.sources.declarative.declarative_stream.DeclarativeStream\"}], \"check\": {\"stream_names\": [\"time_series_daily\"], \"class_name\": \"airbyte_cdk.sources.declarative.checks.check_stream.CheckStream\"}}"}}
{"type": "DEBUG", "message": "Making outbound API request", "data": {"url": "https://www.alphavantage.co/query?apikey=****&symbol=TSLA&datatype=json&outputsize=compact", "request_body": "None", "headers": "{'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}"}}
{"type": "DEBUG", "message": "Receiving response", "data": {"status": "200", "body": "{\n    \"Error Message\": \"This API function () does not exist.\"\n}", "headers": "{'Date': 'Thu, 13 Oct 2022 17:21:15 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Cookie', 'X-Frame-Options': 'SAMEORIGIN', 'Allow': 'GET, HEAD, OPTIONS', 'Via': '1.1 vegur', 'CF-Cache-Status': 'DYNAMIC', 'Server': 'cloudflare', 'CF-RAY': '7599c027a934dd2b-LHR', 'Content-Encoding': 'gzip'}"}}
{"type": "LOG", "log": {"level": "ERROR", "message": "Check failed"}}
{"type": "CONNECTION_STATUS", "connectionStatus": {"status": "FAILED", "message": "'Unable to connect to stream time_series_daily - '"}}

Steps to Reproduce

1. Create connector with low-code CDK named alpha-vantage
2. Generate an Alpha Vantage API key (www.alphavantage.co)
3. Set up the connector with the below:

spec.yaml

documentationUrl: https://docsurl.com
connectionSpecification:
  $schema: http://json-schema.org/draft-07/schema#
  title: Alpha Vantage Spec
  type: object
  required:
    - api_key
    - symbol
  properties:
    api_key:
      title: API Key
      type: string
      description: API Key
      airbyte_secret: true
      order: 0
    symbol:
      title: Symbol
      type: string
      description: Stock symbol (with exchange code)
      examples:
        - AAPL
        - TSCO.LON
      order: 1
    interval:
      title: Interval
      type: string
      description: |
        Time-series data point interval. Required for intraday endpoints.
      enum:
        - 1min
        - 5min
        - 15min
        - 30min
        - 60min
      default: 1min
      order: 2
    slice:
      title: Slice
      type: string
      description: |
        Time-series data point slice. Required for intraday endpoints. Goes as
        far back as 2 years (year2month12).
      pattern: "^year[1-2]month[1-12]$"
      examples:
        - year1month1
        - year2month12
      order: 3
    adjusted:
      title: Adjusted?
      type: boolean
      description: |
        Whether to return adjusted data. Only applicable to intraday endpoints.
      default: false
      order: 4
    outputsize:
      title: Output Size
      type: string
      description: |
          Whether to return full or compact data (the last 100 data points).
      enum:
          - compact
          - full
      default: compact
      order: 5

alpha_vantage.yaml

version: "0.1.0"

definitions:
  schema_loader:
    type: JsonSchema
    file_path: "./source_alpha_vantage/schemas/{{ options['name'] }}.json"
  selector:
    type: RecordSelector
    extractor:
      type: DpathExtractor
      field_pointer:
        - "{{ options['dpath'] }}"
  requester:
    url_base: "https://www.alphavantage.co"
    http_method: "GET"
    authenticator:
      type: NoAuth
    request_options_provider:
      request_parameters:
        apikey: "{{ config['api_key'] }}"
        symbol: "{{ config['symbol'] }}"
        function: "{{ options['function'] }}"
#        function: TIME_SERIES_DAILY
        datatype: json
        outputsize: "{{ config['outputsize'] }}"
  retriever:
    record_selector:
      $ref: "*ref(definitions.selector)"
    paginator:
      type: NoPagination
    requester:
      $ref: "*ref(definitions.requester)"
  base_stream:
    retriever:
      $ref: "*ref(definitions.retriever)"
  time_series_daily_stream:
    $ref: "*ref(definitions.base_stream)"
    $options:
      name: "time_series_daily"
      dpath: "Time Series (Daily)"
      function: "TIME_SERIES_DAILY"
      path: "/query"

streams:
  - "*ref(definitions.time_series_daily_stream)"

check:
  stream_names:
    - "time_series_daily"

4. Run py main.py check --config secrets/config.json --debug

Are you willing to submit a PR?

Not currently.

[natalyjazzviolin]

@bleonard could you assist with this? I know we want to catch these as early as possible for Hacktoberfest :)

[bleonard]

maybe @girarda or @sherifnada can help me out.
My guess is that the check command does not use options from a single stream. It has to be able to work only off of config params.

[brianjlai]

tagging as P2, but also has an in flight PR here: #18050