Low-code CDK: Odd behavior when a stream's path starts with /

[sherifnada]

Current Behavior

This issue tries to describe two problems: one with the developer experience and one with the functionality of the CDK.

I specify a stream whose base_url is https://api.congress.gov/v3 and whose path is /bills.

when running the command, it resulted in some failures in both check and read. --debug mode revealed that we were actually making the HTTP request to https://api.congress.gov/bills not https://api.congress.gov/v3/bill. Changing the path to bill (removing the leading /) resulted in making the request to the correct URL.

I get that this is the "technically" correct behavior of the urlparse python method that we're using under the hood, but even then it was a fairly confusing experience. Really, i would have just guessed that url_base and path are concatenated and the double slashes are removed, that's it. No extra magic.

Expected Behavior

Instead of the current behavior, concatenate url_base and path and remove double forward slashes from the result. That comprises the full URL.

My full YAML configuration for reference was:

version: "0.1.0"
definitions:
  schema_loader:
    type: JsonSchema
    file_path: "./source_congress_dot_gov/schemas/{{ options['name'] }}.json"
  selector:
    type: RecordSelector
    extractor:
      type: DpathExtractor
      field_pointer: []
  requester:
    type: HttpRequester
    name: "{{ options['name'] }}"
    http_method: "GET"
    request_options_provider:
      request_parameters:
        format: json
        api_key: "{{ config['api_key'] }}"
    path: "{{ options['path'] }}"
  retriever:
    type: SimpleRetriever
    $options:
      url_base: "https://api.congress.gov/v3/"
    name: "{{ options['name'] }}"
    primary_key: "{{ options['primary_key'] }}"
    record_selector:
      $ref: "*ref(definitions.selector)"
    requester:
      $ref: "*ref(definitions.requester)"
    paginator:
      type: NoPagination
streams:
  - type: DeclarativeStream
    $options:
      name: "bills"
      path: "/bill"
    primary_key: "number"
    schema_loader:
      $ref: "*ref(definitions.schema_loader)"
    retriever:
      $ref: "*ref(definitions.retriever)"
check:
  type: CheckStream
  stream_names: ["bills"]