Create HTTPConnector interface for all API interactions

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/319

Originally created by @edgarrmondragon on 2022-01-29 01:38:55

---

Summary

Currently, for HTTP taps (REST, GraphQL, etc.), all aspects of making a request go through the stream class. This works well for most access patterns where one endpoint corresponds to one stream. However, there are other use cases that require making requests not with the direct purpose of yielding records.

Proposed benefits

• Another request is required to complement or enrich the data, like when trying to get a complete record after a response that contains only IDs (https://gitlab.com/meltano/sdk/-/issues/93+).

• The API is dynamic and offers metadata endpoints that may inform the tap about available streams (https://gitlab.com/meltano/sdk/-/issues/313+) and/or their schemas (https://gitlab.com/meltano/sdk/-/issues/310+).

• The underlying request is not accessible, making it harder or unclear how to tweak parameters like timeout limits, caching (https://gitlab.com/meltano/sdk/-/issues/237+), etc.

• Async requests are not currently supported. Separation of concerns in an object dedicated to HTTP requests might make it easier for the community to implement and contribute an async connector (https://gitlab.com/meltano/sdk/-/issues/291+ and https://gitlab.com/meltano/sdk/-/issues/184+).

• A single interface for all HTTP sources would make features and bugfixes automatically apply to existing stream types and new ones too (https://gitlab.com/meltano/sdk/-/issues/221+).

Proposal details

• Build a module dedicated to HTTP sources, i.e. singer_sdk.http.

  • singer_sdk.http.connector contains the HTTPConnector class. In the MVC, HTTPConnector would manage aspects that the exiting stream classes do not, like the Session object.

  • singer_sdk.http.streams.base. In the MVC, methods useful to both GraphQL and REST streams can be moved to HTTPStream in this module without introducing breaking changes.

  • singer_sdk.http.streams.rest moved from singer_sdk.streams.rest

  • singer_sdk.http.streams.graphql moved from singer_sdk.streams.grapql

  • singer_sdk.http.authenticators moved from singer_sdk.authenticators

  • singer_sdk.http.paginators new module if https://gitlab.com/meltano/sdk/-/issues/318+ is implemented.

Best reasons not to build

This would not add any new features for existing taps and only counts as addressing maintainability and extensibility (encourage contributions).

[MeltyBot]

View 4 previous comments from the original issue on GitLab