[Bug]: Improve error message when replication key is not present in schema

[edgarrmondragon]

Singer SDK Version

0.17.0

Python Version

NA

Bug scope

Taps (catalog, state, stream maps, etc.)

Operating System

NA

Description

Right now, users only get the following rather unhelpful message:

ValueError: Could not detect type from empty type_dict. Did you forget to define a property in the stream schema?

Maybe the stream could catch ValueError and resurface it with a better message that contains the name of the field.

Code

No response

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

Still relevant

[mjsqu]

Interested in working on this one, just going to jot some notes:

• The message comes from singer_sdk.helpers._typing.EmptySchemaTypeError
• This error is raised in is_datetime_type, called fromsinger_sdk.streams.core.is_timestamp_replication_key:

    @property
    def is_timestamp_replication_key(self) -> bool:
        """Check is replication key is a timestamp.

        Developers can override to `True` in order to force this value, although this
        should not be required in most use cases since the type can generally be
        accurately detected from the JSON Schema.

        Returns:
            True if the stream uses a timestamp-based replication key.
        """
        if not self.replication_key:
            return False
        type_dict = self.schema.get("properties", {}).get(self.replication_key)
        return is_datetime_type(type_dict)

[mjsqu]

Suggested steps to fix:

• add a new Exception to singer_sdk.exceptions, or leverage an existing one if it exists
• check the value of type_dict in the function is_timestamp_replication_key and raise the new Exception if None