You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we have a push source with Redshift Offline Store and DynamoDb Online Store.
We built our view with more than 500 columns. Around 750 columns.
We expected to ingest data in dynamo and redshift when we run fs.push("push_source", df, to=PushMode.ONLINE_AND_OFFLINE)
Current Behavior
Push command raise an issue like [ERROR] ValueError: The input dataframe has columns ..
This issue come from get_table_column_names_and_types method in write_to_offline_store method.
In the method, we check if if set(input_columns) != set(source_columns) and raise the below issue if there are diff.
In case with more than 500 columns we get a diff because source_columns come from get_table_column_names_and_types method result where the result is define by MaxResults parameters.
Hi @beubeu13220 , I think either of the two solutions are good options. I think I'd prefer the NextToken approach simply because it's probably the most stable one.
Would you like to make a PR to add this functionality? We'd be happy to review!
Expected Behavior
Currently, we have a push source with Redshift Offline Store and DynamoDb Online Store.
We built our view with more than 500 columns. Around 750 columns.
We expected to ingest data in dynamo and redshift when we run
fs.push("push_source", df, to=PushMode.ONLINE_AND_OFFLINE)
Current Behavior
Push command raise an issue like
[ERROR] ValueError: The input dataframe has columns ..
This issue come from
get_table_column_names_and_types
method inwrite_to_offline_store
method.In the method, we check if
if set(input_columns) != set(source_columns)
and raise the below issue if there are diff.In case with more than 500 columns we get a diff because source_columns come from
get_table_column_names_and_types
method result where the result is define by MaxResults parameters.Steps to reproduce
Specifications
Possible Solution
In my mind, we have two solutions:
The text was updated successfully, but these errors were encountered: