Saving changes not working when changing destination output format-format type from Avro to Parquet #5907

da-firebolt · 2021-09-08T09:33:02Z

Enviroment

Airbyte version: 0.29.13-alpha
OS Version / Instance: Ubuntu 21.04 VM
Deployment: Kubernetes
Source Connector and version: (if applicable example Salesforce 0.2.3)
Destination Connector and version: (if applicable example Postgres 0.3.3) S3
Severity: Low
Step where error happened: Update connector

Current Behavior

Tell us what happens.
When changing S3 destination output format/format type from Avro to parquet, saving doesn't actually do anything, format type still remains Avro and instead of connection being tested, you're automatically taken to the destinations view, but when you change from Avro to e.g. CSV the change is applied correctly.

Expected Behavior

Tell us what should happen.
When changing S3 destination output format/format type from Avro to parquet, clicking "Save changes and test" button should result in connection being tested and change applied.

Logs

If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.

LOG


replace this with
your long log
output here

Steps to Reproduce

Create a new Destination and use S3 as the Destination type, fill out all the remaining fields including choosing Avro as the output format as well as the format type and click Set up destination when done.
Check that the destination you just created has Avro as its output format, change this to Parquet and change format type to Parquet too, set compression codec to GZIP and leave everything else as it is.
Click "Save changes and test button" and check if this destination now has Parquet as its output format

Are you willing to submit a PR?

Remove this with your answer.

The text was updated successfully, but these errors were encountered:

marcosmarxm · 2021-09-08T20:54:15Z

I recorded a video with the same problem. thanks for reporting this @da-firebolt

Airbyte._.Destinations._.s3-dest.mp4

da-firebolt · 2021-09-09T08:09:51Z

My pleasure @marcosmarxm, here's my video too if it may help!

https://www.loom.com/share/d502aad5c4034784ac1061ff0882c368

tuliren · 2021-09-17T19:33:29Z

I think this is a frontend problem. The nested objects are not saved properly. @jared, FYI.

tuliren · 2021-09-30T09:00:45Z

One more user ran into this issue. I am adding this ticket to the QA milestone for now. I think more people will encounter this in the cloud.

sherifnada · 2021-11-15T06:55:48Z

@jrhizor do you have an idea about ETA for this? this seems like a pretty important one to solve

jamakase · 2021-11-15T20:16:14Z

@jrhizor frontend label was removed from this task, so I assumed it is not about frontend. Just checked it and looks like the problem itself is on server side - it returns 500 error, where we expect logs to be got.

{"message":"Internal Server Error: null","exceptionClassName":"java.lang.IllegalArgumentException","exceptionStack":["java.lang.IllegalArgumentException","\tat com.google.common.base.Preconditions.checkArgument(Preconditions.java:130)","\tat io.airbyte.config.persistence.split_secrets.JsonSecretsProcessor.copySecrets(JsonSecretsProcessor.java:100)","\tat io.airbyte.config.persistence.split_secrets.JsonSecretsProcessor.copySecrets(JsonSecretsProcessor.java:122)","\tat io.airbyte.config.persistence.split_secrets.JsonSecretsProcessor.copySecrets(JsonSecretsProcessor.java:122)","\tat io.airbyte.server.converters.ConfigurationUpdate.destination(ConfigurationUpdate.java:64)","\tat io.airbyte.server.handlers.SchedulerHandler.checkDestinationConnectionFromDestinationIdForUpdate(SchedulerHandler.java:209)","\tat io.airbyte.server.wrapped.ConfigurationApiWrapped.lambda$checkConnectionToDestinationForUpdate$43(ConfigurationApiWrapped.java:553)","\tat io.airbyte.cloud.auth.AuthHelper.execute(AuthHelper.java:101)","\tat io.airbyte.cloud.auth.AuthHelper.callForWorkspace(AuthHelper.java:78)","\tat io.airbyte.server.wrapped.ConfigurationApiWrapped.checkConnectionToDestinationForUpdate(ConfigurationApiWrapped.java:550)","\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)","\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)","\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)","\tat java.base/java.lang.reflect.Method.invoke(Method.java:564)","\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)","\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)","\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)","\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)","\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)","\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)","\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)","\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)","\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)","\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)","\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)","\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)","\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)","\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)","\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)","\tat org.eclipse.jetty.server.Server.handle(Server.java:501)","\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)","\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)","\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)","\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)","\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)","\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)","\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)","\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)","\tat java.base/java.lang.Thread.run(Thread.java:832)"],"rootCauseExceptionClassName":null,"rootCauseExceptionStack":[]}

It's not nice that we swallow this error though. I am going to add few subtasks to fix that problem.

cgardens · 2021-11-16T23:35:16Z

{"message":"Internal Server Error: null","exceptionClassName":"java.lang.IllegalArgumentException","exceptionStack":["java.lang.IllegalArgumentException",
"\tat com.google.common.base.Preconditions.checkArgument(Preconditions.java:130)",
"\tat io.airbyte.config.persistence.split_secrets.JsonSecretsProcessor.copySecrets(JsonSecretsProcessor.java:100)",
"\tat io.airbyte.config.persistence.split_secrets.JsonSecretsProcessor.copySecrets(JsonSecretsProcessor.java:122)",
"\tat io.airbyte.config.persistence.split_secrets.JsonSecretsProcessor.copySecrets(JsonSecretsProcessor.java:122)",
"\tat io.airbyte.server.converters.ConfigurationUpdate.destination(ConfigurationUpdate.java:64)",
"\tat io.airbyte.server.handlers.SchedulerHandler.checkDestinationConnectionFromDestinationIdForUpdate(SchedulerHandler.java:209)",
"\tat io.airbyte.server.wrapped.ConfigurationApiWrapped.lambda$checkConnectionToDestinationForUpdate$43(ConfigurationApiWrapped.java:553)",
"\tat io.airbyte.cloud.auth.AuthHelper.execute(AuthHelper.java:101)",
"\tat io.airbyte.cloud.auth.AuthHelper.callForWorkspace(AuthHelper.java:78)",
"\tat io.airbyte.server.wrapped.ConfigurationApiWrapped.checkConnectionToDestinationForUpdate(ConfigurationApiWrapped.java:550)",
"\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)",
"\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)",
"\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)",
"\tat java.base/java.lang.reflect.Method.invoke(Method.java:564)",
"\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)",
"\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)",
"\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)",
"\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)",
"\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)",
"\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)",
"\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)",
"\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)",
"\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)",
"\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)",
"\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)",
"\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)",
"\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)",
"\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)",
"\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)",
"\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)",
"\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)",
"\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)",
"\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)",
"\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)",
"\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)",
"\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)",
"\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)",
"\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)",
"\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)",
"\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)",
"\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)",
"\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)",
"\tat org.eclipse.jetty.server.Server.handle(Server.java:501)",
"\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)",
"\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)",
"\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)",
"\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)",
"\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)",
"\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
"\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)",
"\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)",
"\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)",
"\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)",
"\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)",
"\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)",
"\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)",
"\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)",
"\tat java.base/java.lang.Thread.run(Thread.java:832)"],"rootCauseExceptionClassName":null,"rootCauseExceptionStack":[]}

boggdan95 · 2021-11-17T23:24:02Z

I'm facing this issue with S3 source. I configurated it as parquet but when I check settings format is csv. My configuration is not saved.

Related issue in slack:
https://airbytehq.slack.com/archives/C021JANJ6TY/p1632988502023900

lmossman · 2021-11-18T17:44:21Z

@da-firebolt @boggdan95 we believe we have a fix for this issue, and we are working on testing and pushing it. Will keep you updated!

lmossman · 2021-11-18T22:29:32Z

@da-firebolt @boggdan95 we have pushed the fix for this issue, so please try updating to v0.32.3-alpha and let us know if you experience any more problems

da-firebolt added the type/bug Something isn't working label Sep 8, 2021

marcosmarxm added the priority/low Low priority label Sep 8, 2021

tuliren added the area/frontend label Sep 17, 2021

tuliren assigned jamakase Sep 17, 2021

jrhizor unassigned jamakase Sep 20, 2021

jrhizor added priority/medium Medium priority and removed priority/low Low priority labels Sep 20, 2021

tuliren added this to the Core 2021-09-29 (qa) milestone Sep 30, 2021

cgardens removed this from the Core 2021-09-04 (qa) milestone Oct 4, 2021

jrhizor added priority/high High priority and removed priority/medium Medium priority labels Oct 4, 2021

jrhizor mentioned this issue Nov 16, 2021

catch all errors when updating connector configs #8029

Closed

jrhizor added area/platform issues related to the platform and removed area/frontend labels Nov 16, 2021

cgardens assigned lmossman Nov 17, 2021

cgardens mentioned this issue Nov 17, 2021

fix issue with oneOf with sub schemas with same key but different type #8034

Merged

cgardens closed this as completed in #8034 Nov 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Saving changes not working when changing destination output format-format type from Avro to Parquet #5907

Saving changes not working when changing destination output format-format type from Avro to Parquet #5907

da-firebolt commented Sep 8, 2021 •

edited

Loading

marcosmarxm commented Sep 8, 2021 •

edited

Loading

da-firebolt commented Sep 9, 2021 •

edited

Loading

tuliren commented Sep 17, 2021

tuliren commented Sep 30, 2021

sherifnada commented Nov 15, 2021

jamakase commented Nov 15, 2021

cgardens commented Nov 16, 2021

boggdan95 commented Nov 17, 2021 •

edited

Loading

lmossman commented Nov 18, 2021

lmossman commented Nov 18, 2021

Saving changes not working when changing destination output format-format type from Avro to Parquet #5907

Saving changes not working when changing destination output format-format type from Avro to Parquet #5907

Comments

da-firebolt commented Sep 8, 2021 • edited Loading

Enviroment

Current Behavior

Expected Behavior

Logs

Steps to Reproduce

Are you willing to submit a PR?

marcosmarxm commented Sep 8, 2021 • edited Loading

da-firebolt commented Sep 9, 2021 • edited Loading

tuliren commented Sep 17, 2021

tuliren commented Sep 30, 2021

sherifnada commented Nov 15, 2021

jamakase commented Nov 15, 2021

cgardens commented Nov 16, 2021

boggdan95 commented Nov 17, 2021 • edited Loading

lmossman commented Nov 18, 2021

lmossman commented Nov 18, 2021

da-firebolt commented Sep 8, 2021 •

edited

Loading

marcosmarxm commented Sep 8, 2021 •

edited

Loading

da-firebolt commented Sep 9, 2021 •

edited

Loading

boggdan95 commented Nov 17, 2021 •

edited

Loading