Fix refresh schema when source_catalog_id is not present on database #20869

marcelopio · 2022-12-23T16:48:35Z

What

The PR #20323 introduced a bug where the source_catalog_id is empty on database you cannot use the refresh schema.
Since it is expected that the source_catalog_id can be empty, this should not happen

2022-12-23 13:47:04 �[1;31mERROR�[m i.a.s.a.ApiHelper(execute):28 - Unexpected Exception
java.util.NoSuchElementException: No value present
	at java.util.Optional.get(Optional.java:143) ~[?:?]
	at io.airbyte.server.handlers.WebBackendConnectionsHandler.webBackendGetConnection(WebBackendConnectionsHandler.java:352) ~[io.airbyte-airbyte-server-0.40.26.jar:?]
	at io.airbyte.server.apis.WebBackendApiController.lambda$webBackendGetConnection$2(WebBackendApiController.java:51) ~[io.airbyte-airbyte-server-0.40.26.jar:?]
	at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:18) ~[io.airbyte-airbyte-server-0.40.26.jar:?]
	at io.airbyte.server.apis.WebBackendApiController.webBackendGetConnection(WebBackendApiController.java:51) ~[io.airbyte-airbyte-server-0.40.26.jar:?]
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.Server.handle(Server.java:501) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

How

Added a guard to check if the object obtained with source_catalog_id is present.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

mfsiega-airbyte · 2022-12-23T17:32:42Z

On PTO so I can't really take a hard look, but:

There's a comment that says over time this should be rarely used because source_catalog_id should always be set. - do you happen to know why this isn't true in your case?
If you don't mind, a unit test that triggers this case would be super helpful @pmossman maybe you can help out here potentially. Without unit tests it's quite likely that the next person to touch this code will introduce another problem :)

mfsiega-airbyte · 2022-12-23T17:33:36Z

(Also added @alovew who IIRC has some familiarity/expertise with this bit of the codebase? If not feel free to disregard.)

marcelopio · 2022-12-23T17:37:33Z

I am still investigating why source_catalog_id is empty. I have a hunch that maybe it is because we use octavia-cli, but some connectors do have source_catalog_id even when deployed with octavia. At first I thought that was our custom connectors, but the S3 and MSSQL also have the same issue.

pmossman · 2022-12-23T17:49:55Z

I'm also on PTO today so I'll tag in @davinchia in case this is urgent

robert-put · 2022-12-23T18:53:03Z

I also ran into this issue and it affected postgres and stitch connections that were setup using the cli but not a postgres connection setup using the UI.
Downgrading to 0.40.25 fixed the issue.
Same logs message as above.
Slack thread: https://airbytehq.slack.com/archives/C021JANJ6TY/p1671746202899689

davinchia · 2022-12-23T23:41:15Z

Sorry for the bug everyone. Most of the team is out and we aren't seeing this internally so I'm going to make sure we come back to this and solve this for everyone the day after Christmas. Thanks all!

mfsiega-airbyte · 2022-12-27T16:04:15Z

Spent a bit of time thinking about this issue. This case can indeed come up if the connection is created without doing schema discovery first - I could indeed imagine that some use cases that use the APIs directly could run into issues.

I'm fine to go ahead with this - if you don't mind @marcelopio I'll add some unit tests here.

Longer-term I think there are some structural changes we can make to the APIs around schema management that will be a big improvement.

marcelopio · 2022-12-27T16:12:22Z

I'm fine to go ahead with this - if you don't mind @marcelopio I'll add some unit tests here.

Sorry I could not add those before. My baby was born! I have no problem at all with you adding them. Thanks!

murat-cetinkaya · 2023-01-03T15:10:16Z

In my case, I get this error if I try to refresh schema for the existing connections (that were created in the previous version). When I create a new connection in 0.40.26 I don't get the error when refreshing schema.
I had to stop upgrading because of this issue. I hope this solution will be available in the next release.
Thanks for the PR @marcelopio 🙏🏼

mfsiega-airbyte · 2023-01-03T15:15:56Z

For ease I opened and merged a separate PR #20928 (with a slightly different solution) - this should be addressed in the next release.

marcelopio · 2023-01-05T22:17:28Z

@mfsiega-airbyte Can I close this then?

robert-put · 2023-01-07T19:05:39Z

Just upgraded and the previous issue was resolved! Appreciate the work!

Fix refresh schema when source_catalog_id is not present on database

e572fb5

marcelopio requested a review from a team as a code owner December 23, 2022 16:48

octavia-squidington-iv added area/platform issues related to the platform area/server community labels Dec 23, 2022

marcosmarxm requested review from pmossman and mfsiega-airbyte December 23, 2022 17:07

mfsiega-airbyte requested a review from alovew December 23, 2022 17:32

marcelopio closed this Jan 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix refresh schema when source_catalog_id is not present on database #20869

Fix refresh schema when source_catalog_id is not present on database #20869

marcelopio commented Dec 23, 2022

mfsiega-airbyte commented Dec 23, 2022

mfsiega-airbyte commented Dec 23, 2022

marcelopio commented Dec 23, 2022

pmossman commented Dec 23, 2022

robert-put commented Dec 23, 2022

davinchia commented Dec 23, 2022

mfsiega-airbyte commented Dec 27, 2022

marcelopio commented Dec 27, 2022

murat-cetinkaya commented Jan 3, 2023

mfsiega-airbyte commented Jan 3, 2023

marcelopio commented Jan 5, 2023

robert-put commented Jan 7, 2023

Fix refresh schema when source_catalog_id is not present on database #20869

Fix refresh schema when source_catalog_id is not present on database #20869

Conversation

marcelopio commented Dec 23, 2022

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

mfsiega-airbyte commented Dec 23, 2022

mfsiega-airbyte commented Dec 23, 2022

marcelopio commented Dec 23, 2022

pmossman commented Dec 23, 2022

robert-put commented Dec 23, 2022

davinchia commented Dec 23, 2022

mfsiega-airbyte commented Dec 27, 2022

marcelopio commented Dec 27, 2022

murat-cetinkaya commented Jan 3, 2023

mfsiega-airbyte commented Jan 3, 2023

marcelopio commented Jan 5, 2023

robert-put commented Jan 7, 2023