413 when creating connection

[jeremy-thomas-roc]

What happened?

When creating a connection between a Postgres source and Snowflake destination, I get a 413 error. I know this is not an adapter issue, because I have configured the same type of connection in this instance without issues. The only difference between the 2 connections is that I have added a replication slot and publication to the Postgres source.

I have also edited the nginx.conf file at /etc/nginx to have this line in the http section: client_max_body_size 0;, which did not change anything, even after restarting nginx.

What did you expect to happen?

The connection to be created successfully

Abctl Version

$ abctl version
# version: v0.12.2

Docker Version

$ docker version
# Client: Docker Engine - Community
 Version:           27.1.2
 API version:       1.46
 Go version:        go1.21.13
 Git commit:        d01f264
 Built:             Mon Aug 12 11:50:58 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.1.2
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       f9522e5
  Built:            Mon Aug 12 11:50:58 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.20
  GitCommit:        8fc6bcff51318944179630522a095cc9dbf9f353
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

OS Version

# On Linux:
$ cat /etc/os-release
# PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

[marcosmarxm]

Hello @jeremy-thomas-roc can you share the complete error message/payload?
Are you able to create other connections maybe using (Source Faker / e2e test destination)?

[jeremy-thomas-roc]

The only error I get is this:

I can create other connections, I have a similar Postgres -> Snowflake connection configured already.

[marcosmarxm]

Can yo check the Network tab -> Developer Tools? It can provide a more complete response from the request.

[jeremy-thomas-roc]

There's the request/response details, it's a pretty generic 413 error for request too large.

[alenoir]

Same here with update google analytics connection and multiple propety_ids. In request payload my syncCatalog -> stream contain 1481 streams, I thinks it's because each property ids create streams (cf #42464)

[alxgrh]

I have the same issue. Error occurs if source DB contains many tables, 1123 in my case. This issue was reported many times
#4086
#18822
#22631

I can connect to the docker container via

docker exec -it airbyte-abctl-control-plane bash

But I'm not sure how to edit and reload nginx conf. I see nginx in a processes list

root@airbyte-abctl-control-plane:/# ps axu | grep nginx.conf
statd       6737  0.0  0.1 122540 12248 ?        S    Sep12   0:00 nginx: master process /usr/bin/nginx -c /etc/nginx/nginx.conf

but neither config or binary files exist on that path. It looks like it's running inside some container environment (container inside container!)

Does anyone know how to restart nginx in Airbyte container?

[alxgrh]

I've solved this problem, having reinstalled the Airbyte with docker-compose - https://docs.airbyte.com/deploying-airbyte/docker-compose

[jeremy-thomas-roc]

I've solved this problem, having reinstalled the Airbyte with docker-compose - https://docs.airbyte.com/deploying-airbyte/docker-compose

Do you mean it worked out-of-the-box, or you were then able to edit the nginx.conf to get it to work correctly?

[alxgrh]

I've solved this problem, having reinstalled the Airbyte with docker-compose - https://docs.airbyte.com/deploying-airbyte/docker-compose

Do you mean it worked out-of-the-box, or you were then able to edit the nginx.conf to get it to work correctly?

It worked out-of-the-box. I didn't do any additional steps.

[japerry911]

adding annotation to ingress-abctl namespace worked for me -

kubectl edit ingress ingress-abctl -n airbyte-abctl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig

then add under metadata (50m = 50 mb, if you need more add higher number, etc..)...

metadata:
  annotations:
     nginx.ingress.kubernetes.io/proxy-body-size: "50m" 

Should work momentarily.. Just a temporarily fix though, I don't know if this is best method, but works for me.

[bgroff]

@japerry911 your method is the correct method. Airbyte has also adjusted these limits in the close PR above.