container build via TCP gives error Content-Type: application/json is not supported. #11012

mattymo · 2021-07-21T18:14:10Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I am trying to build using docker-maven-plugin pointing at podman running in service mode listening on a TCP port. It fails with a content type error. I thought this was already fixed based on the age of the comments in this other issue: fabric8io/docker-maven-plugin#1330

Steps to reproduce the issue:

Run podman in service mode
Run a docker build from another host, specifying DOCKER_HOST pointing to podman

Describe the results you received:
Relevant output from podman in debug mode:

DEBU[0120] Applying tar in /var/lib/containers/storage/overlay/2775bac72cec67a28cdde119e557f74643e82304a6e0148db98d3a9c662c28ad/diff 
DEBU[0123] setting image creation date to 2021-04-08 07:22:24.324057841 +0000 UTC 
DEBU[0123] created new image ID "b14538981bc5d275204434ca07abafa3aab421f7be6a71c6727dd845d0c636c6" 
DEBU[0123] set names of image "b14538981bc5d275204434ca07abafa3aab421f7be6a71c6727dd845d0c636c6" to [docker-hub.mycompany/centos7-with-security-fixes:latest] 
DEBU[0123] saved image metadata "{\"signatures-sizes\":{\"sha256:358470d32fd9c6ec3003ff2472d25feed2d4b3d5a83dcb87c289eb314aef047d\":[]}}" 
DEBU[0123] Pulled candidate docker-hub.mycompany/centos7-with-security-fixes:latest successfully 
DEBU[0123] Looking up image "docker-hub.mycompany/centos7-with-security-fixes:latest" in local containers storage 
DEBU[0123] Trying "docker-hub.mycompany/centos7-with-security-fixes:latest" ... 
DEBU[0123] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@b14538981bc5d275204434ca07abafa3aab421f7be6a71c6727dd845d0c636c6" 
DEBU[0123] Found image "docker-hub.mycompany/centos7-with-security-fixes:latest" as "docker-hub.mycompany/centos7-with-security-fixes:latest" in local containers storage 
DEBU[0123] APIHandler(1fce8af2-ef3e-4826-9063-dea384217f64) -- POST /v1.18/images/create?fromImage=docker-hub.mycompany%2Fcentos7-with-security-fixes&tag=latest END 
DEBU[0123] IdleTracker 0xc0005f6000:idle 1m+0h/1t connection(s) 
DEBU[0123] IdleTracker 0xc0005f6000:active 1m+0h/1t connection(s) 
INFO[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- GET /v1.18/images/internalrepo.mycompany%2Fprj-db%3A21.0.0.0-dev-8123c6c/json BEGIN 
DEBU[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- Header: Accept-Encoding=[gzip,deflate] 
DEBU[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- Header: Accept=[*/*] 
DEBU[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- Header: Content-Type=[application/json] 
DEBU[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- Header: Connection=[Keep-Alive] 
DEBU[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- Header: User-Agent=[Apache-HttpClient/4.5.5 (Java/11.0.8)] 
DEBU[0123] Looking up image "internalrepo.mycompany/prj-db:21.0.0.0-dev-8123c6c" in local containers storage 
DEBU[0123] Trying "internalrepo.mycompany/prj-db:21.0.0.0-dev-8123c6c" ... 
DEBU[0123] Trying "internalrepo.mycompany/prj-db:21.0.0.0-dev-8123c6c" ... 
DEBU[0123] Trying "internalrepo.mycompany/prj-db:21.0.0.0-dev-8123c6c" ... 
INFO[0123] Request Failed(Not Found): failed to find image internalrepo.mycompany/prj-db:21.0.0.0-dev-8123c6c: internalrepo.mycompany/prj-db:21.0.0.0-dev-8123c6c: No such image 
DEBU[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- GET /v1.18/images/internalrepo.mycompany%2Fprj-db%3A21.0.0.0-dev-8123c6c/json END 
DEBU[0123] IdleTracker 0xc0005f6000:idle 1m+0h/1t connection(s) 
DEBU[0124] IdleTracker 0xc0005f6000:active 1m+0h/1t connection(s) 
INFO[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- POST /v1.18/build?dockerfile=Dockerfile&forcerm=1&nocache=0&t=internalrepo.mycompany%2Fprj-db%3A21.0.0.0-dev-8123c6c BEGIN 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- Header: Content-Type=[application/json] 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- Header: Content-Length=[2539520] 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- Header: Connection=[Keep-Alive] 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- Header: User-Agent=[Apache-HttpClient/4.5.5 (Java/11.0.8)] 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- Header: Accept-Encoding=[gzip,deflate] 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- Header: Accept=[*/*] 
INFO[0124] Request Failed(Bad Request): failed to parse query parameter 'Content-Type': "application/json": Content-Type: application/json is not supported. Should be "application/x-tar" 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- POST /v1.18/build?dockerfile=Dockerfile&forcerm=1&nocache=0&t=internalrepo.mycompany%2Fprj-db%3A21.0.0.0-dev-8123c6c END

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.2.2
API Version:  3.2.2
Go Version:   go1.16.4
Built:        Fri Jun 25 23:49:19 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.21.0
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - memory
  - devices
  - freezer
  - net_cls
  - blkio
  - perf_event
  - hugetlb
  - pids
  - net_prio
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.27-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 40
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: file
  hostname: celeste.int.zone
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 3.10.0-1127.10.1.el7.x86_64
  linkmode: dynamic
  memFree: 221974728704
  memTotal: 270205313024
  ociRuntime:
    name: crun
    package: crun-0.20.1-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.20.1
      commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 2195h 11m 16.53s (Approximately 91.46 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.4
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.4
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.2.2
  Built: 1624664959
  BuiltTime: Fri Jun 25 23:49:19 2021
  GitCommit: ""
  GoVersion: go1.16.4
  OsArch: linux/amd64
  Version: 3.2.2

Package info (e.g. output of rpm -q podman or apt list podman):

v3.2.2 (docker image)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

The text was updated successfully, but these errors were encountered:

mheon · 2021-07-21T20:26:41Z

@jwhonce PTAL, especially at the linked Maven plugin issue - it's saying that we don't send Version headers on all requests. I thought I remember us fixing that - but it seems like this is still happening on the latest code?

mattymo · 2021-07-27T06:42:08Z

Is there any update here on this issue?

mheon · 2021-07-30T13:28:08Z

@jwhonce PTAL

jwhonce · 2021-07-30T16:06:42Z

@mattymo @mheon This is as designed from the docker API build documentation, Content-Type has one valid value application/x-tar.

From https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type :

In requests, (such as POST or PUT), the client tells the server what type of data is actually sent.

We validate that header. After reviewing the docker code, they do not. So you may use any value for that header with docker, but only a tar file will be supported hence our validation.

fabric8io/docker-maven-plugin comment asks for a PR to change their behavior.

I'm going to leave this issue open for discussion. /cc @baude @rhatdan

jwhonce · 2021-07-30T16:23:31Z

@mheon As of 87a78c0, the API-Version and Libpod-API-Version headers are written on every response before the endpoint handler code is run. fabric8io/docker-maven-plugin#1330 (comment) pre-dates this change in our code.

Endpoint /build logs an info entry when a client uses the wrong Content-Type for build payload. Given Content-Type is ignored and assumed to be "application/x-tar". Endpoint /libpod/build will fail unless "application/x-tar" or "application/tar" is given for Content-Type. "application/tar" will be logged as an info entry. Fixes containers#11012 Signed-off-by: Jhon Honce <jhonce@redhat.com>

mattymo · 2021-08-23T09:52:52Z

@mheon @jwhonce Please reopen.
I tested on the latest podman master (commit94886d4abad7787cb920614ed26955ca36b13846) and it is still failing on /build endpoint:

docker-maven-plugin error:

12:41:25 [el7] Caused by: io.fabric8.maven.docker.access.hc.http.HttpRequestException: {"cause":"Content-Type: application/json is not supported. Should be \"application/x-tar\"","message":"failed to parse query parameter 'Content-Type': \"application/json\": Content-Type: application/json is not supported. Should be \"application/x-tar\"","response":400} (Bad Request: 400)

podman logs in debug mode:

time="2021-08-23T09:50:59Z" level=info msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- GET /v1.40/images/myimage/json BEGIN"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- Header: Accept=[*/*]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- Header: Content-Type=[application/json]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- Header: Connection=[Keep-Alive]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- Header: User-Agent=[Apache-HttpClient/4.5.5 (Java/11.0.8)]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- Header: Accept-Encoding=[gzip,deflate]"
time="2021-08-23T09:50:59Z" level=debug msg="Looking up image \"myimage\" in local containers storage"
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"myimage\" ..."
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"localhost/myimage:latest\" ..."
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"registry.fedoraproject.org/myimage:latest\" ..."
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"registry.access.redhat.com/myimage:latest\" ..."
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"docker.io/library/myimage:latest\" ..."
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"quay.io/myimage:latest\" ..."
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"docker.io/library/myimage:latest\" ..."
time="2021-08-23T09:50:59Z" level=info msg="Request Failed(Not Found): failed to find image myimage: myimage: No such image"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- GET /v1.40/images/myimage/json END"
time="2021-08-23T09:50:59Z" level=debug msg="IdleTracker 0xc000748000:idle 1m+0h/2t connection(s)"
time="2021-08-23T09:50:59Z" level=debug msg="IdleTracker 0xc000748000:active 1m+0h/2t connection(s)"
time="2021-08-23T09:50:59Z" level=info msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- POST /v1.40/build?dockerfile=Dockerfile&forcerm=1&nocache=0&t=myimage BEGIN"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- Header: Accept=[*/*]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- Header: Content-Type=[application/json]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- Header: Content-Length=[10240]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- Header: Connection=[Keep-Alive]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- Header: User-Agent=[Apache-HttpClient/4.5.5 (Java/11.0.8)]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- Header: Accept-Encoding=[gzip,deflate]"
time="2021-08-23T09:50:59Z" level=info msg="Request Failed(Bad Request): failed to parse query parameter 'Content-Type': \"application/json\": Content-Type: application/json is not supported. Should be \"application/x-tar\""
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- POST /v1.40/build?dockerfile=Dockerfile&forcerm=1&nocache=0&t=myimage END"

mheon · 2021-08-23T13:08:11Z

What Podman version? This should definitely be fixed in 3.3.

mattymo · 2021-08-23T17:12:11Z

@mheon
quay.io/podman/stable@sha256:bc6545c7b3e0d9ab06aa1c7b3303211c15fae50738673834a6f20c5cd1b61814

"Labels": {
    "license": "MIT",
    "name": "fedora",
    "org.opencontainers.image.created": "2021-08-23T08:47:34+00:00",
    "org.opencontainers.image.revision": "2de56a5f41473e8f759972ec568089e9cddc0f98",
    "org.opencontainers.image.source": "https://github.com/containers/podman.git",
    "org.opencontainers.image.version": "3.2.3",
    "vendor": "Fedora Project",
    "version": "34"
}

I don't see any 3.3 builds on quay.io. Is the release coming soon?

mheon · 2021-08-23T17:58:48Z

The release is complete. @TomSweeneyRedHat When will images be rebuilt?

TomSweeneyRedHat · 2021-08-24T01:04:19Z

@mattymo the Podman v3.3.0 (or any new version) lands on quay.io after it gets set to stable on bodhi. After being set to stable, it sometimes can take up to an additional day for the CI system to note that and build the image.

I just checked, and it looks like the Fedora 34 and Fedora 35 variants have gone to stable, but 33 and 32 have not yet. @cevich does the new CI job check Fedora 33? I thought that had been bumped to 34....

cevich · 2021-08-24T20:12:47Z

The multi-arch image build log is here: https://github.com/containers/podman/actions/workflows/multi-arch-build.yaml

@TomSweeneyRedHat the Dockerfile specifies 'latest' Fedora so I think that's F34, no?

Hmm, yes it seems it is pushing v3.2.3 for stable. I just looked and it doesn't appear to have reached the stable repos. yet:

$ podman run -it --rm registry.fedoraproject.org/fedora:latest
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob ecfb9899f4ce done
Copying config 37e5619f4a done
Writing manifest to image destination
Storing signatures
[root@087dc3f5ca63 /]# dnf list available podman
Fedora 34 - x86_64                                     6.2 MB/s |  74 MB     00:12
Fedora 34 openh264 (From Cisco) - x86_64               6.0 kB/s | 2.5 kB     00:00
Fedora Modular 34 - x86_64                             2.5 MB/s | 4.9 MB     00:01
Fedora 34 - x86_64 - Updates                            16 MB/s |  25 MB     00:01
Fedora Modular 34 - x86_64 - Updates                   3.7 MB/s | 4.6 MB     00:01
Last metadata expiration check: 0:00:01 ago on Tue Aug 24 20:10:46 2021.
Available Packages
podman.x86_64                          3:3.2.3-2.fc34                           updates
[root@087dc3f5ca63 /]#

TomSweeneyRedHat · 2021-08-24T20:19:52Z

@cevich I just checked the variant of Fedora used in the build, and it's F34. I just double-checked bodhi and Podman v3.3.0 is stable for F35 and F36. For F33 and F34, they're both still in testing. Apparently, my eyeballs failed me last night when I checked bodhi. So this is working as expected.

cevich · 2021-08-25T11:51:24Z

Ahh, yes that makes sense given F35 isn't released yet 😁

openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 21, 2021

jwhonce added kind/feature Categorizes issue or PR as related to a new feature. remote Problem is in podman-remote and removed kind/bug Categorizes issue or PR as related to a bug. labels Jul 30, 2021

jwhonce self-assigned this Jul 30, 2021

jwhonce mentioned this issue Aug 9, 2021

For compatibility, ignore Content-Type #11167

Merged

openshift-ci bot closed this as completed in #11167 Aug 12, 2021

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

container build via TCP gives error Content-Type: application/json is not supported. #11012

container build via TCP gives error Content-Type: application/json is not supported. #11012

mattymo commented Jul 21, 2021

mheon commented Jul 21, 2021

mattymo commented Jul 27, 2021

mheon commented Jul 30, 2021

jwhonce commented Jul 30, 2021

jwhonce commented Jul 30, 2021

mattymo commented Aug 23, 2021 •

edited

Loading

mheon commented Aug 23, 2021

mattymo commented Aug 23, 2021

mheon commented Aug 23, 2021

TomSweeneyRedHat commented Aug 24, 2021

cevich commented Aug 24, 2021

TomSweeneyRedHat commented Aug 24, 2021

cevich commented Aug 25, 2021

container build via TCP gives error Content-Type: application/json is not supported. #11012

container build via TCP gives error Content-Type: application/json is not supported. #11012

Comments

mattymo commented Jul 21, 2021

mheon commented Jul 21, 2021

mattymo commented Jul 27, 2021

mheon commented Jul 30, 2021

jwhonce commented Jul 30, 2021

jwhonce commented Jul 30, 2021

mattymo commented Aug 23, 2021 • edited Loading

mheon commented Aug 23, 2021

mattymo commented Aug 23, 2021

mheon commented Aug 23, 2021

TomSweeneyRedHat commented Aug 24, 2021

cevich commented Aug 24, 2021

TomSweeneyRedHat commented Aug 24, 2021

cevich commented Aug 25, 2021

mattymo commented Aug 23, 2021 •

edited

Loading