Would it make sense to build hydra statically #374
Comments
|
Sounds good! We could probably provide an additional docker image flavor that uses this feature! |
|
According to this article golang should do that automatically? |
|
Ah, it seems to have changed with Go 1.5. Do you know how this needs to be done with Go 1.7 / 1.8 @dereulenspiegel ? |
|
The command I posted in the initial ticket is a variant of the command how I currently build most of my service under Go 1.7. It creates statically linked binaries (as long as CGo isn't used) without problems and also strips debugging symbols (saving around 30% of the binary size). |
|
That is very likely caused by the base go image. Unfortunately, the
alpine version doesn't work out of the box because we need git and
mercurial for go get. The first step would to get hydra working on
alpine. The hydra binary is (even with debugging symbols) only a few megs.
Am 13.02.2017 um 15:31 schrieb Till Klocke:
…
The command I posted in the initial ticket is a variant of the command
how I currently build most of my service under Go 1.7. It creates
statically linked binaries (as long as CGo isn't used) without
problems and also strips debugging symbols (saving around 30% of the
binary size).
I finally tried your official docker container and was a bit shocked
that clocks in at several hundert MB. I think reducing image size is
very welcome for many users.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#374 (comment)>, or
mute the thread
<https://github.com/notifications/unsubscribe-auth/ADN1evlkkKwuus7sD86VdLXCiGN7b2KPks5rcGk-gaJpZM4L09cE>.
|
|
You could also build the binary outside of docker and then just include it in an image derived from scratch. This way the image only about as big as the hydra binary. At least for many deployments this will be useful. |
|
Building it outside of docker is an option, however it would be kinda tricky with the CI toolchain right now because that works like this:
If we build it outside, it would need to be like this:
This however could be a bit error prone when a part of the CI toolchain fails and there's no easy way to restart only one piece of the chain. Maybe preparing an alpine image that supports Go is one way to go, so basically install git, mercurial and go and be done with it. It would probably reduce the size significantly anyways. Squeezing out the last megs would then be a task for the future. WDYT? |
|
I had a similar issue in the past with our own pipeline. We run a go server that connects to Hydra and is configured with the client id/secret for introspect. I implemented the following pipeline for it (in gitlab):
The final result comes out to a 6 Mb static binary. The only issue is if you have to contact https urls then you also need to include ca-certificates in the Docker image. stages:
- build
- test
- docker_build
- deploy
Verify Build:
artifacts:
paths:
- assets/
- vendor/
- gatekeeper
untracked: true
except:
- tags
image: "golang:1.7.4"
script:
- go get github.com/Masterminds/glide
- glide install -v
- CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo --ldflags="-s" -o gatekeeper
stage: build
tags:
- golang
Unit Test:
image: "golang:1.6.0"
script:
- go test -race -cover $(go list ./... | grep -v vendor)
stage: test
tags:
- golang
API Test:
image: "uadevnet/docker-newman:latest"
script:
- sed -i -e "s/{{USERNAME}}/$USERNAME/g" assets/postman/tests.json
- sed -i -e "s/{{PASSWORD}}/$PASSWORD/g" assets/postman/tests.json
- mkdir -p /etc/ssl/certs
- cp assets/ca-certificates.crt /etc/ssl/certs/
- ./gatekeeper &
- SERVER_PID=$!
- sleep 10
- /usr/bin/newman --collection="assets/postman/tests.json" --environment="assets/postman/environment.json"
- kill $SERVER_PID
stage: test
tags:
- golang
Build Image:
image: "docker:latest"
script:
- docker build -t ${CI_PROJECT_NAME} . 2>&1 | tee build.log
- ID=$(tail -1 build.log | awk '{print $3;}')
- docker login --username="" --password="" $docker_repo
- docker tag $ID "$docker_repo/${CI_PROJECT_NAME}:latest"
- docker push "$docker_repo/${CI_PROJECT_NAME}:latest"
services:
- "docker:dind"
only:
- master
stage: docker_build
Deploy to SUT:
variables:
SPREAD_DIR: $CI_PROJECT_DIR/assets/kubernetes
KUBECFG_SERVER: "https://kubernetes_server"
KUBECFG_CERTIFICATE_AUTHORITY: /certs/kubernetes.ca.crt
KUBECFG_CLIENT_CERTIFICATE: /certs/kubernetes.kubecfg.crt
KUBECFG_CLIENT_KEY: /certs/kubernetes.kubecfg.key
KUBECFG_TOKEN: $token
stage: deploy
image: redspreadapps/gitlabci
only:
- master
script:
- null-script
tags:
- kubedeployFROM scratch
MAINTAINER Michael Golfi <michael.m.golfi@gmail.com>
ADD gatekeeper /
# curl -o ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt
ADD assets/ca-certificates.crt /etc/ssl/certs/
CMD [ "/gatekeeper" ]
EXPOSE 8080 |
|
Awesome, thank you for sharing this! |
|
No problem, glad I could help! |
|
Also thanks from me. |
|
I didn't know the image is actually 1.2GB large (that's a lot, wow). I was able to get alpine working, bringing it down to: ... still a lot but hey, 35% less is a start! |
|
It seems that most of the size comes from the image itself: as well as ~80mb worth of |
|
Sorry, meant to respond much sooner. I think you could pretty easily switch to a scratch image, is there a reason why you'd prefer alpine over scratch? Speaking of scratch, I created a blank Ubuntu 16.04 VM, installed golang 1.7.4, installed docker and ran the following commands: go get github.com/ory/hydra
go get github.com/Masterminds/glide
cd $GOPATH/src/github.com/ory/hydra
glide install
CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo --ldflags="-s" -o hydra
mkdir ~/build
mv hydra ~/build
cd ~/build
wget https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt
# Create following Dockerfile
sudo docker build -t michaelgolfi/hydra .FROM scratch
ADD hydra /hydra
ADD ca-bundle.crt /etc/ssl/certs/ca-certificates.crt
CMD [ "/hydra", "host" ]
EXPOSE 4444ubuntu@default:~/build$ ls -l
total 11060
-rw-rw-r-- 1 ubuntu ubuntu 261889 Mar 12 19:00 ca-bundle.crt
-rw-rw-r-- 1 ubuntu ubuntu 123 Mar 12 19:01 Dockerfile
-rwxrwxr-x 1 ubuntu ubuntu 11057568 Mar 12 18:57 hydra
ubuntu@default:~/build$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
michaelgolfi/hydra latest f6cd972dc84a 8 minutes ago 11.3 MB
ubuntu@default:~/build$ sudo docker run -it michaelgolfi/hydra
INFO[0000] DATABASE_URL not set, connecting to ephermal in-memory database.
WARN[0000] Expected system secret to be at least 32 characters long, got 0 characters.
INFO[0000] Generating a random system secret...
INFO[0000] Generated system secret: 2/(-!5t?CExHdPeFGkZfLqfUD_Px).n=
WARN[0000] WARNING: DO NOT generate system secrets in production. The secret will be leaked to the logs.
INFO[0000] Key pair for signing hydra.openid.id-token is missing. Creating new one.
INFO[0003] Key pair for signing hydra.consent.response is missing. Creating new one.
INFO[0007] Key pair for signing hydra.consent.challenge is missing. Creating new one.
WARN[0010] No clients were found. Creating a temporary root client...
INFO[0010] Temporary root client created.
INFO[0010] client_id: 18779eff-3873-4576-8af8-f6613c8d84be
INFO[0010] client_secret: r)TIer/zKs._cWPH
WARN[0010] WARNING: YOU MUST delete this client once in production, as credentials may have been leaked in your logfiles.
WARN[0010] No TLS Key / Certificate for HTTPS found. Generating self-signed certificate.
INFO[0010] Setting up http server on :4444Off first glance it seems to work correctly as intended... I pushed the image to michaelgolfi/hydra if you want to test it out further. |
|
I brought it down to 700megs for now. The largest layer is probably the git package. Tackled in #396 |
|
@michael-golfi thanks for this, yes scratch makes a lot of sense when building statically. I need to refactor the build pipeline for this however and move the docker push command to travis. |
|
@arekkas If you run into any trouble let me know. I think the steps above from our pipeline should essentially do the trick though I'm not familiar with Travis and that is written for Gitlab. |
aeneasr
modified the milestones:
unplanned,
0.8.0: Resource Owner Password Credentials Grant
|
This took longer than it should have, but they say: better late than never! |
Currently I am building all my GoLang based services as a statically linked binary since they don't use CGo. I noticed that hydra isn't using CGo either. Building a statically linked binary allows to put hydra in docker container without any userland (so basically a scratch container). I think this is a quick and easy improvement.
For reference I tested building hydra with the following command
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -ldflags="-s -w" -o hydraThe text was updated successfully, but these errors were encountered: