Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add ability to start container tests in debug mode and attach a debugger #16887

Merged
merged 10 commits into from
Apr 18, 2023
Merged

add ability to start container tests in debug mode and attach a debugger #16887

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
464c4be
add ability to start container tests in debug mode and attach a debug…
dhiaayachi Apr 5, 2023
9bd2582
Merge branch 'main' into container-test-dbg
dhiaayachi Apr 5, 2023
f85924c
Merge branch 'main' into container-test-dbg
dhiaayachi Apr 11, 2023
f9aa773
add a debug message with the debug port
dhiaayachi Apr 13, 2023
6c9ad62
use pod to get the right port
dhiaayachi Apr 13, 2023
94358e9
fix image used in basic test
dhiaayachi Apr 13, 2023
c67dac8
add more data to identify which container to debug.
dhiaayachi Apr 14, 2023
aa4d214
fix comment
dhiaayachi Apr 17, 2023
8fb470d
rename debugUri to debugURI
dhiaayachi Apr 17, 2023
be7569b
Merge branch 'main' into container-test-dbg
dhiaayachi Apr 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions GNUmakefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,19 @@ dev-build:
rm -f ./bin/consul
cp ${MAIN_GOPATH}/bin/consul ./bin/consul


dev-docker-dbg: dev-docker linux dev-build
@echo "Pulling consul container image - $(CONSUL_IMAGE_VERSION)"
@docker pull consul:$(CONSUL_IMAGE_VERSION) >/dev/null
@echo "Building Consul Development container - $(CONSUL_DEV_IMAGE)"
@# 'consul:local' tag is needed to run the integration tests
dhiaayachi marked this conversation as resolved.
Show resolved Hide resolved
@# 'consul-dev:latest' is needed by older workflows
@docker buildx use default && docker buildx build -t 'consul-dbg:local' -t '$(CONSUL_DEV_IMAGE)' \
--platform linux/$(GOARCH) \
--build-arg CONSUL_IMAGE_VERSION=$(CONSUL_IMAGE_VERSION) \
--load \
-f $(CURDIR)/build-support/docker/Consul-Dev-Dbg.dockerfile $(CURDIR)/pkg/bin/

dev-docker: linux dev-build
@echo "Pulling consul container image - $(CONSUL_IMAGE_VERSION)"
@docker pull consul:$(CONSUL_IMAGE_VERSION) >/dev/null
Expand Down
10 changes: 10 additions & 0 deletions build-support/docker/Consul-Dev-Dbg.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
FROM consul:local
EXPOSE 4000

COPY --from=golang:1.20-alpine /usr/local/go/ /usr/local/go/

ENV PATH="/usr/local/go/bin:${PATH}"

RUN CGO_ENABLED=0 go install -ldflags "-s -w -extldflags '-static'" github.com/go-delve/delve/cmd/dlv@latest

CMD [ "/root/go/bin/dlv", "exec", "/bin/consul", "--listen=:4000", "--headless=true", "", "--accept-multiclient", "--continue", "--api-version=2", "--", "agent", "--advertise=0.0.0.0"]
8 changes: 6 additions & 2 deletions test/integration/consul-container/libs/cluster/builder.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ func NewBuildContext(t *testing.T, opts BuildOptions) *BuildContext {
}

if ctx.consulImageName == "" {
ctx.consulImageName = utils.TargetImageName
ctx.consulImageName = utils.GetTargetImageName()
}
if ctx.consulVersion == "" {
ctx.consulVersion = utils.TargetVersion
Expand Down Expand Up @@ -306,11 +306,15 @@ func (b *Builder) ToAgentConfig(t *testing.T) *Config {
confCopy, err := b.conf.Clone()
require.NoError(t, err)

cmd := []string{"agent"}
if utils.Debug {
cmd = []string{"/root/go/bin/dlv", "exec", "/bin/consul", "--listen=:4000", "--headless=true", "", "--accept-multiclient", "--continue", "--api-version=2", "--", "agent", "--config-file=/consul/config/config.json"}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we always expect the --config-file arg to be /consul/config/config.json? I'm surprised we didn't need this in the args before?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes at least for the base image. I think without dlv it's passed through env vars which dlv don't pass along to the sub process.

}
return &Config{
JSON: string(out),
ConfigBuilder: confCopy,

Cmd: []string{"agent"},
Cmd: cmd,

Image: b.context.consulImageName,
Version: b.context.consulVersion,
Expand Down
18 changes: 18 additions & 0 deletions test/integration/consul-container/libs/cluster/container.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ const disableRYUKEnv = "TESTCONTAINERS_RYUK_DISABLED"
const MaxEnvoyOnNode = 10 // the max number of Envoy sidecar can run along with the agent, base is 19000
const ServiceUpstreamLocalBindPort = 5000 // local bind Port of service's upstream
const ServiceUpstreamLocalBindPort2 = 5001 // local bind Port of service's upstream, for services with 2 upstreams
const debugPort = "4000/tcp"

// consulContainerNode implements the Agent interface by running a Consul agent
// in a container.
Expand Down Expand Up @@ -156,6 +157,20 @@ func NewConsulContainer(ctx context.Context, config Config, cluster *Cluster, po

info AgentInfo
)
if utils.Debug {
for i := 0; i < 10; i++ {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we can replace this with retry-go, which is imported to the integ test:

consul/test/integration/consul-container/libs/cluster/cluster.go

Lines 141 to 157 in 44b3924

if err := goretry.Do(
func() (err error) {
n, err = NewConsulContainer(
context.Background(),
conf,
c,
ports...,
)
return err
},
goretry.Delay(10*time.Second),
goretry.RetryIf(func(err error) bool {
return strings.Contains(err.Error(), "port not found")
}),
); err != nil {
return fmt.Errorf("container %d creating: %s", idx, err)
}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the review @huikang ! I changed that and also added few changes to print some data about the node when printing the debug port to help identify which container we would like to debug in a multi Consul test.

uri, err := podContainer.PortEndpoint(ctx, "4000", "tcp")
if err != nil {
time.Sleep(500 * time.Millisecond)
continue
}
fmt.Printf("\nDebug endpoint:: %s\n\n", uri)
break
}
if err != nil {
return nil, err
}
}
if httpPort > 0 {
for i := 0; i < 10; i++ {
uri, err := podContainer.PortEndpoint(ctx, "8500", "http")
Expand Down Expand Up @@ -565,6 +580,9 @@ func newContainerRequest(config Config, opts containerOpts, ports ...int) (podRe
for _, port := range ports {
pod.ExposedPorts = append(pod.ExposedPorts, fmt.Sprintf("%d/tcp", port))
}
if utils.Debug {
pod.ExposedPorts = append(pod.ExposedPorts, debugPort)
}

// For handshakes like auto-encrypt, it can take 10's of seconds for the agent to become "ready".
// If we only wait until the log stream starts, subsequent commands to agents will fail.
Expand Down
17 changes: 14 additions & 3 deletions test/integration/consul-container/libs/utils/version.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,15 @@ import (
)

var (
TargetImageName string
targetImageName string
TargetVersion string

LatestImageName string
LatestVersion string

FollowLog bool
FollowLog bool

Debug bool
Version_1_14, _ = version.NewVersion("1.14")
)

Expand All @@ -27,13 +29,22 @@ const (
)

func init() {
flag.StringVar(&TargetImageName, "target-image", defaultImageName, "docker image name to be used under test (Default: "+defaultImageName+")")
flag.BoolVar(&Debug, "debug", false, "run consul with dlv to enable live debugging")
flag.StringVar(&targetImageName, "target-image", defaultImageName, "docker image name to be used under test (Default: "+defaultImageName+")")
flag.StringVar(&TargetVersion, "target-version", "local", "docker image version to be used as UUT (unit under test)")

flag.StringVar(&LatestImageName, "latest-image", defaultImageName, "docker image name to be used under test (Default: "+defaultImageName+")")
flag.StringVar(&LatestVersion, "latest-version", "latest", "docker image to be used as latest")

flag.BoolVar(&FollowLog, "follow-log", true, "follow container log in output (Default: true)")

}

func GetTargetImageName() string {
if Debug {
return targetImageName + "-dbg"
}
return targetImageName
}

func DockerImage(image, version string) string {
Expand Down
4 changes: 2 additions & 2 deletions test/integration/consul-container/test/snapshot/snapshot_restore_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ func testSnapShotRestoreForLogStore(t *testing.T, logStore libcluster.LogStore)
NumClients: 0,
BuildOpts: &libcluster.BuildOptions{
Datacenter: "dc1",
ConsulImageName: utils.TargetImageName,
ConsulImageName: utils.GetTargetImageName(),
ConsulVersion: utils.TargetVersion,
LogStore: logStore,
},
Expand Down Expand Up @@ -67,7 +67,7 @@ func testSnapShotRestoreForLogStore(t *testing.T, logStore libcluster.LogStore)
NumClients: 0,
BuildOpts: &libcluster.BuildOptions{
Datacenter: "dc1",
ConsulImageName: utils.TargetImageName,
ConsulImageName: utils.GetTargetImageName(),
ConsulVersion: utils.TargetVersion,
LogStore: logStore,
},
Expand Down
6 changes: 3 additions & 3 deletions test/integration/consul-container/test/upgrade/basic_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ func TestBasic(t *testing.T) {
t.Parallel()

configCtx := libcluster.NewBuildContext(t, libcluster.BuildOptions{
ConsulImageName: utils.TargetImageName,
ConsulVersion: utils.LatestVersion,
ConsulImageName: utils.GetTargetImageName(),
ConsulVersion: utils.TargetVersion,
})

const numServers = 1
Expand All @@ -29,7 +29,7 @@ func TestBasic(t *testing.T) {
Bootstrap(numServers).
ToAgentConfig(t)
t.Logf("Cluster config:\n%s", serverConf.JSON)
require.Equal(t, utils.LatestVersion, serverConf.Version) // TODO: remove
require.Equal(t, utils.TargetVersion, serverConf.Version) // TODO: remove

cluster, err := libcluster.NewN(t, *serverConf, numServers)
require.NoError(t, err)
Expand Down
2 changes: 1 addition & 1 deletion test/integration/consul-container/test/upgrade/fullstopupgrade_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ func TestStandardUpgradeToTarget_fromLatest(t *testing.T) {

run := func(t *testing.T, tc testcase) {
configCtx := libcluster.NewBuildContext(t, libcluster.BuildOptions{
ConsulImageName: utils.TargetImageName,
ConsulImageName: utils.GetTargetImageName(),
ConsulVersion: tc.oldVersion,
})

Expand Down
2 changes: 1 addition & 1 deletion test/integration/consul-container/test/upgrade/healthcheck_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ func testMixedServersGAClient(t *testing.T, majorityIsTarget bool) {
ConsulVersion: utils.LatestVersion,
}
targetOpts = libcluster.BuildOptions{
ConsulImageName: utils.TargetImageName,
ConsulImageName: utils.GetTargetImageName(),
ConsulVersion: utils.TargetVersion,
}

Expand Down