MDB transaction timeout

[armon]

Bug report of a situation triggering an MDB transaction timeout:

https://gist.github.com/cnf/06f79f005e47a0c657c3

[cnf]

This was in docker, running progrium/consul, FYI

[cnf]

This happened twice again, today. I appended the logs to the same gist.

[armon]

Thanks for the extra logs! Odd that you're getting this so often...

[armon]

Having a hard time reproducing this. Can you provide more details on your setup? What disks are you using? Docker flags? etc. Anything that might be useful. A repro case would be the best!

[cnf]

consul config

{
    "datacenter": "seshat",
    "data_dir": "/data",
    "ui_dir": "/ui",
    "node_name": "consul-seshat",
    "domain": "consul",
    "server": true,
    "client_addr": "0.0.0.0",
    "advertise_addr": "172.18.42.1",
    "encrypt": "<redact>",
    "dns_confif": {
    "service_ttl": 1,
    "node_ttl": 1
    },
    "ports": {
        "dns": 53
    },
    "recursor": "8.8.8.8",
    "bootstrap_expect": 1,
    "bootstrap": false,
    "enable_syslog": true,
    "log_level": "warn",
    "statsd_addr": "hekad.node.consul:8125"
}

cat /etc/lsb-release

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"

docker version

Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.3.1
Git commit (client): fa7b24f
OS/Arch (client): linux/amd64
Server version: 1.2.0
Server API version: 1.14
Go version (server): go1.3.1
Git commit (server): fa7b24f

docker info

Containers: 14
Images: 202
Storage Driver: btrfs
Execution Driver: native-0.2
Kernel Version: 3.13.0-36-generic
Operating System: Ubuntu 14.04.1 LTS
Username: <red>
Registry: [https://index.docker.io/v1/]

docker flags

/usr/bin/docker -d -H unix:///var/run/docker.sock --bip 172.18.42.1/16 --dns 172.18.42.1 --dns-search service.consul

docker ps|grep consul

1e198fe7a511 progrium/consul:latest "/bin/start" 4 days ago Up 4 days <port mappings> consul

docker run

docker run \
  -d \
  -p 172.18.42.1:8300:8300 \
  -p 172.18.42.1:8301:8301 \
  -p 172.18.42.1:8301:8301/udp \
  -p 172.18.42.1:8302:8302 \
  -p 172.18.42.1:8302:8302/udp \
  -p 172.18.42.1:8400:8400 \
  -p 172.18.42.1:8500:8500 \
  -p 172.18.42.1:53:53/udp \
  -restart always
  --volume /dev/log:/dev/log \
  --volume /etc/docker/consul:/config \
  --volumes-from consul-data \
  --name consul \
  --hostname consul-seshat \
  progrium/consul

Info

This is running on a VMWare ESXi VM, with regular data stores.