Create a separate CI job for race detection

[gbalint]

We want to keep Swarm's code race free. For that we want to run race detection at least every day on Swarm's code. Go's race detection is nice and neat, but it can increase the test run time significantly (~10x). => We need a separate job for go test -race.

Note: this assumes the code base is race free so we don't just spam people with an always failing job.

Questions:

• Could we use Travis for the task?
• How long does it take to run only Swarm tests with race detection and without?

Note: If the run time is sufficiently low we might want to run race detection for every commit.

Later we might want to run long running tests and benchmarks with the same methods.
(benchmarks: How to keep the tests self-checking?)

[frncmx]

On Dell XPS 13 9350 (Intel i7-6560U CPU @ 2.20GHz, 16GB RAM)

Without Race Detection

time go test -count 1 github.com/ethereum/go-ethereum/swarm...
[...] PASS
real	1m9.517s
user	2m6.279s
sys	0m16.929s

Note: 40% of the time is spent with github.com/ethereum/go-ethereum/swarm/network/stream (49.277s)

With Race Detection

time go test -count 1 -race github.com/ethereum/go-ethereum/swarm...
[...] FAIL
real	11m6.384s
user	20m50.035s
sys	2m3.987s

[frncmx]

By adding the following code snippet to our .travis.yml it's possible to have a separate job to run race detection on only Swarm code base, as a daily Cron job.

Note: A Cron job has to be manually added on Travis UI in the settings menu.

    - name: Race Detector for Swarm # helps the job to stand out
      if: type = cron AND branch = master AND repo = ethersphere/go-ethereum
      os: linux # this is actually the default, but every job mentions
      dist: trusty # this is actually the default, but every job mentions
      go: 1.11.x
      git:
        submodules: false # avoid cloning ethereum/tests
      script: go test -v -race ./swarm...

However the job hits Travis' memory limitations. As the race detector can increase the memory consumption by 10x.

Questions

• Would it be possible for us to fit into 7.5 gigs of memory? Travis env specs
• Is the mem cap/job? => If yes, splitting the race detection into a matrix jobs can help. Otherwise using stages is possible, but I think that would come with a huge change on .travis.yml.

[frncmx]

TestOverlaySim tries to start 16 nodes. Is that too much for Travis as the test always fails the same way?

=== RUN   TestOverlaySim
panic: Could not startup node network for mocker
goroutine 14 [running]:
github.com/ethereum/go-ethereum/p2p/simulations.startStop(0xc00020af00, 0xc000043620, 0x10)
	/home/travis/gopath/src/github.com/ethereum/go-ethereum/p2p/simulations/mocker.go:66 +0x8a0
created by github.com/ethereum/go-ethereum/p2p/simulations.(*Server).StartMocker
	/home/travis/gopath/src/github.com/ethereum/go-ethereum/p2p/simulations/http.go:348 +0x260

[frncmx]

TestOverlaySim was fixed by reducing the number of nodes.
Other memory issues were solved by ethereum/go-ethereum#19004

See #1205 for the remaining issues.