feat(docker): read only root filesystem support

[megglos]

Description

This declares /tmp as a VOLUME in the Dockerfile which makes sure this path is always backed by a volume. As of now spring(tomcat), rocksDb and zeebe itself (hot backup compression) write to /tmp. Given that the hot backup could potentially write larger chunks of data to it I generally think a volume is a reasonable default.

Given that now all directories written to are declared as volumes a read only root filesystem can be enabled without any further setup needed. A test is added to ensure this works going forward.

I would go for backporting this to previous stable versions for alignment and ease of use.

I also verified it works on k8s

apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  containers:
  - name: test
    image: gcr.io/zeebe-io/zeebe:sb-pr-11877
    securityContext:
      runAsUser: 1000
      readOnlyRootFilesystem: true

Which previously failed with write permission errors.

Related issues

closes #11876

Not all items need to be done depending on the issue and the pull request.

Code changes:

• The changes are backwards compatibility with previous versions
• If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

• There are unit/integration tests that verify all acceptance criterias of the issue
• New tests are written to ensure backwards compatibility with further versions
• The behavior is tested manually
• The change has been verified by a QA run
• The impact of the changes is verified by a benchmark

[github-actions]

Test Results

   997 files  ±    0     997 suites  ±0   1h 42m 43s ⏱️ - 8m 30s
8 033 tests +255  8 023 ✔️ +255  10 💤 ±0  0 ❌ ±0 
8 230 runs  +255  8 220 ✔️ +255  10 💤 ±0  0 ❌ ±0 

Results for commit 6a876ba. ± Comparison against base commit ae6dbfc.

This pull request removes 389 and adds 644 tests. Note that renamed tests count towards both.

DmnEvaluationTest If successfully evaluated, the output ‑ Should return a message pack output[6] value={x=1, y=true, z=[1, 2, 3]}
io.camunda.zeebe.engine.processing.bpmn.activity.OutputMappingTest ‑ shouldApplyOutputMapping[0: io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@2771e501]
io.camunda.zeebe.engine.processing.bpmn.activity.OutputMappingTest ‑ shouldApplyOutputMapping[1: io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@3a7abd4]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=BUSINESS_RULE_TASK, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@5d4af473, variables={}]]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=CALL_ACTIVITY, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@486d1868, variables={}]]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=END_EVENT, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@6dd50380, variables={}]]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=EVENT_BASED_GATEWAY, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@18d8da77, variables={correlationKey=value}]]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=EVENT_SUB_PROCESS, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@5653429e, variables={}]]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=EXCLUSIVE_GATEWAY, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@9423344, variables={}]]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=INTERMEDIATE_CATCH_EVENT, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@78c9c38a, variables={correlationKey=value}]]
…

DmnEvaluationTest If successfully evaluated, the output ‑ Should return a message pack output[6] value={z=[1, 2, 3], y=true, x=1}
io.camunda.zeebe.engine.processing.bpmn.activity.OutputMappingTest ‑ shouldApplyOutputMapping[0: io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@1dc9aba0]
io.camunda.zeebe.engine.processing.bpmn.activity.OutputMappingTest ‑ shouldApplyOutputMapping[1: io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@382fba37]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=BUSINESS_RULE_TASK, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@272cec71, variables={}]]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=CALL_ACTIVITY, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@639c5ab4, variables={}]]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=END_EVENT, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@61b11d35, variables={}]]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=EVENT_BASED_GATEWAY, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@63ead376, variables={correlationKey=value}]]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=EVENT_SUB_PROCESS, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@5c883b3f, variables={}]]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=EXCLUSIVE_GATEWAY, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@5834298c, variables={}]]
io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceSupportedElementTest ‑ testProcessInstanceCanStartAtElementType[Scenario[type=INTERMEDIATE_CATCH_EVENT, modelInstance=io.camunda.zeebe.model.bpmn.impl.BpmnModelInstanceImpl@a47a011, variables={correlationKey=value}]]
…

[npepinpe]

Looks good! The main change I can see in k8s is that instead of mounting /tmp to the overlay filesystem, it's now mounted in its own isolated filesystem.

I started a benchmark, though I don't really expect any performance difference, but just to be safe 👍

[github-actions]

Setup

Deployed to measurement-4311871032

camunda-platform:
  zeebe:
    image:
      repository: gcr.io/zeebe-io/zeebe
      tag: meg-11876-read-only-fs-benchmark-6a876ba
  zeebe-gateway:
    image:
      repository: gcr.io/zeebe-io/zeebe
      tag: meg-11876-read-only-fs-benchmark-6a876ba
global:
  image:
    tag: meg-11876-read-only-fs-benchmark-6a876ba

Measurement before

Process Instance Execution Time: p99=2.389 p90=0.815 p50=0.226
Throughput: 148.508 PI/s
Grafana

Chaos injection

Deployed chaos network-latency-5

Measurement after

Process Instance Execution Time: p99=3.756 p90=2.136 p50=0.824
Throughput: 73.306 PI/s
Grafana

Details

See https://github.com/camunda/zeebe/actions/runs/4311871032

[megglos]

bors r+

[ghost]

Build succeeded:

• Test summary

[backport-action]

Successfully created backport PR for stable/8.0:

• [Backport stable/8.0] feat(docker): read only root filesystem support #11887

[backport-action]

Successfully created backport PR for stable/8.1:

• [Backport stable/8.1] feat(docker): read only root filesystem support #11888