Read only file system error creating container

[gcyre]

Issue Report

When creating a Splunk forwarder container through kubernetes, pod gets into CrashLoopBackOff with error Read-only file system

Bug

Container Linux Version

docker://17.9.1

`$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1632.3.0
VERSION_ID=1632.3.0
BUILD_ID=2018-02-14-0338
PRETTY_NAME="Container Linux by CoreOS 1632.3.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"`

BUG_REPORT_URL="https://issues.coreos.com"

Environment

What hardware/cloud provider/hypervisor is being used to run Container Linux?
On-prem kubernetes cluster (v1.9.4)

• 3 master nodes (all vmware)
• 3 worker nodes (2 vmware nodes, 1 physical)

Splunk universal forwarder (splunk/universalforwarder) is being installed as daemonset.
Error is happening on
physical node

Expected Behavior

Splunk forwarder deamonset gets installed on all worker nodes

Actual Behavior

Pods created on vmware worker nodes start as expected but pods created on physical node fails with Read Only file system

Reproduction Steps

1. build physical coreos node
2. add node to kubernetes cluster
3. Create slunk forwarder daemonset

Other Information

chown: changing ownership of ‘/opt/splunk/etc/system/local/inputs.conf’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038/inputs.conf’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038/SPLUNK_FORWARD_SERVER’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/SPLUNK_FORWARD_SERVER’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/..data’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local’: Read-only file system

Feature Request

Environment

What hardware/cloud provider/hypervisor is being used to run Container Linux?

Desired Feature

Other Information

[dghubble]

I think this is the (breaking imo) change Kubernetes v1.9.4 introduced, not related to Container Linux. Effectively, Secret, configMap, downward API, and projected volumes are now read-only. kubernetes/kubernetes#58720

This causes issues for various applications which chown or otherwise manipulate their configs. grafana/grafana-docker#140

[gcyre]

Thanks for the quick response, that worker node does have 1.9.4 installed and you are correct this is a breaking change. :(

I'll contact the vendor and see if there is an update.

thanks again!

[lucab]

@dghubble @gcyre thanks both for the followups. It looks like this is the new behavior introduced to fix CVE-2017-1002102, and it affects all latest k8s releases from 1.7.x to 1.10.x.

I'm closing this as it isn't a ContainerLinux bug, and the behavior reported seems to be the intended one after the security fix. Container images will likely need to be fixed, and the ReadOnlyAPIDataVolumes feature gate can be temporarily used in case of emergencies.