File system performance improvements

[yallop]

Recent Docker versions (17.04 CE Edge onwards) add additional flags to the -v option for docker run that make it possible to specify the consistency requirements for a bind-mounted directory. The flags are

• consistent: Full consistency. The container runtime and the host maintain an identical view of the mount at all times. This is the default.
• cached: The host's view of the mount is authoritative. There may be delays before updates made on the host are visible within a container.
• delegated: The container runtime's view of the mount is authoritative. There may be delays before updates made in a container are visible on the host.

In 17.04 cached is significantly faster than consistent for many workloads. However, delegated currently behaves identically to cached. The Docker for Mac team plans to release an improved implementation of delegated in the future, to speed up write-heavy workloads. We also plan to further improve the performance of cached and consistent.

We will post updates relating to Docker for Mac file sharing performance in comments to this issue. Users interested in news about performance improvements should subscribe here.

To keep the signal-to-noise ratio high we will actively moderate this issue, removing off-topic advertisements, etc. Questions about the implementation, the behaviour, and the design are very welcome, as are realistic benchmarks and real-world use cases.

[joemewes]

A rudimentary real-worked example (Drupal 8):

Docker for Mac : Version 17.05.0-ce-rc1-mac8 (16582)
Mac-mini :: MacOS Sierra : 10.12.2 (16C67)

A simple command line curl test (taken average of 10 calls to URL) Drupal 8 clean install frontend:

old UNISON (custom synced container approach) Volume mount : 0.470s
standard Volume mount: 1.401s
new :cached Volume mount: 0.490

v.easy implementation to add to my compose.yaml files and happy with any delay between host/output using cached on host codebase.

@yallop Is there a rough/expected release date for 17.04 (stable) ?

[yallop]

@yallop Is there a rough/expected release date for 17.04 (stable)

The next stable version will be 17.06, and is likely to be released some time in June. There's been a change to the Docker version numbering scheme recently so that the numbers now indicate the release date, with stable releases every three months (March, June, September, December), and edge releases in the months in between. For example, 17.04 is the Edge release in April 2017 and 17.06 is the stable release in June 2017.

[joemewes]

ok, thanks for that. do you expect 17.06 to contain at least the current edge implementation of :cached?

[yallop]

Yes, that's the plan.

[reinout]

How should this work in a docker-compose.yml file? I tried appending :cached to the existing volume setting:

volumes:
  - .:/srv:cached

... but that got me an error (on OSX):

ERROR: Cannot start service my_service: Mounts denied: 9p: Stack overflow

(Docker version 17.05.0-ce-rc1, build 2878a85)

Note: having a global setting or environment variable to switch my local default to "cached" would also be fine (or rather preferable).

[DanielSchwiperich]

The syntax looks correct @reinout
here's a stripped but working example docker-compose.yml

version: '2'
services:
  php:
    image: php:7.1-fpm
    ports:
      - 9000
    volumes:
      - .:/var/www/project:cached

tested on Docker version 17.05.0-ce-rc1, build 2878a85

[reinout]

Your example works. My own one still not (even after really making sure there were no left-over old mounted volumes). So I rebooted. Afterwards, it worked.

So: a reboot might be needed if you've run a docker-compose before upgrading docker to the latest version. Possibly related: I switched from docker stable to edge.

---

Is there a possibility of a global setting? I don't really want to add this option to the docker-compose.yml that all my linux colleagues are using.

[DanielSchwiperich]

not as far as I know. When you linux colleagues are running edge the flag should work.

Another workaround would be (that's what we do right now for separating linux and mac volume mounting )
to just put the mount settings for mac in a separate file, like docker-compose-mac.yml and then run docker-compose -f docker-compose.yml -f docker-compose.mac.yml up -d

See https://docs.docker.com/compose/extends/

[carn1x]

@reinout You could use an additional compose file to override that of your colleagues? For instance, we have:

docker-compose.yml:

version: '2'

services:

  build_docs:
    image: docs/sphinx
    build: .
    environment:
      - DOCS_NAME='docs'
      - SRC_DIR=src
      - DST_DIR=build
    volumes:
      - "./../../docs/dev:/docs"
    command: /docs/source /docs/build

and volumes-cached.yml:

services:
  build_docs:
    volumes:
      - "./../../docs/dev:/docs:cached"

Which can be run with:

$ docker-compose -f docker-compose.yml -f volumes-cached.yml up

[reinout]

Yes, I could do that. But.... I'd have to do that for each of the 12 docker-compose projects. And I'd have to keep it in sync with changes to the "master" docker-compose.yml.

As an intermediary measure: fine. Long term: no, as it is not very don't-repeat-yourself :-)

If someone wants to enable the "cached" behaviour, that person probably wants to use it for all/most of the dockers. Would it make sense as a config setting in the Docker app itself? In the preferences' "File sharing" tab? (This should probably be its own ticket, I assume?)

[yallop]

@reinout: :cached is supported across platforms, so there should be no issue in adding it directly to compose files.

[reinout]

Provided that everybody uses the latest edge version, right? And it seems a bit strange to add an option that only has effect on osx to everybody's docker-compose.yml.

Anyway, it works for now. I won't drag the signal/noise ratio further down :-)

[matthewjosephtaylor]

mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ time dd if=/dev/zero of=./output bs=8k count=40k; rm ./output
40960+0 records in
40960+0 records out
335544320 bytes transferred in 1.300857 secs (257941007 bytes/sec)

real	0m1.320s
user	0m0.012s
sys	0m0.564s
mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ docker run -it --rm -v "$(pwd):/host-disk" ubuntu /bin/bash 
root@4e9c8bc5e5c1:/# cd /host-disk/
root@4e9c8bc5e5c1:/host-disk# time dd if=/dev/zero of=./output bs=8k count=40k; rm ./output
40960+0 records in
40960+0 records out
335544320 bytes (336 MB, 320 MiB) copied, 10.7496 s, 31.2 MB/s

real	0m10.756s
user	0m0.050s
sys	0m1.090s
root@4e9c8bc5e5c1:/host-disk# exit
exit
mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ docker run -it --rm -v "$(pwd):/host-disk:cached" ubuntu /bin/bash 
root@597dc640bdeb:/# cd /host-disk/
root@597dc640bdeb:/host-disk# time dd if=/dev/zero of=./output bs=8k count=40k; rm ./output
40960+0 records in
40960+0 records out
335544320 bytes (336 MB, 320 MiB) copied, 11.1683 s, 30.0 MB/s

real	0m11.172s
user	0m0.060s
sys	0m1.080s
root@597dc640bdeb:/host-disk# exit
exit
mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ docker run -it --rm -v "$(pwd):/host-disk:delegated" ubuntu /bin/bash 
root@985e4143053b:/# cd /host-disk/
root@985e4143053b:/host-disk# time dd if=/dev/zero of=./output bs=8k count=40k; rm ./output
40960+0 records in
40960+0 records out
335544320 bytes (336 MB, 320 MiB) copied, 12.1589 s, 27.6 MB/s

real	0m12.165s
user	0m0.080s
sys	0m1.000s
root@985e4143053b:/host-disk# exit
exit
mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ docker run -it --rm -v "$(pwd):/host-disk:consistent" ubuntu /bin/bash 
root@3377ae356124:/# cd /host-disk/
root@3377ae356124:/host-disk# time dd if=/dev/zero of=./output bs=8k count=40k; rm ./output
40960+0 records in
40960+0 records out
335544320 bytes (336 MB, 320 MiB) copied, 12.5944 s, 26.6 MB/s

real	0m12.601s
user	0m0.060s
sys	0m0.980s
root@3377ae356124:/host-disk# exit
exit
mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ docker --version
Docker version 17.05.0-ce, build 89658be
mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ 

Perhaps my expectations on how this works are unreasonable, or I'm doing something wrong. Above are some simple disk performance tests and I'm not seeing any differences.

Would be interested in knowing if my expectations, or use of the flag is incorrect.

[barat]

@matthewjosephtaylor ... :cached won't improve dd tests ... for this, you need to check for :delegated to rollout :)

[ToonSpinISAAC]

The issue text says:

delegated: The container runtime's view of the mount is authoritative. There may be delays before updates made in a container are visible on the host.

Does this mean that if I were to use delegated (or cached for that matter) that syncing would strictly be a one-way affair?

In other words, to make my question clearer, let's say I have a codebase in a directory and I mount this directory inside a container using delegated. Does this mean that if I update the codebase on the host, the container will overwrite my changes?

What I understood until now was, that using for instance delegated, would keep syncing "two-way", but make it more efficient from the container to the host, but the text leads me to believe that I may have misunderstood, hence the question.

[geerlingguy]

Just posting another data point:

• Drupal codebase volume with rw,delegated on Docker Edge (for now, it's only using the cached option though): 37 requests/sec
• Drupal codebase volume with rw (and no extra options): 2 requests/sec

Drupal 8 is ~18x faster if you're using a Docker volume to share a host codebase into a container, and it's pretty close to native filesystem performance.

With cached, Drupal and Symfony development are no longer insanely painful with Docker. With delegated, that's even more true, as operations like composer update (which results in many writes) will also be orders-of-magnitude faster!

[kostajh]

I'm seeing good results for running a set of Behat tests on a Drupal 7 site:

On Mac OS, without :cached:

10:02 $ docker exec clientsite_php bin/behat -c tests/behat.yml --tags=~@failing -f progress
...................................................................... 70
...................................................................... 140
...................................................................... 210
...................................................................... 280
.................................................

69 scenarios (69 passed)
329 steps (329 passed)
11m26.20s (70.82Mb)

With :cached:

09:55 $ docker exec clientsite_php bin/behat -c tests/behat.yml --tags=~@failing -f progress
...................................................................... 70
...................................................................... 140
...................................................................... 210
...................................................................... 280
.................................................

69 scenarios (69 passed)
329 steps (329 passed)
4m33.77s (63.33Mb)

On Travis CI (without :cached):

$ docker exec clientsite_php bin/behat -c tests/behat.yml --tags=~@failing -f progress
...................................................................... 70
...................................................................... 140
...................................................................... 210
...................................................................... 280
.................................................
69 scenarios (69 passed)
329 steps (329 passed)
4m7.07s (55.01Mb)

On Travis CI (with :cached):

247.12s$ docker exec cliensite_php bin/behat -c tests/behat.yml --tags=~@failing -f progress
...................................................................... 70
...................................................................... 140
...................................................................... 210
...................................................................... 280
.................................................
69 scenarios (69 passed)
329 steps (329 passed)
4m6.71s (55.01Mb)

Nice work, Docker team! 👏

[carn1x]

Is it expected that both :cached and :delegated can be combined or will they be mutually exclusive?

[dsheets]

@ToonSpinISAAC both :cached and (when it lands) :delegated perform two-way "syncing". The text you cite is saying that, with :cached, the container may read stale data if it has changed on the host and the invalidation event hasn't propagated yet. With :cached, the container will write-through and no new write-write conflicts can occur (POSIX still allows multiple writers). Think of :cached as "read caching". With :delegated, if the container writes to a file that write will win even if an intermediate write has occurred on the host. Container writes can be delayed indefinitely but are guaranteed to persist after the container has successfully exited. flush and similar functionality will also guarantee persistence. Think of :delegated as "read-write caching". Even under :delegated, synchronization happens in both directions and updates may occur rapidly (but don't have to). Additionally, you may overlap :cached and :delegated and :cached semantics will override :delegated semantics. See https://docs.docker.com/docker-for-mac/osxfs-caching/#delegated guarantee 5. If you are using :delegated for source code but your container does not write to your code files (this seems unlikely but maybe it auto-formats or something?), there is nothing to worry about. :delegated is currently the same as :cached but will provide write caching in the future.

@carn1x :cached and :delegated (and :default and :consistent) form a partial order (see https://docs.docker.com/docker-for-mac/osxfs-caching/#semantics). They can't be combined but they do degrade to each other. This allows multiple containers with different requirements to share the same bind mount directories safely.

[lmakarov]

From within a container is there a way to tell which flag was applied for a volume?

I'm getting the same output from mount regardless of the flag used. Is there another way to check?

$ mount | grep osxfs
osxfs on /var/www/project type fuse.osxfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)

Version 17.06.0-rc1-ce-mac13 (18169)
Channel: edge
2425473dc2

[dsheets]

@lmakarov /Applications/Docker.app/Contents/MacOS/com.docker.osxfs state should show you the host directories that are mounted into containers and their mount options. For instance, after I run docker run --rm -it -v ~:/host:cached alpine ash, I then see:

$ /Applications/Docker.app/Contents/MacOS/com.docker.osxfs state
Exported directories:
 - /Users to /Users (nodes_/Users table size: 62)
 - /Volumes to /Volumes (nodes_/Volumes table size: 0)
 - /tmp to /tmp (nodes_/private/tmp table size: 9)
 - /private to /private (nodes_/private table size: 0)
Container-mounted directories:
 - /Users/dsheets into b8f7765665782501bc1a099f1898911b7eb393b08930be638545a55fd06e420e (state=cached)

[ToonSpinISAAC]

Thanks for taking the time to clarify and explain @dsheets!

[gondalez]

Is there any indication from the high sierra dev beta of how this (and docker mac in general) will work under APFS?

Sorry if this is off-topic but it's a question I keep wanting to ask every time I see a new message here :)

[WillSquire]

Unable to get @DanielSchwiperich's example working? Retrieving the error:

invalid spec: .:/var/www/project:cached: unknown option: cached

It doesn't like version being set to 2 either. Perhaps I'm missing something? Currently running 17.06.0-rc2-ce-mac14

[27Bslash6]

@WillSquire What version of docker-compose are you using?

[PhilETaylor]

We're getting some data that the new virtualization framework has speed problems on Intel chips,

That's been my experience to the point I've had it off for a while now.

[ain]

I can confirm the experience of @z1r0-: it does crash with Docker Compose throwing Unsupported opcode: 40 on docker compose up.

I can also confirm it's not specific to Traefik. In my case it happened on a setup that incorporated mysql:5.7 and golang:1.16-alpine.

Same compose file succeeds on 4.2.0.

Full failure log:

{"app_name":"com.apple.Virtualization.VirtualMachine","timestamp":"2021-11-30 21:26:22.00 +0100","app_version":"1.0","slice_uuid":"039ffa84-b59a-363c-87e9-80d6131df3dd","build_version":"59.2","platform":1,"bundleID":"com.apple.Virtualization.VirtualMachine","share_with_app_devs":1,"is_first_party":1,"bug_type":"309","os_version":"macOS 12.0.1 (21A559)","incident_id":"888BC39B-CE6A-4FE4-990D-33F62A141BBE","name":"com.apple.Virtualization.VirtualMachine"}
{
  "uptime" : 81000,
  "procLaunch" : "2021-11-30 20:53:41.9915 +0100",
  "procRole" : "Default",
  "version" : 2,
  "userID" : 501,
  "deployVersion" : 210,
  "modelCode" : "MacBookPro13,1",
  "procStartAbsTime" : 81342686569888,
  "coalitionID" : 11439,
  "osVersion" : {
    "train" : "macOS 12.0.1",
    "build" : "21A559",
    "releaseType" : "User"
  },
  "captureTime" : "2021-11-30 21:26:22.0569 +0100",
  "incident" : "888BC39B-CE6A-4FE4-990D-33F62A141BBE",
  "bug_type" : "309",
  "pid" : 66654,
  "procExitAbsTime" : 81705440307550,
  "cpuType" : "X86-64",
  "procName" : "com.apple.Virtualization.VirtualMachine",
  "procPath" : "\/System\/Library\/Frameworks\/Virtualization.framework\/Versions\/A\/XPCServices\/com.apple.Virtualization.VirtualMachine.xpc\/Contents\/MacOS\/com.apple.Virtualization.VirtualMachine",
  "bundleInfo" : {"CFBundleShortVersionString":"1.0","CFBundleVersion":"59.2","CFBundleIdentifier":"com.apple.Virtualization.VirtualMachine"},
  "buildInfo" : {"ProjectName":"Virtualization","SourceVersion":"59002000000000","BuildVersion":"23"},
  "parentProc" : "launchd",
  "parentPid" : 1,
  "coalitionName" : "com.docker.docker",
  "crashReporterKey" : "6124F52F-9E5D-9582-F29A-952DA77017DE",
  "responsiblePid" : 66639,
  "responsibleProc" : "com.docker.virtualization",
  "wakeTime" : 213,
  "sleepWakeUUID" : "E9CC485F-072B-41C1-8AF1-6D9DCD7D34C9",
  "sip" : "enabled",
  "isCorpse" : 1,
  "exception" : {"codes":"0x0000000000000001, 0x0000000000000000","rawCodes":[1,0],"type":"EXC_BAD_INSTRUCTION","signal":"SIGILL"},
  "termination" : {"flags":0,"code":4,"namespace":"SIGNAL","indicator":"Illegal instruction: 4","byProc":"exc handler","byPid":66654},
  "asi" : {"com.apple.Virtualization.VirtualMachine":["Generic Platform"],"libsystem_c.dylib":["FIXME: \"Unsupported opcode: 40\""]},
  "extMods" : {"caller":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"system":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"targeted":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"warnings":0},
  "faultingThread" : 8,
  "threads" : [{"id":1100785,"frames":[{"imageOffset":25314,"symbol":"__sigsuspend_nocancel","symbolLocation":10,"imageIndex":0},{"imageOffset":82174,"symbol":"_dispatch_sigsuspend","symbolLocation":36,"imageIndex":1},{"imageOffset":82138,"symbol":"_dispatch_sig_thread","symbolLocation":53,"imageIndex":1}]},{"id":1100844,"name":"com.apple.virtualization.thread.raw-disk-io-0","frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":0},{"imageOffset":27273,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":2},{"imageOffset":195602,"imageIndex":3},{"imageOffset":196642,"imageIndex":3},{"imageOffset":25876,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":2},{"imageOffset":8239,"symbol":"thread_start","symbolLocation":15,"imageIndex":2}]},{"id":1100845,"name":"com.apple.virtualization.thread.raw-disk-io-1","frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":0},{"imageOffset":27273,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":2},{"imageOffset":195602,"imageIndex":3},{"imageOffset":196642,"imageIndex":3},{"imageOffset":25876,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":2},{"imageOffset":8239,"symbol":"thread_start","symbolLocation":15,"imageIndex":2}]},{"id":1100846,"name":"com.apple.virtualization.thread.raw-disk-io-2","frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":0},{"imageOffset":27273,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":2},{"imageOffset":195602,"imageIndex":3},{"imageOffset":196642,"imageIndex":3},{"imageOffset":25876,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":2},{"imageOffset":8239,"symbol":"thread_start","symbolLocation":15,"imageIndex":2}]},{"id":1100847,"name":"com.apple.virtualization.thread.raw-disk-io-3","frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":0},{"imageOffset":27273,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":2},{"imageOffset":195602,"imageIndex":3},{"imageOffset":196642,"imageIndex":3},{"imageOffset":25876,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":2},{"imageOffset":8239,"symbol":"thread_start","symbolLocation":15,"imageIndex":2}]},{"id":1100848,"name":"com.apple.virtualization.thread.cpu-0","frames":[{"imageOffset":14383,"symbol":"hv_vcpu_run_until","symbolLocation":16,"imageIndex":4},{"imageOffset":771255,"imageIndex":3},{"imageOffset":774462,"imageIndex":3},{"imageOffset":25876,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":2},{"imageOffset":8239,"symbol":"thread_start","symbolLocation":15,"imageIndex":2}]},{"id":1100849,"name":"com.apple.virtualization.thread.cpu-1","frames":[{"imageOffset":14383,"symbol":"hv_vcpu_run_until","symbolLocation":16,"imageIndex":4},{"imageOffset":771255,"imageIndex":3},{"imageOffset":774462,"imageIndex":3},{"imageOffset":25876,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":2},{"imageOffset":8239,"symbol":"thread_start","symbolLocation":15,"imageIndex":2}]},{"id":1100850,"name":"com.apple.virtualization.thread.virtual-clock","frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":0},{"imageOffset":27273,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":2},{"imageOffset":1063050,"imageIndex":3},{"imageOffset":1063534,"imageIndex":3},{"imageOffset":25876,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":2},{"imageOffset":8239,"symbol":"thread_start","symbolLocation":15,"imageIndex":2}]},{"triggered":true,"id":1106597,"queue":"com.apple.virtualization.virtio.filesystem-device:0","frames":[{"imageOffset":1172245,"imageIndex":3},{"imageOffset":1171790,"imageIndex":3},{"imageOffset":1039127,"imageIndex":3},{"imageOffset":253573,"imageIndex":3},{"imageOffset":225123,"imageIndex":3},{"imageOffset":481335,"imageIndex":3},{"imageOffset":487640,"imageIndex":3},{"imageOffset":11465,"symbol":"_dispatch_client_callout","symbolLocation":8,"imageIndex":1},{"imageOffset":36078,"symbol":"_dispatch_lane_serial_drain","symbolLocation":696,"imageIndex":1},{"imageOffset":38856,"symbol":"_dispatch_lane_invoke","symbolLocation":366,"imageIndex":1},{"imageOffset":79841,"symbol":"_dispatch_workloop_worker_thread","symbolLocation":758,"imageIndex":1},{"imageOffset":12431,"symbol":"_pthread_wqthread","symbolLocation":326,"imageIndex":2},{"imageOffset":8219,"symbol":"start_wqthread","symbolLocation":15,"imageIndex":2}]},{"id":1106801,"frames":[{"imageOffset":8204,"symbol":"start_wqthread","symbolLocation":0,"imageIndex":2}]}],
  "usedImages" : [
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703650295808,
    "size" : 225280,
    "uuid" : "12bd6f13-c452-35ee-9069-51befef29f1a",
    "path" : "\/usr\/lib\/system\/libsystem_kernel.dylib",
    "name" : "libsystem_kernel.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703648739328,
    "size" : 290816,
    "uuid" : "be53a13c-8ce1-3e40-b9bc-98473d3eed3e",
    "path" : "\/usr\/lib\/system\/libdispatch.dylib",
    "name" : "libdispatch.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703650521088,
    "size" : 49152,
    "uuid" : "29a2750e-f31b-3630-8761-242a6bc3e99e",
    "path" : "\/usr\/lib\/system\/libsystem_pthread.dylib",
    "name" : "libsystem_pthread.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4346179584,
    "CFBundleShortVersionString" : "1.0",
    "CFBundleIdentifier" : "com.apple.Virtualization.VirtualMachine",
    "size" : 1294336,
    "uuid" : "039ffa84-b59a-363c-87e9-80d6131df3dd",
    "path" : "\/System\/Library\/Frameworks\/Virtualization.framework\/Versions\/A\/XPCServices\/com.apple.Virtualization.VirtualMachine.xpc\/Contents\/MacOS\/com.apple.Virtualization.VirtualMachine",
    "name" : "com.apple.Virtualization.VirtualMachine",
    "CFBundleVersion" : "59.2"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140712715620352,
    "CFBundleShortVersionString" : "1.0",
    "CFBundleIdentifier" : "com.apple.Hypervisor",
    "size" : 32768,
    "uuid" : "bb2b0157-6297-310c-9c80-9b243a6e234f",
    "path" : "\/System\/Library\/Frameworks\/Hypervisor.framework\/Versions\/A\/Hypervisor",
    "name" : "Hypervisor",
    "CFBundleVersion" : "1"
  }
],
  "sharedCache" : {
  "base" : 140703647293440,
  "size" : 15213641728,
  "uuid" : "3d05845f-3f65-358f-9ebf-2236e772ac01"
},
  "vmSummary" : "ReadOnly portion of Libraries: Total=818.4M resident=0K(0%) swapped_out_or_unallocated=818.4M(100%)\nWritable regions: Total=8.1G written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=8.1G(100%)\n\n                                VIRTUAL   REGION \nREGION TYPE                        SIZE    COUNT (non-coalesced) \n===========                     =======  ======= \nActivity Tracing                   256K        1 \nDispatch continuations            32.0M        1 \nIOKit                              736K        3 \nKernel Alloc Once                    8K        1 \nMALLOC                            38.4M       21 \nMALLOC guard page                   16K        4 \nMemory Tag 240                     8.0G       64 \nSTACK GUARD                         40K       10 \nStack                             5208K       11 \nStack Guard                       64.0M        2 \nVM_ALLOCATE                          8K        2 \n__DATA                            15.2M      240 \n__DATA_CONST                      11.0M      149 \n__DATA_DIRTY                       484K       86 \n__FONT_DATA                          4K        1 \n__LINKEDIT                       635.6M        8 \n__OBJC_RO                         81.5M        1 \n__OBJC_RW                         3120K        2 \n__TEXT                           182.8M      256 \n__UNICODE                          588K        1 \ndyld private memory               1024K        1 \nmapped file                       28.6M        3 \nshared memory                      332K       12 \n===========                     =======  ======= \nTOTAL                              9.1G      880 \n",
  "legacyInfo" : {
  "threadTriggered" : {
    "queue" : "com.apple.virtualization.virtio.filesystem-device:0"
  }
},
  "trialInfo" : {
  "rollouts" : [
    {
      "rolloutId" : "60da5e84ab0ca017dace9abf",
      "factorPackIds" : {

      },
      "deploymentId" : 240000008
    },
    {
      "rolloutId" : "607844aa04477260f58a8077",
      "factorPackIds" : {
        "SIRI_MORPHUN_ASSETS" : "6103050cbfe6dc472e1c982a"
      },
      "deploymentId" : 240000066
    },
    {
      "rolloutId" : "601d9415f79519000ccd4b69",
      "factorPackIds" : {
        "SIRI_TEXT_TO_SPEECH" : "6194416dea7ed64a7812a429"
      },
      "deploymentId" : 240000322
    },
    {
      "rolloutId" : "602ad4dac86151000cf27e46",
      "factorPackIds" : {
        "SIRI_DICTATION_ASSETS" : "614a34c5eae97a2111628dc5"
      },
      "deploymentId" : 240000271
    },
    {
      "rolloutId" : "5fc94383418129005b4e9ae0",
      "factorPackIds" : {

      },
      "deploymentId" : 240000162
    },
    {
      "rolloutId" : "5ffde50ce2aacd000d47a95f",
      "factorPackIds" : {

      },
      "deploymentId" : 240000076
    }
  ],
  "experiments" : [

  ]
},
  "reportNotes" : [
  "thread_get_state(PAGEIN) returned 0x10000003: (ipc\/send) invalid destination port",
  "thread_get_state(EXCEPTION) returned 0x10000003: (ipc\/send) invalid destination port",
  "thread_get_state(FLAVOR) returned 0x10000003: (ipc\/send) invalid destination port"
]
}

On a retry, Docker Compose crashes with unexpected EOF, error modal does not appear and Docker Desktop goes to restart by itself.

[varp]

Correct, virtiofs requires the new virtualization framework, and also correct that the UI is wrong.

I don't see "Enable VirtioFs accelerated directory sharing". I'm on Monterey with Docker Desktop 4.3.1 (72247). @owsygavin could you please help with that?

[owsygavin]

Correct, virtiofs requires the new virtualization framework, and also correct that the UI is wrong.

I don't see "Enable VirtioFs accelerated directory sharing". I'm on Monterey with Docker Desktop 4.3.1 (72247). @owsygavin could you please help with that?

I have an older version than you do:

VirtioFS is under experimental features in preferences:

[owsygavin]

Correct, virtiofs requires the new virtualization framework, and also correct that the UI is wrong.

I don't see "Enable VirtioFs accelerated directory sharing". I'm on Monterey with Docker Desktop 4.3.1 (72247). @owsygavin could you please help with that?

I suspect what's happened is, due to the big fuss with the log4j CVE, your Docker has been updated, which may have updated to a non-dev-build version. To test this, I hit "Software Updates" -> "Update" and after it came back, my experimental virtuoFS option is gone too:

I don't know how to get the docker dev builds; I only got the one linked on this thread, I have no idea how you find "latest Docker dev build with experimental feature support", perhaps a Docker wizard on this thread could advise :-)

[vraravam]

I tried out running docker-compose on a plain vanilla springboot application that has postgres as a dependency. This is what I get:

docker: Error response from daemon: error while creating mount source path '/host_mnt/Users/XXXX/java-template': mkdir /host_mnt/Users/XXXX: permission denied.

I have the Experimental features (both of them enabled), and also the gRPCFuse option.

Any pointers please?

[varp]

Correct, virtiofs requires the new virtualization framework, and also correct that the UI is wrong.

I don't see "Enable VirtioFs accelerated directory sharing". I'm on Monterey with Docker Desktop 4.3.1 (72247). @owsygavin could you please help with that?

I suspect what's happened is, due to the big fuss with the log4j CVE, your Docker has been updated, which may have updated to a non-dev-build version. To test this, I hit "Software Updates" -> "Update" and after it came back, my experimental virtuoFS option is gone too:

I don't know how to get the docker dev builds; I only got the one linked on this thread, I have no idea how you find "latest Docker dev build with experimental feature support", perhaps a Docker wizard on this thread could advise :-)

I think I found docker/roadmap#7 (comment)

[stephen-turner]

We don't have any later dev builds than the one already posted on this thread.

[owsygavin]

We don't have any later dev builds than the one already posted on this thread.

thanks for clearing that up :-) If you want any future builds testing for this feature, feel free to post on this thread and I can give it a go

[stephen-turner]

thanks for clearing that up :-) If you want any future builds testing for this feature, feel free to post on this thread and I can give it a go

Thanks, we appreciate it. I think we might wait until there is a fix for the Unsupported opcode: 40 problem before making a new build, as lots of people are getting blocked by that. It will require a change in MacOS though, so it may depend on their timescale.

[fredericdalleau]

Hey there, I'm pretty sure some of you are already aware that Apple published a beta version of macOS 12.2.
What is less known is that this beta contains a fix for the "opcode 40" problems.
We already gave it a try and our internal tests cases have passed with this version.

If you feel like updating to a macOS beta version, we would love it if you could retest the build at [link] with the new version of the OS.

[christophermclellan]

Thank you to everyone who has provided feedback so far on the virtiofs experimental build that was released on 22-November.

The latest macOS 12.2 release contains fixes to a number of the issues mentioned in this thread, specifically the unsupported opcode and unsupported mknod file type errors. @fredericdalleau has created a new experimental build which can be used with macOS 12.2.

Any feedback that you might be able provide on whether the new build + OS update has improved reliability for you would be very helpful as we move towards releasing this within Docker Desktop master as an Experimental Feature.

To enable virtiofs accelerated directory sharing please follow these steps:

1. Ensure that you are using the latest macOS 12.2 release
2. Download our new experimental build here:
   • Mac (intel)
   • Mac (M1)
3. Go to ‘Preferences > Experimental features’, then:
   • Check the ‘Use the new Virtualization framework’ toggle
   • Check the ‘Enable VirtioFS accelerated directory sharing’ toggle
4. Click ‘Apply & Restart’

Virtualization framework performance issues on Intel

Please note, there are known performance problems with the new virtualization framework on Intel on Monterey (independent of virtiofs). We've reported it to Apple, and we believe that they're working on it.

[back-2-95]

macOS 12.2 was just released.

[christophermclellan]

Thanks @back-2-95 ! I've now edited my comment from Tuesday to reflect that.

[owsygavin]

I've installed MacOS 12.2 and the prototype Docker with these settings

It's absolutely crawling, it's so slow it's pretty much unusable with our large PHP application with lots of vendors.

Here are the numbers, I don't know how to interpret these runes but it may be of utility to a cleverererr person than me!

docker run --rm -ti alpine sh -c "apk add p7zip ; 7z b"

...
RAM size:    1985 MB,  # CPU hardware threads:   6
RAM usage:   1323 MB,  # Benchmark threads:      6

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:       7007   521   1308   6817  |      87193   588   1265   7436
23:       5449   538   1032   5553  |      85882   584   1273   7431
24:       3799   538    759   4085  |      81722   587   1222   7173
25:       4527   538    962   5169  |      78720   582   1205   7006
----------------------------------  | ------------------------------
Avr:             534   1015   5406  |              585   1241   7261
Tot:             559   1128   6334

I can see that it's using virtiofs:

docker run -it --rm -v $(pwd):/vol alpine mount | grep /vol
virtiofs0 on /vol type virtiofs (rw,relatime)

[stephen-turner]

Thanks, @owsygavin. There are known performance problems with the new virtualization framework on Intel on Monterey (independent of virtiofs). We've reported it to Apple, and I believe they're working on it.

(@christophermclellan It could be worth adding that to your earlier message too, we've had a few people report it).

[johnmaguire]

I am running an integration test suite against dependencies (MySQL and Redis) which live in Docker. With MySQL and Redis running on the host, this suite takes approximately 2-3s, the same amount of time it takes on Linux with Docker dependencies.

With Docker for Mac, it takes roughly 12-13s.

With the experimental build above, and both features enabled, on 12.2 w/ M1 Pro, it takes 75.5s.

[johnmaguire]

In keeping with the trend in this thread, I ran the following benchmark.

w/ experimental features:

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,5 CPUs LE)

LE
CPU Freq: - - - - - - - - -

RAM size:    3918 MB,  # CPU hardware threads:   5
RAM usage:    882 MB,  # Benchmark threads:      5

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:      25566   366   6800  24871  |     213299   398   4567  18198
23:      24763   390   6471  25231  |     209774   399   4547  18151
24:      24452   399   6596  26291  |     206299   399   4544  18110
25:      23321   399   6681  26628  |     200921   398   4498  17881
----------------------------------  | ------------------------------
Avr:             388   6637  25755  |              398   4539  18085
Tot:             393   5588  21920

I then tried to disable the experimental features by unchecking the top checkbox ("Use the new Virtualization framework"), which grayed out the bottom one ("Enable VirtioFS accelerated directory sharing") but left it checked. I hit save and Docker for Mac has hung ever since. Restarting results in it hanging with the message "Docker Desktop starting..."

None of the troubleshooting options - "Clean / Purge data", "Reset to factory defaults", and "Uninstall" - appear to be responsive. I downloaded and installed the stable version of Docker for Mac, and the issue persists. Since it's still starting I am unable to modify any settings to try unchecking both checkboxes.

edit w/o experimental features:

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,5 CPUs LE)

LE
CPU Freq: - - - - - - - - -

RAM size:    3933 MB,  # CPU hardware threads:   5
RAM usage:    882 MB,  # Benchmark threads:      5

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:      23257   374   6046  22625  |     196733   396   4237  16785
23:      22883   389   5993  23316  |     193574   397   4216  16749
24:      22070   402   5903  23730  |     188944   397   4173  16587
25:      21050   400   6014  24034  |     185981   398   4156  16552
----------------------------------  | ------------------------------
Avr:             391   5989  23426  |              397   4195  16668
Tot:             394   5092  20047

[PhilETaylor]

I hit save and Docker for Mac has hung ever since. Restarting results in it hanging with the message "Docker Desktop starting..."

None of the troubleshooting options - "Clean / Purge data", "Reset to factory defaults", and "Uninstall" - appear to be responsive. I downloaded and installed the stable version of Docker for Mac, and the issue persists. Since it's still starting I am unable to modify any settings to try unchecking both checkboxes.

Which describes the issue I also had. docker/roadmap#7 (comment)

[johnmaguire]

FWIW I was able to resolve the issue by deleting various Docker directories out of the ~/Library folder and /usr/lib/docker.

[fredericdalleau]

@johnmaguire sorry for the inconvenience. If anyone gets stuck, they remove the settings file ~/Library/Group Containers/group.com.docker/settings.json

[christophermclellan]

Thank you to everyone that has provided feedback on the latest experimental build. Apple have just released the macOS 12.3 beta, in which the known speed problems with the new virtualization framework on Intel + Monterey should now be resolved (for example - the issue that @owsygavin reported here). If those that have previously experienced performance issues on Intel would be willing to try out 12.3 we'd be very appreciative of any feedback.

[sarisia]

Hi, thank you for this amazing work! Performance of bind mount with virtiofs is far better than gRPC-fuse.

Now I found that chmod to bind-mounted directory fails randomly on macOS 12.3 beta1, like:

$ mkdir test
$ chmod 777 test
$ chmod 777 test
chmod: changing permissions of 'test': Operation not permitted
$ chmod 777 test
$ chmod 777 test
chmod: changing permissions of 'test': Operation not permitted
$ chmod 777 test
$ chmod 777 test
chmod: changing permissions of 'test': Operation not permitted
$ chmod 777 test
chmod: changing permissions of 'test': Operation not permitted

Environment:

• macOS 12.3 Beta (21E5196i)
• Intel MacBook Pro 2020
• Docker Desktop 4.4.2 (73862)

I can provide any further informations if needed!

[fredericdalleau]

@sarisia, that's interesting. I can't reproduce here. Could you share your full command line for starting the container, in case the difference is more subtle? Are there other containers running on this host?

[sarisia]

@fredericdalleau

I usually use VSCode devcontainer, but I've confirmed it can be reproduced
with the commands below:

$ cat Dockerfile
FROM ubuntu:latest

RUN apt-get update && apt-get -y install git
RUN groupadd user && \
    useradd -s /bin/bash --gid user -m user

$ docker build -t dockertest .
[+] Building 1.2s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                       0.5s
 => => transferring dockerfile: 579B                                       0.0s
 => [internal] load .dockerignore                                          0.7s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest           0.0s
 => [1/3] FROM docker.io/library/ubuntu:latest                             0.0s
 => CACHED [2/3] RUN apt-get update && apt-get -y install git              0.0s
 => CACHED [3/3] RUN groupadd user &&     useradd -s /bin/bash --gid user  0.0s
 => exporting to image                                                     0.2s
 => => exporting layers                                                    0.0s
 => => writing image sha256:befa7326e73b8f7c3c541cb52092f113bd7e5ee4fbc21  0.1s
 => => naming to docker.io/library/dockertest                              0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

$ mkdir test
$ docker run -it --rm --mount type=bind,src=(pwd)/test,target=/test,consistency=delegated -u user dockertest /bin/bash

user@e66a3e99705e:/$ cd /test
user@e66a3e99705e:/test$ git init
Initialized empty Git repository in /test/.git/
user@e66a3e99705e:/test$ git submodule add https://github.com/aws/aws-iot-device-sdk-embedded-C.git
Cloning into '/test/aws-iot-device-sdk-embedded-C'...
remote: Enumerating objects: 25668, done.
remote: Counting objects: 100% (9029/9029), done.
remote: Compressing objects: 100% (1997/1997), done.
remote: Total 25668 (delta 6917), reused 8726 (delta 6765), pack-reused 16639
Receiving objects: 100% (25668/25668), 32.90 MiB | 5.92 MiB/s, done.
Resolving deltas: 100% (16389/16389), done.
user@e66a3e99705e:/test$ cd aws-iot-device-sdk-embedded-C/
user@e66a3e99705e:/test/aws-iot-device-sdk-embedded-C$ git submodule update --init --recursive
Submodule 'demos/jobs/jobs_demo_mosquitto/libmosquitto' (https://github.com/eclipse/mosquitto.git) registered for path 'demos/jobs/jobs_demo_mosquitto/libmosquitto'
Submodule 'libraries/3rdparty/CMock' (https://github.com/ThrowTheSwitch/CMock) registered for path 'libraries/3rdparty/CMock'
error: chmod on /test/.git/modules/aws-iot-device-sdk-embedded-C/config.lock failed: Operation not permitted
Submodule 'libraries/3rdparty/mbedtls' (https://github.com/ARMmbed/mbedtls) registered for path 'libraries/3rdparty/mbedtls'
Submodule 'libraries/3rdparty/tinycbor' (https://github.com/intel/tinycbor) registered for path 'libraries/3rdparty/tinycbor'
Submodule 'libraries/aws/device-defender-for-aws-iot-embedded-sdk' (https://github.com/aws/device-defender-for-aws-iot-embedded-sdk.git) registered for path 'libraries/aws/device-defender-for-aws-iot-embedded-sdk'
error: chmod on /test/.git/modules/aws-iot-device-sdk-embedded-C/config.lock failed: Operation not permitted
fatal: Failed to register url for submodule path 'libraries/aws/device-shadow-for-aws-iot-embedded-sdk'

Also, I tested both root and non-root user inside the container and
it only happens when using non-root user.

[Peredery]

any news about new build? 😄

[stephen-turner]

Ah yes, sorry, there is a new one at docker/roadmap#7 (comment). We should have posted it here too, that's our mistake. However, as the main conversation is at docker/roadmap#7, I'm actually going to lock this ticket now to keep everything in one place.