Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker mssql-server-linux crash when mounting a volume #136

Closed
SebastianPfliegel opened this issue Jul 28, 2017 · 110 comments
Closed

Docker mssql-server-linux crash when mounting a volume #136

SebastianPfliegel opened this issue Jul 28, 2017 · 110 comments

Comments

@SebastianPfliegel
Copy link

Setup:

Host-OS: Windows 10
Docker: 17.06.0-ce

Command:

docker run -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=strong1Pass!' -e 'MSSQL_PID=Developer' --cap-add SYS_PTRACE -p 1433:1433 -v c:/users/pfliegel.sebastian/docker_vol/mssql:/var/opt/mssql -d --name mssql microsoft/mssql-server-linux:latest

Error:

This is an evaluation version. There are [165] days left in the evaluation period.This program has encountered a fatal error and cannot continue running. The following diagnostic information is available: Reason: 0x00000006 Status: 0x40000015 Message: Kernel bug check Address: 0x6a42b4a0 Parameters: 0x6a62a250 Stacktrace: 000000006a4e83c6 000000006a42b4fb 000000006a41ec9d 000000006a42ae9b 000000006a4e695d 000000006a4e599c 000000006a4e5810 000000006a4e5745 Process: 8 - sqlservr Thread: 12 (application thread 0x1000) Instance Id: 644f9aaa-60cc-444e-92cd-c2c483694108 Crash Id: e47f1b2c-f882-44ca-9a80-6b380e504d2f Build stamp: e83d0295c4b8a8055fc3842cb84be4f864e57a4bf7c38359c4427ace56a27dbf Capturing core dump and information... dmesg: read kernel buffer failed: Operation not permitted No journal files were found. No journal files were found. Attempting to capture a dump with paldumper

When I try to run the container without attaching volumes it works just fine.

Dumplog is attached: log.zip

@sirkkalap
Copy link

This looks a lot similar to another "Kernel bug check" in #99

@SebastianPfliegel
Copy link
Author

The difference is he blames NFS, I use a local Windows drive. It looks like it's an issue introduced in RC1. I'm currently trying to run the same command (different local path of course) from my private notebook (openSUSE Tumbleweed and 17.04.0-ce) and it works fine.

Will try it also with one of the CTPs.

@SebastianPfliegel
Copy link
Author

ctp2-1 is working fine.

@SebastianPfliegel
Copy link
Author

2 hours ago the new rc2 tag has been pushed. Same issue, though ... :(

@kspearrin
Copy link

kspearrin commented Aug 3, 2017

I am having the same issue (I think).

$ docker run -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=<YourStrong!Passw0rd>' -e 'MSSQL_PID=Express' --cap-add SYS_PTRACE -p 1401:1433 -v c:/data/mysql/data:/var/opt/mssql/data -d microsoft/mssql-server-linux
4aee903227eb2c5831a69e8e7d9f817710f984294810a860d9d24c070e0e5676
$ docker ps
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS                                         NAMES
4aee903227eb        microsoft/mssql-server-linux   "/bin/sh -c /opt/m..."   3 seconds ago       Up 2 seconds        0.0.0.0:1401->1433/tcp                        cranky_rosalind

docker ps shows it running initially, however, after a few seconds I can run docker ps again and it is gone. Not sure where I can get any logs?

No issue when running without the volume. This is docker on windows.

Version 17.06.0-ce-win19 (12801)
Channel: stable
c98c1c2

@SebastianPfliegel
Copy link
Author

After getting the dump the container stops, so it is correct that it is gone. You can find the logs at your mounted volume (c:/data/mysql/data).

@kspearrin
Copy link

Here is what's in my volume:

image

Not sure how to open these though.

@kspearrin
Copy link

kspearrin commented Aug 3, 2017

Here is a log from docker:

$ docker-compose up
Creating network "bwcore_default" with the default driver
Creating mssql-server ...
Creating mssql-server
Attaching to mssql-server
mssql-server    | This is an evaluation version.  There are [173] days left in the evaluation period.
mssql-server    | 2017-08-03 12:44:34.57 Server      The licensing PID was successfully processed. The new edition is [Express Edition].
2017-08-03 12:44:34.80 Server      Microsoft SQL Server 2017 (RC2) - 14.0.900.75 (X64)
mssql-seJul 27 2017 08:53:49
mssql-seCopyright (C) 2017 Microsoft Corporation
mssql-seExpress Edition (64-bit) on Linux (Ubuntu 16.04.2 LTS)
2017-08-03 12:44:34.80 Server      UTC adjustment: 0:00
2017-08-03 12:44:34.80 Server      (c) Microsoft Corporation.
2017-08-03 12:44:34.80 Server      All rights reserved.
2017-08-03 12:44:34.80 Server      Server process ID is 4116.
2017-08-03 12:44:34.81 Server      Logging SQL Server messages in file '/var/opt/mssql/log/errorlog'.
2017-08-03 12:44:34.81 Server      Registry startup parameters:
mssql-se -d /var/opt/mssql/data/master.mdf
mssql-se -l /var/opt/mssql/data/mastlog.ldf
mssql-se -e /var/opt/mssql/log/errorlog
2017-08-03 12:44:34.82 Server      SQL Server detected 1 sockets with 2 cores per socket and 2 logical processors per socket, 2 total logical processors; using 2 logical processors based on SQL Server licensing. This is an informational message; no user action is required.
2017-08-03 12:44:34.82 Server      SQL Server is starting at normal priority base (=7). This is an informational message only. No user action is required.
2017-08-03 12:44:34.82 Server      Detected 3143 MB of RAM. This is an informational message; no user action is required.
2017-08-03 12:44:34.83 Server      Using conventional memory in the memory manager.
2017-08-03 12:44:34.99 Server      Buffer pool extension is already disabled. No action is necessary.
2017-08-03 12:44:35.08 Server      InitializeExternalUserGroupSid failed. Implied authentication will be disabled.
2017-08-03 12:44:35.09 Server      Implied authentication manager initialization failed. Implied authentication will be disabled.
2017-08-03 12:44:35.10 Server      Successfully initialized the TLS configuration. Allowed TLS protocol versions are ['1.0 1.1 1.2']. Allowed TLS ciphers are ['ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!DHE-RSA-AES256-GCM-SHA384:!DHE-RSA-AES128-GCM-SHA256:!DHE-RSA-AES256-SHA:!DHE-RSA-AES128-SHA'].
2017-08-03 12:44:35.13 Server      The maximum number of dedicated administrator connections for this instance is '1'
2017-08-03 12:44:35.13 Server      Node configuration: node 0: CPU mask: 0x0000000000000003:0 Active CPU mask: 0x0000000000000003:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.
2017-08-03 12:44:35.14 Server      Using dynamic lock allocation.  Initial allocation of 2500 Lock blocks and 5000 Lock Owner blocks per node.  This is an informational message only.  No user action is required.
2017-08-03 12:44:35.15 Server      In-Memory OLTP initialized on lowend machine.
2017-08-03 12:44:35.20 Server      Database Instant File Initialization: enabled. For security and performance considerations see the topic 'Database Instant File Initialization' in SQL Server Books Online. This is an informational message only. No user action is required.
2017-08-03 12:44:35.21 Server      Query Store settings initialized with enabled = 1,
2017-08-03 12:44:35.21 spid6s      Starting up database 'master'.
2017-08-03 12:44:35.21 Server      Software Usage Metrics is disabled.
2017-08-03 12:44:35.47 spid6s      Service Master Key could not be decrypted using one of its encryptions. See sys.key_encryptions for details.
2017-08-03 12:44:35.48 spid6s      An error occurred during Service Master Key initialization. SQLErrorCode=33095, State=8, LastOsError=0.
2017-08-03 12:44:35.48 spid6s      Converting database 'master' from version 863 to the current version 869.
2017-08-03 12:44:35.48 spid6s      Database 'master' running the upgrade step from version 863 to version 864.
2017-08-03 12:44:35.49 spid6s      Error: 17053, Severity: 16, State: 1.
2017-08-03 12:44:35.49 spid6s      /var/opt/mssql/data/mastlog.ldf: Operating system error 31(A device attached to the system is not functioning.) encountered.
2017-08-03 12:44:36.50 spid6s      Error: 9002, Severity: 17, State: 0.
2017-08-03 12:44:36.50 spid6s      The transaction log for database 'master' is full due to 'NOTHING'.
2017-08-03 12:44:36.51 spid6s      Error: 928, Severity: 20, State: 1.
2017-08-03 12:44:36.51 spid6s      During upgrade, database raised exception 3602, severity 25, state 53, address 0000000404CD1D8E. Use the exception number to determine the cause.
mssql-server exited with code 255

compose file:

version: '3'

services:
  mssql-server:
    image: microsoft/mssql-server-linux:latest
    container_name: mssql-server
    volumes:
      - c:/data/mssql/data:/var/opt/mssql/data
    environment:
      ACCEPT_EULA=Y
      MSSQL_PID=Express
      SA_PASSWORD=MyPassword123456
    ports:
      - "1433:1433"

@SebastianPfliegel
Copy link
Author

You can only see the databases as you are only mounting the data volume instead of the whole mssql volume: /var/opt/mssql (not /var/opt/mssql/data).

@kspearrin
Copy link

kspearrin commented Aug 3, 2017

New compose file:

version: '3'

services:
  mssql-server:
    image: microsoft/mssql-server-linux:latest
    container_name: mssql-server
    volumes:
      - c:/data/mssql:/var/opt/mssql
    environment:
      ACCEPT_EULA=Y
      MSSQL_PID=Express
      SA_PASSWORD=MyPassword123456
    ports:
      - "1433:1433"

Output from docker-compose up:

$ docker-compose up
Creating network "bwcore_default" with the default driver
Creating mssql-server ...
Creating mssql-server
Attaching to mssql-server
mssql-server    | This is an evaluation version.  There are [173] days left in the evaluation period.
mssql-server    | This program has encountered a fatal error and cannot continue running.
mssql-server    | The following diagnostic information is available:
mssql-server    |
mssql-server    |        Reason: 0x00000006
mssql-server    |        Status: 0x40000015
mssql-server    |       Message: Kernel bug check
mssql-server    |       Address: 0x6a42b4b0
mssql-server    |    Parameters: 0x6a62a250
   Stacktrace: 000000006a4e8446 000000006a42b50b 000000006a41ecad
mssql-server    |                000000006a42aeab 000000006a4e69dd 000000006a4e5a1c
mssql-server    |                000000006a4e5890 000000006a4e57c5
      Process: 7 - sqlservr
mssql-server    |        Thread: 11 (application thread 0x1000)
mssql-server    |   Instance Id: ea74a1c5-68e0-4080-bfcf-5c0225e813fa
mssql-server    |      Crash Id: 524793ab-9d77-4d48-aefe-b032583c2dd6
mssql-server    |   Build stamp: a37664e45e4156e76a53fa282fd694cb49f70c2037515f5684e3ce6dfa7549bc
mssql-server    |
mssql-server    | Capturing core dump and information...
mssql-server    | dmesg: read kernel buffer failed: Operation not permitted
mssql-server    | No journal files were found.
mssql-server    | No journal files were found.
mssql-server    | Attempting to capture a dump with paldumper
mssql-server    | WARNING: Capture attempt failure detected
mssql-server    | Attempting to capture a filtered dump with paldumper
mssql-server    | WARNING: Attempt to capture dump failed.  Reference /var/opt/mssql/log/core.sqlservr.7.temp/log/paldumper-debug.log for details
mssql-server    | Attempting to capture a dump with gdb
mssql-server    | WARNING: Unable to capture crash dump with GDB. You may need to
mssql-server    | allow ptrace debugging, enable the CAP_SYS_PTRACE capability, or
mssql-server    | run as root.
mssql-server exited with code 1

Here's a log from c:/data/mssql/log/core.sqlservr.08_03_2017_13_19_14.7.txt

This program has encountered a fatal error and cannot continue running.
The following diagnostic information is available:

       Reason: 0x00000006
       Status: 0x40000015
      Message: Kernel bug check
      Address: 0x6a42b4b0
   Parameters: 0x6a62a250

               
               
               
               
               
   Stacktrace: 000000006a4e8446 000000006a42b50b 000000006a41ecad 
               000000006a42aeab 000000006a4e69dd 000000006a4e5a1c 
               000000006a4e5890 000000006a4e57c5 

      Process: 7 - sqlservr
       Thread: 11 (application thread 0x1000)
  Instance Id: ea74a1c5-68e0-4080-bfcf-5c0225e813fa
     Crash Id: 524793ab-9d77-4d48-aefe-b032583c2dd6
  Build stamp: a37664e45e4156e76a53fa282fd694cb49f70c2037515f5684e3ce6dfa7549bc

Looks like the same problem.

Logs:
log.zip

@kspearrin
Copy link

In the meantime, this seems to work fine.

version: '3'

services:
  mssql-server:
    image: microsoft/mssql-server-linux:latest
    container_name: mssql-server
    volumes:
      - mssql_server_data:/var/opt/mssql/data
    environment:
      ACCEPT_EULA=Y
      MSSQL_PID=Express
      SA_PASSWORD=MyPassword123456
    ports:
      - "1433:1433"

volumes:
  mssql_server_data:

So it appears that this is only a problem when mounting the volume from the host machine.

@SebastianPfliegel
Copy link
Author

And only on Docker for Windows 10. Works splendid on openSUSE as host system.

@benze
Copy link

benze commented Aug 11, 2017

I have the same problem and have reverted to tag ctp2-1, which is working fine.

@SebastianPfliegel
Copy link
Author

@benze: what host system are you running? Local mount or network share?

@dweggemans
Copy link

dweggemans commented Sep 5, 2017

I have this problem as well. It used to work fine like a week ago. Since then a new Docker version was released. I have no idea what version that was though.

Currently on Docker for windows version 17.06.1-ce, build 874a737
Also tried version 17.07.0-ce build 8784753 which doesn't work either
The volume is local.
Edit: no luck with Docker version 17.06.2-ce, build cec0b72 either

I'd appreciate a fix or workaround as reverting to ctp2-1 (on which volumes currently do work) is not really a improvement for me. Recreating and seeding the databases is actually faster on rc2 than remounting on ctp2-1.

@benze
Copy link

benze commented Sep 6, 2017

@aerkefiende: I'm running in a Win 10 box, with local mount drives. Running Docker Engine 17.06.1-ce-win24 (13025). Using Docker for Linux Containers.

@clydeigc
Copy link

Same problem encountered on Win10, Version 17.06.2-ce-win27 (13194), LinuxVM

@twright-msft
Copy link
Collaborator

This is a known bug. It is fixed in the RTM/GA release which will be available soon. Alternatives for now: Use :ctp2-1 or email me for a special preview image. Please only email me if this is a significant problem for you and you can't use :ctp2-1 for some reason. It takes some effort on my side to be able to set you up to pull the special preview image from our private preview registry. Email address is twright @ microsoft Sorry for the inconvenience!

@sangohan
Copy link

sangohan commented Oct 3, 2017

now that 2017-GA is available (and the old ctp2-1 tag is deleted), this issue is resurfacing - it hasn't been fixed at all.

[INFO] DOCKER> [microsoft/mssql-server-linux:2017-GA] "Microsoft SQL Server-confluence": Start container 66f49e17402a
2017-10-03 07:58:26.07 Server      Setup step is copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf'.
2017-10-03 07:58:26.12 Server      Setup step is copying system data file 'C:\templatedata\mastlog.ldf' to '/var/opt/mssql/data/mastlog.ldf'.
2017-10-03 07:58:26.13 Server      Setup step is copying system data file 'C:\templatedata\model.mdf' to '/var/opt/mssql/data/model.mdf'.
2017-10-03 07:58:26.14 Server      Setup step is copying system data file 'C:\templatedata\modellog.ldf' to '/var/opt/mssql/data/modellog.ldf'.
2017-10-03 07:58:26.16 Server      Setup step is copying system data file 'C:\templatedata\msdbdata.mdf' to '/var/opt/mssql/data/msdbdata.mdf'.
2017-10-03 07:58:26.19 Server      Setup step is copying system data file 'C:\templatedata\msdblog.ldf' to '/var/opt/mssql/data/msdblog.ldf'.
2017-10-03 07:58:26.26 Server      Microsoft SQL Server 2017 (RTM) - 14.0.1000.169 (X64)
Aug 22 2017 17:04:49
Copyright (C) 2017 Microsoft Corporation
Developer Edition (64-bit) on Linux (Ubuntu 16.04.3 LTS)
2017-10-03 07:58:26.26 Server      UTC adjustment: 0:00
2017-10-03 07:58:26.26 Server      (c) Microsoft Corporation.
2017-10-03 07:58:26.26 Server      All rights reserved.
2017-10-03 07:58:26.26 Server      Server process ID is 4120.
2017-10-03 07:58:26.26 Server      Logging SQL Server messages in file '/var/opt/mssql/log/errorlog'.
2017-10-03 07:58:26.26 Server      Registry startup parameters:
-d /var/opt/mssql/data/master.mdf
-l /var/opt/mssql/data/mastlog.ldf
-e /var/opt/mssql/log/errorlog
2017-10-03 07:58:26.27 Server      SQL Server detected 1 sockets with 2 cores per socket and 4 logical processors per socket, 4 total logical processors; using 4 logical  user action is required.
2017-10-03 07:58:26.27 Server      SQL Server is starting at normal priority base (=7). This is an informational message only. No user action is required.
2017-10-03 07:58:26.27 Server      Detected 5984 MB of RAM. This is an informational message; no user action is required.
2017-10-03 07:58:26.27 Server      Using conventional memory in the memory manager.
2017-10-03 07:58:26.50 Server      Buffer pool extension is already disabled. No action is necessary.
2017-10-03 07:58:26.66 Server      InitializeExternalUserGroupSid failed. Implied authentication will be disabled.
2017-10-03 07:58:26.66 Server      Implied authentication manager initialization failed. Implied authentication will be disabled.
2017-10-03 07:58:26.66 Server      Successfully initialized the TLS configuration. Allowed TLS protocol versions are ['1.0 1.1 1.2']. Allowed TLS ciphers are 8-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES-RSA-AES128-GCM-SHA256:!DHE-RSA-AES256-SHA:!DHE-RSA-AES128-SHA'].
2017-10-03 07:58:26.70 Server      The maximum number of dedicated administrator connections for this instance is '1'
2017-10-03 07:58:26.70 Server      Node configuration: node 0: CPU mask: 0x000000000000000f:0 Active CPU mask: 0x000000000000000f:0. This message provides a description of the o user action is required.
2017-10-03 07:58:26.71 Server      Using dynamic lock allocation.  Initial allocation of 2500 Lock blocks and 5000 Lock Owner blocks per node.  This is an informational message 
2017-10-03 07:58:26.71 Server      In-Memory OLTP initialized on lowend machine.
2017-10-03 07:58:26.76 Server      Database Instant File Initialization: enabled. For security and performance considerations see the topic 'Database Instant File Initialization' ion is required.
2017-10-03 07:58:26.76 Server      Query Store settings initialized with enabled = 1,
2017-10-03 07:58:26.77 spid6s      Starting up database 'master'.
2017-10-03 07:58:26.77 Server      Software Usage Metrics is disabled.
2017-10-03 07:58:26.95 spid6s      The tail of the log for database master is being rewritten to match the new sector size of 4096 bytes.  3072 bytes at offset 418816 in file /var/
2017-10-03 07:58:27.04 spid6s      Converting database 'master' from version 862 to the current version 869.
2017-10-03 07:58:27.04 spid6s      Database 'master' running the upgrade step from version 862 to version 863.
2017-10-03 07:58:27.07 spid6s      Database 'master' running the upgrade step from version 863 to version 864.
2017-10-03 07:58:27.08 spid6s      Error: 17053, Severity: 16, State: 1.
2017-10-03 07:58:27.08 spid6s      /var/opt/mssql/data/mastlog.ldf: Operating system error 31(A device attached to the system is not functioning.) encountered.
2017-10-03 07:58:28.09 spid6s      Error: 928, Severity: 20, State: 1.
2017-10-03 07:58:28.09 spid6s      During upgrade, database raised exception 9002, severity 17, state 0, address 0000000484D726DE. Use the exception number to determine the cause.
2017-10-03 07:58:28.09 spid6s      Error: 9002, Severity: 17, State: 0.
2017-10-03 07:58:28.09 spid6s      The transaction log for database 'master' is full due to 'NOTHING'.

@dweggemans
Copy link

Volume mappings are working again. I just tried both GA and latest and can confirm that the issue is fixed.

@jertel
Copy link

jertel commented Oct 3, 2017

I'm running into this problem now, after being forced to upgrade to 2017-GA since the CTP2 tags are no longer hosted on docker hub. Same error as sangohan posted a few hours ago.

I found that if I explicitly mount a local directory into the /var/opt/mssql container location then the problem goes away. So it looks like I have a workaround. For those that might be wondering why I didn't already specify a mount point, it's because I use this container for automated testing so there was never a need to persist the test data once the container shutdown.

Also, the problem did not occur locally on Docker for Mac. However it was consistently happening when running on a Local SSD-backed EC2 instance.

@twright-msft
Copy link
Collaborator

OK, so is this issue resolved for you @jertel and @sangohan?

@SebastianPfliegel
Copy link
Author

@twright-msft: works like a charm for me!

Now I only need the option to create a database at startup and I'm happy.

@sangohan
Copy link

sangohan commented Oct 6, 2017

@jertel's workaround seems to work. I am still testing it out on the EC2 agents i use.

@SebastianPfliegel
Copy link
Author

Closing the issue as noone complained so far :)

@benze
Copy link

benze commented Oct 24, 2017

I'm actually still using the ctp-2 container I have on my machine for the next couple of weeks; I can't afford the time right now to validate that the final version works properly.

I'll post my findings as soon as I test it.

Does the final release have the same 180 day eval license restrictions, or has that changed?

@twright-msft
Copy link
Collaborator

The General Availability (GA) release is not timebombed. If you choose MSSQL_PID=Evaluation it will be limited to 180 days, but realistically I dont know why you would use that. If you want to just do demo, eval, testing, dev you can use MSSQL_PID=Developer (which is the default if you don't specify MSSQL_PID).

@Artgit
Copy link

Artgit commented Oct 24, 2017

I ran into exactly the same issue when updated to the latest release a few days ago. My container loses data during restarts. Is there any way to get the fixed image or some previous tag without this issue?

@staff0rd
Copy link

staff0rd commented Apr 9, 2018

Occurs on other mounts as already reported;

1> restore database zapmoney from disk ='/data/zapmoney.1804.bak' with move 'zapmoney' to '/data/zapmoney.mdf', move 'zapmoney_log' to '/data/zapmoney_log.ldf'
2> go
Msg 5149, Level 16, State 3, Server mssql, Line 1
MODIFY FILE encountered operating system error 31(A device attached to the system is not functioning.) while attempting to expand the physical file '/data/zapmoney.mdf'.
Msg 3013, Level 16, State 1, Server mssql, Line 1
RESTORE DATABASE is terminating abnormally.

Windows 10, v1709, 16299.309
Docker version 18.03.0-ce, build 0520e24
NTFS

@vinhboy
Copy link

vinhboy commented Apr 16, 2018

Had the same problem as @staff0rd above using RESTORE DATABASE and WITH MOVE

Solved it by deleting the images and re-installing it.

Host OS type (Windows, Linux incl. which distro): Mac OS X
Host OS version: High Sierra 10.13.4
Docker version: 18.03.0-ce-mac60
File system: APFS

@anikammsft
Copy link

A fix for the SQL startup issue on Docker for Windows when /var/opt/mssql is mounted is scheduled to be released in CU7.

Specifically, this will fix the issue that shows the following in errorlog:

2018-01-10 19:40:40.17 spid6s Database 'master' running the upgrade step from version 862 to version 863.
2018-01-10 19:40:40.21 spid6s Database 'master' running the upgrade step from version 863 to version 864.
2018-01-10 19:40:40.25 spid6s Error: 17053, Severity: 16, State: 1.
2018-01-10 19:40:40.25 spid6s /var/opt/mssql/data/mastlog.ldf: Operating system error 31(A device attached to the system is not functioning.) encountered.

@twright-msft
Copy link
Collaborator

CU7 is the monthly cumulative update release that is scheduled to come out in the middle of next month (May).

@staff0rd
Copy link

@anikammsft @twright-msft any issue or commit links to satisfy my curiosity?

@EmilEriksen
Copy link

I just spent the better part of the day dealing with "Error: 17053/Operating system error 31" when trying to restore a backup on macOS High Sierra 10.13.4. I tried a bunch of solutions but in the end resizing my Docker image size from 16 GB to 32 GB fixed the issue.

@mhellsten
Copy link

I have the same problem with CU11.

Host OS: Ubuntu 18.04 with VirtualBox 5.2.18
Guest OS: Ubuntu 18.04
Docker: 17.12.1-ce

When mounting a volume pointing to a VirtualBox Shared Folder that's hosted on the host machine, SQL Server crashes, even though it's able to create a number of files before doing so.

Mounting volume pointing to a folder on the guest OS instead works, but is not a practical setup as I'd like to have minimal data on the container hosts.

@MagellanTX
Copy link

I have the same issue with CU12. If I mount a volume to CIFS or NFS then SQL crashes after a few minutes. Different error though:

Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Cloud.Sql.Analytics.BoxTelemetry.Util.GetSqmId()
   at Microsoft.Cloud.Sql.Analytics.BoxTelemetry.MachineInfo..ctor(Util util)
   at Microsoft.Cloud.Sql.Analytics.BoxTelemetry.BoxTelemetry.TelemetryCollectorMain(BoxTelemetryOptions options)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Host OS: ESXi 6.0U3
Guest OS: Ubuntu 18.04LTS
Docker Version: 18.09.0-CE

Like some of the other comments, it creates the directory structure just fine in the persistent storage and I can even use the SQL server (until it crashes).

I've tried getting the image from Docker Hub and MCR but both have the same issue. If I remove the persistent volume, the server runs normally.

@MoeTheCoder
Copy link

I'm having the except same issue as @MagellanTX. SQL crashes when using a mounted volume (same error msg). SQL works fine when not used with the volume.

Host OS: Win 10
Docker version: 18.09.0
SQL version: 2017-latest

@Newsfan
Copy link

Newsfan commented Feb 11, 2019

I have the same issue. SQL crashes when using a mounted volume on host machine, but works fine when the volume is on the virtual machine.
Host OS: win 7 enterprise
Docker version : 18.03.0-ce
SQL version: 2017-latest-ubuntu
File system: NTFS

@mirsaeedi
Copy link

mirsaeedi commented Jul 23, 2019

I have this problem too with mcr.microsoft.com/mssql/server:2017-latest and mcr.microsoft.com/mssql/server:2019-CTP3.1-ubuntu on Windows Server 2019 version 1809.

I cannot mount a data volume using the -v flag. docker logs give me following error:

(2017-CU11-ubuntu does not have this problem though)

This program has encountered a fatal error and cannot continue running at Tue Jul 23 17:30:19 2019
The following diagnostic information is available:

       Reason: 0x00000006
      Message: Kernel bug check
      Address: 0x6b448190
   Parameters: 0x10861f670
   Stack Trace:
               000000006b5344ce
               000000006b4481eb
               000000006b4344f6
               000000006b443622
               000000006b5327fc
               000000006b531019
               000000006b575331
        Process: 7 - sqlservr
         Thread: 11 (application thread 0x4)
    Instance Id: 43329e1c-dbf3-42a8-a284-44fbea6ae8c8
       Crash Id: 56a5abba-136b-42c7-9ab2-91dbec5b08ed
    Build stamp: 4a92842108074d858c9f7d8bbbc88360b2a93d7fe17df31bdef8b95495870f81
   Distribution: Ubuntu 16.04.6 LTS
     Processors: 3
   Total Memory: 6227410944 bytes
      Timestamp: Tue Jul 23 17:30:19 2019
     last_errno: 2
last_errno_text: No such file or directory

Ubuntu 16.04.6 LTS
Capturing core dump and information to /var/opt/mssql/log...
dmesg: read kernel buffer failed: Operation not permitted
No journal files were found.
No journal files were found.
Tue Jul 23 17:30:19 UTC 2019 Capturing program information
Tue Jul 23 17:30:20 UTC 2019 Attempting to capture a dump with paldumper
WARNING: Capture attempt failure detected
Attempting to capture a filtered dump with paldumper
WARNING: Attempt to capture dump failed.  Reference /var/opt/mssql/log/core.sqlservr.7.temp/log/paldumper-debug.log for details
Tue Jul 23 17:30:20 UTC 2019 Attempting to capture a dump with gdb
WARNING: Unable to capture crash dump with GDB. You may need to
allow ptrace debugging, enable the CAP_SYS_PTRACE capability, or
run as root.
Tue Jul 23 17:30:20 UTC 2019 Capturing program binaries
Tue Jul 23 17:30:20 UTC 2019 Compressing the dump files

@sirio3mil
Copy link

Same for me, but I can use mcr.microsoft.com/mssql/server:latest but none of 2019 images.

Reported here

@M0nsieurChat
Copy link

Hello, problem solved by upgrading our linux kernel (we applied 5.2)

@Maxxer1
Copy link

Maxxer1 commented Aug 9, 2019

Docker Toolbox on Win7 problem still occurs. My compose file:

services:
mssql1:
container_name: "mssql1"
volumes:
- ./db:/var/opt/mssql
image: "mcr.microsoft.com/mssql/server:2017-latest-ubuntu"
environment:
ACCEPT_EULA: "Y"
SA_PASSWORD: "Admin!Admin"
ports:
- "1434:1433"

     Reason: 0x00000006
    Message: Kernel bug check
    Address: 0x6c047d10
 Parameters: 0x10861f590
Stack Trace:
             000000006c1345c6
             000000006c047d6b
             000000006c03451e
             000000006c043015
             000000006c0431e6
             000000006c132938
             000000006c1314ef
             000000006c1747c1
    Process: 9 - sqlservr
     Thread: 13 (application thread 0x4)
Instance Id: 83ce0eef-5575-4d63-bd21-d52417f5ba9f
   Crash Id: 970c988e-2e1a-4cf4-88d8-1f146d1e6d86
Build stamp: fad5947c34c3f4acfa527fa10979e5c9b3c7dc2811fe1019283ffcb88e85c5c1
Distribution: Ubuntu 16.04.6 LTS
 Processors: 4

@simonpinn
Copy link

Same issue here, running docker desktop 2.1.1.0 on Windows 10 1803 when using a volume I have the following log output:

This program has encountered a fatal error and cannot continue running at Wed Sep  4 01:22:07 2019
The following diagnostic information is available:

         Reason: 0x00000006
        Message: Kernel bug check
        Address: 0x6b047d10
     Parameters: 0x10861f590
    Stack Trace:
                 000000006b1345c6
                 000000006b047d6b
                 000000006b03451e
                 000000006b043015
                 000000006b0431e6
                 000000006b132938
                 000000006b1314ef
                 000000006b1747c1
        Process: 7 - sqlservr
         Thread: 11 (application thread 0x4)
    Instance Id: d2d13881-30cb-4e05-89d3-bf9ea3f5fee4
       Crash Id: dc9088e2-e486-4aae-975b-12e9298b3744
    Build stamp: fad5947c34c3f4acfa527fa10979e5c9b3c7dc2811fe1019283ffcb88e85c5c1
   Distribution: Ubuntu 16.04.6 LTS
     Processors: 2
   Total Memory: 8329093120 bytes
      Timestamp: Wed Sep  4 01:22:07 2019
     Last errno: 2
Last errno text: No such file or directory

@M0nsieurChat - how did you update the kernel?

@stevo-knievo
Copy link

stevo-knievo commented Sep 15, 2019

Same issue here, running docker desktop 2.1.0.2 on macOS 10.14.6 when using a volume I have the following log output:

The following diagnostic information is available:

         Reason: 0x00000006
        Message: Kernel bug check
        Address: 0x6ba47d10
     Parameters: 0x10861f590
    Stack Trace:
                 000000006bb345c6
                 000000006ba47d6b
                 000000006ba3451e
                 000000006ba43015
                 000000006ba431e6
                 000000006bb32938
                 000000006bb314ef
                 000000006bb747c1
        Process: 8 - sqlservr
         Thread: 12 (application thread 0x4)
    Instance Id: d9e59cb2-1d71-40cf-a198-0a75120b7c44
       Crash Id: 358267a4-191d-4ae1-a431-dc895b29a3ed
    Build stamp: fad5947c34c3f4acfa527fa10979e5c9b3c7dc2811fe1019283ffcb88e85c5c1
   Distribution: Ubuntu 16.04.6 LTS
     Processors: 6
   Total Memory: 2095681536 bytes
      Timestamp: Sun Sep 15 18:59:10 2019
     Last errno: 2
Last errno text: No such file or directory

Here my docker-compse file:

version: "3"
services:
  db:
    container_name: mssql-db-server
    image: mcr.microsoft.com/mssql/server:2017-latest-ubuntu
    ports:
      - "1433:1433"
    volumes:
      - ./data:/var/opt/mssql
    environment:
      SA_PASSWORD: "Your_password123" 
      ACCEPT_EULA: "Y"

@tmpbin
Copy link

tmpbin commented Nov 16, 2019

See this:
https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-configure-docker?view=sql-server-ver15#mount-a-host-directory-as-data-volume

Host volume mapping for Docker on Mac with the SQL Server on Linux image is not supported at this time. Use data volume containers instead. This restriction is specific to the /var/opt/mssql directory. Reading from a mounted directory works fine. For example, you can mount a host directory using -v on Mac and restore a backup from a .bak file that resides on the host.

@hades200082
Copy link

I've got this issue using the following run command...

docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=MyStr0ngP@ssw0rd" -e "MSSQL_PID=Developer" --name "MSSQL1" -v D:\DockerMounts\MSSQL:/var/opt/mssql -p 1433:1433 -d mcr.microsoft.com/mssql/server:2017-latest

Host OS: Windows 10 (all up to date)
Docker version : Docker Desktop 2.1.0.5, Engine 19.03.5
SQL version: mcr.microsoft.com/mssql/server:2017-latest
File system: NTFS

Log in mounted folder shows:

This program has encountered a fatal error and cannot continue running at Wed Dec  4 15:11:58 2019
The following diagnostic information is available:

         Reason: 0x00000006
        Message: Kernel bug check
        Address: 0x6b447d60
     Parameters: 0x10861f590
    Stack Trace:
                 000000006b53539e
                 000000006b447dbb
                 000000006b43447e
                 000000006b443025
                 000000006b4431f6
                 000000006b53366c
                 000000006b5321df
                 000000006b5755d1
        Process: 8 - sqlservr
         Thread: 12 (application thread 0x4)
    Instance Id: e20894a2-8425-4e33-9cba-8d2cbbfd4f80
       Crash Id: 1daf583c-2dc9-4b8d-967f-b98b676359ca
    Build stamp: 80fb92dd4d876182c7b8d6d599e31b3f6d91efe87f8c1bb6540962021ef4b921
   Distribution: Ubuntu 16.04.6 LTS
     Processors: 2
   Total Memory: 2096136192 bytes
      Timestamp: Wed Dec  4 15:11:58 2019
     Last errno: 2
Last errno text: No such file or directory

@VishalK-NewcrestImage
Copy link

In the meantime, this seems to work fine.

version: '3'

services:
  mssql-server:
    image: microsoft/mssql-server-linux:latest
    container_name: mssql-server
    volumes:
      - mssql_server_data:/var/opt/mssql/data
    environment:
      ACCEPT_EULA=Y
      MSSQL_PID=Express
      SA_PASSWORD=MyPassword123456
    ports:
      - "1433:1433"

volumes:
  mssql_server_data:

So it appears that this is only a problem when mounting the volume from the host machine.

What if I want to fire multiple container for multiple ms sql server? Then this code does not seem to work.

@suseu
Copy link

suseu commented Jul 9, 2020

See this:
https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-configure-docker?view=sql-server-ver15#mount-a-host-directory-as-data-volume

Host volume mapping for Docker on Mac with the SQL Server on Linux image is not supported at this time. Use data volume containers instead. This restriction is specific to the /var/opt/mssql directory. Reading from a mounted directory works fine. For example, you can mount a host directory using -v on Mac and restore a backup from a .bak file that resides on the host.

As stated in the above link:

Host volume mapping for Docker on Windows does not currently support mapping the complete /var/opt/mssql directory. However, you can map a subdirectory, such as /var/opt/mssql/data to your host machine.

    volumes:
      - D:/Server/MSSQL/data:/var/opt/mssql/data
      - D:/Server/MSSQL/log:/var/opt/mssql/log
      - D:/Server/MSSQL/secrets:/var/opt/mssql/secrets
      - D:/Server/MSSQL/backups:/var/opt/mssql/backups

This volumes work fine.

@petermicuch
Copy link

@suseu does that really work for you? I tried mounting data, log, secrets and even data directory on its own instead of whole mssql directory on my Docker for Windows, but I am still getting error. Without volumes used, this works fine, but nothing is persisted of course. So that comment on MS site seems pretty misleading unless I am doing something wrong. I even use init container in my k8s deployment to set correct owner of the data directory (command: ["sh", "-c", "chown -R 10001:0 /var/opt/mssql/data"])

SQL Server 2019 will run as non-root by default.
This container is running as user mssql.
Your master database file is owned by mssql.
To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
2020-08-10 13:34:26.88 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_replicatedmaster.mdf' to '/var/opt/mssql/data/model_replicatedmaster.mdf'.
2020-08-10 13:34:26.97 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_replicatedmaster.ldf' to '/var/opt/mssql/data/model_replicatedmaster.ldf'.
2020-08-10 13:34:26.97 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_msdbdata.mdf' to '/var/opt/mssql/data/model_msdbdata.mdf'.
2020-08-10 13:34:26.99 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_msdblog.ldf' to '/var/opt/mssql/data/model_msdblog.ldf'.
2020-08-10 13:34:27.12 Server      Microsoft SQL Server 2019 (RTM-CU3) (KB4538853) - 15.0.4023.6 (X64)
        Mar  4 2020 00:59:26
        Copyright (C) 2019 Microsoft Corporation
        Developer Edition (64-bit) on Linux (Ubuntu 18.04.4 LTS) <X64>
2020-08-10 13:34:27.13 Server      UTC adjustment: 0:00
2020-08-10 13:34:27.13 Server      (c) Microsoft Corporation.
2020-08-10 13:34:27.13 Server      All rights reserved.
2020-08-10 13:34:27.14 Server      Server process ID is 44.
2020-08-10 13:34:27.14 Server      Logging SQL Server messages in file '/var/opt/mssql/log/errorlog'.
2020-08-10 13:34:27.14 Server      Registry startup parameters:
         -d /var/opt/mssql/data/master.mdf
         -l /var/opt/mssql/data/mastlog.ldf
         -e /var/opt/mssql/log/errorlog
2020-08-10 13:34:27.15 Server      Error: 17113, Severity: 16, State: 1.
2020-08-10 13:34:27.15 Server      Error 87(The parameter is incorrect.) occurred while opening file '/var/opt/mssql/data/master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.

@petermicuch
Copy link

To be fair, I just tried this using command: docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=MyStrong#Password" -p 1433:1433 -v D:/Data:/var/opt/mssql --name mssql -d mcr.microsoft.com/mssql/server:2017-latest and it works.

What does not work for me is running this in kubernetes cluster on the Docker Desktop for Windows. There I actually mount things like in template below, which I thought should work just like when I execute docker command:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mssql-database
  namespace: default
  labels:
    app: mssql
    track: canary
spec:
  serviceName: mssql-database-headless
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: mssql
      role: master
  template:
    metadata:
      name: mssql-database
      labels:
        app: mssql
        track: canary
        role: master
    spec:
      containers:
      - name: mssql-database
        image: mcr.microsoft.com/mssql/server:2017-latest
        imagePullPolicy: IfNotPresent
        env:
        - name: ACCEPT_EULA
          value: "Y"   
        - name: SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql-db-secret
              key: password
        - name: MSSQL_PID
          value: Developer
        ports:
        - name: mssqlport
          containerPort: 1433
        volumeMounts:
        - name: mssql-database-data
          mountPath: /var/opt/mssql/data
      initContainers:
      - name: mssql-change-owner
        image: busybox:latest
        command: ["sh", "-c", "chown -R 10001:0 /var/opt/mssql/data"]
        volumeMounts:
        - name: mssql-database-data
          mountPath: /var/opt/mssql/data
      volumes:
      - name: mssql-database-data
        hostPath:
          path: /opt/virtual-clinic-mssql/data
          type: DirectoryOrCreate

No matter if I use SQL server 2017 or 2019 image. I always get error as in previous post. Only when I remove volumes, then this works fine, but without persistency in place.

In the cluster that is also hosted in the Hyper-V on windows server 2016 on multiple RedHat VMs, this template works fine as well as in Azure managed k8s cluster. So I wonder what might be the issue in my case with Docker Desktop for Windows.

@petermicuch
Copy link

In case anyone will encounter the same issue as myself, here is solution that worked for me.

I have noticed that Docker Desktop for Windows by default pushes all volumes into /var/lib/k8s-pvs/. So I set path in my volume to /var/lib/k8s-pvs/virtual-clinic-mssql/data instead of /opt/virtual-clinic-mssql/data and the template from my previous post worked fine, there is no error anymore. I also tried with /var/lib/virtual-clinic-mssql/data and it worked. I tried several other directories outside of /var/lib (e.g. /tmp or /opt), but without success. This indicates permission issues, so I will try to follow with MS folks.

@cezarlamann
Copy link

I'm encountering the same issue with the following setup:
docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=Dev#0110" -p 1433:1433 -v "sql1volume:/var/opt/mssql/data" -d --name sql1 mcr.microsoft.com/mssql/server:2019-CU6-ubuntu-18.04
When I hit docker logs -f sql1 I get the following error:

SQL Server 2019 will run as non-root by default.
This container is running as user mssql.
To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
2020-08-12 03:49:54.94 Server      Setup step is copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf'.
2020-08-12 03:49:55.06 Server      ERROR: Setup FAILED copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf':  2(The system cannot find the file specified.)
ERROR: BootstrapSystemDataDirectories() failure (HRESULT 0x80070002)

I'm running Xubuntu 20.04 as host (kernel 5.4.0-42-generic), Docker version 19.03.12, build 48a66213fe from the official repo. Regarding the double quotes on volume: it doesn't matter if I run the command with or without them.

Funny thing is, when I run the same line on Windows, it works. However I don't know exactly what is the docker version over there, because I don't have access to the box at the moment. What I'm sure is that on Windows, I run Windows 10 2004 with Docker on WSL 2 (Ubuntu 20.04)

@LeonardoHabitzreuter
Copy link

Using docker-compose worked for me!

version: '3'
services:
db:
user: root
image: mcr.microsoft.com/mssql/server:2019-latest
container_name: sqlserver
ports:
- 1433:1433
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=some(!)Password
volumes:
- ./volumes/data:/var/opt/mssql/data

@martinandersen3d
Copy link

martinandersen3d commented Jan 19, 2021

SOLUTION:

Thanks @LeonardoHabitzreuter it works

Adding the user: root part works for me.

Running ubuntu on host.

This is my file:

docker-compose.yml

version: "3.7"
services:
  sqlserver:
    image: mcr.microsoft.com/mssql/server:2019-latest
    container_name: sqlserver
    user: root   #####################   <<<<<<<<< IMPORTANT!!!!!  ############
    ports:
    #   host-port:container-port 
      - "1433:1433"
    environment:
      # SA_USERNAME: sa
      SA_PASSWORD: Dev1234!
      ACCEPT_EULA: Y
    volumes:
      - ./volumes/data2:/var/opt/mssql/data

Thanks, has been searhing for hours

@she-trifork
Copy link

she-trifork commented Jan 31, 2024

Don't run your containers as root. Running as root is not a fix, but a dirty hack you can hurt yourself on.

Instead, create the required directories on the volume with proper permissions. I find that this pattern works nicely, though it does make some assumptions about the UID/GID within the container, which is a problem if they change.

I've also added a silly healthcheck because I need the status for depends_on elsewhere.

services:

  mssql:
    image: mcr.microsoft.com/mssql/server:2022-latest
    container_name: mssql
    healthcheck:
      test: pgrep -fx /opt/mssql/bin/sqlservr
      start_period: 10s
    environment:
      - ACCEPT_EULA=no
      - MSSQL_SA_PASSWORD=secret
    ports:
      - 1433:1433
    volumes:
      - mssql-storage:/var/opt/mssql
    depends_on:
      mssql-storage-fix-permissions:
        condition: service_completed_successfully
    profiles:
      - db

  mssql-storage-fix-permissions:
    image: busybox:stable-musl
    command: install -o 10001 -g 10001 -m 775 -d /var/opt/mssql/data -d /var/opt/mssql/log
    volumes:
      - mssql-storage:/var/opt/mssql

volumes:
  mssql-storage:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests