Skip to content
This repository has been archived by the owner on Nov 3, 2024. It is now read-only.

Add support for Microsoft SQL Server #168

Merged
merged 30 commits into from
May 11, 2022
Merged

Add support for Microsoft SQL Server #168

merged 30 commits into from
May 11, 2022

Conversation

drupal-daffie
Copy link
Contributor

@drupal-daffie drupal-daffie commented Oct 19, 2021

The New Solution/Problem/Issue/Bug:

Add support for the Microsoft SQL Server

How this PR Solves The Problem:

This PR add that support

Manual Testing Instructions:

Follow the comments in the readme file and the docker-compose file.

Related Issue Link(s):

Copy link
Member

@rfay rfay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this! It's a great addition. I haven't actually tried it out yet, but will on the next round. If you know other people that use sqlsrv it would be great to get them to review also.

docker-compose-services/sqlsrv/README.md Outdated Show resolved Hide resolved
docker-compose-services/sqlsrv/README.md Outdated Show resolved Hide resolved
```
Host: sqlsrv
User: sa
Password: password!
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the mysql and postgres db setups use db as username and password, could we use that here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We cannot use the password "db" as it would fail the restrictions set by SQL Server. See: https://docs.microsoft.com/en-us/sql/relational-databases/security/password-policy?view=sql-server-ver15

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, but could use the username 'db' true?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default user on SQL Server is the user "sa". I think it stands for "system administrator", a.k.a. the root user on linux. An other user needs to be created with the SQL string "CREATE USER 'db' WITH PASSWORD 'db'". Is that what you would like to do? In this way we can also create the database "db".

Host: sqlsrv
User: sa
Password: password!
Database: master
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it important for it to be named "master"? Again, the default mysql and postgres setups use "db" for the name of the primary database.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not need to be master, it is only the default database for SQL Server. The "db" database needs to be created.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah if 'master' is default database then it makes sense to use it.


## Installing the PHP extensions

The PHP extensions for SQL Server CANNOT be installed by adding them to the `webimage_extra_packages` setting. The problem is that they are not available as a Debian or any other distribution package. The 2 extensions (`sqlsrv` and `pdo_sqlsrv`) need to be compiled and this needs to be done after PHP is installed on the webimage. The following commands need to be copied to the end of the main `config.yaml` file:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do this with a .ddev/web-build/Dockerfile instead - it's far more efficient (only has to be done once, instead of on every ddev start. It's also easier to understand and maintain. https://ddev.readthedocs.io/en/stable/users/extend/customizing-images/#adding-extra-dockerfiles-for-webimage-and-dbimage

Copy link
Contributor Author

@drupal-daffie drupal-daffie Oct 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tried to do that, only when I did that there was no PHP installed yet, therefor it failed. Maybe I did something wrong or there is a way to do it after PHP is installed. For me, I was fantastic that I got it to work in the first place. It took me 2 weeks to get it to work. I just wanted to fix some bugs in the Drupal module sqlsrv.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will be happy to help with this of course. And appreciate you sharing the work that you did!

Copy link
Contributor Author

@drupal-daffie drupal-daffie Oct 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main thing for me was to make it a lot easier for the people after me that would like to use DDEV with SQL Server. For some reason the PHP extension for SQL Server are not available as packages. Really annoying!
I appreciate your help with this very much!

docker-compose-services/sqlsrv/README.md Outdated Show resolved Hide resolved
docker-compose-services/sqlsrv/README.md Outdated Show resolved Hide resolved
- sqlsrv:sqlsrv

volumes:
sqlsystem:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ddev delete can only delete the one volume with the name of the service. Is there a way to get all these on a single volume? It would be better that way...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am more of a programmer than somebody who does production. Therefor if you think that is best then please do so.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't about programming, it's about where the mounts are. Since I know nothing about sqlsrv, I don't know why you are using volumes for all these things. Is is possible that only the database is actually required to be persistent, and that's sqldata, and it could be called sqlsrv? You only need the volumes for stuff that's persistent. On mysql and postgres, that's just the database itself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the SQL Server docker file. Removed the other volumes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, thanks!

@rfay rfay changed the title Add support for the Microsoft SQL Server Add support for Microsoft SQL Server Nov 23, 2021
Copy link
Member

@rfay rfay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I apologize for being so slow on this.It's a valuable contribution and I look forward to getting it in.

  • I'm not clear where the docker image is coming from. I guess not from dockerhub, but I am able to do a docker pull on it. But the parent repo doesn't seem to have any particular information, so makes me wonder.
  • I'm pretty sure this image is amd64 only, so it won't work reliably with mac M1 or other arm64 machines. Please add that caveat in there.
  • Most importantly: Installing the needed packages with post-start hooks is terribly painful and slow, please don't do that. Please change that into a .ddev/web-build/Dockerfile so it only happens once and doesn't have to be done on every ddev start.

I'll try to be more responsive when you update this, and I'll be happy to help you with any issues you run across getting the Dockerfile going. But see https://ddev.readthedocs.io/en/latest/users/extend/customizing-images/#adding-extra-dockerfiles-for-webimage-and-dbimage

Updated docker-compose.sqlsrv.yaml with comment aboutthe fact that the image does not work on Apple M1 hardware. See: microsoft/mssql-docker#734.
@drupal-daffie
Copy link
Contributor Author

For: "I'm not clear where the docker image is coming from. I guess not from dockerhub, but I am able to do a docker pull on it. But the parent repo doesn't seem to have any particular information, so makes me wonder."
This is how Microsoft wants you to do it: "mcr.microsoft.com/mssql/server:2019-CU12-ubuntu-20.04". I have no idea how they generate the image.

For: "I'm pretty sure this image is amd64 only, so it won't work reliably with mac M1 or other arm64 machines. Please add that caveat in there."
There is an issue open for fixing this: microsoft/mssql-docker#734. Added comment to the MR.

For: "Most importantly: Installing the needed packages with post-start hooks is terribly painful and slow, please don't do that. Please change that into a .ddev/web-build/Dockerfile so it only happens once and doesn't have to be done on every ddev start."
I have tried that and did not get it to work and you said: "I will be happy to help with this of course."

@rfay
Copy link
Member

rfay commented Jan 16, 2022

Unfortunately, it looks to me like Microsoft doesn't have msodbcsql17 or mssql-tools in their Debian 11 Bullseye repository. This is what I was working on, but got stuck with

apt-get install -y msodbcsql17 mssql-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package msodbcsql17
E: Unable to locate package mssql-tools
ARG BASE_IMAGE
FROM $BASE_IMAGE

ENV PATH="${PATH}:/opt/mssql-tools/bin"
RUN curl -sSL https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl -sSL -o /etc/apt/sources.list.d/mssql-release.list https://packages.microsoft.com/config/debian/11/prod.list

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y -o Dpkg::Options::="--force-confold" --no-install-recommends --no-install-suggests apt-utils build-essential dialog php-pear php-dev unixodbc-dev

RUN ACCEPT_EULA=Y DEBIAN_FRONTEND=noninteractive apt-get install -y msodbcsql17 mssql-tools
RUN pecl channel-update pecl.php.net && pecl install sqlsrv && pecl install pdo_sqlsrv

RUN for v in 7.3 7.4 8.0 8.1; do \
   echo 'extension=sqlsrv.so' >> /etc/php/${v}/mods-available/sqlsrv.ini"; \
   echo 'extension=pdo_sqlsrv.so' >> /etc/php/${v}/mods-available/pdo_sqlsrv.ini" ; \
done
RUN phpenmod sqlsrv pdo_sqlsrv

It's easy enough to work these through one at a time just using a debian image, that's what I did. docker run -it --rm debian:bullseye and then work from there.

Happy to carry on from here if you know how to get msodbcsql17 and mssql-tools on Debian 11.

@drupal-daffie
Copy link
Contributor Author

Support for Debian 11 is in!

@drupal-daffie
Copy link
Contributor Author

drupal-daffie commented Feb 27, 2022

About #2: When I build a container with the command docker build . I get a lot of information what the build is doing. When I run the command ddev restart I get a lot less information about how the build is doing. Is there a possibility to get more build info with the command ddev restart?

A also got a new problem. It is working fine when I set the php_version to "8.1". When I change it to "8.0" and run ddev restart PHP is not working any more. When I run ddev ssh and than php -v I get:

daffie@ddev-93mssql-web:/var/www/html/web$ php -v
PHP Warning:  PHP Startup: Unable to load dynamic library 'pdo_sqlsrv.so' (tried: /usr/lib/php/20200930/pdo_sqlsrv.so (/usr/lib/php/20200930/pdo_sqlsrv.so: cannot open shared object file: No such file or directory), /usr/lib/php/20200930/pdo_sqlsrv.so.so (/usr/lib/php/20200930/pdo_sqlsrv.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'pdo_sqlsrv.so' (tried: /usr/lib/php/20200930/pdo_sqlsrv.so (/usr/lib/php/20200930/pdo_sqlsrv.so: cannot open shared object file: No such file or directory), /usr/lib/php/20200930/pdo_sqlsrv.so.so (/usr/lib/php/20200930/pdo_sqlsrv.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library 'sqlsrv.so' (tried: /usr/lib/php/20200930/sqlsrv.so (/usr/lib/php/20200930/sqlsrv.so: cannot open shared object file: No such file or directory), /usr/lib/php/20200930/sqlsrv.so.so (/usr/lib/php/20200930/sqlsrv.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'sqlsrv.so' (tried: /usr/lib/php/20200930/sqlsrv.so (/usr/lib/php/20200930/sqlsrv.so: cannot open shared object file: No such file or directory), /usr/lib/php/20200930/sqlsrv.so.so (/usr/lib/php/20200930/sqlsrv.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP 8.0.13 (cli) (built: Nov 22 2021 09:47:37) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.13, Copyright (c) Zend Technologies
    with Zend OPcache v8.0.13, Copyright (c), by Zend Technologies

When I run ddev ssh and than php-config I get:

daffie@ddev-93mssql-web:/var/www/html/web$ php-config
Usage: /usr/bin/php-config [OPTION]
Options:
  --prefix            [/usr]
  --includes          [-I/usr/include/php/20210902 -I/usr/include/php/20210902/main -I/usr/include/php/20210902/TSRM -I/usr/include/php/20210902/Zend -I/usr/include/php/20210902/ext -I/usr/include/php/20210902/ext/date/lib ]
  --ldflags           [-L/usr/lib/php/20210902 ]
  --libs              [-lcrypt   -lresolv -lcrypt -lutil -lrt -lm -ldl  -lxml2 -lssl -lcrypto -lpcre2-8 -lz -lsodium -largon2 -lrt -ldl -lcrypt ]
  --extension-dir     [/usr/lib/php/20210902]
  --include-dir       [/usr/include/php/20210902]
  --man-dir           [/usr/share/man]
  --php-binary        [/usr/bin/php8.1]
  --php-sapis         [cli fpm ]
  --phpapi            [20210902]
  --ini-path          [/etc/php/8.1/cli]
  --ini-dir           [/etc/php/8.1/cli/conf.d]
  --configure-options [--includedir=/usr/include --mandir=/usr/share/man --infodir=/usr/share/info --disable-option-checking --disable-silent-rules --libdir=/usr/lib/x86_64-linux-gnu --libexecdir=/usr/lib/x86_64-linux-gnu --disable-maintainer-mode --disable-dependency-tracking --prefix=/usr --enable-cli --disable-cgi --disable-phpdbg --with-config-file-path=/etc/php/8.1/cli --with-config-file-scan-dir=/etc/php/8.1/cli/conf.d --build=x86_64-linux-gnu --host=x86_64-linux-gnu --config-cache --cache-file=/build/php8.1-8.1.3/config.cache --libdir=${prefix}/lib/php --libexecdir=${prefix}/lib/php --datadir=${prefix}/share/php/8.1 --program-suffix=8.1 --sysconfdir=/etc --localstatedir=/var --mandir=/usr/share/man --disable-all --disable-debug --disable-rpath --disable-static --with-pic --with-layout=GNU --without-pear --enable-filter --with-openssl --with-password-argon2=/usr --with-external-pcre --enable-hash --with-mhash=/usr --with-libxml --enable-session --with-sodium --with-system-tzdata --with-zlib=/usr --with-zlib-dir=/usr --enable-dtrace --enable-pcntl --with-libedit=shared,/usr build_alias=x86_64-linux-gnu host_alias=x86_64-linux-gnu CFLAGS=-g -O2 -ffile-prefix-map=/build/php8.1-8.1.3=. -fstack-protector-strong -Wformat -Werror=format-security -O2 -Wall -pedantic -fsigned-char -fno-strict-aliasing -g]
  --version           [8.1.3]
  --vernum            [80103]

Any idea how to fix this?

@rfay: As always thank you very much for your help. It is very much appreciated.

@rfay
Copy link
Member

rfay commented Feb 27, 2022

Thanks for the hard work on this. MS is not known for their Debian packaging skills, as we all know. Could you please link to their instructions on how to do the debian package installation? That will help for us to try to work with this.

@drupal-daffie
Copy link
Contributor Author

I have created a ticket on the MS PHP driver project.
See: microsoft/msphpsql#1377

@rfay
Copy link
Member

rfay commented Feb 28, 2022

I followed up in microsoft/msphpsql#1377 but note that the current Dockerfile you have here isn't related to their current instructions. Note that packages.microsoft.com isn't being used at all any more, nothing is being installed with apt-get, it's all pecl.

@rfay
Copy link
Member

rfay commented Feb 28, 2022

BTW, I learned how to get appropriate output in a complex Dockerfile situation. After a ddev start do a ddev stop and

cd .ddev
docker-compose -f .ddev-docker-compose-full.yaml build
# Optionally docker-compose -f .ddev-docker-compose-full.yaml build --no-cache

@drupal-daffie
Copy link
Contributor Author

Thank you, that is very helpful information. No more, debugging in the dark.
Also thank you you input on the MS PHP driver issue.

@rfay
Copy link
Member

rfay commented Feb 28, 2022

I might be wrong, but I didn't see links to the sqlsrv repo and the linux installation instructions in your readme. Please just make sure they're provided... for the future.

@drupal-daffie
Copy link
Contributor Author

Added the 2 requested links.

@rfay
Copy link
Member

rfay commented Mar 16, 2022

Are you seeing a path forward on this?

@drupal-daffie
Copy link
Contributor Author

We can do the MR with a release for PHP 8.1 only and a followup for once microsoft/msphpsql#1377 is fixed. Or we can wait until the blocker is fixed. @rfay what do you think is best?

@rfay
Copy link
Member

rfay commented Mar 19, 2022

If that works for you, it's a step forward, just document that it's php8.1 only and move forward.

@truls1502
Copy link
Collaborator

Is it possible to have it supporting PHP 7.3?

@drupal-daffie
Copy link
Contributor Author

User @absci has replied in microsoft/msphpsql#1377 with how to change the used PHP version.

@rfay I have updated the Dockerfile as requested.

@truls1502: I think that you could use PHP version 7.3, but I have not tested it.

@rfay
Copy link
Member

rfay commented May 10, 2022

OK, if you're ready for this and want to bring it in and you think the README shows enough info, we can pull it. I haven't taken another look, but as much time as you've spent on it I imagine it's. I'd love to see it turned into a ddev-get add-on so it's easiest for people, but I'm not sure how much demand there is for sqlsrv, but there is some.

@drupal-daffie
Copy link
Contributor Author

@rfay For me it is ready to be merged. Thanks for all your help with this MR.

@rfay rfay merged commit b249fde into ddev:master May 11, 2022
@rfay
Copy link
Member

rfay commented May 11, 2022

Thanks, hope you'll consider converting this into a ddev-addon, https://ddev.readthedocs.io/en/latest/users/extend/additional-services/#additional-service-configurations-for-ddev - it will make it loads easier to maintain, and certainly make it available to more people.

@NickStees
Copy link

NickStees commented Aug 4, 2022

@drupal-daffie Following the current instructions in the readme.md https://github.com/drud/ddev-contrib/tree/master/docker-compose-services/sqlsrv I am getting a phpize error, Is there something maybe missing form that Readme? Only thing I deviated from was setting it to PHP7.3 but I got the same error when it was PHP 8.0

#21 [drud/ddev-webserver:v1.20.0-inet-built 12/16] RUN pecl -d php_suffix=7.3 install sqlsrv #21 0.514 Xdebug: [Step Debug] Time-out connecting to debugging client, waited: 200 ms. Tried: host.docker.internal:9003 (through xdebug.client_host/xdebug.client_port) :-( #21 35.99 downloading sqlsrv-5.10.1.tgz ... #21 35.99 Starting to download sqlsrv-5.10.1.tgz (193,661 bytes) #21 36.03 .........................................done: 193,661 bytes #21 36.18 34 source files, building #21 36.18 running: phpize7.3 #21 36.18 sh: 1: phpize7.3: not found #21 36.18 ERROR: "phpize" failed #21 ERROR: executor failed running [/bin/sh -c pecl -d php_suffix=7.3 install sqlsrv]: exit code: 1

@NickStees
Copy link

Got past the error by modifying line 13 of https://github.com/drud/ddev-contrib/blob/e8eb363487d764738e48fb41ee4177d473b1fd40/docker-compose-services/sqlsrv/Dockerfile#L13 to
RUN ACCEPT_EULA=Y DEBIAN_FRONTEND=noninteractive apt-get install -y msodbcsql18 mssql-tools php7.3-dev adding php7.3-dev

@rfay
Copy link
Member

rfay commented Aug 4, 2022

Please do a PR @NickStees , also explain about how you did it with 7.3 in the README. Thanks!

@NickStees
Copy link

Thanks will do, I still don't have anything completely working yet, but when I do ill put it all together in a PR.

@rfay
Copy link
Member

rfay commented Aug 4, 2022

It would be great to see this turned into a ddev-get add-on, with configurable PHP version.

@NickStees
Copy link

@rfay Wish I knew how to make it an Add-on, yeah I was thinking why is the PHP version hard coded in that Dockerfile when I was following the readme.

@rfay
Copy link
Member

rfay commented Aug 4, 2022

  1. Making an add-on is easy, see https://github.com/drud/ddev-addon-template
  2. If you want to support multiple PHP versions (and you do) you'll have to either build them (use a for loop in the Dockerfile.sqlsrc) or you'll have to figure out if MS and company have decided to start making binary packages available. Maybe? Really? Not yet? Anyway. Or alternately, you can make a repo that builds them (for both arm64 and amd64, it's not hard) and then you can get the binary packages/tarballs from that repo in your Dockerfile. The good thing about just building it in is that your future need for maintenance is lower. The bad thing is that it already takes forever.

@seebeen
Copy link

seebeen commented Jan 18, 2024

It would be great to see this turned into a ddev-get add-on, with configurable PHP version.

On it.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants