Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker image does not have mysqlclient #29727

Closed
3 tasks done
iercan opened this issue Jul 28, 2024 · 7 comments · Fixed by #29771
Closed
3 tasks done

Docker image does not have mysqlclient #29727

iercan opened this issue Jul 28, 2024 · 7 comments · Fixed by #29771
Assignees
Labels
install:dependencies Installation - Dependencies install:docker Installation - docker container

Comments

@iercan
Copy link
Contributor

iercan commented Jul 28, 2024

Bug description

I'm testing 4.1.0rc1 using a Docker image and noticed it lacks the mysqlclient package that previous versions included. Was this removal intentional? Attempting to install it results in the following error.

superset@4a77196bff67:/app$ pip install mysqlclient==2.1.0
Defaulting to user installation because normal site-packages is not writeable
Collecting mysqlclient==2.1.0
  Downloading mysqlclient-2.1.0.tar.gz (87 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.6/87.6 kB 2.0 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: mysqlclient
  Building wheel for mysqlclient (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [40 lines of output]
      mysql_config --version
      ['10.11.6']
      mysql_config --libs
      ['-L/usr/lib/x86_64-linux-gnu/', '-lmariadb']
      mysql_config --cflags
      ['-I/usr/include/mariadb', '-I/usr/include/mariadb/mysql']
      ext_options:
        library_dirs: ['/usr/lib/x86_64-linux-gnu/']
        libraries: ['mariadb']
        extra_compile_args: ['-std=c99']
        extra_link_args: []
        include_dirs: ['/usr/include/mariadb', '/usr/include/mariadb/mysql']
        extra_objects: []
        define_macros: [('version_info', "(2,1,0,'final',0)"), ('__version__', '2.1.0')]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/MySQLdb
      copying MySQLdb/__init__.py -> build/lib.linux-x86_64-cpython-310/MySQLdb
      copying MySQLdb/_exceptions.py -> build/lib.linux-x86_64-cpython-310/MySQLdb
      copying MySQLdb/connections.py -> build/lib.linux-x86_64-cpython-310/MySQLdb
      copying MySQLdb/converters.py -> build/lib.linux-x86_64-cpython-310/MySQLdb
      copying MySQLdb/cursors.py -> build/lib.linux-x86_64-cpython-310/MySQLdb
      copying MySQLdb/release.py -> build/lib.linux-x86_64-cpython-310/MySQLdb
      copying MySQLdb/times.py -> build/lib.linux-x86_64-cpython-310/MySQLdb
      creating build/lib.linux-x86_64-cpython-310/MySQLdb/constants
      copying MySQLdb/constants/__init__.py -> build/lib.linux-x86_64-cpython-310/MySQLdb/constants
      copying MySQLdb/constants/CLIENT.py -> build/lib.linux-x86_64-cpython-310/MySQLdb/constants
      copying MySQLdb/constants/CR.py -> build/lib.linux-x86_64-cpython-310/MySQLdb/constants
      copying MySQLdb/constants/ER.py -> build/lib.linux-x86_64-cpython-310/MySQLdb/constants
      copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.linux-x86_64-cpython-310/MySQLdb/constants
      copying MySQLdb/constants/FLAG.py -> build/lib.linux-x86_64-cpython-310/MySQLdb/constants
      running build_ext
      building 'MySQLdb._mysql' extension
      creating build/temp.linux-x86_64-cpython-310
      creating build/temp.linux-x86_64-cpython-310/MySQLdb
      gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Dversion_info=(2,1,0,'final',0) -D__version__=2.1.0 -I/usr/include/mariadb -I/usr/include/mariadb/mysql -I/usr/local/include/python3.10 -c MySQLdb/_mysql.c -o build/temp.linux-x86_64-cpython-310/MySQLdb/_mysql.o -std=c99
      error: command 'gcc' failed: No such file or directory
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for mysqlclient
  Running setup.py clean for mysqlclient
Failed to build mysqlclient
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (mysqlclient)

I resolved the issue by installing the necessary packages. A better solution would be to include it as a built-in image.

apt install -y python3-dev default-libmysqlclient-dev build-essential pkg-config

How to reproduce the bug

  1. Run superset via official docker image from https://hub.docker.com/r/apache/superset

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.9

Node version

16

Browser

Chrome

Additional context

No response

Checklist

  • I have searched Superset docs and Slack and didn't find a solution to my problem.
  • I have searched the GitHub issue tracker and didn't find a similar bug report.
  • I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
@dosubot dosubot bot added install:dependencies Installation - Dependencies install:docker Installation - docker container labels Jul 28, 2024
Copy link

dosubot bot commented Jul 28, 2024

Yes, the removal of the mysqlclient package from the Superset version 4.1.0rc1 Docker image was intentional. Superset does not ship bundled with connectivity to databases, and users are required to install the proper database drivers in their environment.

To include the necessary dependencies in the Docker image by default, you can create a requirements-local.txt file in a subdirectory called docker and add the required packages there. Here are the steps:

  1. Create the requirements-local.txt file:

    # Run from the repo root:
    touch ./docker/requirements-local.txt
  2. Add the mysqlclient package to the file:

    echo "mysqlclient" >> ./docker/requirements-local.txt
  3. Launch Superset with Docker Compose:

    docker compose -f docker-compose-non-dev.yml up

If you are running a customized Docker image, rebuild your local image with the new driver baked in:

docker compose build --force-rm

After the rebuild is complete, relaunch Superset by running:

docker compose up

This approach ensures that the mysqlclient package is included in the Docker image and avoids the installation issues you encountered [1].

To continue talking to Dosu, mention @dosu.

@mistercrunch
Copy link
Member

Yes, that's intentional, we try not to bloat our docker images as there's an infinity of drivers that you may or may not want as part of your Superset environment, and there's a maintenance burden associated with that too.

MySQL is pretty well supported as a metadata backend for Superset (we run some of our CI against it), and we may want to improve the documentation to clarify how to do this. In your typical k8s setup you'll need a dockerfile that inherits from ours, add the apt-get packages and pip installs mysqlclient.

@mistercrunch
Copy link
Member

I wrote the comment above and then I was thinking "but wait, how we're able to build the mysql-related matrices in our CI?". Turns out we do have the os-level mysql dep in our Dockerfile here https://github.com/apache/superset/blob/master/Dockerfile#L78, but arguably maybe we shouldn't.

Then our "developement" python requirements, indirectly installs mysqlclient, and that happens in our CI https://github.com/apache/superset/blob/master/requirements/development.in

So I'm unclear why your pip install failed. Are you using one of our dockers? If so I'm pretty sure gcc should be on there and mysqlclient installable at this time.

@rusackas
Copy link
Member

Might this little quagmire be resolved by removing it?

@mistercrunch
Copy link
Member

Related: #29771

@iercan
Copy link
Contributor Author

iercan commented Jul 30, 2024

So I'm unclear why your pip install failed. Are you using one of our dockers? If so I'm pretty sure gcc should be on there and mysqlclient installable at this time.
@mistercrunch

Using the official image, simply run it and try to install. You will see it fails. Because it needs some deps which is not included to image as I mentioned in issue.

root@SUPERSET01:~>docker run -it apache/superset:4.1.0rc1 bash
superset@9ea94806d9c8:/app$
superset@9ea94806d9c8:/app$ pip install mysqlclient
Defaulting to user installation because normal site-packages is not writeable
Collecting mysqlclient
  Downloading mysqlclient-2.2.4.tar.gz (90 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 90.4/90.4 kB 2.4 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [27 lines of output]
      /bin/sh: 1: pkg-config: not found
      /bin/sh: 1: pkg-config: not found
      /bin/sh: 1: pkg-config: not found
      Trying pkg-config --exists mysqlclient
      Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127.
      Trying pkg-config --exists mariadb
      Command 'pkg-config --exists mariadb' returned non-zero exit status 127.
      Trying pkg-config --exists libmariadb
      Command 'pkg-config --exists libmariadb' returned non-zero exit status 127.
      Traceback (most recent call last):
        File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-7gegxxt4/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
        File "/tmp/pip-build-env-7gegxxt4/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-7gegxxt4/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 155, in <module>
        File "<string>", line 49, in get_config_posix
        File "<string>", line 28, in find_package_name
      Exception: Can not find valid pkg-config name.
      Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[notice] A new release of pip is available: 24.1.2 -> 24.2
[notice] To update, run: pip install --upgrade pip

@mistercrunch
Copy link
Member

Oh after digging in the Dockerfile I understand why it's not the case. We're playing tricks to install apt-get's build-essentials package, running pip install on top, and removing that presumably large os dependency package all within a single RUN command to try and keep the docker image layers size down. It's nice in some ways, but quite a pain in the behind in terms of cognitive load + having to play that trick multiple times (once per "target"). Python's c-bindings are great and all, but a source of a lot of problems operationally...

relevant, the RUN statement that plays that trick: https://github.com/apache/superset/blob/master/Dockerfile#L93-L97

Also note that in #29771 I'm fully removing all mysql-related packages from the lean build as we really shouldn't ship a binary with GPL in it, though unclear whether a docker image IS a binary under GPL, but cleaning it out nonetheless.

As a workaround, you should be able to install the mysql deps on top lean by install build-essentials, the mysql apt-get client package, and then running pip install mysqlclient.

It'd be great if someone could add this to the docs, somewhere under the installation section

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
install:dependencies Installation - Dependencies install:docker Installation - docker container
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants