Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/database #52

Merged
merged 15 commits into from
Jul 18, 2023
Merged

Feature/database #52

merged 15 commits into from
Jul 18, 2023

Conversation

quanpython
Copy link
Collaborator

No description provided.

@quanpython quanpython linked an issue Jul 12, 2023 that may be closed by this pull request
@quanpython quanpython marked this pull request as ready for review July 13, 2023 02:20
Copy link
Collaborator

@itutu-tienday itutu-tienday left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add some review comment

docker-compose.dev.yml Outdated Show resolved Hide resolved
docker-compose.dev.yml Outdated Show resolved Hide resolved
alembic.ini Outdated Show resolved Hide resolved
docker-compose.dev.yml Show resolved Hide resolved
@quanpython quanpython linked an issue Jul 14, 2023 that may be closed by this pull request
@ReiHashimoto
Copy link

@quanpython cc: @itutu-tienday

dev container is OK.

But prod container still fails with following errors.
Can you reproduce this?

docker compose up --build
 => ERROR [studio 8/8] RUN alembic upgrade head                                                                                                                    2.4s
------
 > [studio 8/8] RUN alembic upgrade head:
2.269 Traceback (most recent call last):
2.269   File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 613, in connect
2.269     sock = socket.create_connection(
2.269   File "/usr/local/lib/python3.8/socket.py", line 787, in create_connection
2.269     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
2.269   File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
2.269     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
2.269 socket.gaierror: [Errno -2] Name or service not known
2.269 
2.269 During handling of the above exception, another exception occurred:
2.269 
2.269 Traceback (most recent call last):
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect
2.269     return fn()
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 320, in connect
2.269     return _ConnectionFairy._checkout(self)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 884, in _checkout
2.269     fairy = _ConnectionRecord.checkout(pool)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 486, in checkout
2.269     rec = pool._do_get()
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
2.269     return self._create_connection()
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 266, in _create_connection
2.269     return _ConnectionRecord(self)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 381, in __init__
2.269     self.__connect()
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 678, in __connect
2.269     pool.logger.debug("Error on connect(): %s", e)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
2.269     compat.raise_(
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
2.269     raise exception
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 673, in __connect
2.269     self.dbapi_connection = connection = pool._invoke_creator(self)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/create.py", line 578, in connect
2.269     return dialect.connect(*cargs, **cparams)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 598, in connect
2.269     return self.dbapi.connect(*cargs, **cparams)
2.269   File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 353, in __init__
2.269     self.connect()
2.269   File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 664, in connect
2.269     raise exc
2.269 pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'db' ([Errno -2] Name or service not known)")
2.269 
2.269 The above exception was the direct cause of the following exception:
2.269 
2.269 Traceback (most recent call last):
2.269   File "/usr/local/bin/alembic", line 8, in <module>
2.269     sys.exit(main())
2.269   File "/usr/local/lib/python3.8/site-packages/alembic/config.py", line 590, in main
2.269     CommandLine(prog=prog).main(argv=argv)
2.269   File "/usr/local/lib/python3.8/site-packages/alembic/config.py", line 584, in main
2.269     self.run_cmd(cfg, options)
2.269   File "/usr/local/lib/python3.8/site-packages/alembic/config.py", line 561, in run_cmd
2.269     fn(
2.269   File "/usr/local/lib/python3.8/site-packages/alembic/command.py", line 378, in upgrade
2.269     script.run_env()
2.269   File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 569, in run_env
2.269     util.load_python_file(self.dir, "env.py")
2.269   File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
2.269     module = load_module_py(module_id, path)
2.269   File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
2.269     spec.loader.exec_module(module)  # type: ignore
2.269   File "<frozen importlib._bootstrap_external>", line 843, in exec_module
2.269   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2.269   File "studio/alembic/env.py", line 82, in <module>
2.269     run_migrations_online()
2.269   File "studio/alembic/env.py", line 72, in run_migrations_online
2.269     with connectable.connect() as connection:
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3315, in connect
2.269     return self._connection_cls(self, close_with_result=close_with_result)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
2.269     else engine.raw_connection()
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3394, in raw_connection
2.269     return self._wrap_pool_connect(self.pool.connect, _connection)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3364, in _wrap_pool_connect
2.269     Connection._handle_dbapi_exception_noconnection(
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2198, in _handle_dbapi_exception_noconnection
2.269     util.raise_(
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
2.269     raise exception
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect
2.269     return fn()
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 320, in connect
2.269     return _ConnectionFairy._checkout(self)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 884, in _checkout
2.269     fairy = _ConnectionRecord.checkout(pool)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 486, in checkout
2.269     rec = pool._do_get()
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
2.269     return self._create_connection()
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 266, in _create_connection
2.269     return _ConnectionRecord(self)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 381, in __init__
2.269     self.__connect()
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 678, in __connect
2.269     pool.logger.debug("Error on connect(): %s", e)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
2.269     compat.raise_(
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
2.269     raise exception
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 673, in __connect
2.269     self.dbapi_connection = connection = pool._invoke_creator(self)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/create.py", line 578, in connect
2.269     return dialect.connect(*cargs, **cparams)
2.269   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 598, in connect
2.269     return self.dbapi.connect(*cargs, **cparams)
2.269   File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 353, in __init__
2.269     self.connect()
2.269   File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 664, in connect
2.269     raise exc
2.269 sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'db' ([Errno -2] Name or service not known)")
2.269 (Background on this error at: https://sqlalche.me/e/14/e3q8)
------
failed to solve: process "/bin/sh -c alembic upgrade head" did not complete successfully: exit code: 1

sortOptions: SortOptions = Depends(),
db: Session = Depends(get_db),
):
data = paginate(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is no particular reason, it would be good to wrap the search_db_cells() function.

studio/app/optinist/routers/expdb.py Show resolved Hide resolved
options: ExpDbExperimentsSearchOptions = Depends(),
sortOptions: SortOptions = Depends(),
):
data = paginate(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"experiments_share_users" table is not used. Please use it.

options: ExpDbExperimentsSearchOptions = Depends(),
sortOptions: SortOptions = Depends(),
):
data = paginate(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When using the "experiments_share_users" table, an authentication operation (token parsing) process is required to identify the user_id.

Since it would be more efficient to omit the passing of authentication token during development, we would like to have a mode in which the passing of authentication token can be omitted (i.e., a mode in which token and user_id for debugging can be used).

if not exp:
raise HTTPException(status_code=404)
exp.publish_status = int(flag == PublishFlags.on)
db.add(exp)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Data publish" is updating experiments.publish_status, not adding records, but are you creating a new experiments in the following?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exp is an existing experiment object from the database. So db.add(exp) line adds the modified experiment to the database session to update it rather than create a new one.
To avoid misunderstandings, if the experiment object is already associated with the database session, it is better to omit the db.add(exp) line.
Thank you.

db.query(optinist_model.Experiment)
.join(
common_model.Organization,
common_model.Organization.id == optinist_model.Experiment.organization_id,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

users.organizasion_id and also requires a join, do you have any discrepancies in perception?

if not exp:
raise HTTPException(status_code=404)
exp.share_type = data.share_type
db.add(exp)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • "Data publish" is updating experiments.publish_status, not adding records, but are you creating a new experiments in the following?
  • There are multiple data parameters, and for the experiments_share_users table, the operation is to replace all existing data (delete all -> register all).

@quanpython
Copy link
Collaborator Author

@quanpython cc: @itutu-tienday

dev container is OK.

But prod container still fails with following errors. Can you reproduce this?

@ReiHashimoto
Because when run alembic upgrade head, database container haven't started yet.
It is solved by moving alembic upgrade head command to docker-compose.yml 5dcaf13
Thank you.

@ReiHashimoto
Copy link

@quanpython
prod container's problem has been resolved, thank you!

docker-compose.dev.yml Show resolved Hide resolved
docker-compose.yml Show resolved Hide resolved
MYSQL_USER=docker
MYSQL_PASSWORD=docker
ECHO_SQL=True
DATABASE_URL=mysql+pymysql://docker:docker@db/docker
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of the data in DATABASE_URL and MYSQL_* are duplicated, is it possible to manage them smartly in one place?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Just use MYSQL_* is enough because DATABASE_URL is generate from assemble_db_connection function in studio/app/common/db/config.py

Copy link

@ReiHashimoto ReiHashimoto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still required changes exists, approve for now.

@quanpython quanpython force-pushed the feature/database branch 2 times, most recently from 0b4afd1 to 19501db Compare July 18, 2023 10:42
@itutu-tienday itutu-tienday merged commit e913f9f into develop-main Jul 18, 2023
sanglevinh pushed a commit that referenced this pull request Oct 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Database API実装② データベース開発環境整備
3 participants