Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG -psycopg error #364

Closed
tech3371 opened this issue Oct 4, 2024 · 1 comment · Fixed by #365
Closed

BUG -psycopg error #364

tech3371 opened this issue Oct 4, 2024 · 1 comment · Fixed by #365
Assignees
Labels
bug Something isn't working
Milestone

Comments

@tech3371
Copy link
Contributor

tech3371 commented Oct 4, 2024

Description of the issue

We get import error of psycopg on AWS lambda. I initially thought it was due to imap-data-access version0.10.0. Initial debugging pointed to that because I tried creating new lambda layer with imap-data-access version 0.7.0 through 0.10.0 and deploy the CDK each time with new layer. All versions worked besides version 0.10.0. Therefore, I thought that was the issue.

Then to test why version 0.10.0 was causing the issue, I created a local working branch. I did not make any change to imap-data-access beside updating version to 0.10.1rc2 on it. Then I tried creating package locally using this command python -m build --sdist and then copying the zip file to sds-data-manager/lambda_layer/python and update lambda_layer_construct.py to pip install that independently like this

        code_bundle = lambda_.Code.from_asset(
            layer_dependencies_dir,
            bundling=cdk.BundlingOptions(
                image=runtime.bundling_image,
                command=[
                    "bash",
                    "-c",
                    (
                        "pip install -r requirements.txt -t /asset-output/python && "
                        "pip install imap_data_access-0.10.1rc2.tar.gz -t /asset-output/python && "
                        "cp -au . /asset-output/python"
                    ),
                ],
            ),
        )

        super().__init__(
            id=f"{id}-Layer", code=code_bundle, compatible_runtimes=[runtime], **kwargs
        )

When I deployed with above changes through CDK, the issue went away on AWS lambda.

But when everything is deployed through requirements.txt and through GitHub deploy action, it raises this issue on AWS lambda. I am not sure now what the issue is.

Steps to reproduce the issue

Deploy CDK to AWS through GitHub Actions or locally from this snapshot of sds-data-managercommit

Expected behavior (What should happen)

No error

Actual behavior (What does happen)

{
"errorMessage": "No module named 'psycopg2._psycopg'",
"errorType": "ModuleNotFoundError",
"requestId": "150f8d02-8a1c-41c2-bd29-11a9f79778ec",
"stackTrace": [
" File "/var/task/SDSCode/create_schema.py", line 29, in lambda_handler\n Base.metadata.create_all(db.get_engine())\n",
" File "/var/task/SDSCode/database/database.py", line 27, in get_engine\n return create_engine(db_uri)\n",
" File "", line 2, in create_engine\n",
" File "/opt/python/sqlalchemy/util/deprecations.py", line 281, in warned\n return fn(*args, **kwargs) # type: ignore[no-any-return]\n",
" File "/opt/python/sqlalchemy/engine/create.py", line 599, in create_engine\n dbapi = dbapi_meth(**dbapi_args)\n",
" File "/opt/python/sqlalchemy/dialects/postgresql/psycopg2.py", line 690, in import_dbapi\n import psycopg2\n",
" File "/opt/python/psycopg2/init.py", line 51, in \n from psycopg2._psycopg import ( # noqa\n"
]
}

Code Snippet:

Code

Additional notes, affected areas, and suggested fixes

Files and resources that caused this issue could be these:

  1. imap-data-access - version 0.10.0
  2. mismatch of psycopg = "^3.1" and psycopg2-binary = "^2.9.9" in pyproject.toml file
  3. GitHub workflow yaml not installing dependency for psycopg
@tech3371 tech3371 added the bug Something isn't working label Oct 4, 2024
@tech3371 tech3371 added this to IMAP Oct 4, 2024
@tech3371 tech3371 added this to the Oct 2024 milestone Oct 4, 2024
@tech3371 tech3371 linked a pull request Oct 4, 2024 that will close this issue
@tech3371 tech3371 self-assigned this Oct 7, 2024
@tech3371 tech3371 moved this to In Progress in IMAP Oct 7, 2024
@tech3371
Copy link
Contributor Author

tech3371 commented Oct 7, 2024

I don't what the real issue is still. This issue was fixed by re-releasing imap-data-access version 0.10.0 to 0.10.1 https://github.com/IMAP-Science-Operations-Center/imap-data-access/releases/tag/v0.10.1

@github-project-automation github-project-automation bot moved this from In Progress to Done in IMAP Oct 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant