How to Set up Lamby with pg gem – rails + aws lambda + postgresql #133

richardsondx · 2022-11-25T04:54:42Z

It has been really hard to figure out some of the issues I ran into related to setting up postgresql with lamby so far, so I'm going to share some progress here since I'm sure other people might experience these issues and benefit from this.

Issue with pg native extension
I've been able to resolve most of pg installation issues with these lines in my Dockerfile-build:

RUN yum update -y
RUN yum install -y amazon-linux-extras
RUN amazon-linux-extras enable postgresql13
RUN yum install -y  postgresql postgresql-server postgresql-devel

but then I started getting the following error once my stack is deployed.

pg_ext.s issue with rails + aws lambda and postgreqsl

"errorMessage": "libpq.so.5: cannot open shared object file: No such file or directory - /var/task/vendor/bundle/ruby/2.7.0/gems/pg-1.4.5/lib/pg_ext.so",

status: Resolved

In my attempt to resovle the issue above I tried to add the missing library manually in the bin/_build file
source: rubyonjets/jets#104

mkdir -p /var/task/lib
cp -a /usr/lib64/libpq.so.5 /var/task/lib/libpq.so.5
cp -a /usr/lib64/libldap_r-2.4.so.2 /var/task/lib/libldap_r-2.4.so.2
cp -a /usr/lib64/liblber-2.4.so.2 /var/task/lib/liblber-2.4.so.2
cp -a /usr/lib64/libsasl2.so.3 /var/task/lib/libsasl2.so.3
cp -a /usr/lib64/libssl3.so /var/task/lib/
cp -a /usr/lib64/libsmime3.so /var/task/lib/
cp -a /usr/lib64/libnss3.so /var/task/lib/
cp -a /usr/lib64/libnssutil3.so /var/task/lib/

Here is the full error:

2022-11-24T22:28:04.467-05:00 | "errorMessage": "libpq.so.5: cannot open shared object file: No such file or directory - /var/task/vendor/bundle/ruby/2.7.0/gems/pg-1.4.5/lib/pg_ext.so",
-- | --
  | 2022-11-24T22:28:04.467-05:00 | "errorType": "Init<LoadError>",
  | 2022-11-24T22:28:04.467-05:00 | "stackTrace": [
  | 2022-11-24T22:28:04.467-05:00 | "/var/task/vendor/bundle/ruby/2.7.0/gems/pg-1.4.5/lib/pg.rb:49:in `require'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/task/vendor/bundle/ruby/2.7.0/gems/pg-1.4.5/lib/pg.rb:49:in `block in <module:PG>'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/task/vendor/bundle/ruby/2.7.0/gems/pg-1.4.5/lib/pg.rb:37:in `block in <module:PG>'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/task/vendor/bundle/ruby/2.7.0/gems/pg-1.4.5/lib/pg.rb:42:in `<module:PG>'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/task/vendor/bundle/ruby/2.7.0/gems/pg-1.4.5/lib/pg.rb:6:in `<top (required)>'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/runtime/gems/bundler-2.3.25/lib/bundler/runtime.rb:60:in `require'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/runtime/gems/bundler-2.3.25/lib/bundler/runtime.rb:60:in `block (2 levels) in require'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/runtime/gems/bundler-2.3.25/lib/bundler/runtime.rb:55:in `each'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/runtime/gems/bundler-2.3.25/lib/bundler/runtime.rb:55:in `block in require'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/runtime/gems/bundler-2.3.25/lib/bundler/runtime.rb:44:in `each'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/runtime/gems/bundler-2.3.25/lib/bundler/runtime.rb:44:in `require'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/runtime/gems/bundler-2.3.25/lib/bundler.rb:186:in `require'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/task/config/application.rb:19:in `<top (required)>'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/task/app.rb:6:in `require_relative'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/task/app.rb:6:in `<top (required)>'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'",
  | 2022-11-24T22:28:04.467-05:00 | "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'"
  | 2022-11-24T22:28:04.467-05:00 | ]

Line 6 on application.rb was theBundler.require(*Rails.groups) it made me realized that the issue could be because of a mismatch of ruby version in my gemfile and the app configuration since my gemfile had the version:

ruby "2.7.6"
gem 'rails', '~> 7.0.1'

I noticed a couple area in the app that were pointing to the same ruby version listed in the error:

docker-compose.yml environment had this configuration GEM_HOME=/var/task/vendor/bundle/ruby/2.7.0
There were also a couple location in the app that expected 2.7.0. I removed the ruby "2.7.6" mention from my gemfile

So I removed the ruby version from my gemfile ( alternatively you can update all the references – but you will also need to update the image of ruby being loaded most likely since Docker line FROM public.ecr.aws/sam/build-ruby2.7 might be pointing to 2.7.0)

then I got this error:
"errorMessage": "Could not find pg-1.4.5, nio4r-2.5.8, websocket-driver-0.7.5, nokogiri-1.13.9, racc-1.6.0 in locally installed gems",

status: Resolved

Full error:

{
  "errorMessage": "Could not find pg-1.4.5, nio4r-2.5.8, websocket-driver-0.7.5, nokogiri-1.13.9, racc-1.6.0 in locally installed gems",
  "errorType": "Init<Bundler::GemNotFound>",
  "stackTrace": [
    "/var/runtime/gems/bundler-2.3.25/lib/bundler/definition.rb:507:in `materialize'",
    "/var/runtime/gems/bundler-2.3.25/lib/bundler/definition.rb:187:in `specs'",
    "/var/runtime/gems/bundler-2.3.25/lib/bundler/definition.rb:244:in `specs_for'",
    "/var/runtime/gems/bundler-2.3.25/lib/bundler/runtime.rb:18:in `setup'",
    "/var/runtime/gems/bundler-2.3.25/lib/bundler.rb:161:in `setup'",
    "/var/runtime/gems/bundler-2.3.25/lib/bundler/setup.rb:20:in `block in <top (required)>'",
    "/var/runtime/gems/bundler-2.3.25/lib/bundler/ui/shell.rb:136:in `with_level'",
    "/var/runtime/gems/bundler-2.3.25/lib/bundler/ui/shell.rb:88:in `silence'",
    "/var/runtime/gems/bundler-2.3.25/lib/bundler/setup.rb:20:in `<top (required)>'",
    "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'",
    "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'",
    "/var/task/config/boot.rb:3:in `<top (required)>'",
    "/var/task/app.rb:2:in `require_relative'",
    "/var/task/app.rb:2:in `<top (required)>'",
    "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'",
    "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'"
  ]
}

It looks like some of these gems failed to install . I know this because I ssh'd into the container and ran bundle doctor and notice that some of the gem where throwing native extension errors. I figure out a way to resolve it in the container; but it doesn't seem to resolve the issue on lambda.

I added this line to my Dockerfile-build to help install all related lib dependency:

RUN yum install -y gcc ruby-devel zlib-devel libxml2 libxml2-devel libxslt libxslt-devel make
 RUN yum install -y rubygem-nokogiri libxml2-devel libxslt libxslt-devel

and these two line helped in my bin/_build file for nio4r and nokogiri

bundle config build.nio4r --with-cflags="-std=c99"
bundle config build.nokogiri --use-system-libraries

then I got this error:
"libexslt.so.0: cannot open shared object file: No such file or directory

status: Unresolved

Here is the full stacktrace:

"errorMessage": "libexslt.so.0: cannot open shared object file: No such file or directory - /var/task/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.13.9/lib/nokogiri/nokogiri.so",
"errorType": "Init<LoadError>",
"stackTrace": [
"/var/task/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.13.9/lib/nokogiri/extension.rb:30:in `require'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.13.9/lib/nokogiri/extension.rb:30:in `rescue in <top (required)>'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.13.9/lib/nokogiri/extension.rb:4:in `<top (required)>'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.13.9/lib/nokogiri.rb:10:in `require_relative'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.13.9/lib/nokogiri.rb:10:in `<top (required)>'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/loofah-2.19.0/lib/loofah.rb:4:in `require'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/loofah-2.19.0/lib/loofah.rb:4:in `<top (required)>'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/rails-html-sanitizer-1.4.3/lib/rails-html-sanitizer.rb:2:in `require'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/rails-html-sanitizer-1.4.3/lib/rails-html-sanitizer.rb:2:in `<top (required)>'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/actionview-7.0.4/lib/action_view/helpers/sanitize_helper.rb:3:in `require'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/actionview-7.0.4/lib/action_view/helpers/sanitize_helper.rb:3:in `<top (required)>'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/actionview-7.0.4/lib/action_view/helpers.rb:8:in `require'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/actionview-7.0.4/lib/action_view/helpers.rb:8:in `<top (required)>'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/sprockets-rails-3.4.2/lib/sprockets/rails/context.rb:1:in `require'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/sprockets-rails-3.4.2/lib/sprockets/rails/context.rb:1:in `<top (required)>'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/sprockets-rails-3.4.2/lib/sprockets/railtie.rb:10:in `require'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/sprockets-rails-3.4.2/lib/sprockets/railtie.rb:10:in `<top (required)>'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/sprockets-rails-3.4.2/lib/sprockets/rails.rb:3:in `require'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/sprockets-rails-3.4.2/lib/sprockets/rails.rb:3:in `<top (required)>'",
"/var/runtime/gems/bundler-2.3.25/lib/bundler/runtime.rb:73:in `require'",
"/var/runtime/gems/bundler-2.3.25/lib/bundler/runtime.rb:73:in `rescue in block in require'",
"/var/runtime/gems/bundler-2.3.25/lib/bundler/runtime.rb:51:in `block in require'",
"/var/runtime/gems/bundler-2.3.25/lib/bundler/runtime.rb:44:in `each'",
"/var/runtime/gems/bundler-2.3.25/lib/bundler/runtime.rb:44:in `require'",
"/var/runtime/gems/bundler-2.3.25/lib/bundler.rb:186:in `require'",
"/var/task/config/application.rb:19:in `<top (required)>'",
"/var/task/app.rb:6:in `require_relative'",
"/var/task/app.rb:6:in `<top (required)>'",
"/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'",
"/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'"
]

UPDATE: I wasn't able to make it work – Too many dependency to resolves. Let me know if you find a solution. Hopefully what I shared here will make it easier to get to a solution.

The text was updated successfully, but these errors were encountered:

metaskills · 2022-12-05T12:50:08Z

So I would recommend switching to official Ruby images with the RIC installed. I recently refactored the Lamby Cookiecutter to follow this pattern and outlined all the changes. It actually simplifies a lot. Review here.

Should work OOTB, but at the worst, you would have to do a few apt installs.

richardsondx · 2022-12-05T13:21:05Z

@metaskills It seems that there's an issue with that lamby-cookiecutter project
rails-lambda/lamby-cookiecutter#22

metaskills · 2022-12-05T13:30:51Z

Commented there, I think you might have a local cache issue. Seen this before. Lemme know!

richardsondx changed the title ~~Lamby with pg gem ( postgresql )~~ Setting Lamby with pg gem ( postgresql ) Nov 25, 2022

richardsondx changed the title ~~Setting Lamby with pg gem ( postgresql )~~ How to Set up Lamby with pg gem – rails + aws lambda + postgresql Nov 25, 2022

metaskills closed this as completed Dec 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to Set up Lamby with pg gem – rails + aws lambda + postgresql #133

How to Set up Lamby with pg gem – rails + aws lambda + postgresql #133

richardsondx commented Nov 25, 2022 •

edited

Loading

metaskills commented Dec 5, 2022

richardsondx commented Dec 5, 2022

metaskills commented Dec 5, 2022

How to Set up Lamby with pg gem – rails + aws lambda + postgresql #133

How to Set up Lamby with pg gem – rails + aws lambda + postgresql #133

Comments

richardsondx commented Nov 25, 2022 • edited Loading

metaskills commented Dec 5, 2022

richardsondx commented Dec 5, 2022

metaskills commented Dec 5, 2022

richardsondx commented Nov 25, 2022 •

edited

Loading