Geospatial libraries no longer work

[volkanunsal]

In the last 3 days (since our last deploy on August 18th) this library has stopped working for us. At first I thought it might be the same issue as #47, but it's different because the build isn't broken, yet the libraries are not working. This is what I'm seeing from Rails console:

irb(main):002:0> RGeo::Geos.supported?
=> false

These are the buildpacks that I use:

1. https://github.com/mojodna/heroku-buildpack-jemalloc
2. https://github.com/cyberdelia/heroku-geo-buildpack.git
3. heroku/ruby
4. https://github.com/BetterWorks/heroku-buildpack-pgbouncer

And I'm using heroku-16 stack.

Here is the build details.

remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> geos/gdal/proj app detected
remote:        Using geos version: 3.6.1
remote:        Using gdal version: 2.1.3
remote:        Using proj version: 4.9.3
remote: -----> Vendoring geo libraries done
remote: -----> Ruby app detected
remote: -----> Compiling Ruby/Rails
remote: -----> Using Ruby version: ruby-2.3.7

The only difference here is we switched to using Ruby 2.3.7p456 (from 2.3.3p222) since the last build. But I'm not sure why that would be a factor. (I've tried reverting that and it still didn't work.)

I have also tried fixing the buildpack to this commit from August 10 to see if the August 17 commit is the culprit, but that did not work either.

[volkanunsal]

Same issue as #39

[volkanunsal]

@cyberdelia Is there anyone on Heroku team who can help us? The gem is building without error, but the headers don't seem to be available at runtime.

[sds-dubois]

Hi @volkanunsal I was facing the same problem and was able to fix it by deleting the build cache (using purge-cache from heroku-repo).

I'm not exactly sure what happened, but my app failing was on cedar-14 before. Upgrading to heroku-16 and pointing to older versions of this repo didn't do it. But finally clearing the cache and starting a new deployment with heroku-16 and this repo release 1.4 fixed it.

Hope it helps!

[volkanunsal]

@sds-dubois Believe me, I've tried that, and it didn't help. I was already running on heroku-16 before this started happening. And it didn't go back to normal after I fixed the buildpack to the previous commit and purged the build caches.

Thanks for taking the time to reply to this post, btw. Heroku support has been totally unhelpful (and so have the maintainers of this library). I'm not blaming this buildpack or the maintainers. No one else seems to be having any issues with it right now, so it must be working for everyone else. Ostensibly, though, based on the issue history, this buildpack is failing randomly for people for no apparent reason; the Heroku build system is screwing it up, and I have no clue why it is, and how to fix it.

Even if this gets resolved on its own somehow, it can happen again at anytime in the future. There is no logic behind why it happened to me this week. Unless I can definitively understand why it happened, when I use this buildpack again (or any buildpack for that matter) in the future, it's going to be like a roll of the dice. And when the app fails because of this library, it's my head on the line.

If you have the luxury, you shouldn't use this buildpack. I hope I can stop using it. I've wasted an entire week trying to fix this issue already. Now I'm writing everything in Docker containers. It's not a happy time right now.

[volkanunsal]

Heroku support says it might be related to a security update to the platform that was released recently. But that's only a best guess.

[sds-dubois]

I guess you've been googling around for alternatives but just in case, are you aware of this other buildpack ?

Out of curiosity, do you have other Heroku apps with the same config, and if so, are they all failing or just one? Honestly I'm not sure this buildpack is to blame - if it is failing randomly, starting a new deployment should give you good chances of 'fixing' the bug (except if there's something special about your config, maybe other buildpacks that conflict with it?). But Heroku should ensure that every deployment is a fresh start and should behave exactly the same for everyone / every app.

[sds-dubois]

Btw looking back at your buildpack config, I would try and set this one first, to be sure it's not the other buildpack messing up with its installation.

[mbaird]

Ran into this same issue. It's something with this commit -> 77fe3e6

Didn't have time to dig into why, but forking & removing this commit fixed the issue for us.

[volkanunsal]

@mbaird I tried the same -- it didn't work. This has happened to me before, and many others. This buildpack doesn't work one day, and starts working the next. The Heroku stack is changing all the time, and it's hard to say what causes the breakage.

[mbaird]

Hmm. We also rebuilt the geos/gdal/proj binaries, using the latest heroku-16 docker image. May have been related.

[denishaskin]

The libraries stopped working for us yesterday. We had upgraded to the new buildpacks and heroku-16 a couple of weeks ago, everything was fine. Then suddenly a deploy yesterday to staging (whew) resulted in rgeo no longer working:

irb(main):001:0> RGeo::Geos.supported?
=> false

(and any code using rgeo is failing)

We have tried purging the build cache, did not help.

Here's our buildpacks:

=== xx Buildpack URLs
1. https://github.com/cyberdelia/heroku-geo-buildpack.git
2. heroku/ruby

[volkanunsal]

I've successfully switched to the container stack, which is still in beta, but everything seems to be working now. If anyone is interested, I can post my Dockerfile here.

[sds-dubois]

@denishaskin just in case, maybe try specifying the release version (and purge the cache again)

https://github.com/cyberdelia/heroku-geo-buildpack.git#1.4

[davidtrogers]

@volkanunsal yes, please post your Dockerfile

[volkanunsal]

@davidtrogers There are a few of them actually and also a heroku.yml file. I compiled them into a gist. You'd need to modify the various scripts to substitute your own.

The scripts that must be present in your app:

• script/ci install_gems_production
• script/ci precompile
• script/release-tasks.sh
• script/start-puma
• script/start-background-worker

https://gist.github.com/volkanunsal/2410960894e0df4ecc66d707fa022746

The one that needs to sit at the base of your app is this one:

FROM volkanunsal/ruby:2.3.7-node-geos

WORKDIR /app
COPY ./ /app

ENV NODE_ENV 'production'
RUN echo '>> Installing Node packages...' \
  && yarn --ignore-optional

RUN  echo ">> Bundling Ruby gems to ${BUNDLE_PATH}..." \
  && script/ci install_gems_production \
  && apk del .ruby-builddeps

ENV RAILS_ENV="production"
# This is a fake DATABASE_URL...neded for asset compilation to work.
RUN DATABASE_URL=postgis://postgres@db/fake script/ci precompile
RUN rm -rf node_modules

CMD /app/script/start-puma

[denishaskin]

For what it's worth, we've switch to heroku-buildpack-rgeo-prep and it's working again (after purging the build cache and redeploying).

[raphaelpra]

I had the problem going on.

It was fixed by using https://github.com/korukids/heroku-geo-buildpack buildpack. It seems to me that the s3 bucket was somehow corrupted and the package was installing properly.

[abezzub]

Happened for me too after going from ruby 2.4.1 to 2.4.5. Did everyone resolve the issue by not using this buildpack?

[abezzub]

I can confirm that latest commit (77fe3e6) in the buildpack broke the build. Reverting to previous version makes it work.

[mudge]

(Cross-posted from #51.)

Just a warning that https://github.com/korukids/heroku-geo-buildpack is no longer supported and will not be updated as we've switched to using heroku-buildpack-apt to install libgeos-dev and libproj-dev instead.

[yc-yvonne]

Update as of Oct 6, 2020: I ended up solving this issue and also issues with using the apt buildpack by using this one newly built by heroku with no additional config or setup: https://github.com/heroku/heroku-geo-buildpack