Improve the Hugo snap package #9078

jmooring · 2021-10-24T21:03:49Z

Background

The Hugo snap package does not include external dependencies required for certain Hugo features. Under strict confinement these dependencies must be bundled within the snap.

We have an open request for classic confinement which would allow us to remove the dependencies from the snap¹, but this would:

Result in site building errors if the dependencies have not been manually installed
Force users to manually install the dependencies (a non-trivial task in some cases)

We should try to make strict confinement work as much as possible.

Dependencies

External Dependency	v0.88.1	v0.101.0
Asciidoctor	❌	✔️
Embedded Dart Sass	❌	✔️
Git	✔️	✔️
Go	❌	✔️
Node.js	❌²	✔️
Pandoc	❌³	✔️
rst2html	❌	✔️

Testing

Install the extended version of Hugo from the Snap Store.

sudo snap install hugo --channel=extended/stable

Clone this test repository, install the npm packages, and build the site.

git clone https://github.com/jmooring/hugo-snap-test
cd hugo-snap-test
npm install
/snap/bin/hugo

Removing bundled dependencies is not the primary motivation behind our request for classic confinement. ↩
Based on this comment, the Node.js version is too old (v10) on some platforms. We should be bundling v12.20 at a minimum, preferably v16. ↩
The existing snapcraft.yaml includes a part for pandoc, but it is not working. "error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory" ↩

jmooring · 2021-10-26T00:57:47Z

Hugo was granted classic confinement by the Snap Store today:
https://forum.snapcraft.io/t/19892/16

classic confinement granted for hugo - this is now live (ie future uploads of hugo which specify confinement: classic will pass automated review)

Note existing users of the snap will not be automatically upgraded to the classic version - so you should publish an update which is still strictly confined that prompts them to manually refresh to get the classic version (or you could perhaps use a separate track for classic…?)

ie. users will need to manually run: snap refresh --classic hugo

I recommend that we defer this for now, and instead attempt to fix the known issues (see above) with strict confinement. If we can fix all of the missing dependencies, including Go, then I think the only reason to offer classic confinement would be access to the file system outside of $HOME. In this case, I think classic confinement should be provided as a separate track.

Letme · 2022-08-03T10:14:03Z

Ping on this, as it seems like long lasting problem that was never fixed. It also means there is no decent installation option for newer versions of hugo (apt has version 0.92)

anthonyfok · 2022-08-03T16:07:18Z

@Letme Thank you for the heads up! I didn't realize that Ubuntu kinetic (the future 22.10 release) has Hugo stuck on 0.92.2-1 when it is already 0.101.0-1 on Debian. It seems to be due to one of the updated dependencies (aws-sdk-go) not being synchronized from Debian. The Ubuntu developers have just been notified and will hopefully have the issue resolved soon so that you'll get Hugo v0.101.0 in Ubuntu 22.10.

Meanwhile, about this issue: It might be partially solved (Node.js and pandoc are probably updated and working), but I'll need to find time to actually sit down, test, and work on any remaining issues. Sorry for the delay!

@jmooring

Git: - Set GIT_EXEC_PATH and include usr/lib/git-core so that git can find the git-remote-https helper (needed by e.g. "go mod download"). Go: - Put Go in its own snap part, and use stage-snaps instead of build-snaps so that it is included in the final snap. - Set GOCACHE to a writable directory. (In a previous commit, HOME is set to $SNAP_REAL_HOME which is unwritable.) Hugo: - Patch config/security/securityConfig.go "[security.exec] osEnv" whitelist (during snap build) so that external dependencies can use the required environment variables to run properly from within the snap. Asciidoctor: - Replace shebang line in asciidoctor so it can find the ruby executable. - Set RUBYLIB so that Ruby can find its libraries. Caveat/TODO: The Ruby version is hardcoded in our custom RUBYLIB. Embedded Dart Sass: - Download from GitHub and install it, for amd64 and arm64 only. Node.js: - Remove my incomplete include list so that npx is actually installed. - Set npm_config_{cache,init_module,userconfig} to writable locations. Pandoc: - Set pandoc_datadir so that Pandoc can find its data files. rst2html: - Install python3-docutils package for rst2html, rst2html5, etc. - Set PYTHONHOME so that Python can find its libraries. Note that asciidoctor, pandoc and rst2html are not in Hugo’s default "[security.exec] allow" whitelist, and the snap package does not change that default, so they still needed to be whitelisted manually in config.toml if necessary by the end user. Special thanks to Joe Mooring (@jmooring) for meticulously diagnosing the issue and providing a comprehensive test repository at https://github.com/jmooring/hugo-snap-test without which I would not have been able to understand and resolve the issue. Fixes #9078

@jmooring

Git: - Set GIT_EXEC_PATH and include usr/lib/git-core so that git can find the git-remote-https helper (needed by e.g. "go mod download"). Go: - Put Go in its own snap part, and use stage-snaps instead of build-snaps so that it is included in the final snap. - Set GOCACHE to a writable directory. (In a previous commit, HOME is set to $SNAP_REAL_HOME which is unwritable.) Hugo: - Patch config/security/securityConfig.go "[security.exec] osEnv" whitelist (during snap build) so that external dependencies can use the required environment variables to run properly from within the snap. Asciidoctor: - Replace shebang line in asciidoctor so it can find the ruby executable. - Set RUBYLIB so that Ruby can find its libraries. Caveat/TODO: The Ruby version is hardcoded in our custom RUBYLIB. Embedded Dart Sass: - Download from GitHub and install it, for amd64 and arm64 only. Node.js: - Remove my incomplete include list so that npx is actually installed. - Set npm_config_{cache,init_module,userconfig} to writable locations. Pandoc: - Set pandoc_datadir so that Pandoc can find its data files. rst2html: - Install python3-docutils package for rst2html, rst2html5, etc. - Set PYTHONHOME so that Python can find its libraries. Note that asciidoctor, pandoc and rst2html are not in Hugo’s default "[security.exec] allow" whitelist, and the snap package does not change that default, so they still needed to be whitelisted manually in config.toml if necessary by the end user. Special thanks to Joe Mooring (@jmooring) for meticulously diagnosing the issue and providing a comprehensive test repository at https://github.com/jmooring/hugo-snap-test without which I would not have been able to understand and resolve the issue. Fixes gohugoio#9078

@jmooring

Git: - Set GIT_EXEC_PATH and include usr/lib/git-core so that git can find the git-remote-https helper (needed by e.g. "go mod download"). Go: - Put Go in its own snap part, and use stage-snaps instead of build-snaps so that it is included in the final snap. - Set GOCACHE to a writable directory. (In a previous commit, HOME is set to $SNAP_REAL_HOME which is unwritable.) Hugo: - Patch config/security/securityConfig.go "[security.exec] osEnv" whitelist (during snap build) so that external dependencies can use the required environment variables to run properly from within the snap. Asciidoctor: - Replace shebang line in asciidoctor so it can find the ruby executable. - Set RUBYLIB so that Ruby can find its libraries. Caveat/TODO: The Ruby version is hardcoded in our custom RUBYLIB. Embedded Dart Sass: - Download from GitHub and install it, for amd64 and arm64 only. Node.js: - Remove my incomplete include list so that npx is actually installed. - Set npm_config_{cache,init_module,userconfig} to writable locations. Pandoc: - Set pandoc_datadir so that Pandoc can find its data files. rst2html: - Install python3-docutils package for rst2html, rst2html5, etc. - Set PYTHONHOME so that Python can find its libraries. Note that asciidoctor, pandoc and rst2html are not in Hugo’s default "[security.exec] allow" whitelist, and the snap package does not change that default, so they still needed to be whitelisted manually in config.toml if necessary by the end user. Special thanks to Joe Mooring (@jmooring) for meticulously diagnosing the issue and providing a comprehensive test repository at https://github.com/jmooring/hugo-snap-test without which I would not have been able to understand and resolve the issue. Fixes gohugoio#9078

@jmooring

Git: - Set GIT_EXEC_PATH and include usr/lib/git-core so that git can find the git-remote-https helper (needed by e.g. "go mod download"). Go: - Put Go in its own snap part, and use stage-snaps instead of build-snaps so that it is included in the final snap. - Set GOCACHE to a writable directory. (In a previous commit, HOME is set to $SNAP_REAL_HOME which is unwritable.) Hugo: - Patch config/security/securityConfig.go "[security.exec] osEnv" whitelist (during snap build) so that external dependencies can use the required environment variables to run properly from within the snap. Asciidoctor: - Replace shebang line in asciidoctor so it can find the ruby executable. - Set RUBYLIB so that Ruby can find its libraries. Caveat/TODO: The Ruby version is hardcoded in our custom RUBYLIB. Embedded Dart Sass: - Download from GitHub and install it, for amd64 and arm64 only. Node.js: - Remove my incomplete include list so that npx is actually installed. - Set npm_config_{cache,init_module,userconfig} to writable locations. Pandoc: - Set pandoc_datadir so that Pandoc can find its data files. rst2html: - Install python3-docutils package for rst2html, rst2html5, etc. - Set PYTHONHOME so that Python can find its libraries. Note that asciidoctor, pandoc and rst2html are not in Hugo’s default "[security.exec] allow" whitelist, and the snap package does not change that default, so they still needed to be whitelisted manually in config.toml if necessary by the end user. Special thanks to Joe Mooring (@jmooring) for meticulously diagnosing the issue and providing a comprehensive test repository at https://github.com/jmooring/hugo-snap-test without which I would not have been able to understand and resolve the issue. Fixes #9078

anthonyfok · 2022-08-13T18:34:43Z

Dear all,

I believe this has just been fixed for v0.101.0 in the stable branch. Please install/refresh the hugo snap to v0.101.0+git4.d7294751 (13802) in the extended/stable channel.

See #10174 for the pull request for the master branch.

Due to the recently improved security model in Hugo, the following patch needs to be applied to https://github.com/jmooring/hugo-snap-test for the test to pass:

diff --git a/config/_default/config.toml b/config/_default/config.toml
index 3bf5e3a..6306b75 100644
--- a/config/_default/config.toml
+++ b/config/_default/config.toml
@@ -15,3 +15,7 @@ target = 'content'
 
 [params]
 transpiler = 'dartsass'  # dartsass or libsass
+
+[security]
+  [security.exec]
+    allow = ['^asciidoctor$', '^dart-sass-embedded$', '^go$', '^npx$', '^pandoc$', '^postcss$', '^rst2html$']

@jmooring

Git: - Set GIT_EXEC_PATH and include usr/lib/git-core so that git can find the git-remote-https helper (needed by e.g. "go mod download"). Go: - Put Go in its own snap part, and use stage-snaps instead of build-snaps so that it is included in the final snap. - Set GOCACHE to a writable directory. (In a previous commit, HOME is set to $SNAP_REAL_HOME which is unwritable.) Hugo: - Patch config/security/securityConfig.go "[security.exec] osEnv" whitelist (during snap build) so that external dependencies can use the required environment variables to run properly from within the snap. Asciidoctor: - Replace shebang line in asciidoctor so it can find the ruby executable. - Set RUBYLIB so that Ruby can find its libraries. Caveat/TODO: The Ruby version is hardcoded in our custom RUBYLIB. Embedded Dart Sass: - Download from GitHub and install it, for amd64 and arm64 only. Node.js: - Remove my incomplete include list so that npx is actually installed. - Set npm_config_{cache,init_module,userconfig} to writable locations. Pandoc: - Set pandoc_datadir so that Pandoc can find its data files. rst2html: - Install python3-docutils package for rst2html, rst2html5, etc. - Set PYTHONHOME so that Python can find its libraries. Note that asciidoctor, pandoc and rst2html are not in Hugo’s default "[security.exec] allow" whitelist, and the snap package does not change that default, so they still needed to be whitelisted manually in config.toml if necessary by the end user. Special thanks to Joe Mooring (@jmooring) for meticulously diagnosing the issue and providing a comprehensive test repository at https://github.com/jmooring/hugo-snap-test without which I would not have been able to understand and resolve the issue. Fixes #9078

@jmooring

Git: - Set GIT_EXEC_PATH and include usr/lib/git-core so that git can find the git-remote-https helper (needed by e.g. "go mod download"). Go: - Put Go in its own snap part, and use stage-snaps instead of build-snaps so that it is included in the final snap. - Set GOCACHE to a writable directory. (In a previous commit, HOME is set to $SNAP_REAL_HOME which is unwritable.) Hugo: - Patch config/security/securityConfig.go "[security.exec] osEnv" whitelist (during snap build) so that external dependencies can use the required environment variables to run properly from within the snap. Asciidoctor: - Replace shebang line in asciidoctor so it can find the ruby executable. - Set RUBYLIB so that Ruby can find its libraries. Caveat/TODO: The Ruby version is hardcoded in our custom RUBYLIB. Embedded Dart Sass: - Download from GitHub and install it, for amd64 and arm64 only. Node.js: - Remove my incomplete include list so that npx is actually installed. - Set npm_config_{cache,init_module,userconfig} to writable locations. Pandoc: - Set pandoc_datadir so that Pandoc can find its data files. rst2html: - Install python3-docutils package for rst2html, rst2html5, etc. - Set PYTHONHOME so that Python can find its libraries. Note that asciidoctor, pandoc and rst2html are not in Hugo’s default "[security.exec] allow" whitelist, and the snap package does not change that default, so they still needed to be whitelisted manually in config.toml if necessary by the end user. Special thanks to Joe Mooring (@jmooring) for meticulously diagnosing the issue and providing a comprehensive test repository at https://github.com/jmooring/hugo-snap-test without which I would not have been able to understand and resolve the issue. Fixes #9078

github-actions · 2022-09-20T02:17:06Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

jmooring added the Proposal label Oct 24, 2021

This was referenced Oct 24, 2021

Snap installation not compatible with modules gohugoio/hugoDocs#1222

Closed

[Snap] Error: failed to download modules: exec: "go": executable file not found in $PATH #9012

Closed

rst2html tool is not included in snap package #5500

Closed

jmooring added the Feature: Snap Package label Oct 24, 2021

anthonyfok self-assigned this Oct 25, 2021

jmooring mentioned this issue Oct 28, 2021

Postcss error with snap install of hugo v0.88.1+extended #9092

Closed

jmooring mentioned this issue Feb 20, 2022

Add MPR installation instructions gohugoio/hugoDocs#1659

Closed

bep mentioned this issue Aug 3, 2022

Hugo binary seems to have external dependency, despite README.md's claim #10136

Closed

anthonyfok added this to the v0.102.0 milestone Aug 13, 2022

anthonyfok mentioned this issue Aug 13, 2022

snap: Make external dependencies actually work #10174

Merged

bep modified the milestones: v0.102.0, v0.103.0 Aug 28, 2022

anthonyfok closed this as completed in #10174 Aug 29, 2022

github-actions bot added the Outdated label Sep 20, 2022

github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve the Hugo snap package #9078

Improve the Hugo snap package #9078

jmooring commented Oct 24, 2021 •

edited

Loading

jmooring commented Oct 26, 2021 •

edited

Loading

Letme commented Aug 3, 2022

anthonyfok commented Aug 3, 2022

anthonyfok commented Aug 13, 2022

github-actions bot commented Sep 20, 2022

Improve the Hugo snap package #9078

Improve the Hugo snap package #9078

Comments

jmooring commented Oct 24, 2021 • edited Loading

Background

Dependencies

Testing

Footnotes

jmooring commented Oct 26, 2021 • edited Loading

Letme commented Aug 3, 2022

anthonyfok commented Aug 3, 2022

anthonyfok commented Aug 13, 2022

github-actions bot commented Sep 20, 2022

jmooring commented Oct 24, 2021 •

edited

Loading

jmooring commented Oct 26, 2021 •

edited

Loading