Ruby Build Initial Design

.appveyor.yml

@@ -19,6 +19,9 @@ install:
 - "set PATH=%PYTHON%\\Scripts;%PYTHON%\\bin;%PATH%"
 - "%PYTHON%\\python.exe -m pip install -r requirements/dev.txt"
 - "%PYTHON%\\python.exe -m pip install -e ."
+- "set PATH=C:\\Ruby25-x64\\bin;%PATH%"
+- "gem install bundler --no-ri --no-rdoc"
+- "bundler --version"
 
 test_script:
 - "%PYTHON%\\python.exe -m pytest --cov aws_lambda_builders --cov-report term-missing tests/unit tests/functional"

.gitignore

@@ -162,6 +162,9 @@ typings/
 # Output of 'npm pack'
 *.tgz
 
+# Except test file
+!tests/functional/workflows/ruby_bundler/test_data/test.tgz
+
 # Yarn Integrity file
 .yarn-integrity
 

.pylintrc

@@ -221,7 +221,7 @@ notes=FIXME,XXX
 [SIMILARITIES]
 
 # Minimum lines number of a similarity.
-min-similarity-lines=6
+min-similarity-lines=10
 
 # Ignore comments when computing similarities.
 ignore-comments=yes

aws_lambda_builders/workflows/__init__.py

@@ -3,4 +3,5 @@
 """
 
 import aws_lambda_builders.workflows.python_pip
+import aws_lambda_builders.workflows.ruby_bundler
 import aws_lambda_builders.workflows.nodejs_npm

aws_lambda_builders/workflows/ruby_bundler/DESIGN.md

@@ -0,0 +1,84 @@
+# Ruby - Lambda Builder
+
+## Scope
+
+For the basic case, building the dependencies for a Ruby Lambda project is very easy:
+
+```shell
+# ensure you are using Ruby 2.5, for example with rbenv or rvm
+bundle install # if no Gemfile.lock is present
+bundle install --deployment
+zip -r source.zip * # technically handled by `sam package`
+```
+
+The basic scope of a `sam build` script for Ruby would be as a shortcut for this, while performing some housekeeping steps:
+
+- Skipping the initial `bundle install` if a Gemfile.lock file is present.
+- Ensuring that `ruby --version` matches `/^ ruby 2\.5\./`
+- Raising a soft error if there is already a `.bundle` and `vendor/bundle` folder structure, and giving an option to clobber this if desired.
+  - I don't want this to be a default behavior, in case users are using the `vendor` or `.bundle` folder structures for other things and clobbering it could have destructive and unintended side effects.
+
+Having a unified command also gives us the ability to solve once the most common issues and alternative use cases in a way that follows best practices:
+
+1. Including dependencies that have native extensions, and building them in the proper environment.
+   - An open question is how to help users represent binary dependencies, but that's not a Ruby concern per se so it should be solved the same way across all builds.
+2. Building and deploying the user dependencies as a layer rather than as part of the code package.
+   - These also have slightly different folder pathing:
+     - Bundled dependencies are looked for in `/var/task/vendor/bundle/ruby/2.5.0` which is the default result of a `bundle install --deployment` followed by an upload.
+     - Layer dependencies are looked for in `/opt/ruby/gems/2.5.0`, so for a layer option would have to use a `--path` build or transform the folder structure slightly.
+3. Down the road, perhaps providing a way to bundle code as a layer, such as for shared libraries that are not gems. These need to go in the `/opt/ruby/lib` folder structure.
+
+## Challenges
+
+- Ensuring that builds happen in Ruby 2.5.x only.
+- Ensuring that builds that include native extensions happen in the proper build environment.
+
+## Interface/Implementation
+
+Off hand, I envision the following commands as a starting point:
+- `sam build`: Shorthand for the 2-liner build at the top of the document.
+- `sam build --use-container`: Provides a build container for native extensions.
+
+I also envision Ruby tie-ins for layer commands following the same pattern. I don't yet have a mental model for how we should do shared library code as a layer, that may be an option that goes into `sam init` perhaps? Like `sam init --library-layer`? Layer implementations will be solved at a later date.
+
+Some other open issues include more complex Gemfiles, where a user might want to specify certain bundle groups to explicitly include or exclude. We could also build out ways to switch back and forth between deployment and no-deployment modes.
+
+### sam build
+
+First, validates that `ruby --version` matches a `ruby 2.5.x` pattern, and exits if not. When in doubt, container builds will not have this issue.
+
+```shell
+# exit with error if vendor/bundle and/or .bundle directory exists and is non-empty
+bundle install # if no Gemfile.lock is present
+bundle install --deployment
+```
+
+This build could also include an optional cleanout of existing `vendor/bundle` and `.bundle` directories, via the `--clobber-bundle` command or similar. That would behave as follows:
+
+```shell
+rm -rf vendor/bundle*
+rm -rf .bundle*
+bundle install # if no Gemfile.lock is present
+bundle install --deployment
+```
+
+### sam build --use-container
+
+This command would use some sort of container, such as `lambci/lambda:build-ruby2.5`.
+
+```shell
+# exit with error if vendor/bundle and/or .bundle directory exists and is non-empty
+bundle install # if no Gemfile.lock is present
+docker run -v `pwd`:`pwd` -w `pwd` -i -t $CONTAINER_ID bundle install --deployment
+```
+
+This approach does not need to validate the version of Ruby being used, as the container would use Ruby 2.5.
+
+This build could also include an optional cleanout of existing `vendor/bundle` and `.bundle` directories, via the `--clobber-bundle` command or similar. That would behave as follows:
+
+```shell
+rm -rf vendor/bundle*
+rm -rf .bundle*
+bundle install # if no Gemfile.lock is present
+docker run -v `pwd`:`pwd` -w `pwd` -i -t $CONTAINER_ID bundle install --deployment
+```

aws_lambda_builders/workflows/ruby_bundler/__init__.py

@@ -0,0 +1,5 @@
+"""
+Builds Ruby Lambda functions using Bundler
+"""
+
+from .workflow import RubyBundlerWorkflow

aws_lambda_builders/workflows/ruby_bundler/actions.py

@@ -0,0 +1,59 @@
+"""
+Actions for Ruby dependency resolution with Bundler
+"""
+
+import logging
+
+from aws_lambda_builders.actions import BaseAction, Purpose, ActionFailedError
+from .bundler import BundlerExecutionError
+
+LOG = logging.getLogger(__name__)
+
+class RubyBundlerInstallAction(BaseAction):
+
+    """
+    A Lambda Builder Action which runs bundle install in order to build a full Gemfile.lock
+    """
+
+    NAME = 'RubyBundle'
+    DESCRIPTION = "Resolving dependencies using Bundler"
+    PURPOSE = Purpose.RESOLVE_DEPENDENCIES
+
+    def __init__(self, source_dir, subprocess_bundler):
+        super(RubyBundlerInstallAction, self).__init__()
+        self.source_dir = source_dir
+        self.subprocess_bundler = subprocess_bundler
+
+    def execute(self):
+        try:
+            LOG.debug("Running bundle install in %s", self.source_dir)
+            self.subprocess_bundler.run(
+                ['install', '--without', 'development', 'test'],
+                cwd=self.source_dir
+            )
+        except BundlerExecutionError as ex:
+            raise ActionFailedError(str(ex))
+
+class RubyBundlerVendorAction(BaseAction):
+    """
+    A Lambda Builder Action which vendors dependencies to the vendor/bundle directory.
+    """
+
+    NAME = 'RubyBundleDeployment'
+    DESCRIPTION = "Package dependencies for deployment."
+    PURPOSE = Purpose.RESOLVE_DEPENDENCIES
+
+    def __init__(self, source_dir, subprocess_bundler):
+        super(RubyBundlerVendorAction, self).__init__()
+        self.source_dir = source_dir
+        self.subprocess_bundler = subprocess_bundler
+
+    def execute(self):
+        try:
+            LOG.debug("Running bundle install --deployment in %s", self.source_dir)
+            self.subprocess_bundler.run(
+                ['install', '--deployment', '--without', 'development', 'test'],
+                cwd=self.source_dir
+            )
+        except BundlerExecutionError as ex:
+            raise ActionFailedError(str(ex))

aws_lambda_builders/workflows/ruby_bundler/bundler.py

@@ -0,0 +1,57 @@
+"""
+Wrapper around calls to bundler through a subprocess.
+"""
+
+import logging
+
+LOG = logging.getLogger(__name__)
+
+class BundlerExecutionError(Exception):
+    """
+    Exception raised when Bundler fails.
+    Will encapsulate error output from the command.
+    """
+
+    MESSAGE = "Bundler Failed: {message}"
+
+    def __init__(self, **kwargs):
+        Exception.__init__(self, self.MESSAGE.format(**kwargs))
+
+class SubprocessBundler(object):
+    """
+    Wrapper around the Bundler command line utility, encapsulating
+    execution results.
+    """
+
+    def __init__(self, osutils, bundler_exe=None):
+        self.osutils = osutils
+        if bundler_exe is None:
+            if osutils.is_windows():
+                bundler_exe = 'bundler.bat'
+            else:
+                bundler_exe = 'bundle'
+
+        self.bundler_exe = bundler_exe
+
+    def run(self, args, cwd=None):
+        if not isinstance(args, list):
+            raise ValueError('args must be a list')
+
+        if not args:
+            raise ValueError('requires at least one arg')
+
+        invoke_bundler = [self.bundler_exe] + args
+
+        LOG.debug("executing Bundler: %s", invoke_bundler)
+
+        p = self.osutils.popen(invoke_bundler,
+                               stdout=self.osutils.pipe,
+                               stderr=self.osutils.pipe,
+                               cwd=cwd)
+
+        out, err = p.communicate()
+
+        if p.returncode != 0:
+            raise BundlerExecutionError(message=err.decode('utf8').strip())
+
+        return out.decode('utf8').strip()

aws_lambda_builders/workflows/ruby_bundler/utils.py

@@ -0,0 +1,40 @@
+"""
+Commonly used utilities
+"""
+
+import os
+import platform
+import tarfile
+import subprocess
+
+
+class OSUtils(object):
+
+    """
+    Wrapper around file system functions, to make it easy to
+    unit test actions in memory
+    """
+
+    def extract_tarfile(self, tarfile_path, unpack_dir):
+        with tarfile.open(tarfile_path, 'r:*') as tar:
+            tar.extractall(unpack_dir)
+
+    def popen(self, command, stdout=None, stderr=None, env=None, cwd=None):
+        p = subprocess.Popen(command, stdout=stdout, stderr=stderr, env=env, cwd=cwd)
+        return p
+
+    def joinpath(self, *args):
+        return os.path.join(*args)
+
+    @property
+    def pipe(self):
+        return subprocess.PIPE
+
+    def dirname(self, path):
+        return os.path.dirname(path)
+
+    def abspath(self, path):
+        return os.path.abspath(path)
+
+    def is_windows(self):
+        return platform.system().lower() == 'windows'

aws_lambda_builders/workflows/ruby_bundler/workflow.py

@@ -0,0 +1,55 @@
+"""
+Ruby Bundler Workflow
+"""
+
+from aws_lambda_builders.workflow import BaseWorkflow, Capability
+from aws_lambda_builders.actions import CopySourceAction
+from .actions import RubyBundlerInstallAction, RubyBundlerVendorAction
+from .utils import OSUtils
+from .bundler import SubprocessBundler
+
+
+class RubyBundlerWorkflow(BaseWorkflow):
+
+    """
+    A Lambda builder workflow that knows how to build
+    Ruby projects using Bundler.
+    """
+    NAME = "RubyBundlerBuilder"
+
+    CAPABILITY = Capability(language="ruby",
+                            dependency_manager="bundler",
+                            application_framework=None)
+
+    EXCLUDED_FILES = (".aws-sam")
+
+    def __init__(self,
+                 source_dir,
+                 artifacts_dir,
+                 scratch_dir,
+                 manifest_path,
+                 runtime=None,
+                 osutils=None,
+                 **kwargs):
+
+        super(RubyBundlerWorkflow, self).__init__(source_dir,
+                                                  artifacts_dir,
+                                                  scratch_dir,
+                                                  manifest_path,
+                                                  runtime=runtime,
+                                                  **kwargs)
+
+        if osutils is None:
+            osutils = OSUtils()
+
+        subprocess_bundler = SubprocessBundler(osutils)
+        bundle_install = RubyBundlerInstallAction(artifacts_dir,
+                                                  subprocess_bundler=subprocess_bundler)
+
+        bundle_deployment = RubyBundlerVendorAction(artifacts_dir,
+                                                    subprocess_bundler=subprocess_bundler)
+        self.actions = [
+            CopySourceAction(source_dir, artifacts_dir, excludes=self.EXCLUDED_FILES),
+            bundle_install,
+            bundle_deployment,
+        ]

tests/functional/workflows/ruby_bundler/test_ruby_utils.py

@@ -0,0 +1,54 @@
+import os
+import shutil
+import sys
+import tempfile
+
+from unittest import TestCase
+
+from aws_lambda_builders.workflows.ruby_bundler import utils
+
+
+class TestOSUtils(TestCase):
+
+    def setUp(self):
+        self.osutils = utils.OSUtils()
+
+    def test_extract_tarfile_unpacks_a_tar(self):
+        test_tar = os.path.join(os.path.dirname(__file__), "test_data", "test.tgz")
+        test_dir = tempfile.mkdtemp()
+        self.osutils.extract_tarfile(test_tar, test_dir)
+        output_files = set(os.listdir(test_dir))
+        shutil.rmtree(test_dir)
+        self.assertEqual({"test_utils.py"}, output_files)
+
+    def test_dirname_returns_directory_for_path(self):
+        dirname = self.osutils.dirname(sys.executable)
+        self.assertEqual(dirname, os.path.dirname(sys.executable))
+
+    def test_abspath_returns_absolute_path(self):
+        result = self.osutils.abspath('.')
+        self.assertTrue(os.path.isabs(result))
+        self.assertEqual(result, os.path.abspath('.'))
+
+    def test_joinpath_joins_path_components(self):
+        result = self.osutils.joinpath('a', 'b', 'c')
+        self.assertEqual(result, os.path.join('a', 'b', 'c'))
+
+    def test_popen_runs_a_process_and_returns_outcome(self):
+        cwd_py = os.path.join(os.path.dirname(__file__), '..', '..', 'testdata', 'cwd.py')
+        p = self.osutils.popen([sys.executable, cwd_py],
+                               stdout=self.osutils.pipe,
+                               stderr=self.osutils.pipe)
+        out, err = p.communicate()
+        self.assertEqual(p.returncode, 0)
+        self.assertEqual(out.decode('utf8').strip(), os.getcwd())
+
+    def test_popen_can_accept_cwd(self):
+        testdata_dir = os.path.join(os.path.dirname(__file__), '..', '..', 'testdata')
+        p = self.osutils.popen([sys.executable, 'cwd.py'],
+                               stdout=self.osutils.pipe,
+                               stderr=self.osutils.pipe,
+                               cwd=testdata_dir)
+        out, err = p.communicate()
+        self.assertEqual(p.returncode, 0)
+        self.assertEqual(out.decode('utf8').strip(), os.path.abspath(testdata_dir))