Summer lovin' refactorization

* Add multiple boto profile management
* Add data validators
* Add personalized exceptions
* Update all parsers
* Remove obsolete code
* Update cfg file
* Fix a ton of bugs
* Update Dockerfile
* Simplify code in in almost all classes
* Update log file path

.gitignore

@@ -4,3 +4,4 @@ test/*.zip
 *.pyc
 *.zip*
 *.log
+.settings

Dockerfile

@@ -2,13 +2,15 @@ FROM python:3.7.0-stretch
 
 MAINTAINER @iMerica <imerica@me.com>
 
-RUN apt-get update && apt-get install -y zip
+RUN apt-get update && apt-get install -y \
+    zip \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
 
 RUN addgroup --system scar  && adduser --system --group scar
 
 RUN git clone --branch master --depth 1 https://github.com/grycap/scar.git /usr/bin/scar && \
-    pip install -r /usr/bin/scar/requirements.txt && \
-    pip install pyyaml
+    pip install -r /usr/bin/scar/requirements.txt 
 
 RUN touch /scar.log && chown scar /scar.log
 

docs/source/installation.rst

@@ -1,28 +1,23 @@
 Installation
 ============
 
-1) SCAR requires python3, first make sure you have python3 available in your system
+1) SCAR requires python3, pip3 and a configured AWS credentials file in your system.
+   More info about the AWS credentials file can be found `here <https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html>`_.
 
 2) Clone the GitHub repository::
 
     git clone https://github.com/grycap/scar.git
+    cd scar
 
-3) Install the required dependencies:
+3) Install the python required dependencies automatically with the command::
 
-  * `zip <https://linux.die.net/man/1/zip>`_ (linux package)
-  * `Boto3 <https://pypi.org/project/boto3/>`_ (v1.4.4+ is required)
-  * `Tabulate <https://pypi.python.org/pypi/tabulate>`_
-  * `Requests <https://pypi.org/project/requests/>`_
+    sudo pip3 install -r requirements.txt
 
-  You can automatically install the python dependencies by issuing the following command::
+  The last dependency needs to be installed using the apt manager::
+  
+    sudo apt install zip
 
-      sudo pip install -r requirements.txt
 
-  The zip package can be installed using apt::
-
-      sudo apt install zip
-
-
-4) (Optional) Define an alias for increased usability::
+4) (Optional) Define an alias for easier usability::
 
     alias scar='python3 `pwd`/scar.py'

examples/cowsay/README.md

@@ -53,13 +53,13 @@ docker save grycap/minicow > minicow.tar.gz
 2. Create the Lambda function using the 'scar-minicow.yaml' configuration file:
 
 ```sh
-scar init -f scar-cowsay.yaml
+scar init -f scar-minicow.yaml
 ```
 
 3. Execute the Lambda function
 
 ```sh
-scar run -f scar-cowsay.yaml
+scar run -f scar-minicow.yaml
 ```
 
 From the user perspective nothing changed in comparison with the previous execution, but the main difference with the 'standard' lambda deployment is that the container is already available when the function is launched for the first time. Moreover, the function doesn't need to connect to any external repository to download the container, so this is also useful to execute small binaries or containers that you don't want to upload to a public repository.

examples/darknet/README.md

@@ -75,15 +75,8 @@ darknet/output/68f5c9d5-5826-44gr-basc-8f8b23f44cdf/result.out
 
 The files are created in the output folder following the `s3://scar-darknet-bucket/scar-darknet-s3/output/$REQUEST_ID/*.*` structure.
 
-To download the created files you can also use SCAR:
 
-Download an specific file with :
-
-```sh
-scar get -b scar-darknet-bucket -bf scar-darknet-s3/output/68f5c9d5-5826-44gr-basc-8f8b23f44cdf/image-result.png -p /tmp/result.png
-```
-
-Download a folder with:
+To download the created files you can also use SCAR. Download a folder with:
 
 ```sh
 scar get -b scar-darknet-bucket -bf scar-darknet-s3/output -p /tmp/lambda/

requirements.txt

@@ -2,3 +2,4 @@ boto3
 tabulate
 configparser
 requests
+pyyaml

scar.py

@@ -18,8 +18,11 @@
 from src.providers.aws.controller import AWS
 from src.parser.cli import CommandParser
 from src.parser.yaml import YamlParser
+from src.parser.cfgfile import ConfigFileParser
 from src.cmdtemplate import Commands
-import src.logger as logger 
+import src.logger as logger
+import src.exceptions as excp
+import src.utils as utils
 
 class Scar(Commands):
 
@@ -52,24 +55,30 @@ def put(self):
 
     def get(self):
         self.cloud_provider.get()        
+
+    @excp.exception(logger)        
+    def parse_arguments(self):
+        '''
+        Merge the scar.conf parameters, the cmd parameters and the yaml file parameters in a single dictionary.
         
-    def parse_command_arguments(self):
-        args = CommandParser(self).parse_arguments()
-        if hasattr(args, 'func'):
-            if hasattr(args, 'conf_file') and args.conf_file:
-                # Update the arguments with the values extracted from the configuration file
-                args.__dict__.update(YamlParser(args).parse_arguments())
-            self.cloud_provider.parse_command_arguments(args)
-            args.func()
-        else:
-            logger.error("Incorrect arguments: use scar -h to see the options available")
+        The precedence of parameters is CMD >> YAML >> SCAR.CONF
+        That is, the CMD parameter will override any other configuration, 
+        and the YAML parameters will override the SCAR.CONF settings
+        '''
+        merged_args = ConfigFileParser().get_properties()
+        cmd_args = CommandParser(self).parse_arguments()
+        if 'conf_file' in cmd_args['scar'] and cmd_args['scar']['conf_file']:
+            yaml_args = YamlParser(cmd_args['scar']).parse_arguments()
+            merged_args = utils.merge_dicts(yaml_args, merged_args)
+        merged_args = utils.merge_dicts(cmd_args, merged_args)
+        self.cloud_provider.parse_arguments(**merged_args)
+        merged_args['scar']['func']()
 
 if __name__ == "__main__":
     logger.init_execution_trace()
     try:
-        Scar().parse_command_arguments()
+        Scar().parse_arguments()
         logger.end_execution_trace()
-    except Exception as ex:
-        logger.exception(ex)
+    except:
         logger.end_execution_trace_with_errors()
 

src/cmdtemplate.py

@@ -15,6 +15,18 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import abc
+from enum import Enum
+
+class CallType(Enum):
+    INIT = "init"
+    INVOKE = "invoke"
+    RUN = "run"
+    UPDATE = "update"    
+    LS = "ls"
+    RM = "rm"
+    LOG = "log"
+    PUT = "put"
+    GET = "get"  
 
 class Commands(metaclass=abc.ABCMeta):
     ''' All the different cloud provider controllers must inherit 
@@ -57,5 +69,5 @@ def get(self):
         pass    
 
     @abc.abstractmethod
-    def parse_command_arguments(self, args):
+    def parse_arguments(self, args):
         pass    

src/exceptions.py

@@ -13,6 +13,38 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import functools
+from botocore.exceptions import ClientError
+import sys
+
+def exception(logger):
+    '''
+    A decorator that wraps the passed in function and logs exceptions
+    @param logger: The logging object
+    '''
+    def decorator(func):
+        @functools.wraps(func)
+        def wrapper(*args, **kwargs):
+            try:
+                return func(*args, **kwargs)
+            except ClientError as ce:
+                print("There was an exception in {0}".format(func.__name__))
+                print(ce.response['Error']['Message'])
+                logger.exception(ce)
+                sys.exit(1)
+            except ScarError as se:
+                print(se.args[0])
+                logger.exception(se)
+                # Finish the execution if it's an error
+                if 'Error' in se.__class__.__name__:
+                    sys.exit(1)
+            except Exception as ex:
+                print("There was an unmanaged exception in {0}".format(func.__name__))
+                logger.exception(ex)
+                sys.exit(1)
+        return wrapper
+    return decorator
+
 class ScarError(Exception):
     """
     The base exception class for ScarError exceptions.
@@ -26,20 +58,172 @@ def __init__(self, **kwargs):
         Exception.__init__(self, msg)
         self.kwargs = kwargs
 
+################################################
+##             GENERAL EXCEPTIONS             ##
+################################################
+class MissingCommandError(ScarError):
+    """
+    SCAR was launched without a command
+
+    """
+    fmt = "Please use one of the scar available commands (init,invoke,run,update,rm,ls,log,put,get)"
+
+class ScarConfigFileError(ScarError):
+    """
+    The SCAR configuration file does not exist and it has been created
+
+    :ivar file_path: Path of the file
+    """
+    fmt = "Config file '{file_path}' created.\n"
+    fmt += "Please, set a valid iam role in the file field 'role' before the first execution."    
+
+class YamlFileNotFoundError(ScarError):
+    """
+    The yaml configuration file does not exist
+
+    :ivar file_path: Path of the file
+    """
+    fmt = "Unable to find the yaml file '{file_path}'"
+
+class ValidatorError(ScarError):
+    """
+    An error occurred when validating a parameter
+
+    :ivar parameter: Name of the parameter evaluated
+    :ivar parameter_value: Current value of the validated parameter
+    :ivar error_msg: General error message
+    """
+    fmt = "Error validating '{parameter}'.\nValue '{parameter_value}' incorrect.\n{error_msg}"
+
+class ScarFunctionNotFoundError(ScarError):
+    """
+    The called function was not found
+
+    :ivar func_name: Name of the function called
+    """
+    fmt = "Unable to find the function '{func_name}'"
+
+class FunctionCodeSizeError(ScarError):
+    """
+    Function code size exceeds AWS limits
+
+    :ivar code_size: Name of the parameter evaluated
+    """
+    fmt = "Payload size greater than {code_size}.\nPlease reduce the payload size or use an S3 bucket and try again."
+
+class S3CodeSizeError(ScarError):
+    """
+    Function code uploaded to S3 exceeds AWS limits
+
+    :ivar code_size: Name of the parameter evaluated
+    """
+
+    fmt = "Uncompressed image size greater than {code_size}.\nPlease reduce the uncompressed image and try again."
+
+################################################
+##             LAMBDA EXCEPTIONS              ##
+################################################
 class FunctionCreationError(ScarError):
     """
     An error occurred when creating the lambda function.
 
-    :ivar name: Name of the function
+    :ivar function_name: Name of the function
+    :ivar error_msg: General error message    
     """
     fmt = "Unable to create the function '{function_name}' : {error_msg}"        
 
 class FunctionNotFoundError(ScarError):
     """
     The requested function does not exist.
 
-    :ivar name: Name of the function
+    :ivar function_name: Name of the function
     """
-    fmt = "Unable to find the function '{function_name}' : {error_msg}"
+    fmt = "Unable to find the function '{function_name}'"
 
+class FunctionExistsError(ScarError):
+    """
+    The requested function exists.
+
+    :ivar function_name: Name of the function
+    """
+    fmt = "Function '{function_name}' already exists"    
+
+################################################
+##               S3 EXCEPTIONS                ##
+################################################
+class BucketNotFoundError(ScarError):
+    """
+    The requested bucket does not exist.
+
+    :ivar bucket_name: Name of the bucket
+    """
+    fmt = "Unable to find the bucket '{bucket_name}'."
+
+class ExistentBucketWarning(ScarError):
+    """
+    The bucket already exists
+
+    :ivar bucket_name: Name of the bucket
+    """
+    fmt = "Using existent bucket '{bucket_name}'."    
+
+################################################
+##         CLOUDWATCH LOGS EXCEPTIONS         ##
+################################################
+class ExistentLogGroupWarning(ScarError):
+    """
+    The requested log group already exists
+
+    :ivar log_group_name: Name of the log group
+    """
+    fmt = "Using existent log group '{logGroupName}'."
+
+class NotExistentLogGroupWarning(ScarError):
+    """
+    The requested log group does not exists
+
+    :ivar log_group_name: Name of the log group
+    """
+    fmt = "The requested log group '{logGroupName}' does not exist."
+
+################################################
+##           API GATEWAY EXCEPTIONS           ##
+################################################
+class ApiEndpointNotFoundError(ScarError):
+    """
+    The requested function does not have an associated API.
+
+    :ivar function_name: Name of the function
+    """
+    fmt = "Error retrieving API ID for lambda function '{function_name}'\n"
+    fmt += "Looks like he requested function does not have an associated API."
+
+class ApiCreationError(ScarError):
+    """
+    Error creating the API endpoint.
+
+    :ivar api_name: Name of the api
+    """
+    fmt = "Error creating the API '{api_name}'"
+
+class InvocationPayloadError(ScarError):
+    """
+    Error invocating the API endpoint.
+
+    :ivar file_size: Size of the passed file
+    :ivar max_size: Max size allowd of the file
+    """
+    fmt = "Invalid request: Payload size {file_size} greater than {max_size}\n"
+    fmt += "Check AWS Lambda invocation limits in : https://docs.aws.amazon.com/lambda/latest/dg/limits.html"
+
+################################################
+##               IAM EXCEPTIONS               ##
+################################################
+class GetUserInfoError(ScarError):
+    """
+    There was an error gettting the IAM user info
+
+    :ivar error_msg: General error message    
+    """
+    fmt = "Error getting the AWS user information.\n{error_msg}."