Fix #25: using x509 certificate for access control

.travis.yml

@@ -14,9 +14,9 @@ install:
 before_script:
   - sleep 5
 script:
-#  - make docs
   - make testall
   - make pep8
+  - make docs
 #after_success:
 # - coveralls
 matrix:

CHANGES.rst

@@ -1,6 +1,18 @@
 Changes
 *******
 
+current
+=======
+
+* pep8
+* removed unused ``c4i`` option.
+* added ``auth`` option to set authentication method.
+* updated docs for usage of x509 certificates.
+
+New Features:
+
+* Feature #25: using x509 certificates for service authentication.
+
 0.3.5 (2018-03-01)
 ==================
 

README.rst

@@ -22,14 +22,24 @@ Twitcher: A simple OWS Security Proxy
 Twitcher (the bird-watcher)
   *a birdwatcher mainly interested in catching sight of rare birds.* (`Leo <https://dict.leo.org/ende/index_en.html>`_).
 
-Twitcher is a security proxy for Web Processing Services (WPS). The execution of a WPS process is blocked by the proxy. The proxy service provides access tokens (uuid, Macaroons) which needs to be used to run a WPS process. The access tokens are valid only for a short period of time.
+Twitcher is a security proxy for Web Processing Services (WPS). The execution of a WPS process is blocked by the proxy.
+The proxy service provides access tokens (uuid, Macaroons) which needs to be used to run a WPS process.
+The access tokens are valid only for a short period of time.
+In addition one can also use X.509 certificates for WPS client authentication.
 
-The implementation is not restricted to WPS services. It will be extended to more OWS services like WMS (Web Map Service) and CSW (Catalogue Service for the Web) and might also be used for Thredds catalog services.
+The implementation is not restricted to WPS services.
+It will be extended to more OWS services like WMS (Web Map Service) and CSW (Catalogue Service for the Web)
+and might also be used for Thredds catalog services.
+
+Twitcher extensions:
+
+* `Magpie`_ is an AuthN/AuthZ service provided by the `PAVICS`_ project.
 
 Twitcher is a **prototype** implemented in Python with the `Pyramid`_ web framework.
 
 Twitcher is part of the `Birdhouse`_ project. The documentation is on `ReadTheDocs`_.
 
 .. _Pyramid: http://www.pylonsproject.org
-.. _Birdhouse: http://bird-house.github.io
 .. _ReadTheDocs: http://twitcher.readthedocs.io/en/latest/
+.. _Magpie: https://github.com/Ouranosinc/Magpie
+.. _PAVICS: https://ouranosinc.github.io/pavics-sdi/index.html

buildout.cfg

@@ -126,7 +126,8 @@ etc-user = ${deployment:etc-user}
 input = ${buildout:directory}/templates/nginx.conf
 socket = ${config:socket}
 hostname =  ${settings:hostname}
-https_port = ${settings:https-port}
+https-port = ${settings:https-port}
+ssl-verify-client = optional
 
 [pytest]
 recipe = zc.recipe.egg
@@ -141,9 +142,11 @@ eggs =
     sphinx
     ${twitcher:eggs}
 
+[noversions]
+
 [versions]
 birdhousebuilder.recipe.mongodb = 0.4.0
-birdhousebuilder.recipe.nginx = 0.3.6
+birdhousebuilder.recipe.nginx = 0.3.7
 buildout.locallib = 0.3.1
 collective.recipe.environment = 1.1.0
 collective.recipe.template = 2.0

custom.cfg.example

@@ -1,20 +1,10 @@
 [buildout]
 extends = buildout.cfg
 
+## Development options
 # versions = noversions
 
 [settings]
 hostname = localhost
-http-port = 8083
 https-port = 5000
 log-level = WARN
-username =
-password =
-workdir =
-ows-security = true
-ows-proxy = true
-rpcinterface = true
-wps = true
-wps-cfg =
-
-[noversions]

docs/source/changes.rst

@@ -0,0 +1 @@
+.. include:: ../../CHANGES.rst

docs/source/conf.py

@@ -335,5 +335,4 @@
 .. _icclim: http://icclim.readthedocs.io/en/latest/
 .. _PyWPS: http://pywps.org/
 .. _dispel4py: https://github.com/dispel4py/dispel4py
-.. _esgf-pyclient: https://github.com/ESGF/esgf-pyclient
 """

docs/source/index.rst

@@ -1,21 +1,4 @@
-=====================================
-Twitcher: A simple OWS Security Proxy
-=====================================
-
-.. image:: https://travis-ci.org/bird-house/twitcher.svg?branch=master
-   :target: https://travis-ci.org/bird-house/twitcher
-   :alt: Travis Build
-
-Twitcher (the bird-watcher)
-  *a birdwatcher mainly interested in catching sight of rare birds.* (`Leo <https://dict.leo.org/ende/index_en.html>`_).
-
-Twitcher is a security proxy for Web Processing Services (WPS). The execution of a WPS process is blocked by the proxy. The proxy service provides access tokens (uuid, Macaroons) which needs to be used to run a WPS process. The access tokens are valid only for a short period of time.
-
-The implementation is not restricted to WPS services. It will be extended to more OWS services like WMS (Web Map Service) and CSW (Catalogue Service for the Web) and might also be used for Thredds catalog services.
-
-Twitcher is a **prototype** implemented in Python with the `Pyramid`_ web framework.
-
-Twitcher is part of the `Birdhouse`_ project. The documentation is on `ReadTheDocs`_.
+.. include:: ../../README.rst
 
 .. toctree::
    :maxdepth: 1
@@ -26,7 +9,5 @@ Twitcher is part of the `Birdhouse`_ project. The documentation is on `ReadTheDo
    running
    tutorial
    api
+   changes
    appendix
-
-.. _Pyramid: http://pylonsproject.org
-.. _ReadTheDocs: http://twitcher.readthedocs.io/en/latest/

docs/source/tutorial.rst

@@ -211,7 +211,7 @@ Run an ``Exceute`` request:
 
 .. code-block:: sh
 
-    $ curl -k "https://localhost:5000/ows/wps?service=wps&request=execute&identifier=hello&version=1.0.0"
+    $ curl -k "https://localhost:5000/ows/proxy/emu?service=wps&request=execute&identifier=hello&version=1.0.0&datainputs=name=tux"
 
 Now you should get an XML error response with a message that you need to provide an access token (see section above).
 
@@ -241,6 +241,41 @@ In the following example we provide the token as HTTP parameter:
    If you have set enviroment variables with your access token then they will *not* be available in the external service.
 
 
+Use x509 certificates to control client access
+==================================================
+
+Since version 0.3.6 Twitcher is prepared to use x509 certificates for control client access.
+By default it is configured to accept x509 proxy certificates from `ESGF`_.
+
+Register the Emu WPS service at the Twitcher ``OWSProxy`` with ``auth`` option ``cert``:
+
+.. code-block:: sh
+
+   $ bin/twitcherctl -k register --name emu --auth cert http://localhost:8094/wps
+
+The ``GetCapabilities``  and ``DescribeProcess`` requests are not blocked:
+
+.. code-block:: sh
+
+  $ curl -k "https://localhost:5000/ows/proxy/emu?service=wps&request=getcapabilities"
+  $ curl -k "https://localhost:5000/ows/proxy/emu?service=wps&request=describeprocess&identifier=hello&version=1.0.0"
+
+When you run an ``Exceute`` request without a certificate you should get an exception report:
+
+.. code-block:: sh
+
+  $ curl -k "https://localhost:5000/ows/proxy/emu?service=wps&request=execute&identifier=hello&version=1.0.0&datainputs=name=tux"
+
+Now you should get an XML error response with a message that you need to provide a valid X509 certificate.
+
+Get a valid proxy certificate from ESGF, you may use the `esgf-pyclient`_ to run a myproxy logon.
+Let's say your proxy certificate is ``cert.pem``, then run the exceute request again using this certificate:
+
+.. code-block:: sh
+
+  $ curl --cert cert.pem --key cert.pem -k "https://localhost:5000/ows/proxy/emu?service=wps&request=execute&identifier=hello&version=1.0.0&datainputs=name=tux"
+
+
 Use Birdy WPS command line client to run a Process
 ==================================================
 
@@ -358,3 +393,7 @@ If you don't provide a token or the token is invalid then you will get an error
 
    owslib.wps.WPSException : {'locator': 'AccessForbidden', 'code': 'NoApplicableCode', 'text': 'Access token is required to access this service.'}
    WARNING:Error: code=NoApplicableCode, locator=AccessForbidden, text=Access token is required to access this service.
+
+
+.. _ESGF: https://esgf.llnl.gov/
+.. _esgf-pyclient: https://github.com/ESGF/esgf-pyclient

environment.yml

@@ -41,4 +41,4 @@ dependencies:
 - pip:
   - genshi==0.7
   - pyramid-rpc
-  - sphinx-autoapi==0.4.0
+  - sphinx-autoapi

templates/nginx.conf

@@ -1,30 +1,37 @@
-# Phoenix: a pyramid web frontend for WPS 
+# Phoenix: a pyramid web frontend for WPS
 upstream twitcher {
     server unix://${socket} fail_timeout=0;
 }
 
 # https server
 # http://nginx.org/en/docs/http/configuring_https_servers.html
-server 
+server
 {
     listen              ${https_port} ssl;
     server_name         ${hostname};
     ssl_certificate     cert.pem;
     ssl_certificate_key cert.pem;
     ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
     ssl_ciphers         HIGH:!aNULL:!MD5;
-    #ssl_session_cache   shared:SSL:1m;
-    #ssl_session_timeout 1m;
+    ssl_session_cache   shared:SSL:1m;
+    ssl_session_timeout 1m;
+    ssl_client_certificate ${ssl_client_certificate};
+    #ssl_crl               ca.crl;
+    ssl_verify_client      ${ssl_verify_client};
+    ssl_verify_depth       2;
 
     # app
-    location / 
+    location /
     {
         proxy_pass              http://twitcher;
         proxy_set_header        X-Forwarded-Ssl on;
         proxy_set_header        Host $http_host;
         proxy_set_header        X-Real-IP       $remote_addr;
         proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header        X-Forwarded-Protocol $scheme;
+        #proxy_set_header        X-SSL-Client-Cert $ssl_client_cert;
+        proxy_set_header        X-SSL-Client-Verify $ssl_client_verify;
+        proxy_set_header        X-SSL-Client-S-DN $ssl_client_s_dn;
         proxy_redirect          off;
     }
 

twitcher/__init__.py

@@ -1,6 +1,3 @@
-import logging
-logger = logging.getLogger(__name__)
-
 __version__ = '0.3.5'
 
 

twitcher/datatype.py

@@ -8,10 +8,6 @@
 from twitcher.exceptions import AccessTokenNotFound
 
 
-import logging
-logger = logging.getLogger(__name__)
-
-
 class Service(dict):
     """
     Dictionary that contains OWS services. It always has ``'url'`` key.
@@ -39,16 +35,22 @@ def type(self):
     @property
     def public(self):
         """Flag if service has public access."""
+        # TODO: public access can be set via auth parameter.
         return self.get('public', False)
 
     @property
-    def c4i(self):
-        """Flag if service is by climate4impact."""
-        return self.get('c4i', False)
+    def auth(self):
+        """Authentication method: public, token, cert."""
+        return self.get('auth', 'token')
 
     @property
     def params(self):
-        return {'url': self.url, 'name': self.name, 'type': self.type, 'public': self.public, 'c4i': self.c4i}
+        return {
+            'url': self.url,
+            'name': self.name,
+            'type': self.type,
+            'public': self.public,
+            'auth': self.auth}
 
     def __str__(self):
         return self.name

twitcher/owsproxy.py

@@ -54,10 +54,6 @@ def _send_request(request, service, extra_path=None, request_params=None):
     url = service['url']
     if extra_path:
         url += '/' + extra_path
-    if service.get('c4i', False):
-        if 'C4I-Access-Token' in request.headers:
-            LOGGER.debug('using c4i token')
-            url += '/' + request.headers['C4I-Access-Token']
     if request_params:
         url += '?' + request_params
     LOGGER.debug('url = %s', url)

twitcher/owsrequest.py

@@ -44,6 +44,7 @@ def service(self):
 
     @property
     def request(self):
+        # TODO: same name for service request and HTTP request is confusing.
         return self.parser.params['request']
 
     @property

twitcher/owssecurity.py

@@ -9,6 +9,7 @@
 from twitcher.utils import parse_service_name
 from twitcher.owsrequest import OWSRequest
 from twitcher.esgf import fetch_certificate, ESGF_CREDENTIALS
+from twitcher.datatype import Service
 
 import logging
 LOGGER = logging.getLogger("TWITCHER")
@@ -49,32 +50,51 @@ def prepare_headers(self, request, access_token):
                 LOGGER.debug("Prepared request headers.")
         return request
 
+    def verify_access(self, request, service):
+        # TODO: public service access handling is confusing.
+        try:
+            if service.auth == 'cert':
+                self._verify_cert(request)
+            else:  # token
+                self._verify_access_token(request)
+        except OWSAccessForbidden:
+            if not service.public:
+                raise
+
+    def _verify_access_token(self, request):
+        try:
+            # try to get access_token ... if no access restrictions then don't complain.
+            token = self.get_token_param(request)
+            access_token = self.tokenstore.fetch_by_token(token)
+            if access_token.is_expired():
+                raise OWSAccessForbidden("Access token is expired.")
+            # update request with data from access token
+            # request.environ.update(access_token.data)
+            # TODO: is this realy the way we want to do this?
+            request = self.prepare_headers(request, access_token)
+        except AccessTokenNotFound:
+            raise OWSAccessForbidden("Access token is required to access this service.")
+
+    def _verify_cert(self, request):
+        # LOGGER.debug('+++ request headers=%s', request.headers.keys())
+        if not request.headers.get('X-Ssl-Client-Verify', '') == 'SUCCESS':
+            raise OWSAccessForbidden("A valid X.509 client certificate is needed.")
+
     def check_request(self, request):
         if request.path.startswith(protected_path):
-            # TODO: fix this code
+            # TODO: refactor this code
             try:
                 service_name = parse_service_name(request.path)
                 service = self.servicestore.fetch_by_name(service_name)
-                is_public = service.public
                 if service.public is True:
                     LOGGER.warn('public access for service %s', service_name)
             except ServiceNotFound:
-                is_public = False
+                # TODO: why not raising an exception?
+                service = Service(url='unregistered', public=False, auth='token')
                 LOGGER.warn("Service not registered.")
             ows_request = OWSRequest(request)
             if not ows_request.service_allowed():
                 raise OWSInvalidParameterValue(
                     "service %s not supported" % ows_request.service, value="service")
             if not ows_request.public_access():
-                try:
-                    # try to get access_token ... if no access restrictions then don't complain.
-                    token = self.get_token_param(request)
-                    access_token = self.tokenstore.fetch_by_token(token)
-                    if access_token.is_expired() and not is_public:
-                        raise OWSAccessForbidden("Access token is expired.")
-                    # update request with data from access token
-                    # request.environ.update(access_token.data)
-                    request = self.prepare_headers(request, access_token)
-                except AccessTokenNotFound:
-                    if not is_public:
-                        raise OWSAccessForbidden("Access token is required to access this service.")
+                self.verify_access(request, service)

twitcher/store/base.py

@@ -51,7 +51,7 @@ class ServiceStore(object):
 
     def save_service(self, service, overwrite=True):
         """
-        Stores an OWS service with given name in storage.
+        Stores an OWS service in storage.
 
         :param service: An instance of :class:`twitcher.datatype.Service`.
         """