Bugfix/consul service reg fails tls

CHANGELOG.md

@@ -5,6 +5,7 @@
 ### DEPENDENCIES
 
 * The orchestrator requires now at least Terraform 0.11.8 and following Terraform plugins (with corresponding version constraints): `Consul (~> 2.1)`, `AWS (~> 1.36)`, `OpenStack (~> 1.9)`, `Google (~ 1.18)` and `null provider (~ 1.0)`. (Terraform upgrade from 0.9.11 introduced in [GH-82](https://github.com/ystia/yorc/issues/82))
+* Consul version updated to 1.2.3
 
 ### IMPROVEMENTS
 
@@ -13,7 +14,8 @@
 
 ### BUG FIXES
 
-* inputs are not injected into Slurm (srun) jobs ([GH-161](https://github.com/ystia/yorc/issues/161))
+* Inputs are not injected into Slurm (srun) jobs ([GH-161](https://github.com/ystia/yorc/issues/161))
+* Yorc consul service registration fails if using TLS ([GH-153](https://github.com/ystia/yorc/issues/153))
 
 ## 3.1.0-M3 (September 14, 2018)
 

doc/ha.rst

@@ -1,6 +1,8 @@
 Run Yorc in High Availability (HA) mode
 =======================================
 
+.. _yorc_ha_section:
+
 High level view of a typical HA installation
 --------------------------------------------
 

doc/secured.rst

@@ -5,7 +5,9 @@ To run Yorc in secured mode, the following issues have to be addressed:
 
 * Setup a secured Consul cluster
 * Setup a secured Yorc server and configure it to use a secured Consul client
-* Setup Alien4Cloud security and configure it to use the secured Yorc server
+* Setup Alien4Cloud security and configure it to use a secured Yorc server
+
+In the case of Yorc HA setup (see :ref:`yorc_ha_section`), all the Yorc servers composing the cluster need to be secured.
 
 To secure the components listed above, and enable TLS, Multi-Domain (SAN) certificates need to be generated.
 A short list of commands based on openSSL is provided below.
@@ -22,41 +24,40 @@ You might already have one, otherwise, create it using OpenSSL commands below:
 
 Generate certificates signed by your CA
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-You need to generate certificates for all the software component to be secured (Consul, Yorc, Alien4Cloud).
+You need to generate certificates for all the software component to be secured (Consul agents, Yorc servers, Alien4Cloud).
 
-Use the commands below for each component instance (where <IP> represents host's IP address used to connect to):
+Use the commands below for each component instance (where <IP> represents IP address used to connect to the component).
+Replace ``comp`` by a string of your choise corresponding to the components to be secured.
 
 .. code-block:: bash
 
     openssl genrsa -out comp.key 4096
     openssl req -new -sha256 -key comp.key  -subj "/C=FR/O=Atos/CN=127.0.0.1" -reqexts SAN -config <(cat /etc/pki/tls/openssl.cnf <(printf "[SAN]\nsubjectAltName=IP:127.0.0.1,IP:<IP>,DNS:localhost")) -out comp.csr
     openssl x509 -req -in comp.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out comp.pem -days 2048 -extensions SAN -extfile <(cat /etc/pki/tls/openssl.cnf <(printf "[SAN]\nsubjectAltName=IP:127.0.0.1,IP:<IP>,DNS:localhost"))
 
-In the sections below, the ``comp.key`` and ``comp.pem`` files are used to define the different components' configuration.
+In the sections below, the ``comp.key`` and ``comp.pem`` files path are used in the components' configuration file.
 
 Secured Consul cluster Setup
 ----------------------------
-Create a ``consul.key`` and ``consul.pem`` for all the Consul agents within the Consul cluster you setup:
-
- * the server (you may need 3 servers for HA),
- * and the client (you need one client on each host where a Yorc server is running).
+.. note:: You need to generate cerificates for all the Consul agents within the Consul cluster you setup.
 
-Use the above commands and replace <IP> and <SERVER_IP> by the host's IP address.
+In a High Availability cluster, you need to setup at least 3 consul servers, and one consul client on each host where a Yorc server is running. 
 
-Check Consul documentation for details about `agent's configuration <https://www.consul.io/docs/agent/options.html>`_ and `network traffic encryption <https://www.consul.io/docs/agent/encryption.html>`_.
+Check Consul documentation for details about `agent's configuration <https://www.consul.io/docs/agent/options.html>`_.
 
-You may find below a typical configuration file for a consul server:
+You may find below a typical configuration file for a consul server ; to be updated after having generated the ``consul_server.key`` and ``consul_server.pem`` files.
 
 .. code-block:: json
 
     {
-      "domain": "starlings",
+      "domain": "yorc",
       "data_dir": "/tmp/work",
       "client_addr": "0.0.0.0",
       "advertise_addr": "{SERVER_IP}",
       "server": true,
       "bootstrap": true,
-      "encrypt": "{ENCRYPT_KEY}",
+      "ui": true,
+      "encrypt": "{ENCRYPT_KEY},
       "ports": {
         "https": 8543
       },
@@ -67,18 +68,17 @@ You may find below a typical configuration file for a consul server:
       "verify_outgoing": true
     }
 
-And below, one for a consul client.
+And below, a typical configuration file for a consul client.
 
 .. code-block:: json
 
     {
-      "domain": "starlings",
+      "domain": "yorc",
       "data_dir": "/tmp/work",
       "client_addr": "0.0.0.0",
       "advertise_addr": "{IP}",
-      "ui": true,
       "retry_join": [ "{SERVER_IP}" ],
-      "encrypt": "{ENCRYPT_KEY}",
+      "encrypt": "{ENCRYPT_KEY},
       "ports": {
         "https": 8543
       },
@@ -89,14 +89,15 @@ And below, one for a consul client.
       "verify_outgoing": true
     }
 
+In the above example, the encryption is enabled for the gossip traffic inside the Consul cluster. Check Consul documentation for details `network traffic encryption <https://www.consul.io/docs/agent/encryption.html>`_.
 
 You can also consult this `Blog <http://russellsimpkins.blogspot.fr/2015/10/consul-adding-tls-using-self-signed.html>`_. 
 You may found useful information about how to install CA certificate in the OS, in case you get errors about trusting the signing authority.
 
 Secured Yorc Setup
 ------------------
 
-Create a ``yorc-server.key`` and ``yorc-server.pem`` using the above commands and replace <IP> by the host's IP address.
+Generate a ``yorc_server.key`` and ``yorc_server.pem`` using the above commands and replace <IP> by the host's IP address.
 
 Bellow is an example of configuration file with TLS enabled and using the collocated and secured Consul client.
 
@@ -108,7 +109,7 @@ Bellow is an example of configuration file with TLS enabled and using the colloc
             "ca_cert": "{PATH_TO_CA_PEM}",
             "key_file": "{PATH_TO_CONSUL_CLIENT_KEY}",
             "cert_file": "{PATH_TO_CONSUL_CLIENT_PEM}",
-            "address": "127.0.0.1:8543",
+            "address": "127.0.0.1:8543"
         },
         "resources_prefix": "yorc1-",
         "key_file": "{PATH_TO_YORC_SERVER_KEY}",
@@ -126,14 +127,38 @@ Bellow is an example of configuration file with TLS enabled and using the colloc
         }
     }
 
+In the above example SSL verification is enabled for Yorc (ssl_verify set to true). In this case, the Consul Agent must be enabled to use TLS configuration files for HTTP health checks. Otherwise, the TLS handshake may fail.
+You can find below the Consul agent's configuration:
+
+.. code-block:: json
+
+    {
+      "domain": "yorc",
+      "data_dir": "/tmp/work",
+      "client_addr": "0.0.0.0",
+      "advertise_addr": "{IP}",
+      "ui": true,
+      "retry_join": [ "{SERVER_IP}" ],
+      "encrypt": "{ENCRYPT_KEY}",
+      "ports": {
+        "https": 8543
+      },
+      "key_file": "{PATH_TO_CONSUL_CLIENT_KEY}",
+      "cert_file": "{PATH_TO_CONSUL_CLIENT_PEM}",
+      "ca_file": "{PATH_TO_CA_PEM}",
+      "enable_agent_tls_for_checks": true,
+      "verify_incoming_rpc": true,
+      "verify_outgoing": true
+    }
+
 As for Consul, you may need to install CA certificate in the OS, in case you get errors about trusting the signing authority.
 
 Secured Yorc CLI Setup
 ----------------------
 
-If ``ssl_verify`` is enabled for Yorc server the Yorc CLI have to provide a client certificate signed by the Yorc's Certificate Authority.
+If ``ssl_verify`` is enabled for Yorc server, the Yorc CLI have to provide a client certificate signed by the Yorc's Certificate Authority.
 
-So, create a ``yorc-client.key`` and ``yorc-client.pem`` using the above commands and replace <IP> by the host's IP address.
+So, create a ``yorc_client.key`` and ``yorc_client.pem`` using the above commands and replace <IP> by the host's IP address.
 
 Bellow is an example of configuration file with TLS enabled. Refer to :ref:`yorc_config_client_section` for more information.
 

helper/consulutil/service.go

@@ -16,6 +16,7 @@ package consulutil
 
 import (
 	"fmt"
+
 	"github.com/hashicorp/consul/api"
 	"github.com/ystia/yorc/config"
 	"github.com/ystia/yorc/log"
@@ -27,17 +28,35 @@ const YorcService = "yorc"
 // RegisterServerAsConsulService allows to register the Yorc server as a Consul service
 func RegisterServerAsConsulService(cfg config.Configuration, cc *api.Client, chShutdown chan struct{}) error {
 	log.Printf("Register Yorc as Consul Service for node %q", cfg.ServerID)
-	service := &api.AgentServiceRegistration{
+	var service *api.AgentServiceRegistration
+
+	var sslEnabled bool
+	sslEnabled = ((&cfg.KeyFile != nil) && len(cfg.KeyFile) != 0) || ((&cfg.CertFile != nil) && len(cfg.CertFile) != 0)
+
+	var httpString string
+	if sslEnabled {
+		httpString = fmt.Sprintf("https://localhost:%d/health", cfg.HTTPPort)
+		log.Debugf("Register Yorc service with HTTPS check: %s", httpString)
+	} else {
+		httpString = fmt.Sprintf("http://localhost:%d/health", cfg.HTTPPort)
+		log.Debugf("Register Yorc service with HTTP check: %s", httpString)
+	}
+
+	service = &api.AgentServiceRegistration{
 		Name: YorcService,
 		Tags: []string{"server", cfg.ServerID},
 		Port: cfg.HTTPPort,
 		Check: &api.AgentServiceCheck{
-			Name:     "TCP check on yorc port",
-			Interval: "5s",
-			TCP:      fmt.Sprintf("localhost:%d", cfg.HTTPPort),
-			Status:   api.HealthPassing,
+			Name:          "HTTP check on yorc port",
+			Interval:      "5s",
+			HTTP:          httpString,
+			Method:        "GET",
+			TLSSkipVerify: false,
+			Header:        map[string][]string{"Accept": []string{"application/json"}},
+			Status:        api.HealthPassing,
 		},
 	}
+
 	go func() {
 		for {
 			select {

pkg/Dockerfile

@@ -1,4 +1,4 @@
-FROM laurentg/alpine-consul:1.0.4_consul-1.0.6
+FROM ystia/alpine-consul:1.0.4_consul-1.2.3
 ARG TERRAFORM_VERSION
 ARG ANSIBLE_VERSION
 ARG TF_CONSUL_PLUGIN_VERSION
@@ -33,8 +33,7 @@ RUN apk add --update make openssh-client python python-dev py-pip gcc musl-dev l
     umask 0022 && \
     chown -R yorc:yorc /var/yorc && \
     chmod 400 /var/yorc/.ssh/*.pem && \
-
-    # Copy Terraform providers plugins
+    echo "Copy Terraform providers plugins" && \
     cd ${YORC_TERRAFORM_PLUGINS_DIR} && \
     curl -O https://releases.hashicorp.com/terraform-provider-consul/${TF_CONSUL_PLUGIN_VERSION}/terraform-provider-consul_${TF_CONSUL_PLUGIN_VERSION}_linux_amd64.zip && \
     unzip terraform-provider-consul_${TF_CONSUL_PLUGIN_VERSION}_linux_amd64.zip && \
@@ -48,7 +47,7 @@ RUN apk add --update make openssh-client python python-dev py-pip gcc musl-dev l
     unzip terraform-provider-openstack_${TF_OPENSTACK_PLUGIN_VERSION}_linux_amd64.zip && \
     rm -rf ${YORC_TERRAFORM_PLUGINS_DIR}/*.zip && \
     chmod 775 ${YORC_TERRAFORM_PLUGINS_DIR}/* && \
-
+    echo "Cleaning up" && \
     apk del make py-pip python-dev gcc musl-dev libffi-dev openssl-dev && \
     rm -rf /var/cache/apk/* && \
     rm -fr /tmp/*

rest/health.go

@@ -0,0 +1,22 @@
+// Copyright 2018 Bull S.A.S. Atos Technologies - Bull, Rue Jean Jaures, B.P.68, 78340, Les Clayes-sous-Bois, France.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package rest
+
+import "net/http"
+
+func (s *Server) getHealthHandler(w http.ResponseWriter, r *http.Request) {
+	health := Health{"passing"}
+	encodeJSONResponse(w, r, health)
+}

rest/http.go

@@ -142,6 +142,7 @@ func NewServer(configuration config.Configuration, client *api.Client, shutdownC
 
 func (s *Server) registerHandlers() {
 	commonHandlers := alice.New(telemetryHandler, loggingHandler, recoverHandler)
+	s.router.Get("/health", commonHandlers.Append(acceptHandler("application/json")).ThenFunc(s.getHealthHandler))
 	s.router.Post("/deployments", commonHandlers.Append(contentTypeHandler("application/zip")).ThenFunc(s.newDeploymentHandler))
 	s.router.Put("/deployments/:id", commonHandlers.Append(contentTypeHandler("application/zip")).ThenFunc(s.newDeploymentHandler))
 	s.router.Delete("/deployments/:id", commonHandlers.ThenFunc(s.deleteDeploymentHandler))

rest/http_api.md

@@ -690,6 +690,28 @@ Content-Type: application/json
   }
 }
 ```
+## Health
+
+### Get the Yorc service health
+
+This request si made by Consul to check the Yorc service is alive
+
+'Accept' header should be set to 'application/json'.
+
+`GET /health`
+
+**Response**:
+
+```HTTP
+HTTP/1.1 200 Created
+Content-Type: application/json
+```
+
+```json
+{
+  "value": "passing"
+}
+```
 
 ## Registry
 

rest/structs.go

@@ -73,6 +73,11 @@ func newAtomLink(rel, href string) AtomLink {
 	return AtomLink{Rel: rel, Href: href, LinkType: "application/json"}
 }
 
+// Health of a Yorc instance
+type Health struct {
+	Value string `json:"value"`
+}
+
 // Deployment is the representation of a Yorc deployment
 //
 // Deployment's links may be of type LinkRelSelf, LinkRelNode, LinkRelTask, LinkRelOutput.

testutil/helper.go

@@ -45,14 +45,18 @@ func NewTestConsulInstance(t *testing.T) (*testutil.TestServer, *api.Client) {
 		t.Fatalf("Failed to create consul server: %v", err)
 	}
 
-	consulConfig := api.DefaultConfig()
-	consulConfig.Address = srv1.HTTPAddr
+	cfg := config.Configuration{
+		Consul: config.Consul{
+			Address:        srv1.HTTPAddr,
+			PubMaxRoutines: config.DefaultConsulPubMaxRoutines,
+		},
+	}
 
-	client, err := api.NewClient(consulConfig)
+	client, err := cfg.GetNewConsulClient()
 	assert.Nil(t, err)
 
 	kv := client.KV()
-	consulutil.InitConsulPublisher(config.DefaultConsulPubMaxRoutines, kv)
+	consulutil.InitConsulPublisher(cfg.Consul.PubMaxRoutines, kv)
 	return srv1, client
 }