Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create gluster cluster API call doesn't work #400

Closed
fbalak opened this issue Apr 26, 2017 · 24 comments
Closed

Create gluster cluster API call doesn't work #400

fbalak opened this issue Apr 26, 2017 · 24 comments
Assignees

Comments

@fbalak
Copy link

fbalak commented Apr 26, 2017

When I tried creating cluster via API as documented I received job_id but the job is never finished. It's state remains as new. I can't access job's messages either.
Used curl:
curl -X POST -H 'Authorization: Bearer c06b8ede913fde12013828d5210461345b8830d8108c9b47727509397dea4a96' -d '{"sds_name":"gluster","sds_version":"3.9.1", "sds_parameters":{"name":"MyCluster","cluster_id":"mojeclusterid", "public_network":"server_ip/22","cluster_network":"server_ip/22"}, "node_identifier":"ip","node_configuration":{"gl1_ip":{"role": "glusterfs/node", "provisioning_ip":"gl1_ip"},"gl2_ip":{"role":"glusterfs/node", "provisioning_ip":"gl2_ip"},"gl3_ip":{"role":"glusterfs/node", "provisioning_ip":"gl3_ip"}}}' http://server_ip/api/1.0/CreateCluster

From etcd:
{"action":"get","node":{"key":"/queue/752da924-31b2-40dc-bb30-678375aa7629","dir":true,"nodes":[{"key":"/queue/752da924-31b2-40dc-bb30-678375aa7629/status","value":"new","modifiedIndex":43283,"createdIndex":43283},{"key":"/queue/752da924-31b2-40dc-bb30-678375aa7629/payload","value":"{\"job_id\":\"752da924-31b2-40dc-bb30-678375aa7629\",\"status\":\"new\",\"name\":\"CreateCluster\",\"run\":\"tendrl.flows.CreateCluster\",\"type\":\"node\",\"created_from\":\"API\",\"created_at\":\"2017-04-26T12:12:21Z\",\"username\":\"admin\",\"parameters\":{\"TendrlContext.sds_name\":\"gluster\",\"TendrlContext.sds_version\":\"3.9.1\",\"TendrlContext.cluster_name\":\"MyCluster\",\"TendrlContext.cluster_id\":\"mojeclusterid\",\"Node[]\":[\"f83a3427-4e49-47a6-85e6-48f5d9fe8622\",\"8e6f4d79-f7dd-4666-865e-a0e53d4356c2\",\"f73d0f8b-cbcd-432f-9a48-7709f8515c4e\"],\"Cluster.public_network\":\"server_ip/22\",\"Cluster.cluster_network\":\"gl1_ip/22\",\"Cluster.node_configuration\":{\"f83a3427-4e49-47a6-85e6-48f5d9fe8622\":{\"role\":\"glusterfs/node\",\"provisioning_ip\":\"gl1_ip\"},\"8e6f4d79-f7dd-4666-865e-a0e53d4356c2\":{\"role\":\"glusterfs/node\",\"provisioning_ip\":\"gl2_ip\"},\"f73d0f8b-cbcd-432f-9a48-7709f8515c4e\":{\"role\":\"glusterfs/node\",\"provisioning_ip\":\"gl3_ip\"}},\"TendrlContext.integration_id\":\"bcb06eff-bbe8-438c-9dc5-848392726205\"},\"tags\":[\"provisioner/gluster\"]}","modifiedIndex":43284,"createdIndex":43284}],"modifiedIndex":43283,"createdIndex":43283}}
Where server_ip, gl1_ip, gl2_ip and gl3_ip stand for valid machine ip addresses. On gluster machines is installed gluster but they are not in a trusted pool. Tendrl recognizes each of those machines as a single cluster (3 clusters).

Tested with:
tendrl-commons-1.2.3-04_26_2017_14_09_12.noarch
tendrl-dashboard-1.2.3-04_26_2017_04_11_04.noarch
tendrl-api-1.2.3-04_26_2017_01_51_04.noarch
tendrl-node-agent-1.2.3-04_26_2017_00_01_05.noarch
tendrl-node-monitoring-1.2.3-04_26_2017_04_25_04.noarch

@nthomas-redhat
Copy link
Contributor

@fbalak , Need to add provisioner/gluster tag to the node-agent configuration file for gluster nodes. This information was missing in the install doc , have updated now.

@fbalak
Copy link
Author

fbalak commented Apr 26, 2017

@nthomas-redhat ok, thanks. I will later test it.

@fbalak
Copy link
Author

fbalak commented Apr 28, 2017

I tried to create cluster but server returned: {"errors":{"missing":"Unavailable nodes: ip_gl1, ip_gl2, ip_gl3."}}.
provisioner/gluster tag is set in node-agent.conf.yaml on all gluster nodes.

Response from GetNodeList:

{
    "nodes": [
        {
            "tags": "[\"provisioner/ceph\", \"tendrl/node\"]",
            "updated_at": "2017-04-28 07:01:53.899844+00:00",
            "fqdn": "hostname_server",
            "hash": "b30de50a19bc697f769796d8adf8cfa5",
            "machine_id": "20b568c29315446a93ac5a94317d4882",
            "node_id": "eeaa9ccd-7521-4e62-94f2-a9fdc24cc6d0",
            "status": "UP",
            "platform": {
                "updated_at": "2017-04-28 07:01:55.477115+00:00",
                "hash": "bb37045a1f9911856dc2942e72531653",
                "kernel_version": "3.10.0-514.16.1.el7.x86_64",
                "os": "CentOS Linux",
                "os_version": "7.3.1611"
            },
            "tendrlcontext": {
                "hash": "123ce1b18183db31162451fdfe6f0635",
                "integration_id": "",
                "sds_name": "",
                "sds_version": "",
                "updated_at": "2017-04-28 07:01:54.412389+00:00",
                "cluster_id": "",
                "cluster_name": ""
            },
            "stats": {
                "status": "not_monitored",
                "cpu_usage": {
                    "percent_used": "",
                    "updated_at": ""
                },
                "hash": "6a658b11be6f549702e29989759efca1",
                "name": "hostname_server",
                "storage_usage": {
                    "total": "",
                    "percent_used": "",
                    "used": "",
                    "updated_at": ""
                },
                "updated_at": "2017-04-28 07:04:44.487290+00:00",
                "alert_count": "0",
                "cluster_name": "",
                "node_id": "eeaa9ccd-7521-4e62-94f2-a9fdc24cc6d0",
                "role": "[\"provisioner/ceph\", \"tendrl/node\"]",
                "memory_usage": {
                    "total": "",
                    "percent_used": "",
                    "used": "",
                    "updated_at": ""
                }
            }
        },
        {
            "updated_at": "2017-04-28 08:19:09.625183+00:00",
            "fqdn": "hostname_gl2",
            "hash": "25e5f93cf0f3de82c9ef18bfa4a38b0b",
            "machine_id": "49ea978997da482aa62440ded32e80ba",
            "node_id": "be382eb8-15ef-4731-8e16-ec1cf21a37ec",
            "status": "UP",
            "tags": "[\"detected_cluster/c62c7072b3acab2e2e99a12475644c8767332c9d5fed5147a4488812c738e999\", \"provisioner/gluster\", \"detected_cluster/1e81c0115d5a6be62d8a58c7488382aab0b185c45a85d2c26ff866ce27d36d50\", \"tendrl/node\"]",
            "detectedcluster": {
                "detected_cluster_id": "c62c7072b3acab2e2e99a12475644c8767332c9d5fed5147a4488812c738e999",
                "detected_cluster_name": "gluster-c62c7072b3acab2e2e99a12475644c8767332c9d5fed5147a4488812c738e999",
                "hash": "402515a6862e9a056c46d93cdc48b7da",
                "sds_pkg_name": "gluster",
                "sds_pkg_version": "3.9.1",
                "updated_at": "2017-04-28 08:19:09.062761+00:00"
            },
            "tendrlcontext": {
                "sds_name": "",
                "sds_version": "",
                "updated_at": "2017-04-28 07:04:51.044217+00:00",
                "cluster_id": "",
                "cluster_name": "",
                "hash": "123ce1b18183db31162451fdfe6f0635",
                "integration_id": ""
            },
            "platform": {
                "hash": "bb37045a1f9911856dc2942e72531653",
                "kernel_version": "3.10.0-514.16.1.el7.x86_64",
                "os": "CentOS Linux",
                "os_version": "7.3.1611",
                "updated_at": "2017-04-28 07:04:52.108462+00:00"
            },
            "stats": {
                "status": "up",
                "cpu_usage": {
                    "percent_used": "",
                    "updated_at": ""
                },
                "hash": "9e6fdc439e6040286db54a058058feea",
                "memory_usage": {
                    "total": "",
                    "percent_used": "",
                    "used": "",
                    "updated_at": ""
                },
                "storage_usage": {
                    "total": "",
                    "percent_used": "",
                    "used": "",
                    "updated_at": ""
                },
                "updated_at": "2017-04-28 08:19:13.899683+00:00",
                "alert_count": "0",
                "cluster_name": "",
                "node_id": "be382eb8-15ef-4731-8e16-ec1cf21a37ec",
                "role": "[\"detected_cluster/c62c7072b3acab2e2e99a12475644c8767332c9d5fed5147a4488812c738e999\", \"provisioner/gluster\", \"detected_cluster/1e81c0115d5a6be62d8a58c7488382aab0b185c45a85d2c26ff866ce27d36d50\", \"tendrl/node\"]",
                "name": "hostname_gl2"
            }
        },
        {
            "fqdn": "hostname_gl4",
            "hash": "7ca94b0818d7c0ffb9543bc0f5fbbb19",
            "machine_id": "b3f9463dd6c24718b655a6a81f25cc02",
            "node_id": "415c97c6-e6e6-4d79-a182-c52139288476",
            "status": "UP",
            "tags": "[\"detected_cluster/8ce91c83d992e1e0c7596a66ced02a631ff3454edcf102711da07ce50c666d69\", \"provisioner/gluster\", \"detected_cluster/1e81c0115d5a6be62d8a58c7488382aab0b185c45a85d2c26ff866ce27d36d50\", \"tendrl/node\"]",
            "updated_at": "2017-04-28 08:19:09.552558+00:00",
            "tendrlcontext": {
                "hash": "123ce1b18183db31162451fdfe6f0635",
                "integration_id": "",
                "sds_name": "",
                "sds_version": "",
                "updated_at": "2017-04-28 07:04:51.111157+00:00",
                "cluster_id": "",
                "cluster_name": ""
            },
            "platform": {
                "updated_at": "2017-04-28 07:04:52.126745+00:00",
                "hash": "bb37045a1f9911856dc2942e72531653",
                "kernel_version": "3.10.0-514.16.1.el7.x86_64",
                "os": "CentOS Linux",
                "os_version": "7.3.1611"
            },
            "detectedcluster": {
                "detected_cluster_id": "8ce91c83d992e1e0c7596a66ced02a631ff3454edcf102711da07ce50c666d69",
                "detected_cluster_name": "gluster-8ce91c83d992e1e0c7596a66ced02a631ff3454edcf102711da07ce50c666d69",
                "hash": "2e6212f09ccf46a47e4f178ef21fc38c",
                "sds_pkg_name": "gluster",
                "sds_pkg_version": "3.9.1",
                "updated_at": "2017-04-28 08:19:08.966182+00:00"
            },
            "stats": {
                "status": "up",
                "cpu_usage": {
                    "percent_used": "",
                    "updated_at": ""
                },
                "hash": "d72d0fc9d38e082b84a2ca84aac067cc",
                "memory_usage": {
                    "total": "",
                    "percent_used": "",
                    "used": "",
                    "updated_at": ""
                },
                "storage_usage": {
                    "total": "",
                    "percent_used": "",
                    "used": "",
                    "updated_at": ""
                },
                "updated_at": "2017-04-28 08:19:08.679539+00:00",
                "alert_count": "0",
                "cluster_name": "",
                "node_id": "415c97c6-e6e6-4d79-a182-c52139288476",
                "role": "[\"provisioner/gluster\", \"detected_cluster/1e81c0115d5a6be62d8a58c7488382aab0b185c45a85d2c26ff866ce27d36d50\", \"tendrl/node\"]",
                "name": "hostname_gl4"
            }
        },
        {
            "status": "UP",
            "tags": "[\"detected_cluster/834e19270642e08355adfa83bfa611897e02686c6aa9f98325bc1f37ad21fd13\", \"provisioner/gluster\", \"detected_cluster/1e81c0115d5a6be62d8a58c7488382aab0b185c45a85d2c26ff866ce27d36d50\", \"tendrl/node\"]",
            "updated_at": "2017-04-28 08:19:09.413602+00:00",
            "fqdn": "hostname_gl1",
            "hash": "d6de483a869586fa75d704abf3aeee33",
            "machine_id": "0913b6c0281a437cb190641dd8936f32",
            "node_id": "b84fcb9f-881c-4831-84da-c298390aa913",
            "tendrlcontext": {
                "cluster_id": "",
                "cluster_name": "",
                "hash": "123ce1b18183db31162451fdfe6f0635",
                "integration_id": "",
                "sds_name": "",
                "sds_version": "",
                "updated_at": "2017-04-28 07:04:51.164148+00:00"
            },
            "platform": {
                "kernel_version": "3.10.0-514.16.1.el7.x86_64",
                "os": "CentOS Linux",
                "os_version": "7.3.1611",
                "updated_at": "2017-04-28 07:04:52.141230+00:00",
                "hash": "bb37045a1f9911856dc2942e72531653"
            },
            "detectedcluster": {
                "detected_cluster_name": "gluster-834e19270642e08355adfa83bfa611897e02686c6aa9f98325bc1f37ad21fd13",
                "hash": "0257ee455983b2a9e2ceb91b70b4e816",
                "sds_pkg_name": "gluster",
                "sds_pkg_version": "3.9.1",
                "updated_at": "2017-04-28 08:19:08.824097+00:00",
                "detected_cluster_id": "834e19270642e08355adfa83bfa611897e02686c6aa9f98325bc1f37ad21fd13"
            },
            "stats": {
                "status": "up",
                "cpu_usage": {
                    "percent_used": "",
                    "updated_at": ""
                },
                "hash": "0d8a91132ec4f537b04af0f40a95da78",
                "memory_usage": {
                    "total": "",
                    "updated_at": "",
                    "used": "",
                    "percent_used": ""
                },
                "storage_usage": {
                    "total": "",
                    "used": "",
                    "percent_used": "",
                    "updated_at": ""
                },
                "alert_count": "0",
                "updated_at": "2017-04-28 08:19:10.221705+00:00",
                "cluster_name": "",
                "node_id": "b84fcb9f-881c-4831-84da-c298390aa913",
                "role": "[\"detected_cluster/834e19270642e08355adfa83bfa611897e02686c6aa9f98325bc1f37ad21fd13\", \"provisioner/gluster\", \"detected_cluster/1e81c0115d5a6be62d8a58c7488382aab0b185c45a85d2c26ff866ce27d36d50\", \"tendrl/node\"]",
                "name": "hostname_gl1"
            }
        },
        {
            "tags": "[\"detected_cluster/886cc33bfb72a22a819e49da92e51a1e4c00ab3ec02c1f4da26b896dff972e33\", \"provisioner/gluster\", \"detected_cluster/1e81c0115d5a6be62d8a58c7488382aab0b185c45a85d2c26ff866ce27d36d50\", \"tendrl/node\"]",
            "updated_at": "2017-04-28 08:19:09.441534+00:00",
            "fqdn": "hostname_gl3",
            "hash": "aa83855d8669ec571336279d196199bf",
            "machine_id": "01a23324e9d24e9fa3b7eb873e9bb490",
            "node_id": "14208801-e9fe-4875-b3f7-a0f76d1ebdd5",
            "status": "UP",
            "tendrlcontext": {
                "sds_version": "",
                "updated_at": "2017-04-28 07:04:51.165488+00:00",
                "cluster_id": "",
                "cluster_name": "",
                "hash": "123ce1b18183db31162451fdfe6f0635",
                "integration_id": "",
                "sds_name": ""
            },
            "platform": {
                "os": "CentOS Linux",
                "os_version": "7.3.1611",
                "updated_at": "2017-04-28 07:04:52.133383+00:00",
                "hash": "bb37045a1f9911856dc2942e72531653",
                "kernel_version": "3.10.0-514.16.1.el7.x86_64"
            },
            "detectedcluster": {
                "hash": "ea911d69c36312309a973e7d83d7f0cb",
                "sds_pkg_name": "gluster",
                "sds_pkg_version": "3.9.1",
                "updated_at": "2017-04-28 08:19:08.885549+00:00",
                "detected_cluster_id": "886cc33bfb72a22a819e49da92e51a1e4c00ab3ec02c1f4da26b896dff972e33",
                "detected_cluster_name": "gluster-886cc33bfb72a22a819e49da92e51a1e4c00ab3ec02c1f4da26b896dff972e33"
            },
            "stats": {
                "status": "up",
                "cpu_usage": {
                    "percent_used": "",
                    "updated_at": ""
                },
                "hash": "a5da62e3b7ca1e19e7f9a94e1d3e96c9",
                "memory_usage": {
                    "total": "",
                    "percent_used": "",
                    "used": "",
                    "updated_at": ""
                },
                "storage_usage": {
                    "total": "",
                    "used": "",
                    "percent_used": "",
                    "updated_at": ""
                },
                "updated_at": "2017-04-28 08:19:11.460519+00:00",
                "alert_count": "0",
                "cluster_name": "",
                "node_id": "14208801-e9fe-4875-b3f7-a0f76d1ebdd5",
                "role": "[\"detected_cluster/886cc33bfb72a22a819e49da92e51a1e4c00ab3ec02c1f4da26b896dff972e33\", \"provisioner/gluster\", \"detected_cluster/1e81c0115d5a6be62d8a58c7488382aab0b185c45a85d2c26ff866ce27d36d50\", \"tendrl/node\"]",
                "name": "hostname_gl3"
            }
        }
    ],
    "clusters": [
        {
            "cluster_id": "c62c7072b3acab2e2e99a12475644c8767332c9d5fed5147a4488812c738e999",
            "sds_name": "gluster",
            "sds_version": "3.9.1",
            "node_ids": [
                "be382eb8-15ef-4731-8e16-ec1cf21a37ec"
            ]
        },
        {
            "cluster_id": "8ce91c83d992e1e0c7596a66ced02a631ff3454edcf102711da07ce50c666d69",
            "sds_name": "gluster",
            "sds_version": "3.9.1",
            "node_ids": [
                "415c97c6-e6e6-4d79-a182-c52139288476"
            ]
        },
        {
            "cluster_id": "834e19270642e08355adfa83bfa611897e02686c6aa9f98325bc1f37ad21fd13",
            "sds_name": "gluster",
            "sds_version": "3.9.1",
            "node_ids": [
                "b84fcb9f-881c-4831-84da-c298390aa913"
            ]
        },
        {
            "cluster_id": "886cc33bfb72a22a819e49da92e51a1e4c00ab3ec02c1f4da26b896dff972e33",
            "sds_name": "gluster",
            "sds_version": "3.9.1",
            "node_ids": [
                "14208801-e9fe-4875-b3f7-a0f76d1ebdd5"
            ]
        }
    ]
}

Used curl:

curl -X POST -H 'Authorization: Bearer 9d1f9c2aac23ff31162a010909c1485a07ec4b86ad08493851ad3347ea1fb68f' -d '{"sds_name":"gluster","sds_version":"3.9.1", "sds_parameters":{"name":"MyCluster","cluster_id":"4654ac00-e67b-4b74-86a3-e740b1b8cee5", "public_network":"ip_server/22","cluster_network":"ip_server/22"}, "node_identifier":"ip","node_configuration":{"ip_gl1":{"role": "glusterfs/node", "provisioning_ip":"ip_gl1"},"ip_gl2":{"role":"glusterfs/node", "provisioning_ip":"ip_gl2"},"ip_gl3":{"role":"glusterfs/node", "provisioning_ip":"ip_gl3"}}}' http://ip_server/api/1.0/CreateCluster

Tested with:
tendrl-performance-monitoring-1.2.3-04_28_2017_03_02_04.noarch
tendrl-api-1.2.3-04_28_2017_01_51_05.noarch
tendrl-node-agent-1.2.3-04_28_2017_00_01_03.noarch
tendrl-node-monitoring-1.2.3-04_28_2017_04_25_04.noarch
tendrl-commons-1.2.3-04_27_2017_05_13_05.noarch

@r0h4n
Copy link
Contributor

r0h4n commented May 2, 2017

@anivargi

quoting @fbalak
I tried to create cluster but server returned: {"errors":{"missing":"Unavailable nodes: ip_gl1, ip_gl2, ip_gl3."}}.
In what conditions does the tendrl-api return above?

@anivargi
Copy link

anivargi commented May 2, 2017

@r0h4n when the specified node (in this case ip_gl1, ip_gl2, ip_gl3) is not available under /indexes/ip/<ip>

@r0h4n
Copy link
Contributor

r0h4n commented May 2, 2017

@fbalak can you check whether it there's "/indexes/ip/" in etcd for each of node (ip_gl1, ip_gl2, ip_gl3)

@fbalak
Copy link
Author

fbalak commented May 2, 2017

There is no /indexes/ip:
{"action":"get","node":{"key":"/indexes","dir":true,"nodes":[{"key":"/indexes/machine_id","dir":true,"modifiedIndex":11,"createdIndex":11}],"modifiedIndex":11,"createdIndex":11}}

@r0h4n
Copy link
Contributor

r0h4n commented May 2, 2017

@fbalak Please attach a dump of your etcd data. This will need further inspection, also check if there's data in etcd "/networks/"

@fbalak
Copy link
Author

fbalak commented May 2, 2017

There is no networks key:
{"action":"get","node":{"dir":true,"nodes":[{"key":"/nodes","dir":true,"modifiedIndex":15,"createdIndex":15},{"key":"/messages","dir":true,"modifiedIndex":63,"createdIndex":63},{"key":"/queue","dir":true,"modifiedIndex":794,"createdIndex":794},{"key":"/monitoring","dir":true,"modifiedIndex":803,"createdIndex":803},{"key":"/indexes","dir":true,"modifiedIndex":11,"createdIndex":11}]}}

Dumps:
https://gist.github.com/fbalak/3eaa0c13e8617a5f0330a975527b7baf

Dumps from /nodes and /messages are too large. Maybe I can grant you an access to etcd, so you can look for yourself.

@fbalak
Copy link
Author

fbalak commented May 3, 2017

When I tested it today it created job, but it failed. After this GetClusterList API call starts returning 500 Internal Server Error. tendrl-gluster-integration was not installed. There is a response from hostname/api/1.0/jobs/143ecd94-c024-46be-acae-4fb4858fafc9/messages

[
    {
        "publisher": "node_agent",
        "job_id": "143ecd94-c024-46be-acae-4fb4858fafc9",
        "timestamp": "2017-05-03T11:21:57.940685+00:00",
        "caller": {
            "function": "_run",
            "line_no": 157,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "Processing Job 143ecd94-c024-46be-acae-4fb4858fafc9"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "87734077-d0ff-4d45-9171-68d945d03521",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "18dac7c4-3202-427e-b2d6-af500e1b4410"
    },
    {
        "publisher": "node_agent",
        "job_id": "143ecd94-c024-46be-acae-4fb4858fafc9",
        "timestamp": "2017-05-03T11:21:57.949910+00:00",
        "caller": {
            "function": "_run",
            "line_no": 169,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "Running Flow tendrl.flows.CreateCluster"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "87734077-d0ff-4d45-9171-68d945d03521",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "243769f2-2112-418f-ab41-20df1ac212dc"
    },
    {
        "publisher": "node_agent",
        "job_id": "143ecd94-c024-46be-acae-4fb4858fafc9",
        "timestamp": "2017-05-03T11:22:15.096034+00:00",
        "caller": {
            "function": "gluster_create_ssh_setup_jobs",
            "line_no": 153,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/utils.py"
        },
        "payload": {
            "message": "Created SSH setup job c63dfdd9-da39-424d-82ac-e3b804840878 for node b37f62e6-fb1e-499e-8a04-86bd2608e1b1"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "87734077-d0ff-4d45-9171-68d945d03521",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "c00ba0e5-0923-456b-84d6-3279a4c8cf03"
    },
    {
        "publisher": "node_agent",
        "job_id": "143ecd94-c024-46be-acae-4fb4858fafc9",
        "timestamp": "2017-05-03T11:22:16.837857+00:00",
        "caller": {
            "function": "gluster_create_ssh_setup_jobs",
            "line_no": 153,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/utils.py"
        },
        "payload": {
            "message": "Created SSH setup job ffbec6af-1ba1-40e9-8268-f57a63f04ca1 for node aba3d233-ee3c-4724-b098-898b859eeefd"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "87734077-d0ff-4d45-9171-68d945d03521",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "27f23396-76bc-42ba-9a13-3e83577f29ff"
    },
    {
        "publisher": "node_agent",
        "job_id": "143ecd94-c024-46be-acae-4fb4858fafc9",
        "timestamp": "2017-05-03T11:22:16.854841+00:00",
        "caller": {
            "function": "run",
            "line_no": 73,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/__init__.py"
        },
        "payload": {
            "message": "SSH setup completed for all nodes in cluster a820cbef-8d5a-418a-b108-88df2e7f6aa4"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "87734077-d0ff-4d45-9171-68d945d03521",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "c7db350c-8383-4c0b-9dd0-7c1dfeab6ab3"
    },
    {
        "publisher": "node_agent",
        "job_id": "143ecd94-c024-46be-acae-4fb4858fafc9",
        "timestamp": "2017-05-03T11:22:17.764501+00:00",
        "caller": {
            "function": "run",
            "line_no": 110,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/__init__.py"
        },
        "payload": {
            "message": "Creating Gluster Storage Cluster a820cbef-8d5a-418a-b108-88df2e7f6aa4"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "87734077-d0ff-4d45-9171-68d945d03521",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "4c2277c8-9d2e-4139-a005-0afec1a01e00"
    },
    {
        "publisher": "node_agent",
        "job_id": "143ecd94-c024-46be-acae-4fb4858fafc9",
        "timestamp": "2017-05-03T11:22:17.775106+00:00",
        "caller": {
            "function": "create_gluster",
            "line_no": 23,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/gluster_help.py"
        },
        "payload": {
            "message": "Setting up gluster nodes a820cbef-8d5a-418a-b108-88df2e7f6aa4"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "87734077-d0ff-4d45-9171-68d945d03521",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "2e6aeaff-26e0-48ba-b102-d6fd97ca3cc6"
    },
    {
        "publisher": "node_agent",
        "job_id": "ffbec6af-1ba1-40e9-8268-f57a63f04ca1",
        "timestamp": "2017-05-03T11:22:20.430175+00:00",
        "caller": {
            "function": "_run",
            "line_no": 157,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "Processing Job ffbec6af-1ba1-40e9-8268-f57a63f04ca1"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "aba3d233-ee3c-4724-b098-898b859eeefd",
        "cluster_id": null,
        "flow_id": "759e639a-1315-11e7-93ae-92361f002671",
        "message_id": "bcad5322-2dd3-46ab-846f-94e48f448feb"
    },
    {
        "publisher": "node_agent",
        "job_id": "ffbec6af-1ba1-40e9-8268-f57a63f04ca1",
        "timestamp": "2017-05-03T11:22:20.439663+00:00",
        "caller": {
            "function": "_run",
            "line_no": 169,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "Running Flow tendrl.flows.AuthorizeSshKey"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "aba3d233-ee3c-4724-b098-898b859eeefd",
        "cluster_id": null,
        "flow_id": "759e639a-1315-11e7-93ae-92361f002671",
        "message_id": "105bcb92-9007-40bb-83fc-44c07fa9d0df"
    },
    {
        "publisher": "node_agent",
        "job_id": "ffbec6af-1ba1-40e9-8268-f57a63f04ca1",
        "timestamp": "2017-05-03T11:22:21.380604+00:00",
        "caller": {
            "function": "_run",
            "line_no": 187,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "JOB[ffbec6af-1ba1-40e9-8268-f57a63f04ca1]:  Finished Flow tendrl.flows.AuthorizeSshKey"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "aba3d233-ee3c-4724-b098-898b859eeefd",
        "cluster_id": null,
        "flow_id": "759e639a-1315-11e7-93ae-92361f002671",
        "message_id": "5f7a393c-fc8e-4f21-9d3a-b9f4dde66595"
    },
    {
        "publisher": "node_agent",
        "job_id": "c63dfdd9-da39-424d-82ac-e3b804840878",
        "timestamp": "2017-05-03T11:22:21.658159+00:00",
        "caller": {
            "function": "_run",
            "line_no": 157,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "Processing Job c63dfdd9-da39-424d-82ac-e3b804840878"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "b37f62e6-fb1e-499e-8a04-86bd2608e1b1",
        "cluster_id": null,
        "flow_id": "759e639a-1315-11e7-93ae-92361f002671",
        "message_id": "1a0d2e20-e40c-4b84-9b9c-f957849452c0"
    },
    {
        "publisher": "node_agent",
        "job_id": "c63dfdd9-da39-424d-82ac-e3b804840878",
        "timestamp": "2017-05-03T11:22:21.905310+00:00",
        "caller": {
            "function": "_run",
            "line_no": 169,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "Running Flow tendrl.flows.AuthorizeSshKey"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "b37f62e6-fb1e-499e-8a04-86bd2608e1b1",
        "cluster_id": null,
        "flow_id": "759e639a-1315-11e7-93ae-92361f002671",
        "message_id": "ebe18eff-b497-4826-b682-924bc0415143"
    },
    {
        "publisher": "node_agent",
        "job_id": "c63dfdd9-da39-424d-82ac-e3b804840878",
        "timestamp": "2017-05-03T11:22:22.431274+00:00",
        "caller": {
            "function": "_run",
            "line_no": 187,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "JOB[c63dfdd9-da39-424d-82ac-e3b804840878]:  Finished Flow tendrl.flows.AuthorizeSshKey"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "b37f62e6-fb1e-499e-8a04-86bd2608e1b1",
        "cluster_id": null,
        "flow_id": "759e639a-1315-11e7-93ae-92361f002671",
        "message_id": "5212bb96-bba9-4119-8f06-55d89fd5952a"
    },
    {
        "publisher": "node_agent",
        "job_id": "143ecd94-c024-46be-acae-4fb4858fafc9",
        "timestamp": "2017-05-03T11:22:23.134149+00:00",
        "caller": {
            "function": "_run",
            "line_no": 210,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "Job failed Error setting up gluster node: Error setting up gluster node"
        },
        "priority": "error",
        "parent_id": null,
        "node_id": "87734077-d0ff-4d45-9171-68d945d03521",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "b196f1b9-a8d8-4b16-bbee-ccc532f2ee32"
    }
]

CC: @mkudlej

Tested with:
tendrl-node-agent-1.2.3-05_03_2017_00_01_04.noarch
tendrl-api-1.2.3-05_03_2017_01_51_04.noarch
tendrl-commons-1.2.3-05_03_2017_05_13_04.noarch

@nnDarshan
Copy link
Contributor

@fbalak In above comment, the last message says "Job failed Error setting up gluster node: Error setting up gluster node"
During this (gluster node setup) step tendrl tries to:

  1. install glusterfs packages
  2. start the glusterd service
  3. set firewall on the node
    Can you please:
    a. check if glusterfs packages are installed.
    b. If above step has no issues, check if glusterd service is running

@r0h4n
Copy link
Contributor

r0h4n commented May 8, 2017

@fbalak Any updates?

@fbalak
Copy link
Author

fbalak commented May 9, 2017

I have running glusterd service and installed glusterfs packages. I have run same curl command with appropriate access_token and ip addresses and job failed hostname/api/1.0/jobs/7cc9351b-314f-42a3-b491-579bbaf04965/messages:

[
    {
        "publisher": "node_agent",
        "job_id": "7cc9351b-314f-42a3-b491-579bbaf04965",
        "timestamp": "2017-05-09T09:17:10.293758+00:00",
        "caller": {
            "function": "_run",
            "line_no": 157,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "Processing Job 7cc9351b-314f-42a3-b491-579bbaf04965"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "d4d8838a-0d7c-439b-a283-0a3d89d3782d",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "3077efa8-e5ae-4978-b827-f7c962bcc699"
    },
    {
        "publisher": "node_agent",
        "job_id": "7cc9351b-314f-42a3-b491-579bbaf04965",
        "timestamp": "2017-05-09T09:17:10.301429+00:00",
        "caller": {
            "function": "_run",
            "line_no": 169,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "Running Flow tendrl.flows.CreateCluster"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "d4d8838a-0d7c-439b-a283-0a3d89d3782d",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "c97b28c5-97a7-4be2-ab1c-ef037ee65641"
    },
    {
        "publisher": "node_agent",
        "job_id": "7cc9351b-314f-42a3-b491-579bbaf04965",
        "timestamp": "2017-05-09T09:17:30.230712+00:00",
        "caller": {
            "function": "_run",
            "line_no": 210,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "Job failed list.remove(x): x not in list: list.remove(x): x not in list"
        },
        "priority": "error",
        "parent_id": null,
        "node_id": "d4d8838a-0d7c-439b-a283-0a3d89d3782d",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "c3be72b5-f046-4ada-8520-ac8ca866d38f"
    }
]

Tested with:
tendrl-commons-1.2.3-05_09_2017_12_59_26.noarch
tendrl-api-1.2.3-05_09_2017_13_23_12.noarch
tendrl-node-agent-1.2.3-05_09_2017_00_01_05.noarch
tendrl-node-monitoring-1.2.3-05_09_2017_04_25_04.noarch

@r0h4n
Copy link
Contributor

r0h4n commented May 9, 2017

@fbalak For CreateCluster in case of Gluster you need to tag any 1 of the node-agents from the detected_cluster as "provisioner/gluster" (in /etc/tendrl/node-agent/node-agent.conf.yaml"

Please verify this

More details at Pt.4 : https://github.com/Tendrl/documentation/wiki/Tendrl-Package-Installation-Reference#storage-node-installation

@fbalak
Copy link
Author

fbalak commented May 9, 2017

Yes, it is tagged. From /etc/tendrl/node-agent/node-agent.conf.yaml:

...
tags:
  - provisioner/gluster
  - tendrl/node

@r0h4n
Copy link
Contributor

r0h4n commented May 9, 2017

@fbalak Please provide payload for Job 7cc9351b-314f-42a3-b491-579bbaf04965 and also provide NodeContext for all nodes participating in above job's "Node[]" param

@fbalak
Copy link
Author

fbalak commented May 9, 2017

hostname/api/1.0/jobs/7cc9351b-314f-42a3-b491-579bbaf04965:

{
    "job_id": "7cc9351b-314f-42a3-b491-579bbaf04965",
    "status": "failed",
    "flow": "CreateCluster",
    "parameters": {
        "TendrlContext.sds_name": "gluster",
        "TendrlContext.sds_version": "3.9.1",
        "TendrlContext.cluster_name": "MyCluster",
        "TendrlContext.cluster_id": "4654ac00-e67b-4b74-86a3-e740b1b8cee5",
        "Node[]": [
            "cedff733-21a1-4697-a03c-97cf5b71f8ee",
            "06affdcc-b8ca-42da-bfed-c19e4eadb633",
            "4e57745f-a15b-4834-b313-f41a9e33ff30"
        ],
        "Cluster.public_network": "server_ip/22",
        "Cluster.cluster_network": "server_ip/22",
        "Cluster.node_configuration": {
            "cedff733-21a1-4697-a03c-97cf5b71f8ee": {
                "role": "glusterfs/node",
                "provisioning_ip": "gl1_ip"
            },
            "06affdcc-b8ca-42da-bfed-c19e4eadb633": {
                "role": "glusterfs/node",
                "provisioning_ip": "gl2_ip"
            },
            "4e57745f-a15b-4834-b313-f41a9e33ff30": {
                "role": "glusterfs/node",
                "provisioning_ip": "gl3_ip"
            }
        },
        "TendrlContext.integration_id": "9b05638a-31d5-48fd-93db-505350f7df75"
    },
    "created_at": "2017-05-09T09:16:56Z",
    "status_url": "/jobs/7cc9351b-314f-42a3-b491-579bbaf04965/status",
    "messages_url": "/jobs/7cc9351b-314f-42a3-b491-579bbaf04965/messages",
    "output_url": "/jobs/7cc9351b-314f-42a3-b491-579bbaf04965/output"
}
{"action":"get","node":{"key":"/nodes/cedff733-21a1-4697-a03c-97cf5b71f8ee/NodeContext","dir":true,"nodes":[{"key":"/nodes/cedff733-21a1-4697-a03c-97cf5b71f8ee/NodeContext/fqdn","value":"gl1_hostname","modifiedIndex":19966,"createdIndex":19966},{"key":"/nodes/cedff733-21a1-4697-a03c-97cf5b71f8ee/NodeContext/hash","value":"01b0f158da6ad466b1dd72f9a307ce3d","modifiedIndex":19967,"createdIndex":19967},{"key":"/nodes/cedff733-21a1-4697-a03c-97cf5b71f8ee/NodeContext/machine_id","value":"0302ef79ef9f40c2a00f08eebe86b2a7","modifiedIndex":19968,"createdIndex":19968},{"key":"/nodes/cedff733-21a1-4697-a03c-97cf5b71f8ee/NodeContext/node_id","value":"cedff733-21a1-4697-a03c-97cf5b71f8ee","modifiedIndex":19969,"createdIndex":19969},{"key":"/nodes/cedff733-21a1-4697-a03c-97cf5b71f8ee/NodeContext/status","value":"UP","modifiedIndex":19970,"createdIndex":19970},{"key":"/nodes/cedff733-21a1-4697-a03c-97cf5b71f8ee/NodeContext/tags","value":"[\"detected_cluster/e1ead11ed4107966b7de506e265aab935944614f613b9a106179a78ffb92c5a7\", \"provisioner/gluster\", \"gluster/server\", \"tendrl/node\"]","modifiedIndex":19971,"createdIndex":19971},{"key":"/nodes/cedff733-21a1-4697-a03c-97cf5b71f8ee/NodeContext/updated_at","value":"2017-05-09 09:17:19.580096+00:00","modifiedIndex":19972,"createdIndex":19972}],"modifiedIndex":825,"createdIndex":825}}

{"action":"get","node":{"key":"/nodes/06affdcc-b8ca-42da-bfed-c19e4eadb633/NodeContext","dir":true,"nodes":[{"key":"/nodes/06affdcc-b8ca-42da-bfed-c19e4eadb633/NodeContext/updated_at","value":"2017-05-09 09:17:17.086124+00:00","modifiedIndex":19952,"createdIndex":19952},{"key":"/nodes/06affdcc-b8ca-42da-bfed-c19e4eadb633/NodeContext/fqdn","value":"gl2_hostname","modifiedIndex":19946,"createdIndex":19946},{"key":"/nodes/06affdcc-b8ca-42da-bfed-c19e4eadb633/NodeContext/hash","value":"5d8267d4885f2fcd80349e749389d12b","modifiedIndex":19947,"createdIndex":19947},{"key":"/nodes/06affdcc-b8ca-42da-bfed-c19e4eadb633/NodeContext/machine_id","value":"a6b9cac34d1e41bf904ae2b2359916cc","modifiedIndex":19948,"createdIndex":19948},{"key":"/nodes/06affdcc-b8ca-42da-bfed-c19e4eadb633/NodeContext/node_id","value":"06affdcc-b8ca-42da-bfed-c19e4eadb633","modifiedIndex":19949,"createdIndex":19949},{"key":"/nodes/06affdcc-b8ca-42da-bfed-c19e4eadb633/NodeContext/status","value":"UP","modifiedIndex":19950,"createdIndex":19950},{"key":"/nodes/06affdcc-b8ca-42da-bfed-c19e4eadb633/NodeContext/tags","value":"[\"provisioner/gluster\", \"detected_cluster/8595b430f2bd4f028bf9ae5b2a1c2a294805f1df500a2d7815337beb72c58d2f\", \"gluster/server\", \"tendrl/node\"]","modifiedIndex":19951,"createdIndex":19951}],"modifiedIndex":827,"createdIndex":827}}

{"action":"get","node":{"key":"/nodes/4e57745f-a15b-4834-b313-f41a9e33ff30/NodeContext","dir":true,"nodes":[{"key":"/nodes/4e57745f-a15b-4834-b313-f41a9e33ff30/NodeContext/node_id","value":"4e57745f-a15b-4834-b313-f41a9e33ff30","modifiedIndex":19959,"createdIndex":19959},{"key":"/nodes/4e57745f-a15b-4834-b313-f41a9e33ff30/NodeContext/status","value":"UP","modifiedIndex":19960,"createdIndex":19960},{"key":"/nodes/4e57745f-a15b-4834-b313-f41a9e33ff30/NodeContext/tags","value":"[\"detected_cluster/27ff8d41e4a930b91e65d04c77123ff9e697b2fc7f0fdbdd336c7d6d2e9fe102\", \"provisioner/gluster\", \"gluster/server\", \"tendrl/node\"]","modifiedIndex":19961,"createdIndex":19961},{"key":"/nodes/4e57745f-a15b-4834-b313-f41a9e33ff30/NodeContext/updated_at","value":"2017-05-09 09:17:17.911109+00:00","modifiedIndex":19962,"createdIndex":19962},{"key":"/nodes/4e57745f-a15b-4834-b313-f41a9e33ff30/NodeContext/fqdn","value":"gl3_hostname","modifiedIndex":19953,"createdIndex":19953},{"key":"/nodes/4e57745f-a15b-4834-b313-f41a9e33ff30/NodeContext/hash","value":"05c7adc245e459c5596e55f71a3efcc6","modifiedIndex":19954,"createdIndex":19954},{"key":"/nodes/4e57745f-a15b-4834-b313-f41a9e33ff30/NodeContext/machine_id","value":"e8de88aecbce45bab90c9be154b7aecb","modifiedIndex":19955,"createdIndex":19955}],"modifiedIndex":818,"createdIndex":818}}

@r0h4n
Copy link
Contributor

r0h4n commented May 9, 2017

@fbalak The "provisioner/gluster" tag is given to all the 3 nodes, only one node from the detected cluster has to be given that tag, can you retry after making changes and on a new etcd instance?

@fbalak
Copy link
Author

fbalak commented May 10, 2017

Cluster creation still doesn't work for me. Services tendrl-node-agent and glusterd are running.

[
    {
        "publisher": "node_agent",
        "job_id": "fa06d3e8-ef0a-44c3-87cc-99c6191e0520",
        "timestamp": "2017-05-10T07:37:51.361734+00:00",
        "caller": {
            "function": "_run",
            "line_no": 157,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "Processing Job fa06d3e8-ef0a-44c3-87cc-99c6191e0520"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "612473bd-0d31-495d-b748-d047921bf9fd",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "3ecb2277-cdfd-4faa-a402-ed56aa256676"
    },
    {
        "publisher": "node_agent",
        "job_id": "fa06d3e8-ef0a-44c3-87cc-99c6191e0520",
        "timestamp": "2017-05-10T07:37:51.372718+00:00",
        "caller": {
            "function": "_run",
            "line_no": 169,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "Running Flow tendrl.flows.CreateCluster"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "612473bd-0d31-495d-b748-d047921bf9fd",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "1a534ef0-f5a1-4deb-b88c-760021c83281"
    },
    {
        "publisher": "node_agent",
        "job_id": "fa06d3e8-ef0a-44c3-87cc-99c6191e0520",
        "timestamp": "2017-05-10T07:37:53.425478+00:00",
        "caller": {
            "function": "gluster_create_ssh_setup_jobs",
            "line_no": 134,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/utils.py"
        },
        "payload": {
            "message": "Error adding authorized key"
        },
        "priority": "error",
        "parent_id": null,
        "node_id": "612473bd-0d31-495d-b748-d047921bf9fd",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "b2110af1-cbbe-4057-9490-6bc9b97a5dbb"
    },
    {
        "publisher": "node_agent",
        "job_id": "fa06d3e8-ef0a-44c3-87cc-99c6191e0520",
        "timestamp": "2017-05-10T07:37:53.534028+00:00",
        "caller": {
            "function": "run",
            "line_no": 73,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/__init__.py"
        },
        "payload": {
            "message": "SSH setup completed for all nodes in cluster cc353e38-52d4-47f1-8865-7eaebad2a034"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "612473bd-0d31-495d-b748-d047921bf9fd",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "a528f9bd-f125-46e3-b52a-ed70abf83723"
    },
    {
        "publisher": "node_agent",
        "job_id": "fa06d3e8-ef0a-44c3-87cc-99c6191e0520",
        "timestamp": "2017-05-10T07:37:54.156633+00:00",
        "caller": {
            "function": "run",
            "line_no": 110,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/__init__.py"
        },
        "payload": {
            "message": "Creating Gluster Storage Cluster cc353e38-52d4-47f1-8865-7eaebad2a034"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "612473bd-0d31-495d-b748-d047921bf9fd",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "d216b529-f5f3-4680-aae6-5b0f6457b057"
    },
    {
        "publisher": "node_agent",
        "job_id": "fa06d3e8-ef0a-44c3-87cc-99c6191e0520",
        "timestamp": "2017-05-10T07:37:54.166408+00:00",
        "caller": {
            "function": "create_gluster",
            "line_no": 23,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/gluster_help.py"
        },
        "payload": {
            "message": "Setting up gluster nodes cc353e38-52d4-47f1-8865-7eaebad2a034"
        },
        "priority": "info",
        "parent_id": null,
        "node_id": "612473bd-0d31-495d-b748-d047921bf9fd",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "0b6a15e3-fc96-4d2a-93b6-a328e9736e97"
    },
    {
        "publisher": "node_agent",
        "job_id": "fa06d3e8-ef0a-44c3-87cc-99c6191e0520",
        "timestamp": "2017-05-10T07:37:55.868665+00:00",
        "caller": {
            "function": "_run",
            "line_no": 210,
            "filename": "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py"
        },
        "payload": {
            "message": "Job failed Error setting up gluster node: Error setting up gluster node"
        },
        "priority": "error",
        "parent_id": null,
        "node_id": "612473bd-0d31-495d-b748-d047921bf9fd",
        "cluster_id": null,
        "flow_id": "2f94a48a-05d7-408c-b400-e27827f4eacd",
        "message_id": "bce8558f-d820-41a0-ae1f-045e449cbd97"
    }
]

curl command I used:
curl -X POST -H 'Authorization: Bearer 5bbc4967e277d2eabcc5d7202b8d2af3c5485384625fddb67baf293c7f2b55d6' -d '{"sds_name":"gluster","sds_version":"3.9.1", "sds_parameters":{"name":"MyCluster","cluster_id":"4654ac00-e67b-4b74-86a3-e740b1b8cee5", "public_network":"server_ip/22","cluster_network":"server_ip/22"}, "node_identifier":"ip","node_configuration":{"gl1_ip":{"role": "glusterfs/node", "provisioning_ip":"gl1_ip"},"gl2_ip":{"role":"glusterfs/node", "provisioning_ip":"gl1_ip"},"gl3_ip":{"role":"glusterfs/node", "provisioning_ip":"gl1_ip"}}}' http://server_ip/api/1.0/CreateCluster
Where gl1_ip, gl2_ip, gl3_ip and server_ip are ip addresses of hosts.

On gluster node with provisioning appered new tag "provisioner/2c0f4770-2547-4ddf-a328-be8b96f2058f" as seen in output of NodeContext from etcd (other hosts don't have the tag):
{"action":"get","node":{"key":"/nodes/612473bd-0d31-495d-b748-d047921bf9fd/NodeContext","dir":true,"nodes":[{"key":"/nodes/612473bd-0d31-495d-b748-d047921bf9fd/NodeContext/tags","value":"[\"provisioner/gluster\", \"provisioner/2c0f4770-2547-4ddf-a328-be8b96f2058f\", \"tendrl/node\"]","modifiedIndex":18181,"createdIndex":18181},{"key":"/nodes/612473bd-0d31-495d-b748-d047921bf9fd/NodeContext/updated_at","value":"2017-05-10 07:17:23.469265+00:00","modifiedIndex":18182,"createdIndex":18182},{"key":"/nodes/612473bd-0d31-495d-b748-d047921bf9fd/NodeContext/fqdn","value":"fbalak-usm1-gl1.usmqe.lab.eng.brq.redhat.com","modifiedIndex":18176,"createdIndex":18176},{"key":"/nodes/612473bd-0d31-495d-b748-d047921bf9fd/NodeContext/hash","value":"caee55e3680d5da4c5036ab648abc21a","modifiedIndex":18177,"createdIndex":18177},{"key":"/nodes/612473bd-0d31-495d-b748-d047921bf9fd/NodeContext/machine_id","value":"295285c351044601bc9eebf2369e50c6","modifiedIndex":18178,"createdIndex":18178},{"key":"/nodes/612473bd-0d31-495d-b748-d047921bf9fd/NodeContext/node_id","value":"612473bd-0d31-495d-b748-d047921bf9fd","modifiedIndex":18179,"createdIndex":18179},{"key":"/nodes/612473bd-0d31-495d-b748-d047921bf9fd/NodeContext/status","value":"UP","modifiedIndex":18180,"createdIndex":18180}],"modifiedIndex":4006,"createdIndex":4006}}

Tested with:
tendrl-commons-1.2.3-05_10_2017_05_13_05.noarch
tendrl-node-agent-1.2.3-05_10_2017_00_01_05.noarch
tendrl-node-monitoring-1.2.3-05_10_2017_04_25_07.noarch

@r0h4n
Copy link
Contributor

r0h4n commented May 10, 2017

@fbalak the new tag "provisioner/2c0f4770-2547-4ddf-a328-be8b96f2058f" indicates that python-gdeploy and gdeploy were installed on that node. This is good progress

@nnDarshan @shtripat please debug the failures of above job, @fbalak feel free to schedule bluejeans or google hangout with them to debug the issue faster

@fbalak
Copy link
Author

fbalak commented May 10, 2017

@r0h4n ok, thank you.

@nnDarshan
Copy link
Contributor

@fbalak Tendrl tries to create password-less ssh from provisioner node to all the other nodes(including itself).
Tendrl is not able to setup password-less ssh from provisioner node to itself. Does tendrl have permission to write the ssh-keys in authorized_key file ?

@nnDarshan
Copy link
Contributor

@fbalak /v2/keys/messages/events in etcd should have more info about the error.
Can you please search if you have any events with message "Unable to copy authorize key" or "Copying authorize key failed". Or any thing w.r.t authorized key error.

@fbalak
Copy link
Author

fbalak commented May 10, 2017

It seems working now. --> closing
Tested with:
tendrl-gluster-integration-1.2.3-05_10_2017_04_44_16.noarch
tendrl-commons-1.2.3-05_10_2017_15_19_26.noarch
tendrl-node-agent-1.2.3-05_10_2017_13_10_14.noarch
tendrl-node-monitoring-1.2.3-05_10_2017_04_25_07.noarch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants