UI service MVP #95

damienkilgannon · 2017-02-02T01:53:19Z

Pull request to provide the first steps to satisfying #25.

A simple angularjs + Flask UI provides a UI to the user to check status of the Scrapy Cluster and submit crawl requests easily.

The status checks and crawl requests are provided via the rest service.

coveralls · 2017-02-02T14:27:25Z

Coverage increased (+0.02%) to 67.237% when pulling 08d09bc on damienkilgannon:dev into 3f20ea1 on istresearch:dev.

damienkilgannon · 2017-02-02T14:35:23Z

Screen grab of the UI as it is currently ...

madisonb · 2017-02-04T16:19:27Z

@damienkilgannon This is an awesome start, great job! Here is what I have done:

Started up scrapy cluster with docker-compose up -d, the Rest service returns

$ curl localhost:5343
{
  "kafka_connected": true,
  "my_id": "fc6c7bc9ae7e",
  "node_health": "GREEN",
  "redis_connected": true,
  "uptime_sec": 249
}

Installed the python reqs from the /ui folder, and then ran the ui service

$ python ui_service.py
2017-02-04 11:00:05,013 [ui_service] INFO: Running main flask method on port 5000
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)

When I hit the main ui in my browser at localhost:5000, I get the following:

and the logs say

2017-02-04 11:00:09,587 [ui_service] INFO: 'index' endpoint called
127.0.0.1 - - [04/Feb/2017 11:00:09] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [04/Feb/2017 11:00:09] "GET /static/js/app.js HTTP/1.1" 200 -
127.0.0.1 - - [04/Feb/2017 11:00:09] "GET /static/angular_settings.js HTTP/1.1" 200 -
127.0.0.1 - - [04/Feb/2017 11:00:09] "GET /static/css/style.css HTTP/1.1" 200 -
127.0.0.1 - - [04/Feb/2017 11:00:09] "GET /static/img/logo.png HTTP/1.1" 200 -
127.0.0.1 - - [04/Feb/2017 11:00:10] "GET /static/partials/overview.html HTTP/1.1" 200 -
127.0.0.1 - - [04/Feb/2017 11:00:10] "GET /favicon.ico HTTP/1.1" 404 -

I don't think the 404 is the issue here, but perhaps it is.

I clicked through the other pages, but I see they are blank (as far as I can tell). The form submission for a crawl request also doesnt work, but again I am not sure if it is because of my initial error above.

Regardless, I think we should work on this with the goal of it being released with v1.3, given that Milestone 1.2 is so close. I would like to actually merge this into a new branch ui and then create a bunch of tasks/tickets/checkboxes for things we need to do.

Off the top of my head I can think of the following:

Documentation
Unit Testing
Figuring out why Travis CI wants 'python path' set in run_tests.sh
Filling out
- Crawlers
- Kafka
- Redis (do we rename these to "Redis/Kafka Monitor")
Cluster Debug
Docker (main and alpine python images)
Travis Testing
etc from our discussions on UI for displaying information about Cluster #25

From there, you/I can work and make PR's onto the ui branch and then when we are happy with it, merge into dev -> master to be included in an official release.

Thoughts? Really nice initial work!

damienkilgannon · 2017-02-05T21:36:33Z

Sounds, good ... i would think that error is as a result of making request to the rest service at the wrong IP:PORT. The ui is running javascript in your browser which makes GET/POST requests to the rest service, its defaults to localhost and as the ui is running on your browser (not the same host as the rest service) it fails. Need to change to edit ui/static/angular_settings.js to reflect the IP of the virtual machine which the rest service is running on.
Will I move the UI stuff to a new branch on the pull request or do you want to create a new branch first?

madisonb · 2017-02-12T20:27:21Z

Do you have any suggestions for the 404 error? I can hit http://localhost:5343/(which is what my angular settings say) and get a return value, but the UI still doesn't quite function.

EDIT: In the console the following error occurs:

XMLHttpRequest cannot load http://localhost:5343/. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:5000' is therefore not allowed access.

EDIT 2: Everything returns 200 in the network inspector, except for the minified angular js which is a 307

damienkilgannon · 2017-02-13T17:22:03Z

rest/rest_service.py

@@ -142,6 +143,7 @@ def __init__(self, settings_name):
        self.wrapper = SettingsWrapper()
        self.logger = None
        self.app = Flask(__name__)
+        CORS(self.app)


@madisonb the CORS module adds support for cross origin requests

the 'No 'Access-Control-Allow-Origin'' should be not be an issue as the CORS modules adds the required HTTP headers to the HTTP messages to allow cross origin requests.

Following your second EDIT .... are you still getting that error?

damienkilgannon · 2017-02-13T17:38:29Z

@madisonb if you have a look at the above screenshot you will see that my client browser connecting to the UI service, note that the request to the REST service from the UI is not via 'localhost' but the IP of the vagrant machine at 'http://192.168.33.99:5343/'.
On your local machine is the REST and UI service running on your localhost or the vagrant? If they are running on the vagrant you need to edit the 'angular_settings' to reflect the IP of the vagrant machine.

damienkilgannon · 2017-02-15T08:31:21Z

@madisonb does that make sense? I think as the IP of the Vagrant machine is set in the 'Vagrantfile' as 192.168.33.99, I should change the default from localhost -> 192.168.33.99.

madisonb · 2017-02-15T14:37:40Z

@damienkilgannon ah ok, I typically just use docker or run the scripts on my host os in order to test/debug stuff. Can you check to see if the problem works with the following setup (and how do we resolve it)?

run docker-compose up -d, make sure curl from your host os your can curl localhost:5343
Run the ui server from your local machine

This is my setup which ends up complaining.

damienkilgannon · 2017-02-15T14:44:02Z

I will try with that setup later this evening and see how it can be fixed

damienkilgannon · 2017-02-17T10:36:35Z

@madisonb I have replicated the issue and its a CORS problem but unsure why its on issue with docker environment on not vagrant. Will revert back when I have fix.

madisonb · 2017-03-04T05:57:08Z

@damienkilgannon I haven't forgotten about this, would like to get the CORS issue fixed and then can merge this into a ui branch so we can do more work. Any help needed?

damienkilgannon · 2017-03-04T12:03:28Z

@madisonb never got a chance to resolve the problem .... I have time this weekend so going to look at it now.

damienkilgannon · 2017-03-04T13:04:52Z

@madisonb so I made some changes, created a route decorator for the rest service which adds the CORS headers to response in a better way than I had set up before. However that wasn't the problem. The reason it doesn't work with docker is due to the fact that docker pulls the 'rest service' image from the repo which doesn't have the changes I added to allow cross origin requests. For you to run it locally you need to run the rest service and the UI service as they are in my pull request. Does that make sense?

damienkilgannon · 2017-03-04T13:08:27Z

Sorry didn't mean to create that merge request!

… into dev

damienkilgannon · 2017-03-04T17:10:34Z

@madisonb not sure what is causing two of the builds to fail, can you have a look?

coveralls · 2017-03-20T10:39:24Z

Coverage increased (+0.1%) to 65.844% when pulling 71e790a on damienkilgannon:dev into 2956780 on istresearch:dev.

damienkilgannon · 2017-03-20T10:50:16Z

@madisonb I just ran that build job again and all passed. What else would you like me to add in before starting a new branch and merging the pull request into that branch? Is it your plan to have a separate branch for the UI ?

madisonb · 2017-03-20T16:22:43Z

@damienkilgannon My plan of attack is as follows:

Wrap up this PR testing for the 👍
Merge it into a new ui branch where we can continue to add features/unit tests/docs/etc
When ready, merge it into dev to be included as part of 1.3

I have been working hard on wrapping up milestone 1.2 so once that is completed that opens up a lot more time to work on this. However, if you would like to continue adding features to this PR my workflow will not change

damienkilgannon · 2017-03-20T16:24:59Z

Sounds good with me. I will continue working on the PR for the time been.

madisonb · 2017-04-13T22:40:02Z

Closing as I have merged this branch into a new ui branch. All subsequent PR's for UI activity should go there until that branch is merged into dev

Please see #116 for remaining checklist to get the UI merged into dev.

damienkilgannon added 6 commits January 31, 2017 08:35

initial commit ui feature

96ec18a

updated ui-service.py

41a431c

added rest api call

9f36a86

support for CORS

ca08293

simple ui

842e845

mvp

97f2329

damienkilgannon changed the title ~~Dev~~ UI service MVP Feb 2, 2017

damienkilgannon added 4 commits February 2, 2017 02:06

fixed requirements.txt

fd7470d

requirments.txt

1767dcf

Merge remote-tracking branch 'upstream/dev' into dev

197f020

set pythonpath in script

08d09bc

damienkilgannon commented Feb 13, 2017

View reviewed changes

damienkilgannon added 2 commits March 4, 2017 12:59

removed dependency on flask CORS module

9a61f3b

updated requirements

916e481

Merge branch 'dev' into dev

db1cbac

updated rest service tests to work with response object

d65a4d2

Merge branch 'dev' of https://github.com/damienkilgannon/scrapy-cluster…

735401e

… into dev

damienkilgannon and others added 4 commits March 20, 2017 09:43

changed scutils version

3df267e

Merge branch 'dev' into dev

a3427de

bumped scutils version back to 1.2.0dev7

9891770

fix merge conflicts

71e790a

madisonb closed this Apr 13, 2017

madisonb mentioned this pull request Jul 17, 2017

Scrapy-Cluster UI #129

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UI service MVP #95

UI service MVP #95

damienkilgannon commented Feb 2, 2017

coveralls commented Feb 2, 2017 •

edited

Loading

damienkilgannon commented Feb 2, 2017

madisonb commented Feb 4, 2017 •

edited

Loading

damienkilgannon commented Feb 5, 2017

madisonb commented Feb 12, 2017 •

edited

Loading

damienkilgannon Feb 13, 2017 •

edited

Loading

damienkilgannon Feb 13, 2017

damienkilgannon commented Feb 13, 2017

damienkilgannon commented Feb 15, 2017

madisonb commented Feb 15, 2017 •

edited

Loading

damienkilgannon commented Feb 15, 2017

damienkilgannon commented Feb 17, 2017

madisonb commented Mar 4, 2017

damienkilgannon commented Mar 4, 2017

damienkilgannon commented Mar 4, 2017

damienkilgannon commented Mar 4, 2017

damienkilgannon commented Mar 4, 2017

coveralls commented Mar 20, 2017 •

edited

Loading

damienkilgannon commented Mar 20, 2017

madisonb commented Mar 20, 2017

damienkilgannon commented Mar 20, 2017

madisonb commented Apr 13, 2017

UI service MVP #95

UI service MVP #95

Conversation

damienkilgannon commented Feb 2, 2017

coveralls commented Feb 2, 2017 • edited Loading

damienkilgannon commented Feb 2, 2017

madisonb commented Feb 4, 2017 • edited Loading

damienkilgannon commented Feb 5, 2017

madisonb commented Feb 12, 2017 • edited Loading

damienkilgannon Feb 13, 2017 • edited Loading

Choose a reason for hiding this comment

damienkilgannon Feb 13, 2017

Choose a reason for hiding this comment

damienkilgannon commented Feb 13, 2017

damienkilgannon commented Feb 15, 2017

madisonb commented Feb 15, 2017 • edited Loading

damienkilgannon commented Feb 15, 2017

damienkilgannon commented Feb 17, 2017

madisonb commented Mar 4, 2017

damienkilgannon commented Mar 4, 2017

damienkilgannon commented Mar 4, 2017

damienkilgannon commented Mar 4, 2017

damienkilgannon commented Mar 4, 2017

coveralls commented Mar 20, 2017 • edited Loading

damienkilgannon commented Mar 20, 2017

madisonb commented Mar 20, 2017

damienkilgannon commented Mar 20, 2017

madisonb commented Apr 13, 2017

coveralls commented Feb 2, 2017 •

edited

Loading

madisonb commented Feb 4, 2017 •

edited

Loading

madisonb commented Feb 12, 2017 •

edited

Loading

damienkilgannon Feb 13, 2017 •

edited

Loading

madisonb commented Feb 15, 2017 •

edited

Loading

coveralls commented Mar 20, 2017 •

edited

Loading