Limit the number of log lines displayed

[omarkhan]

See OC-1035.

Problem

The interface takes a very long time to load, as it tries to fetch all log entries for all instances and there can be a lot of data sent over the wire.

Solution

This pull request adds a configurable limit on the number of log entries returned in the initial request. You can set the limit using the LOG_LIMIT environment variable (default: 10000).

In addition, loading log entries from the server is deferred until the user actually selects an instance.

Testing

To run the js tests in your development environment, see #39.

To test manually, create a server and instance with some log entries. We can use the test factories for this:

from instance.tests.models.factories.server import OpenStackServerFactory

server = OpenStackServerFactory.create()
server.log_entry_set.create(level='INFO', text='server log message')
server.instance.log_entry_set.create(level='INFO', text='instance log message')

Then, start the dev server with

make rundev

With your web console open, go to http://localhost:5000/ and select the instance we just created. You will see a second request to fetch the log entries.

[smarnach]

Out of scope: Are we using Python 3.5? If we do, the whole function above should be deleted, and we should use heapa.merge() instead. (In Python 3.4, this function doesn't have a key parameter, but it might be worthwhile to rewrite the function anyway.)

[omarkhan]

I don't think we're using python 3.5, as pylint doesn't support it yet. Thanks for the tip about heapq.merge, I didn't know that. Should I revert this?

[omarkhan]

Actually pylint does support python 3.5 now, so we could make the switch. See #6

[smarnach]

For the time being, let's just leave the code the way it is. While it's easy to write a helper function implements the key parameter around the old heapq.merge(), it's rather cumbersome (and slow) to write something that supports the reversed parameter. For reversing, you really need to implement custom comparisons, and can't simply annotate the objects with their keys.

We can clean this up once we're using Python 3.5.

[omarkhan]

I have reverted this change for the time being.

[smarnach]

@omarkhan Argh, that's not really what I meant (though it's really what I wrote). I actually meant: just leave the code the way you have it.

Anyway, doesn't really matter, either way is fine, and we'll make sure to simplify this once we are on Python 3.5.

[smarnach]

@omarkhan Well done, looks good already! I've added a few comments, and will continue testing this tomorrow.

[omarkhan]

@smarnach I have added some manual test instructions. Let me know if you have any problems.

[smarnach]

@omarkhan Tested manually, looks all good now. 👍

[omarkhan]

Thanks @smarnach, rebasing

[omarkhan]

@antoviaque will this deploy automatically?

[antoviaque]

@omarkhan Sadly no, at least not yet - definitely on the list of future improvements: https://opencraft.uservoice.com/forums/280754-opencraft/suggestions/9937191-self-deployment . You will have to do it yourself for now (which you can since updates aren't handled by ansible currently):

• ssh ubuntu@console.opencraft.com
• then access screen -rd
• pull the latest code from master (in a sudo su www-data shell, opencraft lives in /var/www/opencraft)
• then switch to the screen that runs make run, stop it, run the migrations and restart it.
• also restart the make shell to ensure it uses the latest code

Let me know if you have any question.

[smarnach]

@omarkhan I have deployed this to production. (It happened automatically when I deployed my changes.) I recommend trying out http://console.opencraft.com/ with this PR installed; it's great! Thanks for getting this done!

[omarkhan]

Thanks Sven! Yeah it was almost unusable for me before

[antoviaque]

@omarkhan Yes, thank you so much for that fix! Really great :D