Transactionize qiita-ware

[josenavas]

There is not a huge problem on having the system as it is right now but we need to better transactionize qiita-ware. However, this requires re-coding and moving code down to qiita-db which is something that we don't have time to do before SLOAN.

[antgonza]

@josenavas could you make a list of exactly what needs to be done?

[josenavas]

qiita-ware is not transactionized. It cannot be transactionized in the same way that qiita-pet is done because then the function becomes non-pickable.

I'm unsure on how to better approach this issue, and I think that in order to perfectly fix this we need:

1. A better qiita_db api that provides all the information that we need at once. Linked with recent discussions with @wasade this will be possible through a REST api (plugins are gonna be based in REST api).
2. Each function/process in Qiita ware needs to be inspected manually and see which things are atomic and which things are not, this requires a fair amount of developer time. Since there are few refactors that need to happen anyways, I think this can be pushed back until we clear out all the other issues, and potentially solve during those other refactors. (e.g. during the refactor of EBI we can transactionize it)

[wasade]

This is serializable though:

from qiita_db.sql_connection import TRN
import cPickle

def foo():
    with TRN:
        pass

cPickle.dumps(foo)

Or is that pattern not amendable for the compute?

[josenavas]

Yes, but that is what will couple qiita-ware with qiita-db and I though we want to avoid that as much as possible.

[wasade]

Okay, so the goal is for qiita-ware to do processing without any knowledge of the database. More generally, is the goal for qiita-ware to do processing without any knowledge of an external resource? If so, then simple functions should work?

def foo(stuff_to_process):
    pass

[josenavas]

Yes, however the current qiita_ware is not design like that, it is designed to be accessing the database independently (as it is using the objects directly). I think I nice goal for qiita_ware will be to act like this:

1. Do 1 call to the external resource to get all the data and let the external resource know that is going to perform some computation on such data
2. Perform the computation
3. Do 1 call to the external resource to store the results of the computation and let the external resource know that the computation is over.

[wasade]

I like that strategy, however, it's worth thinking carefully about whether the external resource is stateful or not. RESTful interfaces, for instance, are not stateful and each HTTP request can be handled such that the server is not tracking the client. Tracking state adds a series of challenges to resolve, which are totally do able, but more complexity leads to more bugs. A different take on the above is:

1. POST /analysis - have the post return a new and unique resource in the collection (e.g., some int).
2. GET /whatever and repeat until data is collected
3. Do compute
4. PUT /analysis/:id the results

If state is critical, as it maybe for being able to determine if a job failed and needs to be restarted, check pointing, etc, then an option would be to do the stateless interfaces like above and track state in a separate system (i.e., separation of concerns). One of the challenges is that in tracking state, you really want the server to be able to know if the client is still alive, or has a heartbeat, so one model there may be:

1. A worker drops a unique key message box when starting with an expiration of 60s
2. A worker updates its key at least every 60s
3. A worker moves its unique key to a completed bucket

OR

1. The key does not exist in the message box or in the completed bucket

In parallel, a server monitors the buckets with the ability to respawn. I think packages like Celery already do this. I worry doing this ourselves as there probably are a lot of edge cases to cover, but that doesn't mean we shouldn't.

[josenavas]

Thanks @wasade all this is really useful information. I will probably check with you once the implementation of this part becomes closer!

[antgonza]

@josenavas was this closed in #1318?

[josenavas]

Nope, qiita-ware was not transactionize during that process... It was too tricky to do.