Working with transactions

[sebbbastien]

Hi David,

Is it possible to manage "transactions", sets of instructions that must all be executed, or none?

For example, I want to create a "site": a firewall, a router, and add the router as the firewall's default route.
Is it possible to define these three operations as one and the same transaction, if one of them fails, none should be taken into account?

Sometimes it is more coherent to do nothing at all if one fails, rather than managing the partial states.

Do you know if something "native" is available? If not, do you have tracks/ideas to do that in a different way?

Best regards,

--
Sébastien

[gabstopper]

Sorry, accidentally sent too early. Hi Sebastien, I have thought about this and have some ideas, but there are some tricky aspects to doing this. First the SMC API does not support a "check mode" or "what if" or similar mode. So in the example above, you might end up with an engine created, a site and then when you try to add the router it could fail because simply the name already existed. At that point, the engine and site would have already been created so the rollback would essentially be a delete. When attempting to determine what types of changes could be considered atomic, things get a little more tricky (although nothing is impossible). One thing I have done is incorporate `update_or_create` logic with most elements which allows more flexibility when creating elements - i.e. if calling Router.update_or_create(name='foo') and the router 'foo' already exists, it would be modified. Versus Router.create(...) would fail entirely. With that being said I will test with a transaction layer that could be used as a context manager and see if that might be useful. This is a bit of a brainstorm, but I think supporting 'savepoints' would be useful as well where you could specify at which point a rollback should be performed. atomic = Atomic() with atomic as atom: print("First operation..: %s" % atom) try: host = Host.create(name='myhost2', address='1.1.1.1') atom.savepoint(host) ...... do some other stuff ...... except SMCException as e: # When the exception was hit here, the rollback would already be performed pass I do need to think more about what could reasonably be rolled back.. For example, if an element is modified (IP address changed for example) and something in the processing failed, should the IP address also be rolled back? ... I need to give this some more thought but I like the idea.

…

On Thu, Jul 12, 2018 at 6:29 AM, Sébastien ***@***.***> wrote: Hi David, Is it possible to manage "transactions", sets of instructions that must all be executed, or none? For example, I want to create a "site": a firewall, a router, and add the router as the firewall's default route. Is it possible to define these three operations as one and the same transaction, if one of them fails, none should be taken into account? Sometimes it is more coherent to do nothing at all if one fails, rather than managing the partial states. Do you know if something "native" is available? If not, do you have tracks/ideas to do that in a different way? Best regards, Sébastien — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#26>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOIA1VSdcXQ7x3wI4C4jCeEdq0edqGPoks5uFzMjgaJpZM4VMrXh> .

[gabstopper]

Here is a prototype that may work but needs more thought behind it.

Basically I've added a context manager that you can use to decorate a function or to use the 'with ....' statement.
Anything create operations that are performed in that block or decorated function will be considered 'transactions'. In the case of a failure, any existing transactions are rolled back (i.e. deleted) in the order they were added.

from smc.base import transaction
     
@transaction.atomic()
def mytasks():
    for name in range(1, 10):
        Host.create(name='host%s' % name, address='1.1.1.1')
     
    Network.create(name='mynet2', ipv4_network='1.1.1.0/24')
     
    print("Now fail")
    Host.create(name='host2', address='1.1.1.1')
     
 
with transaction.atomic():
    for name in range(1, 10):
        Host.create(name='host%s' % name, address='1.1.1.1')
     
    Network.create(name='mynet2', ipv4_network='1.1.1.0/24')
     
    print("Now fail")
    Host.create(name='host2', address='1.1.1.1')

In addition to these two models, I would opt to implement "savepoints" where you could nest context managers which would then trigger a 'savepoint'. In the example below, the top level context manager creates a bunch of hosts.
If the next operation in the nested context manager fails (creating the network element), the host elements will be preserved (i.e. not rolled back) due to the context manager nesting.
So treat a nested context manager as a 'savepoint' where all previous changes will be preserved.

When the Host.create(name='grace2') is run, only the inner element (the Network) is rolled back because of the inner savepoint logic.

with transaction.atomic():
    for name in range(1, 10):
        Host.create(name='host%s' % name, address='1.1.1.1')
        
    with transaction.atomic(): # <-- Nested context manager, saves all previous actions
        Network.create(name='mynet3', ipv4_network='1.1.1.0/24')
         
    print("Now fail")
    Host.create(name='host2', address='1.1.1.1')

[sebbbastien]

Thanks a lot, I'll try this right now.

I'll keep you informed.

[sebbbastien]

Hi David,

I can't use from smc.base import transaction, neither from master nor from develop.

Can you help me to test this solution?

--
Sébastien

[gabstopper]

Hi Sebastien,
I haven't pushed upstream yet, will need a couple more days as I've got some other updates for the session API as well. Will post back here once the post is up.