Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

as_agent_process requires to serialize whole self scope #129

Closed
maurerle opened this issue Oct 29, 2024 · 5 comments · Fixed by #131
Closed

as_agent_process requires to serialize whole self scope #129

maurerle opened this issue Oct 29, 2024 · 5 comments · Fixed by #131

Comments

@maurerle
Copy link
Collaborator

In my simulation creation, I have a simulation creator which adds a container to the simulation like this as part of a method.

Now that mango 2.x uses dill, this changed quite a bit, as dill tries to pickle the whole self context ? which fails as there are objects which are not serializable (like a generator and a db connection) attached to the world.

Is there some other way to limit the scope required to run as_agent_process as before?

id = "hallo"
def creator(container):
    # creating a new role agent and apply the role of a units operator
    units_operator = UnitsOperator(
        available_markets=list(self.markets.values())
    )
    unit_operator_agent = agent_composed_of(
        units_operator, register_in=container, suggested_aid=str(my_id)
    )
    unit_operator_agent._role_context.data.update(
        { "data": self.mydata}
    )
    container.register(DistributedClockAgent(), suggested_aid=clock_agent_name)
    for unit in units:
        units_operator.add_unit(self.create_unit(**unit))

self.container.as_agent_process(agent_creator=creator)
@rcschrg
Copy link
Member

rcschrg commented Oct 29, 2024

The difference is not exactly dill since 2.X, we use "spawn" instead of "fork" on all platforms. However, this should not lead to the necessity of serializing the whole container. Where exactly do you store the DB connection or the generator? When I recall it right, dill pre-serializes the codec, the agent_creator function, and the mirror_container_function (which is defined in the global scope).

@maurerle
Copy link
Collaborator Author

The problem comes from calling "self.dosomething" in the creator function.
This requires to serialize the self context (as it is used in the creator function), which might fail.

This can be fixed, by accessing the values in the self context before, storing it in variables, and serializing them in the creator function:

id = "hallo"
units_operator = UnitsOperator(
    available_markets=list(self.markets.values())
)
for unit in units:
    units_operator.add_unit(self.create_unit(**unit))
update_dict = { "data": self.mydata}
def creator(container):
    # creating a new role agent and apply the role of a units operator
    unit_operator_agent = agent_composed_of(
        units_operator, register_in=container, suggested_aid=str(my_id)
    )
    unit_operator_agent._role_context.data.update(
        update_dict
    )
    container.register(DistributedClockAgent(), suggested_aid=clock_agent_name)

async def main():
    await self.container.as_agent_process(agent_creator=creator)

asyncio.run(main())

Though the container.as_agent_process function still requires to have a running loop and needs to be awaited. This is unexpected.

Finally the above snippet does not work, as even if the loop is running, the self.container does not have a context yet.
So we need to further delay the creation of the tasks in the mirror container as well?

@maurerle
Copy link
Collaborator Author

An example of tests I would like to succeed are:
maurerle@eaccade

Though I am not sure what the most clever way to do this is..

@rcschrg
Copy link
Member

rcschrg commented Oct 30, 2024

Ah I see. Regarding the first problem of the serialization, I do not think there is way around this, and this may be a good thing honestly.

Regarding the second, that's tough. You would need to delay spawning the process as a whole or stop it before it does anything. The problem here is that registering requires IPC, and IPC is currently handled using async pipes. This would lead to the requirement of non-async agent registration, which is possible with polling. We could also implement something like late agent processes, which are spawned while the container is activated. This could be a little bit messy, though, as we always want to ensure that the agent lifecycle remains correct (on_register before on_start before on _ready ...)

@maurerle
Copy link
Collaborator Author

  1. I second this
  2. I would like to have the delayed agent processes, though this also does not work nicely with pre_hook_reserve_aid..

I drafted something in #131 though I do not quite know how we can fix the registration of agent ids there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants