Add new compiler backend

[jlowin]

This is a large PR that restructures the entire orchestration backend without touching the user-facing objects.

Pipeline:

• a flow or task is run
• a Controller object is created. The Controller is given a few parameters (the tasks to complete, the agents assigned, a flow) and its job is to orchestrate resources to achieve those goals
• On each invocation, the Controller does the following:
  • looks for any tasks in the flow that are upstream of its assigned tasks and are ready to run
  • creates a pool of candidate agents assigned to those tasks
  • looks to see if one of the candidates is already selected as the active agent
  • if not, selects a new active agent
  • creates an AgentContext consisting of the agent itself, the agent's assigned tasks, and a sequence of events relevant to those tasks
  • creates tools for the agent based on the active tasks
  • compiles the events history into a series of LLM messages
  • runs the agent's LLM on those messages
  • calls any tools
  • determines if a new agent has been selected

The key to this backend is that all activity is recorded as a sequence of events that can optionally be persisted. These events represent everything that's happened in the workflow. When it's time to run an agent, we compile the events into LLM messages appropriate for that agent. For example, that agent's own responses are given the assistant role, and other agents' responses are given the user role with a system preamble explaining who the speaker is. Anecdotally this is far superior to the current approach in which all agents share the assistant role.

In addition, the compilation only includes events that are relevant to the agent's current objective. This means that if an agent is selected to work on task A, and in the next invocation a different agent is selected to work on task B, and A and B are completely independent, then the second agent will not be shown any of the first agent's responses, because they have no bearing on its work.

In essence, we compile a specialized view of the workflow universe for every agent invocation.