Test suite

[gagb]

[gagb]

@afourney related to any hallucination metrics we end up using: https://arxiv.org/pdf/2305.14292.pdf

[gagb]

@afourney I was talking to @sonichi and we think it might be a good idea for the test suite to support the examples in the OG AutoGen paper. What do you think @qingyun-wu is also interested in helping implement this.

[afourney]

This sounds like a great idea. I'll finish up some documentation and submit a PR for the testbed to live under samples/tools (I'm not wedded to this location!) In any case, the system is super general, and we can try to reproduce results targeting any older or future version of autogen. Each test lives in a fresh docker container, and I can parameterize which version of the library get installed. @qingyun-wu let me know if you want to pursue this. Right now it lives here: https://github.com/microsoft/autogen/tree/testbed

[gagb]

@afourney question: what is the boundary between the test suite and the agentchat pytest tests in the repo?

[afourney]

@gagb I believe they serve different purposes. It's an imperfect analogy, but maybe the difference between unit tests vs. ML benchmarks.

The pytest suite -- to the best of my understanding -- is there to make sure the code generally works and that new PRs don't break things.

The testbed is more about generating trace data for engineering and scientific analysis. In the future, metrics will be computed from that trace data. But basically, each scenario is repeated numerous times, and in each trial begins from a fixed set of initial conditions in fresh Docker conainers. All environments, filesystems and etc. are isolated so that tests can't impact one another.

The pytest will tell you that function calling is working, or that the GroupChatManage is able to route messages appropriately, etc.

The testbed will tell you that, when running the same prompt 10 times, the agents failed to produce the desired artifact twice, that 3 of the 8 artifacts made incorrect assumptions about the current date, that on average the agents exchanged 6 messages, and that code extraction failed once.

[gagb]

This makes sense! Thank you! Also the test suite is more comprehensive and expensive so the entirety of it shouldn't be part of the build. Or maybe it should be at a lower cadence.

[afourney]

You would know. :) The bills go to you haha