More providers for testing

[jamadeo]

Also run them in parallel

[Copilot]

Pull request overview

This PR expands the set of providers covered by scripts/test_providers.sh and changes the test harness to run provider/model combinations in parallel for faster feedback. It also adds conditional inclusion of many providers based on environment variables or installed CLIs.

Changes:

• Extend the PROVIDERS list and add conditional blocks for Databricks, Azure OpenAI, AWS Bedrock, GCP Vertex AI, Snowflake, Venice, LiteLLM, Ollama, SageMaker TGI, GitHub Copilot, ChatGPT Codex, and several CLI-based providers.
• Refactor the test runner to build a job list, execute tests in parallel with a configurable MAX_PARALLEL, and collect results from per-job temporary directories.
• Preserve and adapt the existing success/allowed-failure logic to work with the new parallel execution model.

[Copilot]

meta_file and the corresponding write to $RESULTS_DIR/meta_$idx are never read anywhere, so this extra file creation is dead code and can be removed to simplify the parallel runner loop.

Suggested change

	meta_file="$RESULTS_DIR/meta_$idx"
	echo "$provider|$model" > "$meta_file"

[jamadeo]

As of now, this is what's added but not run:

⚠️  Skipping Databricks tests (DATABRICKS_HOST and DATABRICKS_TOKEN required)
⚠️  Skipping Azure OpenAI tests (AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_DEPLOYMENT_NAME required)
⚠️  Skipping AWS Bedrock tests (AWS_REGION and AWS_PROFILE or AWS credentials required)
⚠️  Skipping GCP Vertex AI tests (GCP_PROJECT_ID required)
⚠️  Skipping Snowflake tests (SNOWFLAKE_HOST and SNOWFLAKE_TOKEN required)
⚠️  Skipping Venice tests (VENICE_API_KEY required)
⚠️  Skipping LiteLLM tests (LITELLM_API_KEY required)
⚠️  Skipping Ollama tests (OLLAMA_HOST required or ollama must be installed)
⚠️  Skipping SageMaker TGI tests (SAGEMAKER_ENDPOINT_NAME and AWS_REGION required)
⚠️  Skipping GitHub Copilot tests (OAuth setup required - run 'goose configure' first)
⚠️  Skipping ChatGPT Codex tests (OAuth setup required - run 'goose configure' first)
⚠️  Skipping Claude Code CLI tests ('claude' CLI tool required)
⚠️  Skipping Codex CLI tests ('codex' CLI tool required)
⚠️  Skipping Gemini CLI tests ('gemini' CLI tool required)
⚠️  Skipping Cursor Agent tests ('cursor-agent' CLI tool required)

[jamadeo]

Running one alone before the rest concurrently is kind of silly, but solves two small pain points. If you immediately start N concurrent sessions, then:

• locally, you'll get the keychain password prompt N times, even if you click "always allow"
• in CI, they all try to create the sqlite database at the same time, and that leads to things breaking. Maybe we should fix that but it also seems highly unlikely to happen in the real world

[DOsinga]

it's also useful since most likely if you break something it is broken for all providers. this way it tells you that immediately maybe

[michaelneale]

yeah the faster inner loop - I think is reasonable

[michaelneale]

nice - proof of the pudding is in the eating, lets get this in.

Also note the time diff:

Code mode helping a lot?

[jamadeo]

Code mode helping a lot?

I think the main difference there is that the other one also runs a compaction test which takes 4-5 min right now. Maybe that should be split out of sequence