Investigate why a small RDS instance crashes during index

[acidjunk]

I had to run the index on my mac: as indexing on AWS didn't produce any output after, "Mounting the agent":

2025-09-22 20:25:19 [info     ] Starting the orchestrator      version=4.5.0a2
2025-09-22 20:25:22 [debug    ] Using orjson
2025-09-22 20:25:22 [info     ] WebSocketManager object configured, all methods referencing `websocket_manager` should work. [orchestrator.websocket]
2025-09-22 20:25:22 [info     ] DistLockManager object configured, all methods referencing `distlock_manager` should work. [orchestrator.distlock]
2025-09-22 20:25:22 [info     ] Database object configured, all methods referencing `db` should work. [orchestrator.db]
2025-09-22 20:25:22 [info     ] Mounting the agent             [orchestrator.agentic_app]

[timfdev]

Theory: Upsert batches are too big, solution would be to reduce the chunk_size parameter in the Indexer. Going to need aws access to test this.

[acidjunk]

Next sprint.

[timfdev]

In #1096 the max_batch_size is now configurable , that should make it easy to test if the embedding api is the bottleneck. If not it might be the upsert batch size.