How to run e2e tests

[mattlindsey]

Are there instructions or hints on how to get the e2e tests to run locally. I'm on a Mac and all 50 tests fail like this:

[mattlindsey]

I discovered 25 of the 50 tests succeed if I run ops server in a separate terminal. Do I run it with RAILS_ENV=test or something?
Here are a few failures:

[oxaronick]

Hey @mattlindsey, there is no documentation for this right now, but let me jot down a few notes off the top of my head, and once they work for you I'll turn it into documentation.

For context, the e2e specs aren't like other RSpec tests in that they just make HTTP calls to Archyve's APIs. They are making HTTP calls and asserting that the response code, body, headers, etc. are what is expected. Sometimes they also then go back to the Archyve API to check that a side effect of the HTTP call was carried out correctly; for example, after hitting POST /v1/chat, an e2e spec might go check that a Conversation with two messages was actually created in the database.

So, to run them:

1. have Archyve and OPP running, with the usual ops server
2. have a Collection in Archyve called "Testing", with at least one document in it

• I like to have KG enabled on the Collection, and have at least one uploaded file Document and one web Document in the Collection

4. have the active server pointing to a running instance of Ollama with llama3.1:8b in it

• this is an odd choice - that string is hard-coded in a spec file and could be changed to llama3.1:latest to be more likely to pass for the average Ollama user

5. run ops e2e

Keep in mind that the tests are just making HTTP requests, and aren't aware of the internal workings of Archyve. So if you have a test fail saying it got a 500 back, that means Archyve returned a 500 to the spec, not that Archyve got a 500 from Ollama or ChromaDB.

LMK how it goes!

[mattlindsey]

@oxaronick Those instructions worked pretty well. In step 4 I assume active server means the chat model, So I updated that on the Archyve Settings page.

I did get 4 failures, usually getting 500 instead of 200, but I'm not going to worry about it:

50 examples, 4 failures

Failed examples:

rspec ./spec/e2e/api/version_spec.rb:8 # /v1/version returns a valid version
rspec ./spec/e2e/opp/chat_spec.rb:31 # opp/api/chat creates messages in the database
rspec ./spec/e2e/opp/completions_spec.rb:18 # opp/v1/chat/completions when client is Open WebUI returns a chat response
rspec ./spec/e2e/opp/completions_spec.rb:47 # opp/v1/chat/completions when client is Huggingface ChatUI returns a chat response

One of them:

  4) opp/v1/chat/completions when client is Huggingface ChatUI returns a chat response
     Failure/Error: expect(call.code).to eq(200)

       expected: 200
            got: 500

       (compared using ==)
     # ./spec/e2e/opp/completions_spec.rb:48:in `block (3 levels) in <top (required)>'

[nickthecook]

That's good for a first run!

The version test failure is likely because Archyve gets its version from a hit tag and you don't have any of those in your clone. You can run git fetch --tags <origin>, where origin is what you called my repo in your fork.

For the others, when the tests get a 500 you will see some errors in your ops server output. Hopefully there are some clues there.

The active server is actually something that you can see on the Settings page. There is an "active" column in the server list. If you only have one server you don't need to worry about that.

Thanks for the feedback!