Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to pass various models to set_llm. #60

Merged
merged 6 commits into from
Dec 5, 2024

Conversation

krystian8207
Copy link
Member

Closes #59

Copy link
Member

@maciekbanas maciekbanas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finished review. Basic logic looks good. I have two questions regarding get_gitlab_pat() handler and failure in creating turn object in process_content() function when testing.

R/set_repos.R Outdated
@@ -47,7 +53,13 @@ set_gitlab_repos <- function(gitai,
GitStats::set_gitlab_host(
host = host,
repos = repos,
token = get_gitlab_pat(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure about this approach. Basically GitStats looks by itself for best-fit access key when token parameter is set to NULL (searches through all keys that start with GITLAB_*). Now, if you leave token parameter empty, tests will pass if you have any key starting with GITLAB_* that gives access to public GitLab API. If you will use approach with get_gitlab_pat(), test will fail if user does not have explicit GITLAB_PAT key as one giving access to public GitLab.

My suggestion would be to create token parameter on the level of set_*_repos() function which value would be then passed to GitStats set_*_host() function (here is issue for that: #58). By default it would stay NULL. For tests we could define it e.g. as GITLAB_PAT_PUBLIC. We could still leave it empty and pack it into withr the way I do in GitStats to test pulling default token:

test_that("When empty token for GitLab, GitStats pulls default token", {
  skip_on_cran()
  expect_snapshot(
    withr::with_envvar(new = c("GITLAB_PAT" = Sys.getenv("GITLAB_PAT_PUBLIC")), {
      test_gitstats <- create_gitstats() %>%
        set_gitlab_host(
          orgs = "mbtests"
        )
    })
  )
})

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I wasn't aware of this functionality of GitStats.
We agreed to add skipping tests directly in the GitStats-related methods.

my_project <-
initialize_project("gitai_test_project") |>
test_that("[integration test] processing content have proper output structure", {
my_project <- initialize_project("gitai_test_project") |>
set_llm() |>
set_prompt(system_prompt = "Say 'Hi there!' only and nothing else.")

result <- process_content(gitai = my_project, content = "")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests locally fail here, as turn object is NULL. It is created with last_turn() method, not sure if it should be somehow mocked?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It happened due to overwriting elmer:::Chat (indirectly by modification of ChatMocked).
Fixed it by using inherited R6 Class.

@maciekbanas
Copy link
Member

The example from README also fails on turn object. Maybe something has been changed in elmer recently?

image

Copy link
Member

@maciekbanas maciekbanas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now works great. Thank you.


my_project <-
initialize_project("gitai_test_project") |>
test_that("[integration test] processing content have proper output structure", {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain why we need to add this [integration test] prefixes to our tests?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's optional, just wanted to distinct the ones we're supposed to run with external keys.
I can revert it if you want.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. I was wondering if this is for some automatic processing I'm not aware about.
So I would remove those prefixes, as they seem to be nothing more then flag comments that can easily be out of sync with the code. Instead (we don't have to do that in this PR though) I would add a helper function in the utils like:
skip_if_integration_tests_should_be_skipped() (or sth shorter but that is what I found most informative so far)
This function could be added at the beginning of all integration tests. Inside the function can have skip_on_cran() and additionally it can skip test if some environmental variable is set to TRUE, like GITAI_SKIP_INTEGRATION_TESTS.
It would be easy to switch off all integration tests with that while developing locally if needed or inside CI/CD.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, gonna remove the prefixes then.

@krystian8207 krystian8207 merged commit 7e1d3da into main Dec 5, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

add more LLM backends to set_llm()
3 participants