-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow to pass various models to set_llm. #60
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Finished review. Basic logic looks good. I have two questions regarding get_gitlab_pat()
handler and failure in creating turn
object in process_content()
function when testing.
R/set_repos.R
Outdated
@@ -47,7 +53,13 @@ set_gitlab_repos <- function(gitai, | |||
GitStats::set_gitlab_host( | |||
host = host, | |||
repos = repos, | |||
token = get_gitlab_pat(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure about this approach. Basically GitStats
looks by itself for best-fit access key when token
parameter is set to NULL
(searches through all keys that start with GITLAB_*
). Now, if you leave token
parameter empty, tests will pass if you have any key starting with GITLAB_*
that gives access to public GitLab API. If you will use approach with get_gitlab_pat()
, test will fail if user does not have explicit GITLAB_PAT
key as one giving access to public GitLab.
My suggestion would be to create token
parameter on the level of set_*_repos()
function which value would be then passed to GitStats
set_*_host()
function (here is issue for that: #58). By default it would stay NULL
. For tests we could define it e.g. as GITLAB_PAT_PUBLIC
. We could still leave it empty and pack it into withr
the way I do in GitStats
to test pulling default token:
test_that("When empty token for GitLab, GitStats pulls default token", {
skip_on_cran()
expect_snapshot(
withr::with_envvar(new = c("GITLAB_PAT" = Sys.getenv("GITLAB_PAT_PUBLIC")), {
test_gitstats <- create_gitstats() %>%
set_gitlab_host(
orgs = "mbtests"
)
})
)
})
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, I wasn't aware of this functionality of GitStats.
We agreed to add skipping tests directly in the GitStats-related methods.
my_project <- | ||
initialize_project("gitai_test_project") |> | ||
test_that("[integration test] processing content have proper output structure", { | ||
my_project <- initialize_project("gitai_test_project") |> | ||
set_llm() |> | ||
set_prompt(system_prompt = "Say 'Hi there!' only and nothing else.") | ||
|
||
result <- process_content(gitai = my_project, content = "") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tests locally fail here, as turn
object is NULL
. It is created with last_turn()
method, not sure if it should be somehow mocked?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It happened due to overwriting elmer:::Chat (indirectly by modification of ChatMocked).
Fixed it by using inherited R6 Class.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now works great. Thank you.
|
||
my_project <- | ||
initialize_project("gitai_test_project") |> | ||
test_that("[integration test] processing content have proper output structure", { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you explain why we need to add this [integration test] prefixes to our tests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's optional, just wanted to distinct the ones we're supposed to run with external keys.
I can revert it if you want.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it. I was wondering if this is for some automatic processing I'm not aware about.
So I would remove those prefixes, as they seem to be nothing more then flag comments that can easily be out of sync with the code. Instead (we don't have to do that in this PR though) I would add a helper function in the utils like:
skip_if_integration_tests_should_be_skipped() (or sth shorter but that is what I found most informative so far)
This function could be added at the beginning of all integration tests. Inside the function can have skip_on_cran() and additionally it can skip test if some environmental variable is set to TRUE, like GITAI_SKIP_INTEGRATION_TESTS.
It would be easy to switch off all integration tests with that while developing locally if needed or inside CI/CD.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, gonna remove the prefixes then.
Closes #59