Assure created model of OpenAIWrapper is in config_list #847

yiranwu0 · 2023-12-02T23:32:37Z

Why are these changes needed?

When use create(..., model=<model_name>), assure that <model_name> exists in config_list.
matching <model_name> with the config in config_list.

Note:
Switching **config and **self._config_list[i] of full_config = {**config, **self._config_list[i]} doesn't work for azure. No matter what model is passed in, the deployed model is determined. For example, when creating azure, the api to use will be converted to https://somedeloyment.openai.azure.com/gpt4, if the param has model=gpt 3.5, it will still use gpt4.

Also, when user passes in config_list, they are limiting what models to use and we should assure that calling to model not shown in config_list will result in error.

Related issue number

#819

Checks

I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

codecov-commenter · 2023-12-02T23:34:04Z

Codecov Report

Attention: 11 lines in your changes are missing coverage. Please review.

Comparison is base (55b9d0b) 26.65% compared to head (ca02914) 38.11%.

Files	Patch %	Lines
autogen/oai/client.py	26.66%	8 Missing and 3 partials ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #847       +/-   ##
===========================================
+ Coverage   26.65%   38.11%   +11.46%     
===========================================
  Files          28       28               
  Lines        3722     3736       +14     
  Branches      846      850        +4     
===========================================
+ Hits          992     1424      +432     
+ Misses       2658     2187      -471     
- Partials       72      125       +53

Flag	Coverage Δ
unittests	`38.06% <26.66%> (+11.46%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

afourney · 2023-12-03T05:24:05Z

@kevin666aa This is a broader issue, and is related to #178. It might be worth standardizing this approach.

More recently, I bet it's also related to #460

sonichi · 2023-12-04T05:14:08Z

For Azure endpoints, a better way is proposed in #820. Would it solve the azure openai problem?

yiranwu0 · 2023-12-04T05:34:44Z

For Azure endpoints, a better way is proposed in #820. Would it solve the azure openai problem?

By looking at the doc, I think switching it might solve the problem of azure endpoint.

At the same time, this PR will also address the problem that assures that creations will be made with intended model. At the same time, assure that the corresponding model is in config_list, which is a stricter force on the consistency of param model and model in config_list.

sonichi

The PR seems to be based on strong, unjustified assumptions.
I think the only necessary change is to skip mismatching model at line 241. Mismatching means: both self._config_list[i] and config contain "model" and they are not compatible.
Detecting compatibility is not trivial though.
If this is hard to do, a simple way is to raise an error when model is specified in both self._config_list[i] and config, as it causes confusion which model to use.

sonichi · 2023-12-04T06:45:22Z

autogen/oai/client.py

        openai_config, extra_kwargs = self._separate_openai_config(base_config)
        if type(config_list) is list and len(config_list) == 0:
            logger.warning("openai client was provided with an empty config_list, which may not be intended.")
+        if isinstance(config_list, list) and any((c.get("model") is None for c in config_list)):


This check shouldn't be here because one can provide the model in base_config and omit model in config_list.

sonichi · 2023-12-04T06:47:54Z

autogen/oai/client.py

+    @staticmethod
+    def _check_model_in_config_list(model, config_list=None):
+        if model is None or config_list is None:
+            return
+
+        if not any(c.get("model", None) == model for c in config_list):
+            raise ValueError(
+                f"model {model} is not found in the config_list. Please add the model with the corresponding api_key when creating the OpenAIWrapper."
+            )
+


This check doesn't make sense as config_list is used to update the base_config. It's valid to have model in the base_config and not specified in config_list.

Co-authored-by: Qingyun Wu <qingyun.wu@psu.edu>

gitguardian · 2024-07-20T21:22:07Z

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
12853598	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`eff19ac`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`06a0a5d`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`0524c77`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`d7ea410`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`e43a86c`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`841ed31`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`802f099`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`9a484d8`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`e973ac3`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`89650e7`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`e07b06b`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`abe4c41`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`7362fb9`	.github/workflows/dotnet-release.yml	View secret
12853599	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`abad9ff`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`c7bb588`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`b97b99d`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
12853602	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
11616921	Triggered	Generic High Entropy Secret	`a86d0fd`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`394561b`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`3eac646`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`f45b553`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`6563248`	notebook/agentchat_agentops.ipynb	View secret
12853598	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853598	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`954ca45`	.github/workflows/dotnet-build.yml	View secret
12853599	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853599	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`3b79cc6`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`11baa52`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	`3b79cc6`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`11baa52`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`11baa52`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`3b79cc6`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret

and 16 others.

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

assure created model in config_list

af8bdd2

yiranwu0 had a problem deploying to openai1 December 2, 2023 23:32 — with GitHub Actions Failure

yiranwu0 added 2 commits December 3, 2023 16:58

Merge remote-tracking branch 'origin/main' into fixcreate

15da7b5

update

ca97d16

yiranwu0 had a problem deploying to openai1 December 4, 2023 04:45 — with GitHub Actions Failure

yiranwu0 added 2 commits December 3, 2023 23:58

update

7910423

Merge remote-tracking branch 'origin/main' into fixcreate

ef4085a

yiranwu0 had a problem deploying to openai1 December 4, 2023 04:58 — with GitHub Actions Failure

yiranwu0 had a problem deploying to openai1 December 4, 2023 05:06 — with GitHub Actions Failure

yiranwu0 requested review from BeibinLi, afourney, gagb, qingyun-wu, sonichi and thinkall December 4, 2023 05:07

yiranwu0 marked this pull request as ready for review December 4, 2023 05:08

update

ca02914

yiranwu0 had a problem deploying to openai1 December 4, 2023 05:35 — with GitHub Actions Failure

sonichi reviewed Dec 4, 2023

View reviewed changes

yiranwu0 mentioned this pull request Dec 27, 2023

[Bug]: CompressibleAgents read llm_config["model"] directly rather than from the config_list -- this is almost always the wrong thing to do #1073

Closed

the-xentropy mentioned this pull request Feb 4, 2024

Autogen always uses ChatGPT 4, it ignores the config list. #460

Closed

gagb removed their request for review March 8, 2024 23:55

whiskyboy pushed a commit to whiskyboy/autogen that referenced this pull request Apr 17, 2024

Update .NET documentation links (microsoft#847)

eb417f7

Co-authored-by: Qingyun Wu <qingyun.wu@psu.edu>

thinkall closed this Aug 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Assure created model of OpenAIWrapper is in config_list #847

Assure created model of OpenAIWrapper is in config_list #847

Uh oh!

yiranwu0 commented Dec 2, 2023

Uh oh!

codecov-commenter commented Dec 2, 2023 •

edited

Loading

Uh oh!

afourney commented Dec 3, 2023

Uh oh!

sonichi commented Dec 4, 2023

Uh oh!

yiranwu0 commented Dec 4, 2023

Uh oh!

sonichi left a comment

Uh oh!

sonichi Dec 4, 2023

Uh oh!

sonichi Dec 4, 2023

Uh oh!

gitguardian bot commented Jul 20, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Assure created model of OpenAIWrapper is in config_list #847

Assure created model of OpenAIWrapper is in config_list #847

Uh oh!

Conversation

yiranwu0 commented Dec 2, 2023

Why are these changes needed?

Related issue number

Checks

Uh oh!

codecov-commenter commented Dec 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

afourney commented Dec 3, 2023

Uh oh!

sonichi commented Dec 4, 2023

Uh oh!

yiranwu0 commented Dec 4, 2023

Uh oh!

sonichi left a comment

Choose a reason for hiding this comment

Uh oh!

sonichi Dec 4, 2023

Choose a reason for hiding this comment

Uh oh!

sonichi Dec 4, 2023

Choose a reason for hiding this comment

Uh oh!

gitguardian bot commented Jul 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov-commenter commented Dec 2, 2023 •

edited

Loading

gitguardian bot commented Jul 20, 2024 •

edited

Loading