[FIX] Allow AAD Auth for AzureContentFilterScorer

[riedgar-ms]

Description

The AzureContentFilterScorer required an API key for use, which does not conform to all security policies.

Tests and Documentation

Corresponding notebook and .py file updated. However, the Python file can't be run since it contains a bare await. And 'fixing' this by switching to asyncio.run() then means the notebook doesn't work, since Jupyter must be creating an event loop.

[nina-msft]

Hey Richard! Thanks for this PR. We don't currently have unit testing around use_aad_auth but you will see that some notebooks will have examples that expose the parameter (such as

PyRIT/doc/code/targets/2_gpt_4o_target.py

Line 49 in 11a64f7

target = AzureOpenAIGPT4OChatTarget(use_aad_auth=False)

).

I'd recommend adding an example in this doc: https://github.com/Azure/PyRIT/blob/main/doc/code/scoring/2_azure_content_safety_scorers.py#L33

...where you show how to use AzureContentFilterScorer with the use_aad_auth option. You can set this to False when actually submitting it, but test locally with it set to True to make sure its functioning as expected.

[riedgar-ms]

Just trying to update the notebook, I'm getting an error when running:

Error fetching data from table PromptMemoryEntries: (duckdb.duckdb.BinderException) Binder Error: Table "PromptMemoryEntries" does not have a column named "original_prompt_id"
LINE 1: ...MemoryEntries_converted_value_sha256", "PromptMemoryEntries".original_prompt_i...
                                                  ^
[SQL: SELECT "PromptMemoryEntries".id AS "PromptMemoryEntries_id", "PromptMemoryEntries".role AS "PromptMemoryEntries_role", "PromptMemoryEntries".conversation_id AS "PromptMemoryEntries_conversation_id", "PromptMemoryEntries".sequence AS "PromptMemoryEntries_sequence", "PromptMemoryEntries".timestamp AS "PromptMemoryEntries_timestamp", "PromptMemoryEntries".labels AS "PromptMemoryEntries_labels", "PromptMemoryEntries".prompt_metadata AS "PromptMemoryEntries_prompt_metadata", "PromptMemoryEntries".converter_identifiers AS "PromptMemoryEntries_converter_identifiers", "PromptMemoryEntries".prompt_target_identifier AS "PromptMemoryEntries_prompt_target_identifier", "PromptMemoryEntries".orchestrator_identifier AS "PromptMemoryEntries_orchestrator_identifier", "PromptMemoryEntries".response_error AS "PromptMemoryEntries_response_error", "PromptMemoryEntries".original_value_data_type AS "PromptMemoryEntries_original_value_data_type", "PromptMemoryEntries".original_value AS "PromptMemoryEntries_original_value", "PromptMemoryEntries".original_value_sha256 AS "PromptMemoryEntries_original_value_sha256", "PromptMemoryEntries".converted_value_data_type AS "PromptMemoryEntries_converted_value_data_type", "PromptMemoryEntries".converted_value AS "PromptMemoryEntries_converted_value", "PromptMemoryEntries".converted_value_sha256 AS "PromptMemoryEntries_converted_value_sha256", "PromptMemoryEntries".original_prompt_id AS "PromptMemoryEntries_original_prompt_id" 
FROM "PromptMemoryEntries" 
WHERE "PromptMemoryEntries".id IN ($1::UUID)]
[parameters: ('3fdb7e3f-2158-4dff-b497-a241a1ae9ce7',)]
(Background on this error at: https://sqlalche.me/e/20/f405)
Traceback (most recent call last):
  File "C:\Users\riedgar\AppData\Local\miniconda3\envs\pyrit-310\lib\site-packages\sqlalchemy\engine\base.py", line 1970, in _exec_single_context
    self.dialect.do_execute(
  File "C:\Users\riedgar\AppData\Local\miniconda3\envs\pyrit-310\lib\site-packages\sqlalchemy\engine\default.py", line 924, in do_execute
    cursor.execute(statement, parameters)
  File "C:\Users\riedgar\AppData\Local\miniconda3\envs\pyrit-310\lib\site-packages\duckdb_engine\__init__.py", line 162, in execute
    self.__c.execute(statement, parameters)
duckdb.duckdb.BinderException: Binder Error: Table "PromptMemoryEntries" does not have a column named "original_prompt_id"
LINE 1: ...MemoryEntries_converted_value_sha256", "PromptMemoryEntries".original_prompt_i...
                                                  ^

I don't think that's my fault.... at least due to the changes I've made.

[romanlutz]

Just trying to update the notebook, I'm getting an error when running:

Error fetching data from table PromptMemoryEntries: (duckdb.duckdb.BinderException) Binder Error: Table "PromptMemoryEntries" does not have a column named "original_prompt_id"

LINE 1: ...MemoryEntries_converted_value_sha256", "PromptMemoryEntries".original_prompt_i...

                                                  ^

[SQL: SELECT "PromptMemoryEntries".id AS "PromptMemoryEntries_id", "PromptMemoryEntries".role AS "PromptMemoryEntries_role", "PromptMemoryEntries".conversation_id AS "PromptMemoryEntries_conversation_id", "PromptMemoryEntries".sequence AS "PromptMemoryEntries_sequence", "PromptMemoryEntries".timestamp AS "PromptMemoryEntries_timestamp", "PromptMemoryEntries".labels AS "PromptMemoryEntries_labels", "PromptMemoryEntries".prompt_metadata AS "PromptMemoryEntries_prompt_metadata", "PromptMemoryEntries".converter_identifiers AS "PromptMemoryEntries_converter_identifiers", "PromptMemoryEntries".prompt_target_identifier AS "PromptMemoryEntries_prompt_target_identifier", "PromptMemoryEntries".orchestrator_identifier AS "PromptMemoryEntries_orchestrator_identifier", "PromptMemoryEntries".response_error AS "PromptMemoryEntries_response_error", "PromptMemoryEntries".original_value_data_type AS "PromptMemoryEntries_original_value_data_type", "PromptMemoryEntries".original_value AS "PromptMemoryEntries_original_value", "PromptMemoryEntries".original_value_sha256 AS "PromptMemoryEntries_original_value_sha256", "PromptMemoryEntries".converted_value_data_type AS "PromptMemoryEntries_converted_value_data_type", "PromptMemoryEntries".converted_value AS "PromptMemoryEntries_converted_value", "PromptMemoryEntries".converted_value_sha256 AS "PromptMemoryEntries_converted_value_sha256", "PromptMemoryEntries".original_prompt_id AS "PromptMemoryEntries_original_prompt_id" 

FROM "PromptMemoryEntries" 

WHERE "PromptMemoryEntries".id IN ($1::UUID)]

[parameters: ('3fdb7e3f-2158-4dff-b497-a241a1ae9ce7',)]

(Background on this error at: https://sqlalche.me/e/20/f405)

Traceback (most recent call last):

  File "C:\Users\riedgar\AppData\Local\miniconda3\envs\pyrit-310\lib\site-packages\sqlalchemy\engine\base.py", line 1970, in _exec_single_context

    self.dialect.do_execute(

  File "C:\Users\riedgar\AppData\Local\miniconda3\envs\pyrit-310\lib\site-packages\sqlalchemy\engine\default.py", line 924, in do_execute

    cursor.execute(statement, parameters)

  File "C:\Users\riedgar\AppData\Local\miniconda3\envs\pyrit-310\lib\site-packages\duckdb_engine\__init__.py", line 162, in execute

    self.__c.execute(statement, parameters)

duckdb.duckdb.BinderException: Binder Error: Table "PromptMemoryEntries" does not have a column named "original_prompt_id"

LINE 1: ...MemoryEntries_converted_value_sha256", "PromptMemoryEntries".original_prompt_i...

                                                  ^

I don't think that's my fault.... at least due to the changes I've made.

Did you try deleting the local DuckDB in the results folder? Might be related to recent schema changes...