Skip to content

Commit

Permalink
New tutorial on prompt tunning and evaluation (#524)
Browse files Browse the repository at this point in the history 
# Description

Add a new tutorial on prompt tunning and evaluation, belong with a chat flow and an eval flow. Make small modifies in promptflow/Readme.md file, lowercase 'Prompt Flow' to 'prompt flow' and delete the old version logo.

# All Promptflow Contribution checklist:
- [x] **The pull request does not introduce [breaking changes]**
- [x] **CHANGELOG is updated for new features, bug fixes or other
significant changes.**
- [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).**

## General Guidelines and Best Practices
- [x] Title of the pull request is clear and informative.
- [x] There are a small number of commits, each of which have an
informative message. This means that previously merged commits do not
appear in the history of the PR. For more information on cleaning up the
commits in your PR, [see this
page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).

### Testing Guidelines
- [x] Pull request includes test coverage for the included changes.

---------

Co-authored-by: ChenJieting <chenjieting1208@163.com>
Co-authored-by: Philip Gao <yigao@microsoft.com>
Co-authored-by: Jieting Chen <chenjieting@microsoft.com>
Co-authored-by: ChenJieting <40321821+ChenJieting@users.noreply.github.com>
  • Loading branch information
@jiaochenlu @ChenJieting @crazygao
5 people authored Nov 13, 2023
1 parent 3479c16 commit 86aed8f
Show file tree
Hide file tree
Showing 34 changed files with 1,374 additions and 48 deletions.
92 changes: 92 additions & 0 deletions .github/workflows/samples_flows_chat_chat_math_variant.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# This code is autogenerated.
# Code is generated by running custom script: python3 readme.py
# Any manual changes to this file may cause incorrect behavior.
# Any manual changes will be overwritten if the code is regenerated.

name: samples_flows_chat_chat_math_variant
on:
schedule:
- cron: "35 20 * * *" # Every day starting at 4:35 BJT
pull_request:
branches: [ main ]
paths: [ examples/flows/chat/chat-math-variant/**, examples/*requirements.txt, .github/workflows/samples_flows_chat_chat_math_variant.yml ]
workflow_dispatch:

env:
IS_IN_CI_PIPELINE: "true"

jobs:
samples_readme_ci:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Setup Python 3.9 environment
uses: actions/setup-python@v4
with:
python-version: "3.9"
- name: Generate config.json
run: echo '${{ secrets.TEST_WORKSPACE_CONFIG_JSON_CANARY }}' > ${{ github.workspace }}/examples/config.json
- name: Prepare requirements
working-directory: examples
run: |
if [[ -e requirements.txt ]]; then
python -m pip install --upgrade pip
pip install -r requirements.txt
fi
- name: Prepare dev requirements
working-directory: examples
run: |
python -m pip install --upgrade pip
pip install -r dev_requirements.txt
- name: Refine .env file
working-directory: examples/flows/chat/chat-math-variant
run: |
AOAI_API_KEY=${{ secrets.AOAI_API_KEY_TEST }}
AOAI_API_ENDPOINT=${{ secrets.AOAI_API_ENDPOINT_TEST }}
AOAI_API_ENDPOINT=$(echo ${AOAI_API_ENDPOINT//\//\\/})
if [[ -e .env.example ]]; then
echo "env replacement"
sed -i -e "s/<your_AOAI_key>/$AOAI_API_KEY/g" -e "s/<your_AOAI_endpoint>/$AOAI_API_ENDPOINT/g" .env.example
mv .env.example .env
fi
- name: Create run.yml
working-directory: examples/flows/chat/chat-math-variant
run: |
gpt_base=${{ secrets.AOAI_API_ENDPOINT_TEST }}
gpt_base=$(echo ${gpt_base//\//\\/})
if [[ -e run.yml ]]; then
sed -i -e "s/\${azure_open_ai_connection.api_key}/${{ secrets.AOAI_API_KEY_TEST }}/g" -e "s/\${azure_open_ai_connection.api_base}/$gpt_base/g" run.yml
fi
- name: Azure Login
uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
- name: Extract Steps examples/flows/chat/chat-math-variant/README.md
working-directory: ${{ github.workspace }}
run: |
python scripts/readme/extract_steps_from_readme.py -f examples/flows/chat/chat-math-variant/README.md -o examples/flows/chat/chat-math-variant
- name: Cat script
working-directory: examples/flows/chat/chat-math-variant
run: |
cat bash_script.sh
- name: Run scripts
working-directory: examples/flows/chat/chat-math-variant
run: |
export aoai_api_key=${{secrets.AOAI_API_KEY_TEST }}
export aoai_api_endpoint=${{ secrets.AOAI_API_ENDPOINT_TEST }}
export test_workspace_sub_id=${{ secrets.TEST_WORKSPACE_SUB_ID }}
export test_workspace_rg=${{ secrets.TEST_WORKSPACE_RG }}
export test_workspace_name=${{ secrets.TEST_WORKSPACE_NAME_CANARY }}
bash bash_script.sh
- name: Pip List for Debug
if : ${{ always() }}
working-directory: examples/flows/chat/chat-math-variant
run: |
pip list
- name: Upload artifact
if: ${{ always() }}
uses: actions/upload-artifact@v3
with:
name: artifact
path: examples/flows/chat/chat-math-variant/bash_script.sh
92 changes: 92 additions & 0 deletions .github/workflows/samples_flows_evaluation_eval_chat_math.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# This code is autogenerated.
# Code is generated by running custom script: python3 readme.py
# Any manual changes to this file may cause incorrect behavior.
# Any manual changes will be overwritten if the code is regenerated.

name: samples_flows_evaluation_eval_chat_math
on:
schedule:
- cron: "52 19 * * *" # Every day starting at 3:52 BJT
pull_request:
branches: [ main ]
paths: [ examples/flows/evaluation/eval-chat-math/**, examples/*requirements.txt, .github/workflows/samples_flows_evaluation_eval_chat_math.yml ]
workflow_dispatch:

env:
IS_IN_CI_PIPELINE: "true"

jobs:
samples_readme_ci:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Setup Python 3.9 environment
uses: actions/setup-python@v4
with:
python-version: "3.9"
- name: Generate config.json
run: echo '${{ secrets.TEST_WORKSPACE_CONFIG_JSON_CANARY }}' > ${{ github.workspace }}/examples/config.json
- name: Prepare requirements
working-directory: examples
run: |
if [[ -e requirements.txt ]]; then
python -m pip install --upgrade pip
pip install -r requirements.txt
fi
- name: Prepare dev requirements
working-directory: examples
run: |
python -m pip install --upgrade pip
pip install -r dev_requirements.txt
- name: Refine .env file
working-directory: examples/flows/evaluation/eval-chat-math
run: |
AOAI_API_KEY=${{ secrets.AOAI_API_KEY_TEST }}
AOAI_API_ENDPOINT=${{ secrets.AOAI_API_ENDPOINT_TEST }}
AOAI_API_ENDPOINT=$(echo ${AOAI_API_ENDPOINT//\//\\/})
if [[ -e .env.example ]]; then
echo "env replacement"
sed -i -e "s/<your_AOAI_key>/$AOAI_API_KEY/g" -e "s/<your_AOAI_endpoint>/$AOAI_API_ENDPOINT/g" .env.example
mv .env.example .env
fi
- name: Create run.yml
working-directory: examples/flows/evaluation/eval-chat-math
run: |
gpt_base=${{ secrets.AOAI_API_ENDPOINT_TEST }}
gpt_base=$(echo ${gpt_base//\//\\/})
if [[ -e run.yml ]]; then
sed -i -e "s/\${azure_open_ai_connection.api_key}/${{ secrets.AOAI_API_KEY_TEST }}/g" -e "s/\${azure_open_ai_connection.api_base}/$gpt_base/g" run.yml
fi
- name: Azure Login
uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
- name: Extract Steps examples/flows/evaluation/eval-chat-math/README.md
working-directory: ${{ github.workspace }}
run: |
python scripts/readme/extract_steps_from_readme.py -f examples/flows/evaluation/eval-chat-math/README.md -o examples/flows/evaluation/eval-chat-math
- name: Cat script
working-directory: examples/flows/evaluation/eval-chat-math
run: |
cat bash_script.sh
- name: Run scripts
working-directory: examples/flows/evaluation/eval-chat-math
run: |
export aoai_api_key=${{secrets.AOAI_API_KEY_TEST }}
export aoai_api_endpoint=${{ secrets.AOAI_API_ENDPOINT_TEST }}
export test_workspace_sub_id=${{ secrets.TEST_WORKSPACE_SUB_ID }}
export test_workspace_rg=${{ secrets.TEST_WORKSPACE_RG }}
export test_workspace_name=${{ secrets.TEST_WORKSPACE_NAME_CANARY }}
bash bash_script.sh
- name: Pip List for Debug
if : ${{ always() }}
working-directory: examples/flows/evaluation/eval-chat-math
run: |
pip list
- name: Upload artifact
if: ${{ always() }}
uses: actions/upload-artifact@v3
with:
name: artifact
path: examples/flows/evaluation/eval-chat-math/bash_script.sh
52 changes: 52 additions & 0 deletions .github/workflows/samples_getstarted_flowasfunction.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# This code is autogenerated.
# Code is generated by running custom script: python3 readme.py
# Any manual changes to this file may cause incorrect behavior.
# Any manual changes will be overwritten if the code is regenerated.

name: samples_getstarted_flowasfunction
on:
schedule:
- cron: "17 21 * * *" # Every day starting at 5:17 BJT
pull_request:
branches: [ main ]
paths: [ examples/**, .github/workflows/samples_getstarted_flowasfunction.yml ]
workflow_dispatch:

env:
IS_IN_CI_PIPELINE: "true"

jobs:
samples_notebook_ci:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Generate config.json
run: echo '${{ secrets.TEST_WORKSPACE_CONFIG_JSON_CANARY }}' > ${{ github.workspace }}/examples/config.json
- name: Azure Login
uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
- name: Setup Python 3.9 environment
uses: actions/setup-python@v4
with:
python-version: "3.9"
- name: Prepare requirements
run: |
python -m pip install --upgrade pip
pip install -r ${{ github.workspace }}/examples/requirements.txt
pip install -r ${{ github.workspace }}/examples/dev_requirements.txt
- name: Create Aoai Connection
run: pf connection create -f ${{ github.workspace }}/examples/connections/azure_openai.yml --set api_key="${{ secrets.AOAI_API_KEY_TEST }}" api_base="${{ secrets.AOAI_API_ENDPOINT_TEST }}"
- name: Create new Aoai Connection
run: pf connection create -f ${{ github.workspace }}/examples/connections/azure_openai.yml --set api_key="${{ secrets.AOAI_API_KEY_TEST }}" api_base="${{ secrets.AOAI_API_ENDPOINT_TEST }}" name=new_ai_connection
- name: Test Notebook
working-directory: examples/tutorials/get-started
run: |
papermill -k python flow-as-function.ipynb flow-as-function.output.ipynb
- name: Upload artifact
if: ${{ always() }}
uses: actions/upload-artifact@v3
with:
name: artifact
path: examples/tutorials/get-started
92 changes: 92 additions & 0 deletions ...orkflows/samples_tutorials_flow_fine_tuning_evaluation_promptflow_quality_improvement.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# This code is autogenerated.
# Code is generated by running custom script: python3 readme.py
# Any manual changes to this file may cause incorrect behavior.
# Any manual changes will be overwritten if the code is regenerated.

name: samples_tutorials_flow_fine_tuning_evaluation_promptflow_quality_improvement
on:
schedule:
- cron: "15 19 * * *" # Every day starting at 3:15 BJT
pull_request:
branches: [ main ]
paths: [ examples/**, .github/workflows/samples_tutorials_flow_fine_tuning_evaluation_promptflow_quality_improvement.yml ]
workflow_dispatch:

env:
IS_IN_CI_PIPELINE: "true"

jobs:
samples_readme_ci:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Setup Python 3.9 environment
uses: actions/setup-python@v4
with:
python-version: "3.9"
- name: Generate config.json
run: echo '${{ secrets.TEST_WORKSPACE_CONFIG_JSON_CANARY }}' > ${{ github.workspace }}/examples/config.json
- name: Prepare requirements
working-directory: examples
run: |
if [[ -e requirements.txt ]]; then
python -m pip install --upgrade pip
pip install -r requirements.txt
fi
- name: Prepare dev requirements
working-directory: examples
run: |
python -m pip install --upgrade pip
pip install -r dev_requirements.txt
- name: Refine .env file
working-directory: examples/tutorials/flow-fine-tuning-evaluation
run: |
AOAI_API_KEY=${{ secrets.AOAI_API_KEY_TEST }}
AOAI_API_ENDPOINT=${{ secrets.AOAI_API_ENDPOINT_TEST }}
AOAI_API_ENDPOINT=$(echo ${AOAI_API_ENDPOINT//\//\\/})
if [[ -e .env.example ]]; then
echo "env replacement"
sed -i -e "s/<your_AOAI_key>/$AOAI_API_KEY/g" -e "s/<your_AOAI_endpoint>/$AOAI_API_ENDPOINT/g" .env.example
mv .env.example .env
fi
- name: Create run.yml
working-directory: examples/tutorials/flow-fine-tuning-evaluation
run: |
gpt_base=${{ secrets.AOAI_API_ENDPOINT_TEST }}
gpt_base=$(echo ${gpt_base//\//\\/})
if [[ -e run.yml ]]; then
sed -i -e "s/\${azure_open_ai_connection.api_key}/${{ secrets.AOAI_API_KEY_TEST }}/g" -e "s/\${azure_open_ai_connection.api_base}/$gpt_base/g" run.yml
fi
- name: Azure Login
uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
- name: Extract Steps examples/tutorials/flow-fine-tuning-evaluation/promptflow-quality-improvement.md
working-directory: ${{ github.workspace }}
run: |
python scripts/readme/extract_steps_from_readme.py -f examples/tutorials/flow-fine-tuning-evaluation/promptflow-quality-improvement.md -o examples/tutorials/flow-fine-tuning-evaluation
- name: Cat script
working-directory: examples/tutorials/flow-fine-tuning-evaluation
run: |
cat bash_script.sh
- name: Run scripts
working-directory: examples/tutorials/flow-fine-tuning-evaluation
run: |
export aoai_api_key=${{secrets.AOAI_API_KEY_TEST }}
export aoai_api_endpoint=${{ secrets.AOAI_API_ENDPOINT_TEST }}
export test_workspace_sub_id=${{ secrets.TEST_WORKSPACE_SUB_ID }}
export test_workspace_rg=${{ secrets.TEST_WORKSPACE_RG }}
export test_workspace_name=${{ secrets.TEST_WORKSPACE_NAME_CANARY }}
bash bash_script.sh
- name: Pip List for Debug
if : ${{ always() }}
working-directory: examples/tutorials/flow-fine-tuning-evaluation
run: |
pip list
- name: Upload artifact
if: ${{ always() }}
uses: actions/upload-artifact@v3
with:
name: artifact
path: examples/tutorials/flow-fine-tuning-evaluation/bash_script.sh
8 changes: 6 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -83,15 +83,19 @@ Interact with your chatbot by running: (press `Ctrl + C` to end the session)
pf flow test --flow ./my_chatbot --interactive
```

**Core value: ensuring "High Quality” from prototype to production**

Explore our [**15-minute tutorial**](examples/tutorials/flow-fine-tuning-evaluation/promptflow-quality-improvement.md) that guides you through prompt tuning ➡ batch testing ➡ evaluation, all designed to ensure high quality ready for production.

Next Step! Continue with the **Tutorial** 👇 section to delve deeper into prompt flow.

## Tutorial 🏃‍♂️

Prompt Flow is a tool designed to **build high quality LLM apps**, the development process in prompt flow follows these steps: develop a flow, improve the flow quality, deploy the flow to production.
Prompt flow is a tool designed to **build high quality LLM apps**, the development process in prompt flow follows these steps: develop a flow, improve the flow quality, deploy the flow to production.

### Develop your own LLM apps

#### VS Code Extension<img src="examples/tutorials/quick-start/media/logo_pf.png" alt="logo" width="25"/>
#### VS Code Extension

We also offer a VS Code extension (a flow designer) for an interactive flow development experience with UI.

Expand Down
Loading

0 comments on commit 86aed8f

Please sign in to comment.