Skip to content

Commit

Permalink
Update pf component example (#2469)
Browse files Browse the repository at this point in the history
# Description

Please add an informative description that covers that changes made by
the pull request and link all relevant issues.

# All Promptflow Contribution checklist:
- [ ] **The pull request does not introduce [breaking changes].**
- [ ] **CHANGELOG is updated for new features, bug fixes or other
significant changes.**
- [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).**
- [ ] **Create an issue and link to the pull request to get dedicated
review from promptflow team. Learn more: [suggested
workflow](../CONTRIBUTING.md#suggested-workflow).**

## General Guidelines and Best Practices
- [ ] Title of the pull request is clear and informative.
- [ ] There are a small number of commits, each of which have an
informative message. This means that previously merged commits do not
appear in the history of the PR. For more information on cleaning up the
commits in your PR, [see this

page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).

### Testing Guidelines
- [ ] Pull request includes test coverage for the included changes.

# Description

Please add an informative description that covers that changes made by
the pull request and link all relevant issues.

# All Promptflow Contribution checklist:
- [ ] **The pull request does not introduce [breaking changes].**
- [ ] **CHANGELOG is updated for new features, bug fixes or other
significant changes.**
- [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).**
- [ ] **Create an issue and link to the pull request to get dedicated
review from promptflow team. Learn more: [suggested
workflow](../CONTRIBUTING.md#suggested-workflow).**

## General Guidelines and Best Practices
- [ ] Title of the pull request is clear and informative.
- [ ] There are a small number of commits, each of which have an
informative message. This means that previously merged commits do not
appear in the history of the PR. For more information on cleaning up the
commits in your PR, [see this
page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).

### Testing Guidelines
- [ ] Pull request includes test coverage for the included changes.

---------

Co-authored-by: Philip Gao <yigao@microsoft.com>
  • Loading branch information
alainli0928 and crazygao authored Apr 1, 2024
1 parent a700ec0 commit 3af352f
Show file tree
Hide file tree
Showing 14 changed files with 803 additions and 229 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,20 @@
# Any manual changes to this file may cause incorrect behavior.
# Any manual changes will be overwritten if the code is regenerated.

name: samples_flowinpipeline_pipeline
name: samples_runflowwithpipeline_pipeline
on:
schedule:
- cron: "28 19 * * *" # Every day starting at 3:28 BJT
- cron: "33 20 * * *" # Every day starting at 4:33 BJT
pull_request:
branches: [ main ]
paths: [ examples/tutorials/flow-in-pipeline/**, examples/flows/standard/web-classification/**, .github/workflows/samples_flowinpipeline_pipeline.yml, examples/requirements.txt, examples/connections/azure_openai.yml ]
paths: [ examples/tutorials/run-flow-with-pipeline/**, examples/flows/standard/web-classification/**, .github/workflows/samples_runflowwithpipeline_pipeline.yml, examples/requirements.txt, examples/connections/azure_openai.yml ]
workflow_dispatch:

env:
IS_IN_CI_PIPELINE: "true"

jobs:
samples_flowinpipeline_pipeline:
samples_runflowwithpipeline_pipeline:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
Expand All @@ -43,12 +43,12 @@ jobs:
- name: Create Aoai Connection
run: pf connection create -f ${{ github.workspace }}/examples/connections/azure_openai.yml --set api_key="${{ secrets.AOAI_API_KEY_TEST }}" api_base="${{ secrets.AOAI_API_ENDPOINT_TEST }}"
- name: Test Notebook
working-directory: examples/tutorials/flow-in-pipeline
working-directory: examples/tutorials/run-flow-with-pipeline
run: |
papermill -k python pipeline.ipynb pipeline.output.ipynb
- name: Upload artifact
if: ${{ always() }}
uses: actions/upload-artifact@v3
with:
name: artifact
path: examples/tutorials/flow-in-pipeline
path: examples/tutorials/run-flow-with-pipeline
3 changes: 2 additions & 1 deletion docs/cloud/azureai/use-flow-in-azure-ml-pipeline.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Use flow in Azure ML pipeline job
In practical scenarios, flows fulfill various functions. For example, consider an offline flow specifically designed to assess the relevance score for communication sessions between humans and agents. This flow is triggered nightly and processes a substantial amount of session data. In such a context, Parallel component and AzureML pipeline emerge as the optimal choices for handling large-scale, highly resilient, and efficient offline batch requirements.

After you have developed and tested the flow in [init and test a flow](../../how-to-guides/init-and-test-a-flow.md), this guide will help you learn how to use a flow as a parallel component in a pipeline job on AzureML, so that you can integrate the created flow with existing pipelines and process a large amount of data.
Once you’ve developed and thoroughly tested your flow using the guidelines in the [init and test a flow](../../how-to-guides/init-and-test-a-flow.md) section, this guide will walk you through utilizing your flow as a parallel component within an AzureML pipeline job.

:::{admonition} Pre-requirements
To enable this feature, customer need to:
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,8 +109,8 @@
------|--------|-------------
| [quickstart.ipynb](tutorials/get-started/quickstart.ipynb) | [![samples_getstarted_quickstart](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_quickstart.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_quickstart.yml) | A quickstart tutorial to run a flow and evaluate it. |
| [quickstart-azure.ipynb](tutorials/get-started/quickstart-azure.ipynb) | [![samples_getstarted_quickstartazure](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_quickstartazure.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_quickstartazure.yml) | A quickstart tutorial to run a flow in Azure AI and evaluate it. |
| [pipeline.ipynb](tutorials/flow-in-pipeline/pipeline.ipynb) | [![samples_flowinpipeline_pipeline](https://github.com/microsoft/promptflow/actions/workflows/samples_flowinpipeline_pipeline.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_flowinpipeline_pipeline.yml) | Create pipeline using components to run a distributed job with tensorflow |
| [flow-as-function.ipynb](tutorials/get-started/flow-as-function.ipynb) | [![samples_getstarted_flowasfunction](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_flowasfunction.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_flowasfunction.yml) | This guide will walk you through the main scenarios of executing flow as a function. |
| [pipeline.ipynb](tutorials/run-flow-with-pipeline/pipeline.ipynb) | [![samples_runflowwithpipeline_pipeline](https://github.com/microsoft/promptflow/actions/workflows/samples_runflowwithpipeline_pipeline.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_runflowwithpipeline_pipeline.yml) | Create pipeline using components to run a distributed job with tensorflow |
| [cloud-run-management.ipynb](tutorials/run-management/cloud-run-management.ipynb) | [![samples_runmanagement_cloudrunmanagement](https://github.com/microsoft/promptflow/actions/workflows/samples_runmanagement_cloudrunmanagement.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_runmanagement_cloudrunmanagement.yml) | Flow run management in Azure AI |
| [run-management.ipynb](tutorials/run-management/run-management.ipynb) | [![samples_runmanagement_runmanagement](https://github.com/microsoft/promptflow/actions/workflows/samples_runmanagement_runmanagement.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_runmanagement_runmanagement.yml) | Flow run management |
| [connection.ipynb](connections/connection.ipynb) | [![samples_connections_connection](https://github.com/microsoft/promptflow/actions/workflows/samples_connections_connection.yml/badge.svg?branch=main)](https://github.com/microsoft/promptflow/actions/workflows/samples_connections_connection.yml) | Manage various types of connections using sdk |
Expand Down
220 changes: 0 additions & 220 deletions examples/tutorials/flow-in-pipeline/pipeline.ipynb

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: my-env
channels:
- conda-forge
dependencies:
- python=3.9
- pip
- pip:
- mlflow
- azureml-core
- azure-ai-ml>=1.12.0
- azureml-dataset-runtime[pandas,fuse]
- azureml-telemetry
- mltable>=1.2.0
- pandas
- pillow
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Copyright (c) Microsoft. All rights reserved.
# Licensed under the MIT license.

import os
import pandas as pd
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--input_data_file", type=str)
parser.add_argument("--output_data_folder", type=str)

args, _ = parser.parse_known_args()

input_df = pd.read_json(args.input_data_file, lines=True)

# data preparation, e.g. data sampling, data cleaning, etc.
processed_data = input_df.sample(n=20, replace=True, random_state=1)

# export data into output folder
output_file_path = os.path.join(args.output_data_folder, "processed_data.csv")
processed_data.to_csv(output_file_path, index=False, header=True)
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json

type: command

name: data_prep
display_name: data preparation
version: 0.0.1

inputs:
input_data_file:
type: uri_file

outputs:
output_data_folder:
type: uri_folder

code: ./

environment:
image: mcr.microsoft.com/azureml/inference-base-2004:latest
conda_file: ./conda.yaml

command: >-
python data-prep.py
--input_data_file ${{inputs.input_data_file}}
--output_data_folder ${{outputs.output_data_folder}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: my-env
channels:
- conda-forge
dependencies:
- python=3.9
- pip
- pip:
- mlflow
- azureml-core
- azure-ai-ml>=1.12.0
- azureml-dataset-runtime[pandas,fuse]
- azureml-telemetry
- mltable>=1.2.0
- pandas
- pillow
Loading

0 comments on commit 3af352f

Please sign in to comment.