Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updates on registry endpoints related doc #541

Merged
merged 4 commits into from
Aug 7, 2022
Merged

Updates on registry endpoints related doc #541

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d13c30f
Updates on registry endpoints related doc
blrchen Aug 1, 2022
60dae88
Add back legacy registry configs and call out deprecate soon
blrchen Aug 1, 2022
25c05c6
Resolve code review comments
blrchen Aug 5, 2022
102b7a7
Enhance deprecation section
blrchen Aug 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions docs/how-to-guides/feathr-configuration-and-env.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,10 @@ Feathr will get the configurations in the following order:
| ONLINE_STORE__REDIS__PORT | Redis port number to access Redis cluster. | Required if using Redis as online store. |
| ONLINE_STORE__REDIS__SSL_ENABLED | Whether SSL is enabled to access Redis cluster. | Required if using Redis as online store. |
| REDIS_PASSWORD | Password for the Redis cluster. | Required if using Redis as online store. |
| FEATURE_REGISTRY__PURVIEW__PURVIEW_NAME | Configure the name of the purview endpoint. | Required if using Purview as the endpoint. |
| FEATURE_REGISTRY__PURVIEW__DELIMITER | See [here](#FEATURE_REGISTRY__PURVIEW__DELIMITER) for more details. | Required |
| FEATURE_REGISTRY__PURVIEW__TYPE_SYSTEM_INITIALIZATION | Controls whether the type system (think this as the "schema" for the registry) will be initialized or not. Usually this is only required to be set to `True` to initialize schema, and then you can set it to `False` to shorten the initialization time. | Required |
| FEATURE_REGISTRY__API_ENDPOINT | Specifies registry endpoint. | Required if using registry service. |
| FEATURE_REGISTRY__PURVIEW__PURVIEW_NAME | Configure the name of the purview endpoint. | Required if using Purview directly without registry service. Deprecate soon, see [here](#deprecation) for more details.|
| FEATURE_REGISTRY__PURVIEW__DELIMITER | See [here](#FEATURE_REGISTRY__PURVIEW__DELIMITER) for more details. | Required if using Purview directly without registry service. Deprecate soon, see [here](#deprecation) for more details.|
| FEATURE_REGISTRY__PURVIEW__TYPE_SYSTEM_INITIALIZATION | Controls whether the type system (think this as the "schema" for the registry) will be initialized or not. Usually this is only required to be set to `True` to initialize schema, and then you can set it to `False` to shorten the initialization time. | Required if using Purview directly without registry service. Deprecate soon, see [here](#deprecation) for more details.|

# Explanation for selected configurations

Expand Down Expand Up @@ -121,6 +122,11 @@ Another use case is to use `instance_pool_id`, where instead of creating the Spa

Other advanced settings includes `idempotency_token` to guarantee the idempotency of job run requests, etc.

## Deprecation

This section is about feature deprecation in Feathr. Deprecated features are still available but planned for removal. Please migrate to the supported features as early as possible.
- Connect to Purview directly without registry service is deprecated soon. Please migrate to use registry service and select Purview as registry provider.

## FEATURE_REGISTRY__PURVIEW__DELIMITER

Delimiter indicates that how the project name, feature names etc. are delimited. By default it will be '__'. this is for global reference (mainly for feature sharing). For example, when we setup a project called foo, and we have an anchor called 'taxi_driver' and the feature name is called 'f_daily_trips'. the feature will have a global unique name called 'foo__taxi_driver__f_daily_trips'
Expand Down
14 changes: 7 additions & 7 deletions docs/samples/fraud_detection_demo.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,12 +184,10 @@
"os.environ['spark_config__azure_synapse__dev_url'] = f'https://{synapse_workspace_url}.dev.azuresynapse.net'\n",
"os.environ['spark_config__azure_synapse__pool_name'] = 'spark31'\n",
"os.environ['spark_config__azure_synapse__workspace_dir'] = f'abfss://{adls_fs_name}@{adls_account}.dfs.core.windows.net/feathr_project'\n",
"os.environ['feature_registry__purview__purview_name'] = f'{purview_name}'\n",
"os.environ['online_store__redis__host'] = redis_host\n",
"os.environ['online_store__redis__port'] = redis_port\n",
"os.environ['online_store__redis__ssl_enabled'] = redis_ssl\n",
"os.environ['REDIS_PASSWORD']=redis_password\n",
"os.environ['feature_registry__purview__purview_name'] = f'{purview_name}'\n",
"feathr_output_path = f'abfss://{adls_fs_name}@{adls_account}.dfs.core.windows.net/feathr_output'"
]
},
Expand Down Expand Up @@ -998,9 +996,6 @@
"notebookOrigID": 1891349682974490,
"widgets": {}
},
"interpreter": {
"hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
},
"kernelspec": {
"display_name": "Python 3.10.4 64-bit",
"language": "python",
Expand All @@ -1018,8 +1013,13 @@
"pygments_lexer": "ipython3",
"version": "3.10.4"
},
"orig_nbformat": 4
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "6eea572ac5b43246b7c51fa33510c93fb6df4c34b515a6e4994c858f44841967"
}
}
},
"nbformat": 4,
"nbformat_minor": 0
}
}
2 changes: 0 additions & 2 deletions docs/samples/product_recommendation_demo.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,12 +184,10 @@
"os.environ['spark_config__azure_synapse__dev_url'] = f'https://{synapse_workspace_url}.dev.azuresynapse.net'\n",
"os.environ['spark_config__azure_synapse__pool_name'] = 'spark31'\n",
"os.environ['spark_config__azure_synapse__workspace_dir'] = f'abfss://{adls_fs_name}@{adls_account}.dfs.core.windows.net/feathr_project'\n",
"os.environ['feature_registry__purview__purview_name'] = f'{purview_name}'\n",
"os.environ['online_store__redis__host'] = redis_host\n",
"os.environ['online_store__redis__port'] = redis_port\n",
"os.environ['online_store__redis__ssl_enabled'] = redis_ssl\n",
"os.environ['REDIS_PASSWORD']=redis_password\n",
"os.environ['feature_registry__purview__purview_name'] = f'{purview_name}'\n",
"feathr_output_path = f'abfss://{adls_fs_name}@{adls_account}.dfs.core.windows.net/feathr_output'"
]
},
Expand Down
2 changes: 1 addition & 1 deletion docs/samples/product_recommendation_demo_advanced.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

7 changes: 1 addition & 6 deletions feathr_project/feathrcli/data/feathr_user_workspace/nyc_driver_demo.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,12 +162,10 @@
"os.environ['spark_config__azure_synapse__dev_url'] = f'https://{synapse_workspace_url}.dev.azuresynapse.net'\n",
"os.environ['spark_config__azure_synapse__pool_name'] = 'spark31'\n",
"os.environ['spark_config__azure_synapse__workspace_dir'] = f'abfss://{adls_fs_name}@{adls_account}.dfs.core.windows.net/feathr_project'\n",
"os.environ['feature_registry__purview__purview_name'] = f'{purview_name}'\n",
"os.environ['online_store__redis__host'] = redis_host\n",
"os.environ['online_store__redis__port'] = redis_port\n",
"os.environ['online_store__redis__ssl_enabled'] = redis_ssl\n",
"os.environ['REDIS_PASSWORD']=redis_password\n",
"os.environ['feature_registry__purview__purview_name'] = f'{purview_name}'\n",
"feathr_output_path = f'abfss://{adls_fs_name}@{adls_account}.dfs.core.windows.net/feathr_output'"
]
},
Expand Down Expand Up @@ -236,10 +234,7 @@
" port: 6380\n",
" ssl_enabled: True\n",
"feature_registry:\n",
" purview:\n",
" type_system_initialization: true\n",
" purview_name: 'feathrazuretest3-purview1'\n",
" delimiter: '__'\n",
" api_endpoint: \"https://feathr-sql-registry.azurewebsites.net/api/v1\"\n",
blrchen marked this conversation as resolved.
Show resolved Hide resolved
"\"\"\"\n",
"tmp = tempfile.NamedTemporaryFile(mode='w', delete=False)\n",
"with open(tmp.name, \"w\") as text_file:\n",
Expand Down