-
Notifications
You must be signed in to change notification settings - Fork 69
Add guide for adding projects to ZenML repository #239
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,224 @@ | ||
# Adding Projects to ZenML Projects Repository | ||
|
||
<!-- Note: This guide is primarily for ZenML internal development use --> | ||
|
||
This guide explains how to add your ZenML project to this repository and make it available through the ZenML Projects platform. | ||
|
||
## 📋 Requirements | ||
|
||
Every project added to this repository must include: | ||
|
||
1. **requirements.txt file** - Contains all Python dependencies | ||
2. **Dockerfile** (if special configuration is needed) | ||
3. **Backend configuration** - YAML file in the zenml-projects-backend repository | ||
|
||
## 🐳 Docker Configuration | ||
|
||
### When to Include a Dockerfile | ||
|
||
Include a custom Dockerfile **only if** your project requires: | ||
- Special system dependencies | ||
- Custom environment variables | ||
- Additional configuration beyond Python packages | ||
- Specific build steps | ||
|
||
### Dockerfile Structure | ||
|
||
If your project needs a custom Dockerfile, follow this exact structure: | ||
|
||
```dockerfile | ||
ARG ZENML_VERSION=latest | ||
ARG PROJECT_DIR_NAME | ||
ARG EXTENSION_VERSION=0.1.6 | ||
|
||
FROM zenmldocker/zenml-codespace:${ZENML_VERSION} | ||
|
||
# Set build arguments again for use in subsequent commands | ||
ARG PROJECT_DIR_NAME | ||
ARG EXTENSION_VERSION | ||
|
||
# Set the working directory for the project | ||
WORKDIR /home/coder/extensions/zenml-io.zenml-tutorial-${EXTENSION_VERSION}-universal/pipelines | ||
|
||
# Copy the specific project's requirements file | ||
COPY ./${PROJECT_DIR_NAME}/requirements.txt /tmp/requirements.txt | ||
|
||
# Install project-specific dependencies using uv for faster installation | ||
RUN uv pip install --system --no-cache -r /tmp/requirements.txt && \ | ||
rm /tmp/requirements.txt | ||
|
||
# Enable tutorial content for this specific project | ||
ENV ZENML_ENABLE_TUTORIAL=true | ||
|
||
# Default command can be overridden in docker run or docker-compose | ||
# CMD ["python", "run.py"] # Or your project's typical entrypoint | ||
``` | ||
|
||
### When No Dockerfile is Needed | ||
|
||
If your project only requires Python dependencies listed in `requirements.txt`, **do not include a Dockerfile**. The projects backend will automatically build your project using the generic Dockerfile available at: | ||
[https://github.com/zenml-io/zenml-projects-backend/blob/main/.docker/project.Dockerfile](https://github.com/zenml-io/zenml-projects-backend/blob/main/.docker/project.Dockerfile) | ||
|
||
## 🔧 Backend Integration | ||
|
||
### YAML Configuration File | ||
|
||
Once your project is added to this repository, you **must** create a corresponding YAML configuration file in the `zenml-projects-backend` repository. This file should include all the following attributes: | ||
|
||
```yaml | ||
project_id: your-project-id | ||
name: Your Project Name | ||
description: A comprehensive description of what your project does and its key features | ||
github: https://github.com/zenml-io/zenml-projects/tree/main/your-project-directory | ||
preview_image: https://github.com/your-org/your-project/raw/main/assets/preview.png | ||
main_image_link: https://public-flavor-logos.s3.eu-central-1.amazonaws.com/projects/[PROJECT_NUMBER].jpg | ||
details: | | ||
Detailed description of your project including: | ||
|
||
### What It Does | ||
|
||
Explain the main functionality and purpose of your project | ||
|
||
### Key Features | ||
|
||
- Feature 1: Description | ||
- Feature 2: Description | ||
- Feature 3: Description | ||
|
||
### How It Works | ||
|
||
Step-by-step explanation of how users can use your project | ||
|
||
### Architecture | ||
|
||
Brief overview of the technical architecture | ||
|
||
environment_variables: | ||
CUSTOM_VAR: 'value' | ||
ANOTHER_VAR: 'another_value' | ||
|
||
stack: | ||
orchestrator: default | ||
artifact_store: default | ||
# Add other stack components if needed | ||
|
||
tags: | ||
- domain (e.g., llmops, cv, mlops) | ||
- technology (e.g., pytorch, tensorflow, hugging-face) | ||
- use-case (e.g., classification, forecasting, nlp) | ||
|
||
tools_used: | ||
- zenml | ||
- pytorch | ||
- pandas | ||
- scikit-learn | ||
# Add all major tools/libraries used | ||
|
||
pipelines: | ||
- name: Main Pipeline Name | ||
description: Description of what this pipeline does | ||
- name: Secondary Pipeline Name | ||
description: Description of secondary pipeline (if applicable) | ||
|
||
architecture_diagram: assets/architecture-diagram.png # Optional | ||
|
||
codespace: | ||
enabled: true | ||
cpu: 4 | ||
memory: 8 | ||
``` | ||
|
||
### Required YAML Attributes | ||
|
||
| Attribute | Description | Required | | ||
|-----------|-------------|----------| | ||
| `project_id` | Unique identifier for your project | ✅ | | ||
| `name` | Display name of your project | ✅ | | ||
| `description` | Short description (1-2 sentences) | ✅ | | ||
| `github` | GitHub URL to your project directory | ✅ | | ||
| `preview_image` | URL to preview image | ✅ | | ||
| `main_image_link` | S3 URL to main project image | ✅ | | ||
| `details` | Detailed markdown description | ✅ | | ||
| `environment_variables` | Custom environment variables | ❌ | | ||
| `stack` | ZenML stack configuration | ✅ | | ||
| `tags` | Categorization tags | ✅ | | ||
| `tools_used` | List of technologies used | ✅ | | ||
| `pipelines` | List of pipelines in the project | ✅ | | ||
| `architecture_diagram` | Path to architecture diagram | ❌ | | ||
| `codespace` | Codespace configuration | ❌ | | ||
| `versions` | Version information | ❌ | | ||
|
||
## 🖼️ Image Requirements | ||
|
||
### Main Project Image | ||
|
||
The `main_image_link` should point to an image stored in the S3 bucket: | ||
``` | ||
https://public-flavor-logos.s3.eu-central-1.amazonaws.com/projects/[PROJECT_NUMBER].jpg | ||
``` | ||
safoinme marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
**Image Requirements:** | ||
- Format: JPG or PNG | ||
- Recommended size: 1200x630 pixels | ||
- High quality and representative of your project | ||
- Upload to the S3 bucket before adding to YAML | ||
|
||
### Preview Image | ||
|
||
The `preview_image` can be: | ||
- Stored in your project's assets directory | ||
- Hosted on GitHub (using raw.githubusercontent.com URLs) | ||
- External hosting (ensure it's reliable) | ||
|
||
## 🚀 Deployment Process | ||
|
||
### Step-by-Step Process | ||
|
||
1. **Add Project to This Repository** | ||
- Create your project directory | ||
- Include `requirements.txt` | ||
- Add `Dockerfile` if needed | ||
- Include comprehensive README.md | ||
- Test your project locally | ||
|
||
2. **Upload Main Image to S3** | ||
- Upload your main project image to the S3 bucket | ||
- Note the project number for the YAML file | ||
|
||
3. **Create Backend Configuration** | ||
- Fork the `zenml-projects-backend` repository | ||
- Create the YAML configuration file | ||
- Submit a pull request | ||
|
||
4. **Automatic Sync** | ||
- Once both PRs are merged to main | ||
- GitHub Actions will automatically sync the project list with Webflow | ||
- Your project will appear on the ZenML Projects website | ||
|
||
### Testing Your Setup | ||
|
||
Before submitting your PR, ensure: | ||
- [ ] Your project runs locally with ZenML | ||
- [ ] All dependencies are listed in `requirements.txt` | ||
- [ ] Dockerfile builds successfully (if included) | ||
- [ ] README.md is comprehensive and clear | ||
- [ ] All required YAML attributes are filled out | ||
- [ ] Images are accessible and high quality | ||
|
||
## 📝 Example Projects | ||
|
||
Refer to existing projects in this repository for examples: | ||
- [ZenML Support Agent](zenml-support-agent/) - LLMOps project | ||
- [Computer Vision End-to-End](end-to-end-computer-vision/) - CV project | ||
- [Credit Scorer](credit-scorer/) - Traditional ML project | ||
|
||
## 🆘 Getting Help | ||
|
||
If you need help adding your project: | ||
- Join our [Slack community](https://zenml.io/slack) | ||
- Open an issue in this repository | ||
- Check the [ZenML documentation](https://docs.zenml.io/) | ||
|
||
## 📜 License | ||
|
||
By contributing your project, you agree that it will be licensed under the Apache License Version 2.0, consistent with this repository's license. |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.