Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: ✨ Add validator #108

Merged
merged 6 commits into from
Nov 22, 2024
Merged

feat: ✨ Add validator #108

merged 6 commits into from
Nov 22, 2024

Conversation

slugb0t
Copy link
Member

@slugb0t slugb0t commented Nov 22, 2024

Summary by Sourcery

Add a new CWL Validator service with API endpoints for validating CWL and CITATION.cff files, and update deployment workflows to include this service.

New Features:

  • Introduce a new CWL Validator API with endpoints for validating CWL and CITATION.cff files.

Enhancements:

  • Add a new deployment configuration for the validator service using Kamal and Docker.

Deployment:

  • Update deployment workflows to include a new 'deploy-validator' job for staging and main environments.

Documentation:

  • Add a README file for the CWL Validator with setup and running instructions.

Copy link

Thank you for submitting this pull request! We appreciate your contribution to the project. Before we can merge it, we need to review the changes you've made to ensure they align with our code standards and meet the requirements of the project. We'll get back to you as soon as we can with feedback. Thanks again!

Copy link

sourcery-ai bot commented Nov 22, 2024

Reviewer's Guide by Sourcery

This PR adds a new validator service to the application, including GitHub workflow updates and API implementation. The validator service is implemented as a Flask application that provides endpoints for validating CWL and CITATION.cff files.

Sequence diagram for the CWL file validation process

sequenceDiagram
    participant User
    participant API
    participant Validator
    User->>API: POST /validate-cwl
    API->>Validator: Validate CWL file
    Validator-->>API: Validation result
    API-->>User: Response with validation result
Loading

Sequence diagram for the CITATION.cff file validation process

sequenceDiagram
    participant User
    participant API
    participant Validator
    User->>API: POST /validate-citation
    API->>Validator: Validate CITATION.cff file
    Validator-->>API: Validation result
    API-->>User: Response with validation result
Loading

Class diagram for the new validator API

classDiagram
    class Api {
        +title: str
        +description: str
        +doc: str
    }
    class Resource {
    }
    class HelloEverynyan {
        +get()
    }
    class Up {
        +get()
    }
    class ValidateCWL {
        +post()
    }
    class ValidateCitation {
        +post()
    }
    Api <|-- HelloEverynyan
    Api <|-- Up
    Api <|-- ValidateCWL
    Api <|-- ValidateCitation
    Resource <|-- HelloEverynyan
    Resource <|-- Up
    Resource <|-- ValidateCWL
    Resource <|-- ValidateCitation
    note for Api "This is the main API class for the validator service"
Loading

File-Level Changes

Change Details Files
Added a new validator service with Flask-based API endpoints
  • Created Flask application with REST API endpoints for CWL and CITATION.cff validation
  • Implemented health check endpoint for Kamal deployment
  • Added CORS configuration for specific domains
  • Set up Docker configuration for containerization
validator/app.py
validator/apis/__init__.py
validator/Dockerfile
validator/requirements.txt
validator/config.py
Updated GitHub workflow configurations for deployment
  • Added deployment steps for the validator service
  • Configured Ruby and Kamal setup
  • Added Azure Container Registry login steps
  • Implemented Docker Buildx setup for caching
.github/workflows/deploy-main.yml
.github/workflows/deploy-staging.yml
Updated bot username references in API endpoints
  • Changed bot username from 'codefair-app[bot]' to 'codefair-io[bot]'
ui/server/api/[owner]/[repo]/cwl-validation/rerun.post.ts
ui/server/api/[owner]/[repo]/license/custom_title.put.ts
ui/server/api/[owner]/[repo]/release/zenodo/index.post.ts
ui/server/api/[owner]/[repo]/rerun.post.ts
Added configuration files for development and deployment
  • Added Kamal deployment configuration
  • Set up linting configurations for Python
  • Added development environment configuration files
validator/config/deploy.yml
validator/.pylint.ini
validator/.pydocstyle.ini
validator/.flaskenv
validator/docker-compose.yaml

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time. You can also use
    this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Copy link

Thanks for making updates to your pull request. Our team will take a look and provide feedback as soon as possible. Please wait for any GitHub Actions to complete before editing your pull request. If you have any additional questions or concerns, feel free to let us know. Thank you for your contributions!

Copy link

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @slugb0t - I've reviewed your changes and they look great!

Here's what I looked at during the review
  • 🟡 General issues: 2 issues found
  • 🟢 Security: all looks good
  • 🟢 Testing: all looks good
  • 🟡 Complexity: 1 issue found
  • 🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment on lines +39 to +42
For developer mode:

```bash
python3 app.py --host $HOST --port $PORT
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: The commands shown for developer mode and production mode are identical, which seems incorrect

Consider clarifying the actual differences between running in developer mode versus production mode, as they currently show the same command.


You will need the following installed on your system:

- Python 3.8+
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: Python version requirement is inconsistent in the documentation

The README mentions Python 3.8+ as a requirement, but later specifies Python 3.10 in the Anaconda instructions. Please clarify the actual version requirements.

return ":)"

@api.route("/validate-cwl", endpoint="validate-cwl")
class ValidateCWL(Resource):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (complexity): Consider extracting common validation and error handling logic into a base class to reduce code duplication.

The validation classes contain significant duplication in their error handling and execution patterns. This can be simplified by extracting common logic into a base class while preserving specific behaviors:

class BaseValidator:
    def validate_file_path(self, file_path):
        if not file_path:
            return {"message": "Validation Error", "error": "file_path is required"}, 400
        if ".." in file_path or file_path.startswith("/"):
            return {"message": "Validation Error", "error": "Invalid file path"}, 400
        if re.search(r'[;&|]', file_path):
            return {"message": "Validation Error", "error": "Invalid characters in file path"}, 400
        return None

    def execute_validation(self, cmd, clean_output_fn=None):
        try:
            result = subprocess.run(cmd, capture_output=True, text=True, check=True)
            stdout = clean_output_fn(result.stdout) if clean_output_fn else result.stdout
            stderr = clean_output_fn(result.stderr) if clean_output_fn else result.stderr
            return {"message": "valid", "output": stdout, "error": stderr}, 200
        except subprocess.CalledProcessError as e:
            # Handle error output
            return {"message": "invalid", "output": e.stdout, "error": e.stderr}, 400
        except Exception as e:
            return {"message": "An unexpected error occurred", "error": str(e)}, 500

class ValidateCWL(Resource, BaseValidator):
    def post(self):
        file_path = api.payload.get('file_path')
        validation_error = self.validate_file_path(file_path)
        if validation_error:
            return validation_error

        cmd = ["cwltool", "--validate", file_path]
        return self.execute_validation(cmd, clean_output=self.clean_output)

    @staticmethod
    def clean_output(output):
        # Existing clean_output implementation

This refactoring:

  1. Eliminates duplicate validation code
  2. Centralizes subprocess execution and error handling
  3. Allows customization through method overrides
  4. Makes it easier to add new validators

@slugb0t slugb0t merged commit b06674c into main Nov 22, 2024
3 checks passed
Copy link

Thanks for closing this pull request! If you have any further questions, please feel free to open a new issue. We are always happy to help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants