feat: ✨ Codefair v3.2.0

[slugb0t]

Summary by Sourcery

Introduce metadata and license validation capabilities, enhance the dashboard UI, and integrate the validator service with the Codefair app. Add support for validating codemeta.json and CITATION.cff files using a new schema and validation endpoints. Improve metadata handling and set up deployment configurations for the validator service using Kamal.

New Features:

• Introduce metadata and license validation capabilities, allowing for re-validation of metadata files and licenses through the UI.
• Add support for validating codemeta.json and CITATION.cff files using a new schema and validation endpoints.
• Enhance the dashboard UI to display metadata validation results and provide options for re-validation.
• Implement a new schema for codemeta.json validation and validation logic for CITATION.cff files.

Bug Fixes:

• Fix issues related to filtering what is considered a CWL file in the repository.

Enhancements:

• Improve metadata handling by introducing functions to convert and validate metadata from codemeta.json and CITATION.cff files.
• Integrate the validator service with the Codefair app, moving it to the current repository.
• Add CI workflows for deploying the validator service using Kamal and Docker.
• Set up deployment configurations for the validator service using Kamal, including Docker and Azure Container Registry integrations.

Build:

• Add a new schema for codemeta.json validation and implement validation logic for CITATION.cff files.

CI:

• Add CI workflows for deploying the validator service using Kamal and Docker.

Deployment:

• Set up deployment configurations for the validator service using Kamal, including Docker and Azure Container Registry integrations.

Documentation:

• Update the CHANGELOG.md to document the new features and fixes in version 3.2.0.

[fairdataihub-bot]

Thank you for submitting this pull request! We appreciate your contribution to the project. Before we can merge it, we need to review the changes you've made to ensure they align with our code standards and meet the requirements of the project. We'll get back to you as soon as we can with feedback. Thanks again!

[sourcery-ai]

Reviewer's Guide by Sourcery

This PR implements comprehensive metadata validation capabilities, including new validation endpoints for codemeta.json and CITATION.cff files, enhanced UI features for displaying validation results, and improved error handling and logging functionality.

Sequence diagram for metadata validation process

sequenceDiagram
    participant User
    participant UI
    participant Server
    participant MetadataValidator
    participant Database
    User->>UI: Request metadata validation
    UI->>Server: Send validation request
    Server->>MetadataValidator: Validate metadata
    MetadataValidator->>Database: Fetch existing metadata
    Database-->>MetadataValidator: Return metadata
    MetadataValidator->>Database: Update validation status
    Database-->>MetadataValidator: Confirmation
    MetadataValidator-->>Server: Validation result
    Server-->>UI: Display validation result
    UI-->>User: Show validation status

Loading

Class diagram for metadata validation functions

classDiagram
    class MetadataValidator {
        +validateMetadata(metadataInfo, fileType, repository)
        +applyDbMetadata(existingMetadataEntry, metadata)
        +applyCodemetaMetadata(codemeta, metadata, repository)
        +applyCitationMetadata(citation, metadata, repository)
    }
    class CodemetaConverter {
        +convertCodemetaForDB(codemetaContent, repository)
    }
    class CitationConverter {
        +convertCitationForDB(citationContent, repository)
    }
    class Database {
        +update(metadata)
        +findUnique(query)
    }
    MetadataValidator --> CodemetaConverter
    MetadataValidator --> CitationConverter
    MetadataValidator --> Database

Loading

File-Level Changes

Change	Details	Files
Added metadata validation functionality with new validation endpoints	Implemented validation endpoints for codemeta.json and CITATION.cff files Added schema-based validation for codemeta.json files Created validation result display in the UI Added re-validation capabilities for metadata files	validator/apis/__init__.py validator/codemeta-schema.json ui/pages/dashboard/[owner]/[repo]/view/metadata-validation.vue
Enhanced metadata handling and conversion functionality	Added functions to convert and validate metadata from different file formats Implemented metadata merging logic for multiple sources Added error handling for metadata conversion process Created validation status tracking in database	bot/metadata/index.js
Improved logging system with new Logwatch utility	Created Logwatch class for centralized logging Added support for different log levels and formats Implemented async logging to external endpoint Added JSON logging capabilities	bot/utils/logwatch.js
Updated UI components for metadata validation	Added validation results display page Implemented re-validation buttons and workflows Enhanced dashboard with validation status indicators Added tooltips for validation status information	ui/pages/dashboard/[owner]/[repo]/index.vue ui/server/api/[owner]/[repo]/dashboard.get.ts
Database schema updates for validation support	Added validation message columns to CodeMetadata table Created migration scripts for new columns Updated database interfaces for validation data	bot/prisma/migrations/20241114193552_metadata_validation_messages/migration.sql ui/prisma/migrations/20241114193552_metadata_validation_messages/migration.sql

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[fairdataihub-bot]

Thanks for making updates to your pull request. Our team will take a look and provide feedback as soon as possible. Please wait for any GitHub Actions to complete before editing your pull request. If you have any additional questions or concerns, feel free to let us know. Thank you for your contributions!

[what-the-diff]

PR Summary

• GitHub Actions Workflows

  • This PR introduces a new job, migrate-db, to our deployment workflows. This task ensures our databases get updated and modified correctly whenever there are changes in the main or staging phases of our project.

• Changelog Updates

  • We've updated the CHANGELOG.md to note the enhancements brought by version v3.2.0, including improvements to metadata validation, UI, CI workflows, and service integrations.

• Webhook API Improvements

  • We've simplified and cleaned up the code that initializes the private keys used in our bot’s webhook functionality, making it more efficient.

• CWL File Handling Enhancements

  • We enhanced our CWL file tracking and management, making it easier to keep track of CWL file states. It also includes improved validation methods to fetch existing repository data.

• License Validation Functionality

  • The updated validateLicense function can now be used across the application, improving the versatility and use of this key function.

• Prisma Schema Updates

  • We've added new fields to the CodeMetadata model to store validation messages. Other updates in various models are aimed at consistently structuring and clarifying their use.

• New Prettier Configuration

  • A new code formatting configuration is introduced to maintain consistent code structure across the project.

• New SQL Migration Script

  • We’ve created a new database migration script, adding new columns to the CodeMetadata table to support new features.

• Package Configuration Updates

  • Our project setup file now includes a command for formatting code and introduces a new dependency for better code formatting.

---

• Added a new utility class Logwatch

  • This tool will help us log various operations on different levels and support us in logging data in the JSON format.

• Improvements to Rendering Logic

  • We’ve revised the logic of how we present repository templates and pull request handling for licenses and metadata validation.

• Updated Dockerfile

  • Minor update to command lines related to migration tasks.

• Expanded Type Declarations

  • New component declarations help to use additional 'naive-ui' features.

• Created Metadata Validation View

  • This new feature in the Dashboard allows users to view metadata validation results. It includes collapsible sections to view details for CITATION.cff and codemeta.json.

• New SQL Migration File

  • The database migration script provides additional storage for validation messages.

---

• Updated Prisma Schema

  • This PR improves the structure and clarity of our models, along with extending the User model and LicenseRequest.

• Enhanced Metadata Validation

  • The PR introduces a new API endpoint for metadata-validation that will help in validating the code metadata and guiding authors to define their roles in the code-metadata API.

• License Custom Title API

  • We've simplified the code by removing unnecessary type imports.

• Rerun API Enhancements

  • There’s improved validation for incoming requests. This enhancement makes it easier to handle multiple rerun types.

• Validator Improvements

  • We've updated and expanded the validation functionality, ensuring code metadata meets the correct criteria and validations.

• General Code Cleanups

  • General improvements to error handling, messaging, and API response structuring have been made across the project.

---

• Added a new JSON schema file for codemeta-schema.json

  • This new schema will help ensure our codemeta.json files meet rigorous standards.

• Updated requirements.txt

  • This file, specifying our project dependencies, now includes jsonschema as a required library.

[sourcery-ai]

Hey @slugb0t - I've reviewed your changes - here's some feedback:

Overall Comments:

• Consider standardizing the error handling pattern across the codebase - in some places errors are logged and rethrown while in others they are just logged. A consistent approach would make the code more maintainable.

Here's what I looked at during the review

• 🟢 General issues: all looks good
• 🟡 Security: 2 issues found
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

🚨 issue (security): Use absolute path resolution for file access

Using relative paths for file access can be unsafe. Consider using path.join() with __dirname or an absolute path to ensure reliable and secure file resolution.

[sourcery-ai]

🚨 issue (security): Avoid hardcoded fallback endpoint URL

Using a hardcoded endpoint URL as fallback could lead to unintended data exposure. Consider requiring an explicit endpoint URL.

[sourcery-ai]

suggestion (code-quality): Avoid unneeded ternary statements (simplify-ternary)

Suggested change

	const license = licenseRequest.data.license ? true : false;
	const license = !!licenseRequest.data.license;

Explanation

It is possible to simplify certain ternary statements into either use of an || or !.
This makes the code easier to read, since there is no conditional logic.

[sourcery-ai]

suggestion (code-quality): Merge nested if conditions (merge-nested-ifs)

Suggested change

	if (metadata.authors) {
	if (convertedCodemeta.authors.length > 0) {
	const updatedAuthors = convertedCodemeta.authors.map((author) => {
	// Find a matching author in metadata
	const foundAuthor = metadata.authors.find(
	(existingAuthor) =>
	existingAuthor?.familyName === author?.familyName &&
	existingAuthor?.givenName === author?.givenName,
	);
	if (foundAuthor) {
	// Merge roles, avoiding duplicates based on `role` and `startDate`
	if (!foundAuthor?.roles) {
	foundAuthor.roles = [];
	}
	const mergedRoles = [
	...foundAuthor.roles,
	...author.roles.filter(
	(newRole) =>
	!foundAuthor.roles.some(
	(existingRole) =>
	existingRole.role === newRole.role &&
	existingRole.startDate === newRole.startDate,
	),
	),
	];
	// Merge and prioritize data from `author`
	return {
	...foundAuthor,
	...author,
	affiliation: author.affiliation || foundAuthor.affiliation || "",
	email: author.email || foundAuthor.email || "",
	uri: author.uri || foundAuthor.uri || "",
	roles: mergedRoles,
	};
	}
	// If no match, return the current author from convertedCodemeta
	return author;
	});
	// Merge updated authors with any authors in metadata not present in convertedCodemeta
	const nonUpdatedAuthors = metadata.authors.filter(
	(existingAuthor) =>
	!convertedCodemeta.authors.some(
	(author) =>
	author.familyName === existingAuthor.familyName &&
	author.givenName === existingAuthor.givenName,
	),
	);
	metadata.authors = [...nonUpdatedAuthors, ...updatedAuthors];
	}
	}
	if (metadata.authors && convertedCodemeta.authors.length > 0) {
	const updatedAuthors = convertedCodemeta.authors.map((author) => {
	// Find a matching author in metadata
	const foundAuthor = metadata.authors.find(
	(existingAuthor) =>
	existingAuthor?.familyName === author?.familyName &&
	existingAuthor?.givenName === author?.givenName,
	);
	if (foundAuthor) {
	// Merge roles, avoiding duplicates based on `role` and `startDate`
	if (!foundAuthor?.roles) {
	foundAuthor.roles = [];
	}
	const mergedRoles = [
	...foundAuthor.roles,
	...author.roles.filter(
	(newRole) =>
	!foundAuthor.roles.some(
	(existingRole) =>
	existingRole.role === newRole.role &&
	existingRole.startDate === newRole.startDate,
	),
	),
	];
	// Merge and prioritize data from `author`
	return {
	...foundAuthor,
	...author,
	affiliation: author.affiliation || foundAuthor.affiliation || "",
	email: author.email || foundAuthor.email || "",
	uri: author.uri || foundAuthor.uri || "",
	roles: mergedRoles,
	};
	}
	// If no match, return the current author from convertedCodemeta
	return author;
	});
	// Merge updated authors with any authors in metadata not present in convertedCodemeta
	const nonUpdatedAuthors = metadata.authors.filter(
	(existingAuthor) =>
	!convertedCodemeta.authors.some(
	(author) =>
	author.familyName === existingAuthor.familyName &&
	author.givenName === existingAuthor.givenName,
	),
	);
	metadata.authors = [...nonUpdatedAuthors, ...updatedAuthors];
	}

Explanation

Reading deeply nested conditional code is confusing, since you have to keep track of which
conditions relate to which levels. We therefore strive to reduce nesting where
possible, and the situation where two if conditions can be combined using
and is an easy win.

[sourcery-ai]

issue (code-quality): We've found these issues:

• Use f-string instead of string concatenation [×3] (use-fstring-for-concatenation)
• Simplify conditional into switch-like form [×3] (switch)

[fairdataihub-bot]

Thanks for closing this pull request! If you have any further questions, please feel free to open a new issue. We are always happy to help!