Skip to content

Commit

Permalink
feat: ✨ Codefair v3.2.0 (#115)
Browse files Browse the repository at this point in the history 
* refactor: ♻️ transfer validation api to codefair repo (#106)

* 🚚 chore: rename `validator-api` to `validator`

* 👷 ci: add validator deploy

* 👷 ci: add validator healthcheck

* 👷 ci: add validator deploy for prod

* refactor: ♻️ update codefair name

* feat: ✨ Metadata + license validators + bug patches (#107)

* merge: 🔀 merge main to staging

* refactor: ♻️ 🐛 Codefair 3.1.0 (#89)

* fix: 🐛 patch removing pull_request.closed bug on metadata pr merge

* fix: 🐛 don't update firstPublished in the codemeta file + better logs

* fix: 🐛 patch error on reading db entry when it doesn't exist

* chore: 🔊 better log on successful Zenodo publish

* style: 🎨 margins between text and icon

* feat: ✨ links to the list of tools used for codefair in /codefair

* refactor: ✨ ♻️ abstract the license validation fn

* doc: 🔖 update changelog

* feat: ✨ ♻️ abstraction to zenodo workflow

* wip: 🚧 abstracting zenodo workflow

* feat: ✨ add help link in the Zenodo release page

* refactor: ♻️ ✨ abstract out github release functions

* refactor: 🔊 ♻️ improve error tracing in metadata file

* chore: 🔨 remove todo

* refactor: ✨ ♻️ abstracted Zenodo workflow (bot)

* wip: 🚧 save custom license content when no assertion is provided

* ✨ feat: add support for custom licenses (#90)

* ✨ feat: add support for custom licenses

* 🐛 fix: reset citation license field

* 🐛 fix: update zenodo workflow for custom licenses

* fix: 🐛 license is valid with custom license

* feat: ✨ custom license template

* ✨ feat: update codemeta, license and cwl ui paths (#91)

* ✨ feat: update meta paths for consistency

* 🚚 fix: updatel icense request path

* ✨ feat: add support for custom license title (#92)

* refactor: ♻️ ✨ new patches for license, cwl, metadata UI's

* refactor: ✨ ♻️ apply migrations from UI to bot

* feat: ✨ request custom license title from user

* refactor: ♻️ hide message box when custom license title is blank

* refactor: ♻️ issue dashboard requests custom title

* feat: ✨ Saving custom title (#93)

* feat: ✨ save custom title option

* refactor: :recyle: reuse saveLicenseDraft for storing custom license title

* refactor: ♻️ join draft and save title buttons together

* refactor: ♻️ use naive-ui's properties for styling

* chore: 🔨 remove typo

* chore: 🔨 remove unused variables

* feat: ✨ toasts for custom title saved button

* refactor: ♻️ function rename

* feat: ✨ create Zenodo metadata fn handles custom licenses

* feat: ✨ custom license reference for archival workflow

* refactor: ♻️ stop creating identifiers for new db entries

* refactor: ♻️ 🐛 revert allowing custom licenses for zenodo

* refactor: ♻️ Custom License Support (#94)

* refactor: ♻️ ✨ custom title saved updates dashboard, disable create release btn when custom license

* refactor: ♻️ block progress of fair release if license is custom

* feat: ✨ listener for when custom license title is saved through UI

* feat: ✨ prevent fair release if not fair compliant

* refactor: ♻️ improve license validation and update template text for custom licenses

* refactor: ♻️ re-render issue from ui side

* refactor: ♻️ update to message explaining custom licenses

* db calls running in parallel

* fix: 🐛 prevent Zendo section from being seen when the license is custom

* fix: 🐛 trim license content before comparison

---------

Co-authored-by: Sanjay Soundarajan <sanjay.16@live.com>

* refactor: ♻️ safety checks for protected middleware

* refacotr: ♻️ update protectRoute to redirect to login when no user or session

* feat: :fix: redirect to login if not signed in

* refactor: ♻️ remove import

* chore: 🔊 🔨 remove logs

* docs: 📝 update changelog

* fix: 🐛 check license content all times

* fix: 🐛 update on zenodo ui page

* feat: ✨ delete branch after pr has been closed/merged

* chore: 🔨 spacing in changelog

* wip: 🚧 individual validation requests for license and metadata

* feat: ✨ license validation endpoint

* wip: 🚧 creating updated metadata validation

* wip: 🚧 patches for metadata revalidation workflow

* wip: 🚧 validation for codemeta

* doc: 🔖 jsdoc comments on fns

* fix: 🐛 template renderer will stop using old links (#99)

* fix: 🐛 rerender template will stop using old links

* doc: 📝 update CHANGELOG

* doc: 📝 update metadata

* fix: 🐛 preserve authors and contributors from codemeta

* feat: ✨ re-validated codemeta from repo dashboard

* fix: 🐛 await metadata validation

* feat: ✨ codemeta validation extended with microservice validator

* refactor: ♻️ update timestamps to be unix format

* feat: ✨ clean privatekey env before use

* refactor: ♻️ update cwl validation endpoint

* wip: 🚧 send validation message to db

* feat: ✨ create migrations for metadata validation messages

* wip: 🚧 creating json schema for codemeta.json validations

* 👷 ci: setup deployment environments (#101)

* 👷 ci: setup for staging env (#103)

Co-authored-by: Sanjay Soundarajan <sanjay.16@live.com>
Co-authored-by: slugb0t <wheresdorian@gmail.com>
fix: 🐛 template renderer will stop using old links (#99)

* fix: 🐛 adjust env variables used after env changes

* 👷 ci: cleanup dockerfile

* feat: ✨ codemeta schema first draft

* 👷 ci: cleanup dockerfile

* 👷 ci: cleanup dockerfile

* 👷 ci: cleanup dockerfile

* 👷 ci: setup for staging env

* 👷 ci: setup for staging env

* 👷 ci: setup for staging env

* 👷 ci: setup for staging env

* 👷 ci: cleanup dockerfile

* 👷 ci: fix staging deploys (#104)

* wip: 🚧 abstracting cwl workflow

* refactor: ♻️ update codemeta schema to validate against the raw codemeta.json files

* refactor: ♻️ update codemeta schema for additional fields

* feat: ✨ validating raw content of codemeta.json file for 3.0

* refactor: ♻️ create required fields for codemeta schema

* wip: 🚧 testing validation responses for db

* handle codemeta.json 3.0 and 2.0 versioning

* fix: 🐛 correct branch watch for staging deployment action

* doc: 🔖 update changelog for 3.2.0 release

* feat: ✨ align bot prisma with ui

* fix: 🐛 bug patch for cwl workflow

* feat: ✨ metadata workflow update

* refactor: ♻️ preserving authors and contributors

* wip: 🚧 update the metadata workflow after discussions

* fix: 🐛 don't create cwlobject twice

* refactor: ♻️ send 200 status when invalid codemeta

* refactor: ♻️ update codemeta validation schema

* refactor: ♻️ do not revalidate metadata files for push events unless metadata files are updated themselves

* wip: 🚧 seperate rerun validation with regather information

* wip: 🚧 remove command from issue dashboard if error occurs (allows retry)

* refactor: ♻️ update the issue body after successful validation (remove the command from the issue body)

* refactor: ♻️ add additional key to codemeta schema

* refactor: ♻️ remove the regather options from the UI as validation is still needed when regathering

* fix: 🐛 pr button updated with new links

* fix: 🐛 patch variable declaration not in scope

* refactor: ♻️ update on the dropdown icons

* refactor: ♻️ update the keys of the cwl object

* refactor: ♻️ update the getcwlfiles function

---------

Co-authored-by: Sanjay Soundarajan <sanjay.16@live.com>

* fix: 🐛 use the correct env variable for private key

* fix: 🐛 Codefair v3.2.0 finalization (#113)

* fix: 🐛 correct cwlObject variables in push event

* fix: 🐛 better error message in metadata workflow + collect missing fields from codemeta.json

* fix: 🐛 apply await to async function call

* refactor: ♻️ removed unused imports

* refactor: ♻️ add try catch in then statement of promise

* fix: 🐛 missing fields in codemeta gathering + add error handling for convertCitationForDB fn

* refactor: ♻️ add try catch to renderer to preserve errors

* fix: 🐛 preserve metadata validation results in db

* wip: 🚧 validation results being displayed through the repo dashboard

* wip: 🚧 displaying metadata validation results in seperate pages

* feat: ✨ validation page for codemeta

* fix: 🐛 patch fundingCode not being preserved from codemeta

* wip: 🚧 final design for validation results

* feat: ✨ add view validation results conditionally

* test: ⚗️ ensuring validation progresses accordingly

* feat: ✨ popover messages for metadata badges

* feat: ✨ if metadata file fails to parse then update validation message

* fix: 🐛 update schema to handle 2.0 and 3.0 authors

* style: 🎨 formatting

* fix: 🐛 add releaseNotes key to schema for 3.0 versions

* refactor: ✏️ change wording

* fix: 🐛 validate metadata file that was updated only on push events

* chore: 📝 remove test logs

* fix: 🐛 migrate db during deployment

* fix: 🐛 revert

* fix: 🐛 typo on dockerfile

* refactor: ♻️ use npm instead of yarn for migration

* refactor: ♻️ access db url during prisma build

* refactor: ♻️ apply node env

* 💚 ci: copy package json

* 💚 ci: revert arg

* 💚 ci: add prisma migrate to ci

* 💚 ci: add prisma migrate to ci

* 👷 ci: update dockerfile

* refactor: ♻️ use deployed validator

* fix: 🐛 when license is updated validate metadata files

* fix: 🐛 validation returns codemeta version on errors

* 🐛 fix: add basic logwatch

* fix: 🐛 add @type key for 2.0 codemeta.json files

* refactor: ♻️ updated required fields for codemeta schmea

* 🐛 fix: add logwatch support

* 🐛 fix: add json to logwatch

* fix: 🐛 patch json parse issue

* fix: 🐛 update logging parameters

* refactor: ♻️ point a folder above for codemeta-schema

* refactor: ♻️ update dockerfile to include schema

* fix: 🐛 ensure authors have roles key

* refactor: ♻️ 🔊 convert some consola logs to logwatch

* fix: 🐛 roles are optional in save request

* chore: 🔊 remove dev logs

* doc: 📝 update changelog

* refactor: ♻️ revert roles key being optional

* fix: 🐛 add logwatch.start method

* fix: 🐛 ensure contrib and authros have roles

* fix: 🐛 CWL handles multiple installations

* refactor: ♻️ update log error

---------

Co-authored-by: Sanjay Soundarajan <sanjay.16@live.com>
  • Loading branch information
@slugb0t @megasanjay
slugb0t and megasanjay authored Dec 10, 2024
1 parent d9834e3 commit d17b463
Show file tree
Hide file tree
Showing 32 changed files with 2,889 additions and 1,234 deletions.
24 changes: 23 additions & 1 deletion .github/workflows/deploy-main.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -202,4 +202,26 @@ jobs:

# Deploy the app for all other times
- run: kamal redeploy
# - run: kamal redeploy --verbose
# - run: kamal redeploy --verbose

migrate-db:
runs-on: ubuntu-latest
environment: stg
defaults:
run:
working-directory: ui
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}

steps:
- uses: actions/checkout@v3

- uses: actions/setup-node@v3
with:
node-version: 20

- run: yarn install

- run: yarn prisma:generate

- run: yarn prisma:migrate:deploy
24 changes: 23 additions & 1 deletion .github/workflows/deploy-staging.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -202,4 +202,26 @@ jobs:

# Deploy the app for all other times
- run: kamal redeploy
# - run: kamal redeploy --verbose
# - run: kamal redeploy --verbose

migrate-db:
runs-on: ubuntu-latest
environment: stg
defaults:
run:
working-directory: ui
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}

steps:
- uses: actions/checkout@v3

- uses: actions/setup-node@v3
with:
node-version: 20

- run: yarn install

- run: yarn prisma:generate

- run: yarn prisma:migrate:deploy
17 changes: 17 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,23 @@ All notable changes the Codefair App will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## v3.2.0 - 12-10-2024

### Added

- Introduce metadata and license validation capabilities, allows for re-validation of metadata files and licenses through the UI.
- Add support for validation codemeta.json and CITATION.cff files using a new schema and validation endpoints.
- Improve metadata handling by introducing functions to convert and validate metadata from codemeta.json and CITATION.cff files.
- Enhance the dashboard UI to display metadata validation results and provide options for re-validation.
- Add CI workflows for deploying the validator service using Kamal and Docker.
- Set up deployment configurations for the validator service using Kamal, including Docker and Azure Container Registry integrations.
- A new schema for codemeta.json validation and implement validation logic for CITATION.cff files.
- Validator service moved to current repository and integrated with the Codefair app.

## Fixed

- Fixed issues related to filtering what is consider a cwl file in the repository.

## v3.1.1 - 11-12-2024

### Fixed
Expand Down
3 changes: 2 additions & 1 deletion bot/api/github/webhooks.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,12 @@
const { createNodeMiddleware, createProbot } = require("probot");

const app = require("../../index.js");
const privateKey = process.env.GH_APP_PRIVATE_KEY.replace(/\\n/g, "\n");

const probot = createProbot({
overrides: {
appId: process.env.APP_ID,
privateKey: process.env.GH_APP_PRIVATE_KEY.replace(/\\n/g, "\n"),
privateKey,
secret: process.env.WEBHOOK_SECRET,
},
});
Expand Down
3 changes: 1 addition & 2 deletions bot/archival/index.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -386,7 +386,7 @@ export async function createZenodoMetadata(codemetadata, repository) {
// creators: zenodoCreators,
// access_right: zenodoMetadata.zenodo_metadata.accessRight,
// publication_date: new_date,
// // TODO: Ask user for language
//
// rights: [
// {
// description: {en: existingLicense?.license_content},
Expand Down Expand Up @@ -467,7 +467,6 @@ export async function uploadReleaseAssetsToZenodo(
repository,
tagVersion,
) {
// TODO: Add try catches for each fetch
const startTime = performance.now();
if (draftReleaseAssets.length > 0) {
for (const asset of draftReleaseAssets) {
Expand Down
175 changes: 100 additions & 75 deletions bot/cwl/index.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,24 +18,28 @@ const CODEFAIR_DOMAIN = process.env.CODEFAIR_APP_DOMAIN;
* @param {String} repoName - Repository name
* @returns {Array} - Array of CWL files in the repository
*/
export function getCWLFiles(context, owner, repoName) {
export function getCWLFiles(context, owner, repository) {
return new Promise((resolve, reject) => {
consola.info("Checking for CWL files in the repository...");

const cwlFiles = [];
const cwlObject = {
contains_cwl_files: false,
files: [],
removed_files: [],
}

const searchDirectory = async function (path) {
try {
const repoContent = await context.octokit.repos.getContent({
owner,
path,
repo: repoName,
repo: repository.name,
});

for (const file of repoContent.data) {
const fileSplit = file.name.split(".");
if (file.type === "file" && fileSplit.includes("cwl")) {
cwlFiles.push(file);
if (file.type === "file" && file.name.endsWith(".cwl")) {
cwlObject.files.push(file);
}
if (file.type === "dir") {
await searchDirectory(file.path);
Expand All @@ -44,7 +48,7 @@ export function getCWLFiles(context, owner, repoName) {
} catch (error) {
if (error.status === 404) {
// Repository is empty
resolve(cwlFiles);
resolve(cwlObject);
return;
}
consola.error(
Expand All @@ -57,8 +61,26 @@ export function getCWLFiles(context, owner, repoName) {

// Call the async function and handle its promise
searchDirectory("")
.then(() => {
resolve(cwlFiles);
.then(async () => {
try {
// Check if the db entry exists for the repository
const existingCWL = await dbInstance.cwlValidation.findUnique({
where: {
repository_id: repository.id,
}
});

if (existingCWL && existingCWL?.contains_cwl_files) {
cwlObject.contains_cwl_files = existingCWL.contains_cwl_files;
}

cwlObject.contains_cwl_files = cwlObject.files.length > 0;

resolve(cwlObject);
} catch (error) {
console.log("Error getting CWL files:", error);
throw new Error("Error getting the CWL files: ", JSON.stringify(error), { cause: error });
}
})
.catch(reject);
});
Expand All @@ -71,7 +93,7 @@ export function getCWLFiles(context, owner, repoName) {
*/
export async function validateCWLFile(downloadUrl) {
try {
const response = await fetch("https://cwl-validate.codefair.io/validate", {
const response = await fetch("https://cwl-validate.codefair.io/validate-cwl", {
body: JSON.stringify({
file_path: downloadUrl,
}),
Expand Down Expand Up @@ -164,81 +186,84 @@ export async function applyCWLTemplate(
}

consola.start("Validating CWL files for", repository.name);
// Validate each CWL file from list
for (const file of subjects.cwl.files) {
const fileSplit = file.name.split(".");

if (fileSplit.includes("cwl")) {
const downloadUrl =
file?.commitId && !privateRepo
? file.download_url.replace("/main/", `/${file.commitId}/`)
: file.download_url; // Replace the branch with the commit id if commit id is available and the repo is public

const [isValidCWL, validationMessage] =
await validateCWLFile(downloadUrl);

if (!isValidCWL && validOverall) {
// Overall status of CWL validations is invalid
validOverall = false;
}

if (!isValidCWL) {
failedCount += 1;
}

const [modifiedValidationMessage, lineNumber1, lineNumber2] =
replaceRawGithubUrl(validationMessage, downloadUrl, file.html_url);

// Add the line numbers to the URL if they exist
if (lineNumber1) {
file.html_url += `#L${lineNumber1}`;
if (lineNumber2) {
file.html_url += `-L${lineNumber2}`;
// Validate each CWL file from list\
consola.info(`Validating ${JSON.stringify(subjects.cwl)} CWL files`);
if (subjects.cwl.files.length > 0) {
for (const file of subjects.cwl.files) {
const fileSplit = file.name.split(".");

if (fileSplit.includes("cwl")) {
const downloadUrl =
file?.commitId && !privateRepo
? file.download_url.replace("/main/", `/${file.commitId}/`)
: file.download_url; // Replace the branch with the commit id if commit id is available and the repo is public

const [isValidCWL, validationMessage] =
await validateCWLFile(downloadUrl);

if (!isValidCWL && validOverall) {
// Overall status of CWL validations is invalid
validOverall = false;
}
}

// Create a new object for the file entry to be added to the db
const newDate = new Date();
cwlFiles.push({
href: file.html_url,
last_modified: newDate,
last_validated: newDate,
path: file.path,
validation_message: modifiedValidationMessage,
validation_status: isValidCWL ? "valid" : "invalid",
});

// Apply the validation file count to the analytics collection on the db
const analyticsCollection = dbInstance.analytics;
await analyticsCollection.upsert({
create: {
cwl_validated_file_count: 1, // Start count at 1 when creating
id: repository.id, // Create a new record if it doesn't exist
},
update: {
cwl_validated_file_count: {
increment: 1,

if (!isValidCWL) {
failedCount += 1;
}

const [modifiedValidationMessage, lineNumber1, lineNumber2] =
replaceRawGithubUrl(validationMessage, downloadUrl, file.html_url);

// Add the line numbers to the URL if they exist
if (lineNumber1) {
file.html_url += `#L${lineNumber1}`;
if (lineNumber2) {
file.html_url += `-L${lineNumber2}`;
}
}

// Create a new object for the file entry to be added to the db
const newDate = Math.floor(Date.now() / 1000);
cwlFiles.push({
href: file.html_url,
last_modified: newDate,
last_validated: newDate,
path: file.path,
validation_message: modifiedValidationMessage,
validation_status: isValidCWL ? "valid" : "invalid",
});

// Apply the validation file count to the analytics collection on the db
const analyticsCollection = dbInstance.analytics;
await analyticsCollection.upsert({
create: {
cwl_validated_file_count: 1, // Start count at 1 when creating
id: repository.id, // Create a new record if it doesn't exist
},
},
where: {
id: repository.id,
},
});

// Add the file to the table content of the issue dashboard
tableContent += `| ${file.path} | ${isValidCWL ? "✔️" : "❌"} |\n`;

consola.success(
`File: ${file.path} is ${isValidCWL ? "valid" : "invalid"}`,
);
update: {
cwl_validated_file_count: {
increment: 1,
},
},
where: {
id: repository.id,
},
});

// Add the file to the table content of the issue dashboard
tableContent += `| ${file.path} | ${isValidCWL ? "✔️" : "❌"} |\n`;

consola.success(
`File: ${file.path} is ${isValidCWL ? "valid" : "invalid"}`,
);
}
}
}

// Entry does not exist in the db, create a new one (no old files exist, first time seeing cwl files)
if (!existingCWL) {
await dbInstance.cwlValidation.create({
data: {
contains_cwl_files: subjects.cwl.contains_cwl,
contains_cwl_files: subjects.cwl.contains_cwl_files,
files: cwlFiles,
identifier,
overall_status: validOverall ? "valid" : "invalid",
Expand Down
Loading

0 comments on commit d17b463

Please sign in to comment.