Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: 15.3.0 #375

Merged
merged 63 commits into from
Dec 2, 2024
Merged

feat: 15.3.0 #375

merged 63 commits into from
Dec 2, 2024

Conversation

aaronm-2112
Copy link
Member

@aaronm-2112 aaronm-2112 commented Dec 2, 2024

Summary by Sourcery

Introduce features to manage dataset structure by purging non-existent files and deleting empty folders. Enhance server status tracking and user feedback during initialization. Implement firewall checks to detect network issues. Update build scripts for folder-based Python builds and adjust CI workflows to include a new branch for folder builds.

New Features:

  • Introduce a feature to purge non-existent files from the dataset structure before generating manifest files.
  • Add functionality to recursively delete empty folders from the dataset structure.
  • Implement a check for external firewall interference that may block access to Pennsieve.

Enhancements:

  • Enhance the server status handling by introducing a global variable to track server live status and updating it based on server events.
  • Improve the user interface for initializing SODA's background services with animations and better feedback on potential network issues.

Build:

  • Update build scripts to use a folder build approach for Python files instead of a single file build.

CI:

  • Add 'folder-build' branch to the CI workflows for build and deployment on Linux, Mac, and Windows.

aaronm-2112 and others added 30 commits October 8, 2024 13:35
fairdataihub-bot and others added 22 commits November 8, 2024 00:00
feat: detect network interference between SODA and server and SODA and pennsieve
feat: one folder build option for packaging SODA
@aaronm-2112 aaronm-2112 requested a review from bvhpatel as a code owner December 2, 2024 22:53
Copy link

Thank you for submitting this pull request! We appreciate your contribution to the project. Before we can merge it, we need to review the changes you've made to ensure they align with our code standards and meet the requirements of the project. We'll get back to you as soon as we can with feedback. Thanks again!

Copy link
Contributor

sourcery-ai bot commented Dec 2, 2024

Reviewer's Guide by Sourcery

This PR implements several key features to improve file handling and network connectivity detection. The main changes include a new system to detect and handle non-existent files in the dataset structure, empty folder cleanup, and network connectivity/firewall detection mechanisms.

Sequence diagram for handling non-existent files in dataset structure

sequenceDiagram
    participant User
    participant App
    participant FileSystem
    User->>App: Initiate dataset structure processing
    App->>FileSystem: Check existence of files
    FileSystem-->>App: Return file existence status
    alt File does not exist
        App->>User: Notify about non-existent files
        App->>App: Remove non-existent file references
    end
Loading

Sequence diagram for server live status check and handling

sequenceDiagram
    participant App
    participant Server
    participant User
    App->>Server: Check server live status
    Server-->>App: Return live status
    alt Server is not live
        App->>User: Notify potential network issues
        App->>App: Attempt to restart server
    end
Loading

Class diagram for new firewall check module

classDiagram
    class CheckFirewall {
        +clientBlockedByExternalFirewall(url: string) bool
        +blockedMessage: string
        +hostFirewallMessage: string
    }
    class Axios
    CheckFirewall --> Axios : uses
    class ErrorHandler
    CheckFirewall --> ErrorHandler : uses
Loading

File-Level Changes

Change Details Files
Added functionality to detect and handle non-existent files in dataset structure
  • Implemented recursive file existence checking
  • Added user notification for non-existent files
  • Automatically removes references to missing files
  • Added empty folder detection and cleanup
src/renderer/src/scripts/guided-mode/guided-curate-dataset.js
Implemented server status monitoring and firewall detection system
  • Added global server status tracking
  • Implemented server live status checks
  • Added event handlers for various server failure scenarios
  • Created firewall detection utility functions
src/main/index.js
src/renderer/src/scripts/others/renderer.js
src/renderer/src/scripts/check-firewall/checkFirewall.js
Updated build configuration and deployment process
  • Changed Python build from onefile to folder structure
  • Updated deployment scripts for all platforms
  • Modified electron builder configuration
  • Updated version numbers and API version handling
package.json
electron-builder.yml
src/pyflask/startup/minimumApiVersion.py
.github/workflows/Build-and-deploy-linux.yml
.github/workflows/Build-and-deploy-mac.yml
.github/workflows/Build-and-deploy-win.yml

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time. You can also use
    this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Copy link

Thanks for making updates to your pull request. Our team will take a look and provide feedback as soon as possible. Please wait for any GitHub Actions to complete before editing your pull request. If you have any additional questions or concerns, feel free to let us know. Thank you for your contributions!

Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @aaronm-2112 - I've reviewed your changes - here's some feedback:

Overall Comments:

  • Please remove debug console.log statements before merging (e.g. in renderer.js and globals.js)
Here's what I looked at during the review
  • 🟡 General issues: 1 issue found
  • 🟢 Security: all looks good
  • 🟢 Testing: all looks good
  • 🟡 Complexity: 1 issue found
  • 🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@@ -331,12 +336,27 @@ const createPyProc = async () => {
pyflaskProcess.stderr.on("data", (data) => {
const logOutput = `[pyflaskProcess stderr] ${data.toString()}`;
sessionServerOutput += `${logOutput}`;
global.serverLive = false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Consolidate server status management into a single function

The serverLive status is being set to false in multiple places. Consider creating a dedicated function to manage server state changes to improve maintainability and consistency.

          setServerStatus(false);

@@ -5416,6 +5416,131 @@ window.openPage = async (targetPageID) => {
}
}

const purgeNonExistentFilesFromDatasetStructure = async (datasetStructure) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (complexity): Consider consolidating the recursive tree traversals into a single pass operation.

The current implementation performs three separate recursive tree traversals that could be reduced to two while maintaining the same functionality. Here's how to simplify it:

const handleNonExistentFiles = async (currentStructure, currentPath = "") => {
  const nonExistentFiles = [];
  const files = currentStructure?.files || {};

  // Handle files at current level
  for (const fileName in files) {
    const fileData = files[fileName];
    if (fileData.type === "local" && !window.fs.existsSync(fileData.path)) {
      nonExistentFiles.push(`${currentPath}${fileName}`);
      delete files[fileName];
      log.info(`Deleting reference to non-existent file: ${currentPath}${fileName}`);
    }
  }

  // Recurse into folders
  const folders = currentStructure?.folders || {};
  for (const folderName in folders) {
    const subResults = await handleNonExistentFiles(
      folders[folderName], 
      `${currentPath}${folderName}/`
    );
    nonExistentFiles.push(...subResults);
  }

  return nonExistentFiles;
};

This combines the file collection and deletion into a single pass, reducing complexity while maintaining the same functionality. The folder cleanup can remain as a separate pass since it needs to run after all file operations are complete.

Comment on lines +411 to +412
const status = await ipcRenderer.invoke("get-server-live-status");
return status;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (code-quality): Inline variable that is immediately returned (inline-immediately-returned-variable)

Suggested change
const status = await ipcRenderer.invoke("get-server-live-status");
return status;
return await ipcRenderer.invoke("get-server-live-status");


ExplanationSomething that we often see in people's code is assigning to a result variable
and then immediately returning it.

Returning the result directly shortens the code and removes an unnecessary
variable, reducing the mental load of reading the function.

Where intermediate variables can be useful is if they then get used as a
parameter or a condition, and the name can act like a comment on what the
variable represents. In the case where you're returning it from a function, the
function name is there to tell you what the result is, so the variable name
is unnecessary.

const folderObjIsEmpty = (folderStructure) => {
// Check if there are no files at this level
const hasFiles = folderStructure.files && Object.keys(folderStructure.files).length > 0;
if (hasFiles) return false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (code-quality): Use block braces for ifs, whiles, etc. (use-braces)

Suggested change
if (hasFiles) return false;
if (hasFiles) {


ExplanationIt is recommended to always use braces and create explicit statement blocks.

Using the allowed syntax to just write a single statement can lead to very confusing
situations, especially where subsequently a developer might add another statement
while forgetting to add the braces (meaning that this wouldn't be included in the condition).

Copy link

sonarqubecloud bot commented Dec 2, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@aaronm-2112 aaronm-2112 merged commit 8002ce9 into staging Dec 2, 2024
8 of 15 checks passed
Copy link

Thanks for closing this pull request! If you have any further questions, please feel free to open a new issue. We are always happy to help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants