The Renku UI is the web-based UI for Renku and it requires a RenkuLab deployment. You can check the Administrator's Guide to get more information.
The Renku UI is made up of two subcomponents, which are each packaged as Docker containers and are deployed using Kubernetes.
| UI Client | React-based front-end |
| UI Server | Express-based back-end |
To develop on this codebase, you will want the following tools installed.
| node | Node JavaScript execution environment |
| npm | Node package manager |
| nvm | NVM or some similar tool for managing Node versions |
Node is a requirement; when you install node, you will probably also get npm automatically. Though not absolutely necessary, we recommend using a node version manager like nvm to manage node/npm installations.
However you get it, you need the node version specified in the Dockerfile.
You don't need much to start developing, at least for the UI Client. If you are content with testing your changes using Cypress, no additional tools are strictly needed. The UI Server requires a full RenkuLab deployment. You can find more information in the UI server section.
To start your journey, clone this repository and install the dependencies using npm:
$ npm install
We have a coding guidelines document that explains how to write code for this repository. We suggest you read it before starting to develop.
Please note that not all the code in the repository follows the guidelines since they were introduced later in the project. We are working on refactoring the codebase to make it consistent, and guidelines will be enforced for all new code.
We use Prettier to format the codebase to keep the syntax consistent. We enforce that by checking every pull request and the CI pipelines will fail in case of formatting issues.
You can manually format the code by running npm run format in the client,
server or tests folder.
This repository is configured to automatically format the files on
commit through git hooks.
To set them up, we use Husky.
You can install Husky and the git hooks by running npm install from the
base folder on this repository.
If you use VS Code, you can also install the
esbenp.prettier-vscode extension to run Prettier each time you save files.
After installing it, go to the workspace settings, search for
editor.defaultFormatter and set it to Prettier, then search for
editor.formatOnSave and turn on "Format on save".
We use Conventional Commits to format our commit messages. This allows us to automatically generate changelogs.
Please stick to the following rules to keep messages as consistent as possible:
- Start the commit message with a type from the following list (taken from the
Angular convention:
build,ci,docs,feat,fix,perf,refactor,style,test.
- Do not add any scope.
- Write a concise description of the change starting with a verb in the present tense (imperative mood).
- Use the body of the commit message to provide more context about the change. This won't be necessary for small changes/fixes or version bumps.
- Include a footer if you wish to reference issues or include co-authors. This is not required.
- Mention breaking changes clearly by adding the following:
- Exclamation mark after the type.
- Description of the breaking change, starting with
BREAKING CHANGE:, as the last line in the body. If you need a newer version of a component that has not yet been released, mention the version as[component_name] > [current_version]. Should the final version number be wrong, we can amend it in the GitHub release notes.
- Include a reference to the PR number at the end of the description (first line). If you use "squash-merge" on GitHub, this will be added automatically.
Here are a couple of examples:
feat!: add a new feature (#1234)
This is a longer description of the feature that was added.
BREAKING CHANGE: this commit breaks something
fix #123
build: bump the react version to 18.1 (#2345)
We encourage squashing commits when merging a PR and expanding the body to include relevant details. This will make the commit history cleaner and easier to understand. You can keep separate commits if they are logically separate (please rebase the PR before merging), although it's best to have separate PRs in that case.
Please pay extra attention to following the guidelines when squash-merging a PR. It's easy to mess up since GitHub pre-compiles the commit message.
To tag a release, please manually run the Prepare Release GitHub action. This action
will create a new PR bumping the version number wherever needed in the code.
We loosely follow the recommendations from Semantic Versioning
when tagging releases. We don't expose APIs so the distinction between major and minor
is subtle. Try to follow these rules:
- If the release includes only bug fixes or minor changes (i.e. no
featcommits), bump thepatchversion. - When introducing at least one new feature or significant change to an existing
feature, bump the
minorversion. - We keep the
majorversion for significant changes that overhaul big chunks of the UI, like a re-design, a re-styling, or URL restructuring.
Once the Release PR is merged, you can go to the
Releases section and
click the Draft a new release" button. Manually enter a new tag matching the version
number created by the GitHub action and fill in the release notes using the "Generate
release notes" button.
Please give it a quick read to ensure the notes are correct. If you make a bugfix release for a previous version, you might need to adjust the target or add/remove entries.
If you want to develop and test your changes against a real RenkuLab deployment or if you wish to develop on the UI Server, then you will need a few more tools to deal with the infrastructure.
| kubectl | K8s command-line tool |
| telepresence | Tool for redirecting requests to your local development environment |
Kubectl and telepresence will allow you to inject code running on your development machine into a K8s deployment. These two tools will be sufficient if you do not need to deploy the renku application into K8s yourself. If you do need to do that, then you will additionally need:
| docker | For building containers |
| helm | For packaging things for K8s |
The UI client is the React-based front-end for RenkuLab. Development started in 2017, and we have striven to change our development style to reflect the evolving best practices around the tools we use.
Not all code conforms to the guidelines and best practices laid out in this document and you might find older code using deprecated technologies. That will be refactored in the future.
We use TypeScript in all new code but we still have a lot of JavaScript code.
This is a list of the main tools we use in the client and you need to know to develop it.
| Framework | Purpose |
|---|---|
| React | Reactive component framework |
| Redux | Centralized state management |
| Redux Toolkit | State integration and fetching/caching tools |
| Bootstrap | CSS framework based on a responsive grid |
Mind that we try to take advantage of the latest features of React and Redux, and to integrate the two tools as seamlessly as possible.
For this reason, please follow these simple best practices when dealing with new React components:
- Use
useStateto manage component-local state - Use
useEffectfor handling the component lifecycle
When dealing with the state, we follow the redux-toolkit style for interacting with Redux. If you are not familiar with this, follow the link to see the tutorial.
In short:
- Global application state is kept in a single, global Redux store.
- Features should have slices that encapsulate the state they need and add the slices into the global store.
- Use slice-specific selector hooks (e.g.
useWorkflowsSelector),
or the
useSelectorhook to access global state. - Use the
useDispatchhook to make changes to the state in components.
Also, please use RTK Query to interact with backend services. It greatly simplifies writing the code to fetch and cache data, as well as handling the transitions through the request lifecycle.
Based on these suggestions from Redux, we decided to use a few folders to bundle functions and components together, broadly following the "Feature folders" style.
Here are the folders in /client/src where to place new components:
features: create/use sub-folders to contain files identifying single features. These sometimes correspond to Renku abstractions, like "Projects", "Datasets", "Sessions", or to cross-entity features such as "Search" and "Dashboard". Wherever relevant, please add*.api.tsfiles containg the RTK queries and*.slice.tsfiles for slices.components: add here components that can be reused in different contexts. If something is clearly a shared component (e.g.RenkuAlert), put it here. If it's not obvious, and currently used by just one component, you can leave it in thefeaturefolder (follow the principle: do not over-engineer it too early). Mind that we also store most of the temporary values in the Redux store, so you can define actions here if necessary.utils: put here anything generic that doesn't fall into the previous categories (e.g. constants, helper functions, wrappers).
Picking the perfect place isn't always straightforward and our current folder structure still has many outdated components that don't follow the convention. We plan to move them when already touching the code for other changes.
We use CSS modules to apply CSS styles locally and avoid leaking styles to the whole web application. No additional configuration is needed since Create React App supports CSS modules out of the box.
When a node has a class name that is either computed dynamically or is comprised of two or more classes, use the classnames package (idiomatically imported typically as cx) to construct the class name string.
If a component requires a large package, it can be loaded on demand by using the lazy() function from React.
Here is an example:
- The component itself: NotebookRender.tsx
- The wrapper allowing for lazy loading: LazyNotebookRender.tsx
In this case, we save ~950kB from being included in the final bundle.
We split testing into multiple categories. All these tests are required to pass before we merge a PR.
We use jest only for unit testing.
Unit tests are used to test individual functions. We don't have a fixed rule for
the amount of coverage we require, but we try to cover all the helper functions and
the most important functions used by components. Note, we do not test components
using jest/jsdom anymore. Tests that require a browser/DOM environment are implemented
either in Storybook or Cypress (see below). There might be legacy tests that still
use jest with components.
Files containing unit tests are named *.test.ts. Usually, they refer to a specific
file in the src folder, so you can find the tests for src/*/MyComponent.tsx
in src/*/MyComponent.test.ts.
You can run the unit tests manually using the following command in the client subfolder:
$ cd client
$ npm test
We use Storybook to test single React components in isolation. This tool allows us to create interactive stories and to keep track of all the reusable components we implemented. Mind that this is a recent introduction so you can expect quite a few components to be missing.
Storybook files are named *.stories.tsx and refer to specific components.
The overhead of writing stories is rather low, so we encourage you to write them
for all new components.
Whether to write stories for specific components depends on the size and re-usability
of the component. The more simple and reusable a component is, the more important
it is to have a story for it.
Be sure to check out the full documentation here.
You can run component tests using the following command:
$ npm run storybook-compile-and-test
If you wish to check the components, you can use the Storybook interface. To start it, run:
$ npm run storybook
This should also open your browser automatically. If it doesn't, you can visit http://localhost:6006 to see the Storybook interface.
Mind that we deploy Storybook automatically in each RenkuLab deployment. You can access it at
https://<renkulab-url>/storybook/.
Here are a few rules to follow when writing new stories:
- Use the
titleproperty in the story definition to group related stories by categories. You can find an example incomponents/buttons/buttonWithMenu. - It's good to showcase different variations of your component. Use multiple stories to demonstrate how props and states affect the component and how that helps in serving different use cases.
- Provide a clear and concise description for each story; feel free to include details on the usage whenever necessary.
- Use
Argsto tweak props' values, making the components interactive so that users can play with them. - Include stories that demonstrate responsiveness across different devices wherever it's relevant.
- You can use
addon-reduxfor state management on components that require to get data from the Redis store.
For testing interactions between multiple components or slices of pages, as well as some common scenarios (or uncommon, like specific error cases), we use Cypress.
Cypress tests are located in the tests folder and are named *.spec.ts. We don't have a
fixed naming convention for them, but we try to group tests based on features instead of
single components. For this reason, some components might be tested in multiple spec files,
while others get less coverage.
Keep in mind we test real scenarios and we try to simulate users interacting with the platform. Since our goal is to catch non-obvious regressions, we mock backend responses in different scenarios, including errors and other edge cases. They do occasionally occur in our platform, especially since users can change a lot of things in their projects and break them in creative ways, so the best we can do is handle errors gracefully and provide meaningful feedback to the user so that they can recover whenever possible.
Cypress is also a great tool to run components in the browser and debug them. Most of the time, you don't need the whole platform to develop or update a component, so you can use Cypress to develop the UI without a dedicated RenkuLab deployment.
You can run Cypress tests using the following command:
$ cd tests
$ npm run e2e
Here are other information that might be useful.
We are using Craco to override default settings from Create React App. Since CRA is not actively maintained, we might move away soon from this stack.
In the meanwhile, mind that Jest cannot be run outside of Craco's context;
if you ever need to debug your tests using advanced features like node --expose-gc
or node --inspect-brk, you have to reference Jest from
./node_modules/@craco/craco/dist/bin/jest.
The following example shows how to check for memory leaks:
$ node --expose-gc ./node_modules/@craco/craco/dist/bin/jest --runInBand --logHeapUsage *
As already mentioned, you don't strictly need a full RenkuLab deployment for the client.
Using Cypress, you can test your changes against a mocked backend; otherwise, you can run
the client locally and point to a development instance like https://dev.renku.ch or one
of the CI deployments made on each PR.
If you are part of the Renku team, you should have full access to the development infrastructure.
In this case, you can use Telepresence to develop the UI
in a realistic setting. The client folder includes a run-telepresence.sh script that
is tailored for the SDSC development cluster.
Try to run it to get more instructions:
$ cd client
$ ./run-telepresence.sh
Telepresence replaces the UI client pod in the target Kubernetes instance. All the traffic is then redirected to a local process, making changes to files almost immediately available in your development RenkuLab instance.
The script allows configuration of certain aspects of the deployment through environment
variables. Take a look at the script to see all the options that are available.
The script generates a config.json file into the client/public folder, and you can
modify this file with a text editor and reload the browser to test out different
configuration settings.
The webpack-bundle-analyzer plugin can be used to analyze the final bundle and see which Node packages take up a lot of space.
Use the build:analyze script to start it:
$ npm run build:analyzeflowchart LR
subgraph L1
A(/)-->DA(/datasets)
A-->HE(/help)
A-->LOOUT(/logout)
A-->PR(/projects)
A-->SEA(/search)
A-->SE(/sessions)
end
subgraph L2
PR-->PR1(/new)
PR-->PRID(/:id)
DA-->DAID(/:id)
HE-->HE1(/changes)
HE-->HE2(/docs)
HE-->HE3(/features)
HE-->HE4(/status)
end
subgraph L3
PRID-->PRID1(/overview)
PRID-->PRFI(/files)
DAID-->DAID1(/add)
PRID-->PRIDDA(/datasets)
PRID-->PRID_WORKFLOWS(/workflows)
PRID-->PRSE(/sessions)
PRID-->PRID_SETTINGS(/settings)
end
subgraph L4
PRID1-->PRID11(/overview/stats)
PRID1-->PRID12(/overview/commits)
PRFI-->PRFI1(/blob/:file-path)
PRFI-->PRFI2(/lineage/:file-path)
PRIDDA-->PRIDDA1(/:dataset-name)
PRIDDA-->PRIDDA2(/new)
PRID_WORKFLOWS-->PRID_WORKFLOWS_ID(/:worfklow-id)
PRSE-->PRSE1(/new?autostart=1)
PRSE-->PRSE2(/new)
PRSE-->PRSE3(/show/:session-id)
PRID_SETTINGS-->PRID_SETTINGS_SESSIONS(/sessions)
end
subgraph L5
PRIDDA1-->PRIDDA11(/modify)
end
External links map
flowchart LR
A(/)-->B(https://renku.readthedocs.io/en/stable/*)
A-->C(https://github.com/SwissDataScienceCenter/*)
A-->G(https://gitter.im/SwissDataScienceCenter/renku)
A-->H(https://renku.discourse.group)
A-->D(https://datascience.ch)
A-->E(https://www.epfl.ch)
A-->F(https://ethz.ch)
The UI server is the Express-based back-end for the UI client. All code is written in TypeScript.
The main responsibilities of the server include:
- Managing access tokens.
- Creating a WebSocket client that can invoke APIs on behalf of the user.
- Storing temporary data for the client (e.g. recent searches and recently visited projects or).
Though the server is the first recipient of service requests from the client, in most cases the server just forwards requests to the appropriate service with the access tokens attached. For this reason, the codebase is much smaller and simpler than the client.
| Framework | Purpose |
|---|---|
| Express | Route and respond to HTTP requests |
| Morgan | Logging middleware for express |
| WS | WebSocket framework |
We use Express to handle HTTP requests and WS to handle WebSocket connections.
We have an abstraction for storage in the storage folder. Currently, we support
only Redis, and we do not currently to use the UI server to store non-volatile data.
We use Redis to store user tokens and temporary data.
The authentication logic is in the authentication folder, including the middleware
to add tokens to the queries.
The WebSocket logic is in the websocket folder. There is a fixed structure for the
messages between the server and the client. Should you need to add support for new
messages, please add a new function in the handlers section and configure the handler
in the index file.
Finally, we configure the routes in the routes folder. There is little to change
there since the Server should have very limited logic to handle requests.
As with the client, we use Jest for unit tests. We should improve the coverage since it's pretty low at the moment.
You can manually run tests using the following commands:
$ cd server
$ npm test
Here are other information that might be useful.
We don't have any abstraction of the resources needed by the UI server, so we rely on an existing deployment of RenkuLab to run it.
If you are part of the Renku team, you should have full access to the development infrastructure and you can use Telepresence to develop the UI server.
Try to run the run-telepresence.sh script to get more instructions:
$ cd server
$ ./run-telepresence.sh
Telepresence replaces the UI server pod in the target Kubernetes instance. All the traffic is then redirected to a local process, making changes to files almost immediately available in your development RenkuLab instance.