This guide will help you understand our infrastructure stack and how we maintain our platforms. While this guide does not have exhaustive details for all operations, it could be used as a reference for your understanding of the systems.
Let us know, if you have feedback or queries, and we will be happy to clarify.
This repository is continuously built, tested and deployed to separate sets of infrastructure (Servers, Databases, CDNs, etc.).
This involves three steps to be followed in sequence:
- New changes (both fixes and features) are merged into our primary development branch (
master) via pull requests. - These changes are run through a series of automated tests.
- Once the tests pass we release the changes (or update them if needed) to deployments on our infrastructure.
Typically, master (the default development branch) is merged into the production-staging branch once a day and is released into an isolated infrastructure.
This is an intermediate release for our developers and volunteer contributors. It is also known as our "staging" or "beta" release.
It is identical to our live production environment at freeCodeCamp.org, other than it using a separate set of databases, servers, web-proxies, etc. This isolation lets us test ongoing development and features in a "production" like scenario, without affecting regular users of freeCodeCamp.org's main platforms.
Once the developer team @freeCodeCamp/dev-team is happy with the changes on the staging platform, these changes are moved every few days to the production-current branch.
This is the final release that moves changes to our production platforms on freeCodeCamp.org.
We employ various levels of integration and acceptance testing to check on the quality of the code. All our tests are done through software like Travis CI and Azure Pipelines.
We have unit tests for testing our challenge solutions, Server APIs and Client User interfaces. These help us test the integration between different components.
Note
We are also in the process of writing end user tests which will help in replicating real world scenarios like updating an email or making a call to the API or third-party services.
Together these tests help in preventing issues from repeating themselves and ensure we do not introduce a bug while working on another bug or a feature.
We have configured continuous delivery software to push changes to our development and production servers.
Once the changes are pushed to the protected release branches, a build pipeline is automatically triggered for the branch. The build pipelines are responsible for building artifacts and keeping them in a cold storage for later use.
The build pipeline goes on to trigger a corresponding release pipeline if it completes a successful run. The release pipelines are responsible for collecting the build artifacts, moving them to the servers and going live.
Status of builds and releases are available here.
Currently, only members on the developer team can push to the production branches. The changes to the production-* branches can land only via fast-forward merge to the upstream.
Note
In the upcoming days we would improve this flow to be done via pull-requests, for better access management and transparency.
-
Configure your remotes correctly.
git remote -v
Results:
origin git@github.com:raisedadead/freeCodeCamp.git (fetch) origin git@github.com:raisedadead/freeCodeCamp.git (push) upstream git@github.com:freeCodeCamp/freeCodeCamp.git (fetch) upstream git@github.com:freeCodeCamp/freeCodeCamp.git (push) -
Make sure your
masterbranch is pristine and in sync with the upstream.git checkout master git fetch --all --prune git reset --hard upstream/master
-
Check that the Travis CI is passing on the
masterbranch for upstream.The continuous integration tests should be green and PASSING for the
masterbranch.Checking status on Travis CI (screenshot)
If this is failing you should stop and investigate the errors.
-
Confirm that you are able to build the repository locally.
npm run clean-and-develop -
Move changes from
mastertoproduction-stagingvia a fast-forward mergegit checkout production-staging git merge master git push upstream[!NOTE] You will not be able to force push and if you have re-written the history in anyway these commands will error out.
If they do, you may have done something incorrectly and you should just start over.
The above steps will automatically trigger a run on the build pipeline for the production-staging branch. Once the build is complete, the artifacts are saved as .zip files in a cold storage to be retrieved and used later.
The release pipeline is triggered automatically when a fresh artifact is available from the connected build pipeline. For staging platforms, this process does not involve manual approval and the artifacts are pushed to the Client CDN and API servers.
[!TIP|label:Estimates] Typically the build run takes ~20-25 minutes to complete followed by the release run which takes ~15-20 mins for the client, and ~5-10 mins for the API to be available live. From code push to being live on the staging platforms the whole process takes ~35-45 mins in total.
The process is mostly the same as the staging platforms, with a few extra checks in place. This is just to make sure, we do not break anything on freeCodeCamp.org which can see hundreds of users using it at any moment.
| Do NOT execute these commands unless you have verified that everything is working on the staging platform. You should not bypass or skip any testing on staging before proceeding further. |
|---|
-
Make sure your
production-stagingbranch is pristine and in sync with the upstream.git checkout production-staging git fetch --all --prune git reset --hard upstream/production-staging
-
Move changes from
production-stagingtoproduction-currentvia a fast-forward mergegit checkout production-current git merge production-staging git push upstream[!NOTE] You will not be able to force push and if you have re-written the history in anyway these commands will error out.
If they do, you may have done something incorrectly and you should just start over.
The above steps will automatically trigger a run on the build pipeline for the production-current branch. Once a build artifact is ready, it will trigger a run on the release pipeline.
[!TIP|label:Estimates] Typically the build run takes ~20-25 minutes to complete.
Additional Steps for Staff Action
One a release run is triggered, members of the developer staff team will receive an automated manual intervention email. They can either approve or reject the release run.
If the changes are working nicely and have been tested on the staging platform, then it can be approved. The approval must be given within 4 hours of the release being triggered before getting rejected automatically. A staff can re-trigger the release run manually for rejected runs, or wait for the next cycle of release.
For staff use:
| Check your email for a direct link or go to the release dashboard after the build run is complete. |
|---|
Once one of the staff members approves a release, the pipeline will push the changes live to freeCodeCamp.org's production CDN and API servers. They typically take ~15-20 mins for the client, and ~5 mins for the API servers to be available live.
[!TIP|label:Estimates] The release run typically takes ~15-20 mins for each client instance, and ~5-10 mins for each API instance to be available live. From code push to being live on the production platforms the whole process takes ~90-120 mins in total (not counting the wait time for the staff approval).
Here is the current test, build and deployment status of the codebase.
| Type | Branch | Status | Dashboard |
|---|---|---|---|
| CI Tests | master |
 |
Go to status dashboard |
| CI Tests | production-staging |
 |
Go to status dashboard |
| Build Pipeline | production-staging |
Go to status dashboard | |
| Release Pipeline | production-staging |
Go to status dashboard | |
| CI Tests | production-current |
 |
Go to status dashboard |
| Build Pipeline | production-current |
Go to status dashboard | |
| Release Pipeline | production-current |
Go to status dashboard |
We welcome you to test these releases in a "public beta testing" mode and get early access to upcoming features to the platforms. Sometimes these features/changes are referred to as next, beta, staging, etc. interchangeably.
Your contributions via feedback and issue reports will help us in making the production platforms at freeCodeCamp.org more resilient, consistent and stable for everyone.
We thank you for reporting bugs that you encounter and help in making freeCodeCamp.org better. You rock!
Currently a public beta testing version is available at:
Note
The domain name is different than freeCodeCamp.org. This is intentional to prevent search engine indexing and avoid confusion for regular users of the platform.
The current version of the platform is always available at freeCodeCamp.org.
The dev-team merges changes from the production-staging branch to production-current when they release changes. The top commit should be what you see live on the site.
You can identify the exact version deployed by visiting the build and deployment logs available in the status section. Alternatively you can also ping us in the contributors chat room for a confirmation.
There are some known limitations and tradeoffs when using the beta version of the platform.
-
All data / personal progress on these beta platforms
will NOT be saved or carried overto production.Users on the beta version will have a separate account from the production. The beta version uses a physically separate database from production. This gives us the ability to prevent any accidental loss of data or modifications. The dev team may purge the database on this beta version as needed.
-
Deployment is expected to be frequent and in rapid iterations, sometimes multiple times a day. As a result there will be unexpected downtime at times or broken functionality on the beta version.
-
The beta site is and always has been to augment local development and testing, nothing else. It's not a promise of what’s coming, but a glimpse of what is being worked upon.
-
We use a test tenant for freecodecamp.dev on Auth0, and hence do not have the ability to set a custom domain. This makes it so that all the redirect callbacks and the login page appear at a default domain like:
https://freecodecamp-dev.auth0.com/. This does not affect the functionality is as close to production as we can get.
Please open fresh issues for discussions and reporting bugs. You can label them as release: next/beta for triage.
You may send an email to dev[at]freecodecamp.org if you have any queries. As always all security vulnerabilities should be reported to security[at]freecodecamp.org instead of the public tracker and forum.