Skip to content

Site Architecture

Jump to bottom
Clinton edited this page Aug 10, 2022 · 64 revisions

Table of Contents

🥞 Stack

At a high level, docs.newrelic.com is built with React and MDX. It uses Gatsby as a static site generator and is hosted on Gatsby Cloud.

🎸 JAMstack

JAM stands for Javascript, APIs & Markup. A core part of the site architecture is Gatsby which is used as a static-site generator. Pages are built by combining Markup (MDX) and Javascript (React) and hosted as static HTML files.

There is no back-end or CMS powering the site.

🛠 Tools

Tool Description
Gatsby Static site generator
newrelic/gatsby-theme-newrelic Shared New Relic components and site configuration (see below)
MDX Markdown + React components for authoring content
React Custom components & view layer
GraphQL Querying content within the site
Emotion CSS in JS (styles)
Github Actions Task automation (see below)
Gatsby Cloud Hosting & deployments (see below)
Yarn Package management & task runner

🔌 Gatsby plugins

In addition to the technologies listed above, we use a number of Gatsby plugins. In addition to established third-party plugins, we've also created a few custom plugins to handle specific tasks. For a complete list of plugins, check out our package.json file.

🎨 Gatsby theme

newrelic/gatsby-theme-newrelic

New Relic has developed a Gatsby theme which includes components and configuration shared across several website. The Open Source and Developer site both make use of this theme. For more information on how to use the components and configuration, refer to the README.

🌐 Hosting and deployment

Gatsby Cloud Hosting

The site is currently hosted on Gatsby Cloud (and is connected to the repository via a webhook). To deploy a new version of the site, simply merge into main (our mainline branch). Gatsby Cloud will:

  1. Build the site by running yarn build:production.
  2. If the build succeeds, deploy the new version.
  3. If the build fails, it will keep the previous version deployed.

Additionally, whenever a pull request is created, a "preview" environment will be spun up. A link for this environment can be found as a comment on your pull request once it finishes building. You can monitor it via the Gatsby Build Service job, located in the Checks section.

For more information about our CD settings, check out the site settings in Gatsby Cloud. For information about other CI processes, see Github actions below.

Gatsby Cloud Bot

Gatsby Cloud uses the Gatsby Cloud Slack Bot to send event notifications to 2 different internal Slack channels:

  • #docs_site_deploys
  • #dev-enablement-bots

The Bot will notify users of:

  • Build successes
  • Build failures
  • Deploy successes
  • Deploy failures

🖥 Local development

To run the project locally, you will need the following dependencies:

  • Node 12.x
  • Yarn

To get the project running locally, follow these steps:

  1. Clone the repository.
  2. Install the dependencies by running yarn.
  3. Start the development server by running yarn start.

The first time you start the server on a machine, it will take a while to run. Subsequent runs should be faster (Gatsby caching).

For more information, see CONTRIBUTING.md.

🗂 Directory structure

Here is a list of the commonly-used directories (see below for a description of each):

.github/workflows/
codemods/
gatsby/
plugins/
rehype-plugins/
scripts/actions/
src/
  | @newrelic/gatsby-theme-newrelic/
  | announcements/
  | components/
  | content/
      | docs/
      | whats-new/
  | data-dictionary/events/
  | data/
  | hooks/
  | i18n/
      | content/
      | nav/
      | translations/
  | layouts/
  | manual-edits/
  | nav/
  | pages/
  | templates/
  | utils/
Path Description
.github/workflows/ Configuration for our Github actions (see below)
codemods/ Files & utilities used when migrating from the previous site
gatsby/ Gatsby lifecycle functions
plugins/ Custom Gatsby plugins
rehype-plugins/ Plugins used by rehype (HTML AST parser)
scripts/actions/ Node scripts used by Github actions
src/@newrelic/gatsby-theme-newrelic/ Shadowing (replacing) parts of our theme
src/announcements/ MDX files that power our Announcement Banner
src/components/ Docs site-specific React components
src/content/docs/ Main site content (MDX)
src/content/whats-new/ Whats new posts (grouped by date)
src/data-dictionary/events/ Content that powers the Attribute Dictionary
src/data/ Misc. site configuration and state management
src/i18n/content/ Site content in languages other than English
src/i18n/nav/ JSON files for navigation in other languages
src/i18n/translations JSON files for individual translation strings
src/layouts/ React components that control the overall site layout
src/manual-edits/ Content that was adjusted manually during pre-migration development
src/nav/ YAML files for configuring the site navigation
src/pages/ Unique top-level pages (e.g. the 404 page)
src/templates/ React components that control page-specific layout
src/utils/ Constants and common helper utilities

🤖 Github actions

We use Github actions to perform a number of CI-related tasks. The workflow files are defined via YAML in .github/workflows. Many of them rely on scripts that can be found in scripts/actions/.

NOTE: The translated-related jobs are documented further in an internal document.

Gatsby Build Service

Workflow file Trigger
Provided by the Gatsby Cloud integration PRs to and commits pushed to develop and main

Our integration with Gatsby Cloud provides a build job to ensure that the site can be built before a merge and provides a preview build of the branch. Once the build has finished, the gatsby-cloud bot will leave a comment on the associated PR with links to the preview and logs. You can always view the build progress by finding the job under the Checks section of your PR, clicking Details, and then following the link to Gatsby Cloud.

Build with Swiftype content

Workflow file Trigger
build-with-swiftype-content.yml Not currently running

This will run at regular intervals and gather "related resources" for each page using Swiftype. This is done by running a script (yarn run build:related-content), checking in some additional files, and pushing the new content to main. We have a similar action for the developer website that runs every day.

Add slugs to translation queue

Workflow file Trigger
get-slugs-to-translate.yml A pull request, with MDX files, is merged into main

This action looks at any PR that contains MDX files. If any of the MDX files have translate in frontmatter, it will save the path to the file in a "to be translated" queue. This queue is sent to our translation vendor via another action (see below).

All of the translation-related workflows use AWS DynamoDB to store state between job runs.

Send content to be translated

Workflow file Trigger
send-content-to-translate.yml Not currently running

This will run at regular intervals and send everything in the "to be translated" queue to our translation vendor. This allows us to batch translation requests rather than sending a new request for every edit.

All of the translation-related workflows use AWS DynamoDB to store state between job runs.

Check status of translation jobs

Workflow file Trigger
check-translations-and-deserialize.yml Not currently running

This will run at regular intervals and check if there are any pages that have been successfully translated by our vendor. If any translation jobs are complete, this will pull down the new content, de-serialize it, and create a new pull request with the translated content.

All of the translation-related workflows use AWS DynamoDB to store state between job runs.

Release

Workflow file Trigger
release.yml A pull request is merged into main

Creates a GitHub release whose release notes contain descriptions of changes, and the author that made the change.

Sync data dictionary

Workflow file Trigger
sync-data-dictionary.yml Daily (12am)

Syncs the attribute definition data from the repository to the data dictionary service.

Validate Pull Request

Workflow file Trigger
validate-pr.yml Pull request created into develop or main

Builds the site, runs a test suite, and lints the code. This is run on every pull request and is used as an automated way to check if a pull request is "passing".

Auto-assign reviewer

Workflow file Trigger
auto_assign.yml Pull request created or marked ready-for-review, and also labeled from_tw

Automatically assigns a member of the documentation team as reviewer.

✅ Content Translation workflow

Below is a high level explanation of the docs site translation workflow. If you need further details on this flow, and work for New Relic, please contact the Developer Enablement team.

  • 😀 A PR is opened with a MDX file that contains a translate flag
  • 😀 The PR is merged into the main branch
    • 🤖 A Github action is run to save the file to a queue
    • 🤖 Once per week, a Github action runs to send the files to our translation vendor
    • 🤖 Once per day, a Github action checks the status of open translation jobs
    • 🤖 If a job is complete, a Github action makes a PR with the new content
  • 😀 The PR is reviewed and merged into main
    • 🤖 The new translated content is automatically deployed

✨ "Docs as a Service" services / dependencies

Docs Eng Site Architecture@2x

View source on New Relic Whimsical

Diagram of dependent docs services and NR UI. View/edit on Excalidraw

Docs JSON

We generate a JSON file for every MDX file in src/content/docs with a body field containing an HTML string. That HTML string is a generic conversion of MDX. Those files used by the Docs Library Service to stitch that data into NerdGraph, which is then queried/rendered by the nr1-help-xp nerdlet. That nerdlet supports searching and rendering all docs content.

Example JSON file: https://docs.newrelic.com/docs/new-relic-solutions/get-started/intro-new-relic.json

Data dictionary JSON

The docs site generates one giant JSON file with all event and attribute data for the Data Dictionary Service to sync down daily.

Endpoint is https://docs.newrelic.com/data-dictionary-1.json

The Docs site has JSON and XML data endpoints that can be used however you see fit. These endpoints serve up docs content in a standard easy to use format.

Release notes JSON

We generate one big JSON file with all agent release note data in it. This is fed to a service (unclear which) that I believe is stitched into Nerdgraph at docs.agentReleases

https://docs.newrelic.com/api/agent-release-notes.json

Nerdgraph query example

agentReleases(agentName: BROWSER) {
  date
  version
}

What's New NR1 Announcements

What's new announcements highlight new product features in the New Relic One Platform.

https://docs.newrelic.com/api/nr1/content/nr1-announcements.json

This endpoint can be used to fetch the entire NR1 announcement.

https://docs.newrelic.com/api/nr1/content/nr1-announcements/ids.json

This endpoint can be used to fetch just the ID of the NR1 announcement.

Release Notes RSS Feed

All Agents have an XML endpoint that can be used for RSS readers to fetch release notes data.

Example XML endpoint

https://docs.newrelic.com/docs/release-notes/agent-release-notes/{agent_type}/feed.xml

These endpoints can be used to fetch all release notes for an agent type.

Security Bulletins RSS Feed

All security bulletin content is available in an RSS feed.

https://docs.newrelic.com/docs/security/new-relic-security/security-bulletins/feed.xml

This endpoint can be used to fetch all security bulletins.

What's New RSS Feed

All What's New content in src/content/whats-new is available in an RSS feed.

https://docs.newrelic.com/whats-new/feed.xml

Clone this wiki locally