Skip to content

Site Architecture

Jump to bottom
Clinton edited this page Aug 30, 2023 · 64 revisions

!! This has a new home in Confluence !!

Table of Contents

🥞 Stack

At a high level, docs.newrelic.com is built with React and MDX. It uses Gatsby as a static site generator and is hosted on Gatsby Cloud.

🎸 JAMstack

JAM stands for Javascript, APIs & Markup. A core part of the site architecture is Gatsby which is used as a static-site generator. Pages are built by combining Markup (MDX) and Javascript (React) and hosted as static HTML files.

There is no back-end or CMS powering the site.

🛠 Tools

Tool Description
Gatsby Static site generator
newrelic/gatsby-theme-newrelic Shared New Relic components and site configuration (see below)
MDX Markdown + React components for authoring content
React Custom components & view layer
GraphQL Querying content within the site
Emotion CSS in JS (styles)
Github Actions Task automation (see below)
Gatsby Cloud Hosting & deployments (see below)
Yarn Package management & task runner

🔌 Gatsby plugins

In addition to the technologies listed above, we use a number of Gatsby plugins. In addition to established third-party plugins, we've also created a few custom plugins to handle specific tasks. For a complete list of plugins, check out our package.json file.

🎨 Gatsby theme

newrelic/gatsby-theme-newrelic

New Relic has developed a Gatsby theme which includes components and configuration shared across several website. The Open Source and Developer site both make use of this theme. For more information on how to use the components and configuration, refer to the README.

🌐 Hosting and deployment

Gatsby Cloud Hosting

The site is currently hosted on Gatsby Cloud (and is connected to the repository via a webhook). To deploy a new version of the site, simply merge into main (our mainline branch). Gatsby Cloud will:

  1. Build the site by running yarn build:production.
  2. If the build succeeds, deploy the new version.
  3. If the build fails, it will keep the previous version deployed.

Additionally, whenever a pull request is created, a "preview" environment will be spun up. A link for this environment can be found as a comment on your pull request once it finishes building. You can monitor it via the Gatsby Build Service job, located in the Checks section.

For more information about our CD settings, check out the site settings in Gatsby Cloud. For information about other CI processes, see Github actions below.

Gatsby Cloud Bot

Gatsby Cloud uses the Gatsby Cloud Slack Bot to send event notifications to 2 different internal Slack channels:

  • #doc_eng_bots

The Bot will notify users of:

  • Build successes
  • Build failures
  • Deploy successes
  • Deploy failures

🖥 Local development

To run the project locally, you will need the following dependencies:

  • Node 16.x
  • Yarn

To get the project running locally, follow these steps:

  1. Clone the repository.
  2. Install the dependencies by running yarn.
  3. Start the development server by running yarn start.

The first time you start the server on a machine, it will take a while to run. Subsequent runs should be faster (Gatsby caching).

For more information, see CONTRIBUTING.md.

🗂 Directory structure

Here is a list of the commonly-used directories (see below for a description of each):

.github/workflows/
gatsby/
plugins/
scripts/actions/
src/
  | @newrelic/gatsby-theme-newrelic/
  | announcements/
  | components/
  | content/
      | docs/
      | whats-new/
  | data-dictionary/events/
  | data/
  | hooks/
  | i18n/
      | content/
      | nav/
      | translations/
  | layouts/
  | nav/
  | pages/
  | templates/
  | utils/
Path Description
.github/workflows/ Configuration for our Github actions (see below)
gatsby/ Gatsby lifecycle functions
plugins/ Custom Gatsby plugins
scripts/actions/ Node scripts used by Github actions
src/@newrelic/gatsby-theme-newrelic/ Shadowing (replacing) parts of our theme
src/announcements/ MDX files that power our Announcement Banner
src/components/ Docs site-specific React components
src/content/docs/ Main site content (MDX)
src/content/whats-new/ Whats new posts (grouped by date)
src/data-dictionary/events/ Content that powers the Attribute Dictionary
src/data/ Misc. site configuration and state management
src/i18n/content/ Site content in languages other than English
src/i18n/nav/ JSON files for navigation in other languages
src/i18n/translations JSON files for individual translation strings
src/layouts/ React components that control the overall site layout
src/nav/ YAML files for configuring the site navigation
src/pages/ Unique top-level pages (e.g. the 404 page)
src/templates/ React components that control page-specific layout
src/utils/ Constants and common helper utilities

🤖 Github actions

We use Github actions to perform a number of CI-related tasks. The workflow files are defined via YAML in .github/workflows. Many of them rely on scripts that can be found in scripts/actions/.

NOTE: The translated-related jobs are documented further in an internal document.

Gatsby Build Service

Workflow file Trigger
Provided by the Gatsby Cloud integration PRs to and commits pushed to develop and main

Our integration with Gatsby Cloud provides a build job to ensure that the site can be built before a merge and provides a preview build of the branch. Once the build has finished, the gatsby-cloud bot will leave a comment on the associated PR with links to the preview and logs. You can always view the build progress by finding the job under the Checks section of your PR, clicking Details, and then following the link to Gatsby Cloud.

Add slugs to translation queue

Workflow file Trigger
get-slugs-to-translate.yml A pull request, with MDX files, is merged into main

This action looks at any PR that contains MDX files. If any of the MDX files have translate in frontmatter, it will save the path to the file in a "to be translated" queue. This queue is sent to our translation vendor via another action (see below).

All of the translation-related workflows use AWS DynamoDB to store state between job runs.

Send content to be translated

Workflow file Trigger
send-content-to-translate.yml Not currently running

This will run at regular intervals and send everything in the "to be translated" queue to our translation vendor. This allows us to batch translation requests rather than sending a new request for every edit.

All of the translation-related workflows use AWS DynamoDB to store state between job runs.

Check status of translation jobs

Workflow file Trigger
check-translations-and-deserialize.yml Not currently running

This will run at regular intervals and check if there are any pages that have been successfully translated by our vendor. If any translation jobs are complete, this will pull down the new content, de-serialize it, and create a new pull request with the translated content.

All of the translation-related workflows use AWS DynamoDB to store state between job runs.

Release

Workflow file Trigger
release.yml A pull request is merged into main

Creates a GitHub release whose release notes contain descriptions of changes, and the author that made the change.

Sync data dictionary

Workflow file Trigger
sync-data-dictionary.yml Daily (12am)

Syncs the attribute definition data from the repository to the data dictionary service.

Validate Pull Request

Workflow file Trigger
validate-pr.yml Pull request created into develop or main

Builds the site, runs a test suite, and lints the code. This is run on every pull request and is used as an automated way to check if a pull request is "passing".

Auto-assign reviewer

Workflow file Trigger
auto_assign.yml Pull request created or marked ready-for-review, and also labeled from_tw

Automatically assigns a member of the documentation team as reviewer.

Generate Agent Release Notes

  • On push to main:
    • Converts a select amount of .mdx release notes files on docs website and turns them into a JSON file
    • Uploads JSON file to S3 bucket with custom domain name
    • NOTE: Known issue where release notes data is updated before latest release note is published

✅ Content Translation workflow

Below is a high level explanation of the docs site translation workflow. If you need further details on this flow, and work for New Relic, please contact the Developer Enablement team.

  • 😀 A PR is opened with a MDX file that contains a translate flag
  • 😀 The PR is merged into the main branch
    • 🤖 A Github action is run to save the file to a queue
    • 🤖 Once per week, a Github action runs to send the files to our translation vendor
    • 🤖 Once per day, a Github action checks the status of open translation jobs
    • 🤖 If a job is complete, a Github action makes a PR with the new content
  • 😀 The PR is reviewed and merged into main
    • 🤖 The new translated content is automatically deployed

✨ "Docs as a Service" services / dependencies

Docs Eng Site Architecture@2x (3)

View source on New Relic Whimsical

Doc page embeds

We no longer generate JSON for every MDX file in src/content/docs. Those embed URLs are used by the Docs Library Service, which is stitched into NerdGraph, which is then queried/rendered by the help-xp nerdlet. That nerdlet supports searching and rendering all docs content. That nerdlet is imminently migrating away from Docs Library Service to use SearchUnify and only display links to docs. Once they migrate, we will remove embed pages.

Example embed: https://docs.newrelic.com/docs/new-relic-solutions/get-started/intro-new-relic/embed

Data dictionary JSON

The docs site generates one giant JSON file with all event and attribute data for the Data Dictionary Service to sync down daily.

The data dictionary is stitched into NerdGraph and is used by Query Builder UI to display event and attribute definitions.

Endpoint is https://docs.newrelic.com/data-dictionary-1.json

Release notes JSON

The Docs site has a directory with all agent release notes. These notes are consumed and formatted into a JSON object for New Relic to reference as it comes to updates to various releases. The notes are hosted at this url: https://docs-assets.newrelic-external.com/release-notes-json which is a custom domain name pointing to an S3 bucket.

The release notes JSON is generated and placed into the S3 via this workflow: https://github.com/newrelic/docs-website/actions/workflows/agent-release-notes.yml.

Dependencies

  • The Agent update recommendation services uses this endpoint to store the agent release notes json in NRDB via the service apm-services/agent-update-recommendations

  • Nerdgraph hits this endpoint and adds the data to be queryable in the NR api via this file in GHE in the after/nerd-graph repo: apps/http_services/lib/services/doc_site_api.ex

    • A key examples using this data in nerdgraph are the open-source/nr1-install-newrelic repo.
    • Another example is the onyx/shared-component-agent-update

  • Dashboard that shows number of requests

Nerdgraph query example

docs {
    agentReleases(agentName: NODEJS) {
      eolDate
      date
    }
  }

What's New NR1 Announcements

What's new announcements highlight new product features in the New Relic One Platform.

https://docs.newrelic.com/api/nr1/content/nr1-announcements.json

This endpoint can be used to fetch the entire NR1 announcement.

https://docs.newrelic.com/api/nr1/content/nr1-announcements/ids.json

This endpoint can be used to fetch just the ID of the NR1 announcement.

Release Notes RSS Feed

All Agents have an XML endpoint that can be used for RSS readers to fetch release notes data.

Example XML endpoint

https://docs.newrelic.com/docs/release-notes/agent-release-notes/{agent_type}/feed.xml

These endpoints can be used to fetch all release notes for an agent type.

Security Bulletins RSS Feed

All security bulletin content is available in an RSS feed.

https://docs.newrelic.com/docs/security/new-relic-security/security-bulletins/feed.xml

This endpoint can be used to fetch all security bulletins.

What's New RSS Feed

All What's New content in src/content/whats-new is available in an RSS feed.

https://docs.newrelic.com/whats-new/feed.xml

Clone this wiki locally