Skip to content
This repository has been archived by the owner on Dec 15, 2021. It is now read-only.

Revise build process to reduce amount of file moves and preparation #155

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Revise build process to reduce amount of file moves and preparation #155

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
611fd8e
Revise build process to reduce amount of file moves and preparation
Apr 2, 2020
828125c
Update scripts/processsourcefiles.sh
evankanderson Apr 3, 2020
4e90dfc
Merge branch 'master' into rebuild
RichieEscarez Jul 25, 2020
6c9b924
Try to get the archive builds to work
RichieEscarez Jul 25, 2020
29af6cd
Reduce the need to set versions for each release in the scripts
RichieEscarez Aug 1, 2020
072a706
Ignore the "archived" docs that get cloned locally (localbuild.sh)
RichieEscarez Aug 1, 2020
56e0e0e
a few changes
RichieEscarez Aug 1, 2020
192a8a1
fix typo from copy/paste
RichieEscarez Aug 1, 2020
c5f0138
Add the .md and README logic to our readfile.md shortcode
RichieEscarez Aug 8, 2020
757e242
Add a comment to explain what we are doing here
RichieEscarez Aug 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@ node_modules/
public/
content/
resources/
docs/
.DS_Store
19 changes: 18 additions & 1 deletion layouts/_default/content.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,24 @@
<div class="td-content">
<h1>{{ .Title }}</h1>
{{ with .Params.description }}<div class="lead">{{ . | markdownify }}</div>{{ end }}
{{ .Content }}
{{ $out := .Content}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you also test the local build too? Early on there was a lot of discrepancies between how Netlify built using Hugo and how the default local Hugo server built our content, thus the move to "building" the same way in both places.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I've been using local build to test.

{{ if not ( in (slice "README.md" "_index.md") .File.LogicalName ) }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this exclude hardcoded links to other non-Knative Repo's README.md files? Im not sure we still have or need to accommodate those links that point out to another Repo's README file today but maybe we should avoid changing those URLs if or when someone does add on in the future?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is determining if this file's name is "README.md" or "_index.md". But I've been adding "index.md" files, so I probably need to update this.

Thanks for asking!

<!-- non-README.md files will be rendered one level below normal, fix links in them -->
{{ $out = $out | replaceRE "href=\"\\.\\./" "href=\"../../" }}
{{ $out = $out | replaceRE "href=\"\\./" "href=\"../"}}
{{ end }}
<!-- For SEO and aesthetics purposes, clean up GitHub URLs; this is
repeating the hugo "Pretty URLs" pattern on _relative_ links in the
source material.

The regexp ([^:\"]*(/[^\"]*)?) matches:
- Zero or more non-: or " charaters (: is needed for an absolute URL)
- Followed by a "/" and zero or more non " characters
(which would end the href attribute)
-->
{{ $out = $out | replaceRE "href=\"([^:\"]*(/[^\"]*)?)\\.md\"" "href=\"$1\""}}
{{ $out = $out | replaceRE "href=\"([^:\"]*(/[^\"]*)?)README\"" "href=\"$1\"" }}
{{ $out | safeHTML}}
{{ if (and (not .Params.hide_feedback) (.Site.Params.ui.feedback.enable) (.Site.GoogleAnalytics)) }}
{{ partial "feedback.html" .Site.Params.ui.feedback }}
<br />
Expand Down
6 changes: 5 additions & 1 deletion layouts/shortcodes/readfile.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,11 @@ Parameters:
{{ else }}

{{/* If HTML or Markdown. For Markdown`{{%...%}}`, don't send content to processor again (use safeHTML). */}}
{{ $.Scratch.Get "filepath" | readFile | safeHTML }}
{{/* Look for part of the Markdown link syntax ']' and remove '.md' or 'README' from within those links. */}}
{{ $out := $.Scratch.Get "filepath" | readFile }}
{{ $out = $out | replaceRE "]([^)]+).md" "]$1" }}
{{ $out = $out | replaceRE "]([^)]+)README" "]$1" }}
{{ $out | safeHTML }}

{{ end }}

Expand Down
3 changes: 3 additions & 0 deletions scripts/convert-repo-ulrs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
# THIS FILE IS USED BY THE 'processsourcefiles.sh' SCRIPT TO CONVERT URLS TO RELATIVE #
#######################################################################################

# TODO(Evan): move this processing into golang, probably via
# layouts/_default/content.html and/or shortcodes.

echo '------ CONVERT FULLY QUALIFIED REPO URLS TO RELATIVE ------'
# Convert fully-qualified URLs that are used in the Knative GitHub repos source files
# to link from repo to repo, into relative URLs for publishing to the knative.dev site.
Expand Down
6 changes: 6 additions & 0 deletions scripts/docs-version-settings.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,11 @@
# This should be set to the latest docs release/branch
DEFAULTBRANCH="release-0.16"

# Latest release version number
LATESTVERSION="16"
# Total number of past versions to publish
NUMOFVERSIONS="3"
OLDESTVERSION=$((LATESTVERSION-NUMOFVERSIONS))

# An optional value that you can locally override for local builds/testing
DEFAULTFORK="knative"
81 changes: 36 additions & 45 deletions scripts/processsourcefiles.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,25 +30,38 @@ then
mv content/en/docs content/en/development
# DOCS BRANCHES
echo '------ Cloning all docs releases ------'

# TODO(Evan): Avoid the need for `mv` above and make the below just ap
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to use mv because I couldnt figure out how to make "git clone" happy when trying to extract only the /docs/ folders out of each knative/docs archive branch (there are all the other files and folders in the repo that we dont want to end up in the site build).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I clone the docs outside of /content/en, and then symlink deeper into the tree. Since the clone isn't in /content/en, it isn't published at all.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice. this then renders my concern below needless (content not in the publishing path wont get picked up automatically)

# set of clone + symlink commands

# Get versions of released docs from their branches in "$FORK"/docs
echo 'The /docs/ section is built from the' "$BRANCH" 'branch of' "$FORK"
# Latest version is defined in website/scripts/docs-version-settings.sh
# If this is a PR build, then build that content as the latest release (assume PR preview builds are always from "latest")
git clone --quiet -b "$BRANCH" https://github.com/"$FORK"/docs.git temp/release/latest
# Only copy and keep the "docs" folder from all branched releases:
mv temp/release/latest/docs content/en/docs
echo 'Getting the archived docs releases from branches in:' "$FORK"'/docs'
###############################################################
# Template for next release:
#git clone -b "release-[VERSION#]" https://github.com/"$FORK"/docs.git temp/release/[VERSION#]
#mv temp/release/[VERSION#]/docs content/en/[VERSION#]-docs
###############################################################
git clone --quiet -b "release-0.15" https://github.com/"$FORK"/docs.git temp/release/v0.15
mv temp/release/v0.15/docs content/en/v0.15-docs
git clone --quiet -b "release-0.14" https://github.com/"$FORK"/docs.git temp/release/v0.14
mv temp/release/v0.14/docs content/en/v0.14-docs
git clone --quiet -b "release-0.13" https://github.com/"$FORK"/docs.git temp/release/v0.13
mv temp/release/v0.13/docs content/en/v0.13-docs
mkdir -p docs # -p won't fail if the file exists
r=$OLDESTVERSION
while [[ $r -le $LATESTVERSION ]]
do
CLONE="docs/release-0.${r}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be a variable too for that day when Knative graduates to a MAJOR version?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When that happens, I hope to rewrite more of this anyway.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ive changed this to use the values set in https://github.com/knative/website/blob/master/scripts/docs-version-settings.sh (so on release time, this file does not need to get updated to include the recent version number)

if [[ -e "$CLONE" ]]
then
echo "Skipping ${CLONE}, a copy of those archived docs already exists."
ln -s ../../"$CLONE"/docs content/en/v0."$r"-docs #always recreate symlinks because /content/en always gets wiped out
else
echo 'Getting docs from: release-0.'"${r}"
git clone --quiet -b "release-0.${r}" "https://github.com/${FORK}/docs.git" "$CLONE"
if [ "$r" = "$LATESTVERSION" ]
then
echo 'The /docs/ section is built from:' "$FORK"'/release-0.'"${r}"
mv "$CLONE"/docs content/en/docs
rm -rf "$CLONE" #always get the latest release
else
# Only use the "docs" folder from all branches/releases
ln -s ../../"$CLONE"/docs content/en/v0."$r"-docs
fi
fi
(( r = r + 1 ))
done

elif [ "$BUILDSINGLEBRANCH" = "true" ]
then
Expand All @@ -64,6 +77,9 @@ else
echo '------ BUILDING ONLY FROM YOUR LOCAL KNATIVE/DOCS CLONE ------'
echo 'Copying local clone of knative/docs into the /docs folder under:'
pwd

# TODO(Evan): switch this to just be symlinks?

cp -r ../docs content/en/
if [ -d "../community" ]; then
echo 'Also copying the local clone of knative/community into the /community/contributing folder.'
Expand All @@ -77,6 +93,7 @@ if [ "$LOCALBUILD" = "false" ]
then
echo '------ Cloning contributor docs ------'
# COMMUNITY
# TODO / Known issue: Auto PR builds will fail if remote fork excludes knative/community -> https://app.netlify.com/sites/knative/deploys
echo 'Getting Knative contributor guidelines from the master branch of' "$FORK"'/community'
git clone --quiet -b master https://github.com/"$FORK"/community.git temp/community
# Move files into existing "contributing" folder
Expand All @@ -97,40 +114,14 @@ source scripts/convert-repo-ulrs.sh
#########################################################
# Process content in .md files (MAKE RELATIVE LINKS WORK)
# We want users to be able view and use the source files in GitHub as well as on the site.
# Therefore, the following changes need to be made to all docs files prior to Hugo site build.
# Convert GitHub enabled source, into HUGO supported content:
# - For all Markdown files under the /content/ directory:
# - Skip all:
# - Markdown link with fully qualified HTTP(s) URL is 'external'
# - GitHub file (.git* files)
# - non-docs directories
# - Ignore Hugo site related files (avoid "readfile" shortcodes):
# - _index.md files (Hugo 'section' files)
# - API shell files (serving-api.md, eventing-contrib-api.md, eventing-api.md)
# - For all remaining Markdown files:
# - Remove all '.md' file extensions from within Markdown links "[]()"
# - For SEO convert README to index:
# - Replace all in-page URLS from "README.md" to "index.html"
# - Rename all files from "README.md" to "index.md"
# - Adjust relative links by adding additional depth:
# - Exclude all README.md & _index.md files
# - Convert './' to '../'
# - Convert '../' to '../../'
echo 'Converting all links in GitHub source files to Hugo supported relative links...'
# Convert relative links to support Hugo
find . -type f -path '*/content/*.md' ! -name '*_index.md' ! -name '*README.md' \
! -name '*serving-api.md' ! -name '*eventing-contrib-api.md' ! -name '*eventing-api.md' \
! -name '*build-api.md' ! -name '*.git*' ! -path '*/.github/*' ! -path '*/hack/*' \
! -path '*/node_modules/*' ! -path '*/test/*' ! -path '*/themes/*' ! -path '*/vendor/*' \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im very close to Go illiterate but I didnt see these files obviously ignored up in the Hugo layout?

For anything here that is explicitly called out, its because Hugo will build that file into the site and add empty/no title Nav entries into our TOC. They rendered as blank spaces and are clickable but since they lacked front matter (to either add the missing meta or explicitly exclude it from the build) they get added to the site in unwanted and conflicting ways.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I figured out what these actually mapped to and made sure that they rendered correctly as best as I could tell.

-exec sed -i '/](/ { s#(\.\.\/#(../../#g; s#(\.\/#(../#g; }' {} +
# Convert all relative links from README.md to index.html
find . -type f -path '*/content/*.md' ! -name '_index.md' \
-exec sed -i '/](/ { /http/ !{s#README\.md#index.html#g;s#\.md##g} }' {} +
#
# See layouts/_default/content.html for how this is done.

###############################################
# Process file names (HIDE README.md FROM URLS)
# For SEO, dont use "README" in the URL
# (convert them to index.md OR use a "readfile" shortcode to nest them within a _index.md section file)
# See also https://github.com/gohugoio/hugo/issues/4691 which would resolve this.
#
# Notes about past docs versions:
# v0.6 and earlier doc releases: Use the "readfile" shortcodes to nest all README.md files within the _index.md files.
Expand All @@ -152,7 +143,7 @@ find . -type f -path '*/content/*/*/*' -name 'README.md' \
! -path '*/contributing/*' ! -path '*/v0.6-docs/*' ! -path '*/v0.5-docs/*' \
! -path '*/v0.4-docs/*' ! -path '*/v0.3-docs/*' ! -path '*/.github/*' ! -path '*/hack/*' \
! -path '*/node_modules/*' ! -path '*/test/*' ! -path '*/themes/*' ! -path '*/vendor/*' \
-execdir bash -c 'if [ -e _index.md -o -e index.md ]; then echo "_index.md exists - skipping ${PWD#*/}"; else mv "$1" "${1/\README/\index}"; fi' -- {} \;
-execdir bash -c 'if [ -e index.md -o -e _index.md ]; then echo "index.md or _index.md exists - skipping ${PWD#*/}"; else mv "$1" "${1/README/index}"; fi' -- {} \;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesnt apply here but I thought I'd mention that Hugo does weird stuff when both an index.md and an _index.md file exist in the same directory (the build only works as expected when only one or the other exist -> an index.md file can be used and reside alone in a folder if thats the only markdown file being build from that folder).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just avoiding overwriting an index.md with README.md. This actually went in #161 to support knative/docs#2377


# GET HANDCRAFTED SITE LANDING PAGE
echo 'Copying the override files into the /content/ folder'
Expand Down