Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

isom-1719 non-prod sites can be indexed by search engine #987

Conversation

adriangohjw
Copy link
Contributor

Problem

Closes https://linear.app/ogp/issue/ISOM-1719/test-sites-can-be-indexed-by-search-engine

while staging sites have disallow in robots.txt, according to google

A page that's disallowed in robots.txt can still be indexed if linked to from other sites.
While Google won't crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google search results. To properly prevent your URL from appearing in Google search results, password-protect the files on your server, use the noindex meta tag or response header, or remove the page entirely.

Solution

Breaking Changes

  • Yes - this PR contains breaking changes
    • Details ...
  • No - this PR is backwards compatible

Improvements:

  • add shouldBlockIndexing based on site's environment
  • pass in true/false into meta.noIndex for /page.tsx based on shouldBlockIndexing

Tests

trigger a new codebuild, and inspect the HTML. It should/should not have <meta name="robots" content="noindex"> based on the table below

type of site should have noindex in HTML
PROD site no
PROD staging site yes
STG/UAT site yes

@adriangohjw adriangohjw added the bug Something isn't working label Jan 13, 2025
@adriangohjw adriangohjw self-assigned this Jan 13, 2025
@adriangohjw adriangohjw requested a review from a team as a code owner January 13, 2025 17:48
Copy link

linear bot commented Jan 13, 2025

@datadog-opengovsg
Copy link

datadog-opengovsg bot commented Jan 13, 2025

Datadog Report

Branch report: adriangohjw/isom-1719-test-sites-can-be-indexed-by-search-engine
Commit report: 748d4af
Test service: isomer-studio

✅ 0 Failed, 257 Passed, 36 Skipped, 47s Total Time
➡️ Test Sessions change in coverage: 1 no change

Comment on lines 114 to 119
export const shouldBlockIndexing = (
environment: IsomerPageSchemaType["site"]["environment"],
): boolean => {
return environment === "staging"
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shuold we make this more restrictive? we might hvae environments that are non-production which shouldn't be indexed also like vapt or uat

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right now we only support production / staging (generated from our infra)

image

we can make it more generic to return true if it DOES NOT equate to production since we don't know how many non-production sites are there (could have more than staging, vapt, uat in the future too)

WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea sgtm!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated in e810697, thanks!

@adriangohjw adriangohjw enabled auto-merge (squash) January 14, 2025 10:01
@adriangohjw adriangohjw merged commit 63e410a into main Jan 14, 2025
18 checks passed
@adriangohjw adriangohjw deleted the adriangohjw/isom-1719-test-sites-can-be-indexed-by-search-engine branch January 14, 2025 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants