Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uncover and Fix Cyclic Dependency, Version-Control Vercel Configs, Fix Turbo Graph. #2502

Merged
merged 8 commits into from
Apr 28, 2023

Conversation

Alfred-Mountfield
Copy link
Contributor

@Alfred-Mountfield Alfred-Mountfield commented Apr 28, 2023

🌟 What is the purpose of this PR?

This fixes a number of issues uncovered within the repository while trying to get #2493 in.

  • There was a cyclic dependency between packages that share GraphQL types (@apps/hash-api, @local/hash-graphql-shared, etc.). This was largely hidden because they were within GraphQL configs, and relied on the file-system, which meant that the dependencies were not within the package.json and not visible to turborepo.

    • While a circular dependency between packages is implicitly a problem by itself, it also surfaces when we try to rely on things like turbo prune
    • This was fixed by hoisting the types to a shared package as they should have been, rather than being defined within an 'app' and imported elsewhere
  • The configuration for our Vercel deployments was managed purely through Vercel's UI. This means that the configuration (such as build commands) was not version-controlled, and importantly it is shared between all new deployments. Thus, if a PR requires a change to the configuration, that change will break all other in-flight PRs.

    • This is fixed by including a vercel.json in the individual projects. This vercel.json works because the root directory is set within the project configuration
    image
    • For the hash-frontend, additional shell scripts have been added in preparation of requiring some more complicated setup, including installing Python on the Vercel machine.
  • Requirements were missing from the README such as Python and Java

  • The changes introduced in Introduce the block design system #2362 accidentally missed some required updates to the Turbo graph. These have been fixed in this PR, and:

  • The Pull Request template has been updated (to this one) to result in an updated checklist of things to look at pre-merge, which now includes a section on Turborepo.

🔗 Related links

🚫 Blocked by

N/A

🔍 What does this change?

Please see commits.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • requires changes to docs
    • the README has been updated to include missing dependencies
    • the pull request template has been updated to contain a bigger check-list

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • affected the execution graph, and the turbo.json's have been updated to reflect this

⚠️ Known issues

  • Our frontend build is eating up the cache on Vercel. The webpack cache is filling up with hundreds of .pack files that persist across runs instead of being overridden. This means we can have well over 8GB of disk being used up which results in failed deployments. This can be mitigated by manually redeploying the Vercel deployment through the UI and choosing not to re-use the cache. - Slack Thread (internal)
  • Attempting to use turbo prune within the Vercel build results in an internal server error. This is probably due to us deleting some of Vercel's files that they are putting in the same root directory as the source code. The attempt at using git to remove only git tracked files did not seem to work.

🐾 Next steps

🛡 What tests cover this?

  • Tooling changes, installations, lints, etc. Basically all tests, so please look at CI

❓ How to test this?

  • See CI and deployments
  • Attempt various turbo graphs locally should you wish (such as running the storybook)

📹 Demo

This PR

@codecov
Copy link

codecov bot commented Apr 28, 2023

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 56.08%. Comparing base (6a32aeb) to head (249cefc).
Report is 2979 commits behind head on main.

Files with missing lines Patch % Lines
apps/hash-api/src/graphql/create-apollo-server.ts 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2502      +/-   ##
==========================================
- Coverage   56.29%   56.08%   -0.21%     
==========================================
  Files         340      330      -10     
  Lines       26380    26624     +244     
  Branches      384      425      +41     
==========================================
+ Hits        14850    14933      +83     
- Misses      11529    11686     +157     
- Partials        1        5       +4     
Flag Coverage Δ
antsi 0.37% <ø> (ø)
backend-integration-tests 3.66% <ø> (ø)
deer 68.40% <ø> (+0.30%) ⬆️
error-stack 82.56% <ø> (ø)
hash-graph 61.06% <ø> (ø)
hash-status 2.70% <ø> (ø)
sarif 100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

- [ ] modifies a **block** that will need publishing via GitHub action once merged
- [ ] does not modify any publishable blocks or libraries, or modifications do not need publishing
- [ ] I am unsure / need advice
- [ ] **⚠️ This is a placeholder, please uncomment at least one of the lines following this one and then delete this. ⚠️**
Copy link
Contributor Author

@Alfred-Mountfield Alfred-Mountfield Apr 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opted for this pattern to try and solve your personal bugbear, @CiaranMn, about the number of "incomplete tasks" that appear for given PRs. This way we should be able to make sure that every checkbox has been checked

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before
image

After
image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

omg yes, this was bothering me so much

Comment on lines 1 to 24
#!/usr/bin/env bash

# Setup TurboRepo and get a pruned src folder and lockfile

echo "Installing turbo"
yarn global add turbo

# TODO: investigate why producing a pruned repo results in a broken Vercel build

#echo "Producing pruned repo"
#turbo prune --scope='@apps/hash-frontend'
#
#echo "Deleting contents of non-pruned dir to save space"
#git ls-files -z | xargs -0 rm -f
#git ls-tree --name-only -d -r -z HEAD | sort -rz | xargs -0 rm -rf
#
#echo "Moving pruned repo back to root"
#mv out/* .
#rm out -r

# Install the pruned dependencies

echo "Installing yarn dependencies"
HUSKY=0 yarn install --frozen-lockfile --prefer-offline --force --build-from-source
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use for these files becomes a lot more apparent in #2493

@Alfred-Mountfield Alfred-Mountfield force-pushed the am/build-improvements branch 2 times, most recently from db17779 to e000177 Compare April 28, 2023 15:07
CiaranMn
CiaranMn previously approved these changes Apr 28, 2023
Copy link
Contributor

@thehabbos007 thehabbos007 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GH fooled me, I wanted to add these comments individually

apps/hash-frontend/vercel-install.sh Show resolved Hide resolved
apps/hash-frontend/vercel-build.sh Outdated Show resolved Hide resolved
Co-authored-by: Ahmad Sattar <thehabbos007@gmail.com>
@Alfred-Mountfield Alfred-Mountfield added this pull request to the merge queue Apr 28, 2023
Merged via the queue into main with commit 45acb52 Apr 28, 2023
@Alfred-Mountfield Alfred-Mountfield deleted the am/build-improvements branch April 28, 2023 17:06
@vilkinsons vilkinsons added type/eng > frontend Owned by the @frontend team area/tests > integration New or updated integration tests and removed area: frontend labels Jul 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/apps > hash* Affects HASH (a `hash-*` app) area/apps > hash-api Affects the HASH API (app) area/infra Relates to version control, CI, CD or IaC (area) area/tests > integration New or updated integration tests type/eng > frontend Owned by the @frontend team
Development

Successfully merging this pull request may close these issues.

5 participants