Ideas for Improving Performance

[tristan-orourke]

React Router Data Loading

• React Router now has tools to avoid waterfall chains of component renders, which also block data fetching.
• Notes on further optimization here: https://remix.run/blog/lazy-loading-routes
• This will make the biggest difference if we're waiting for data to fetch, to render components, to fetch data, to render components. Look for a long waterfall in the Network tab.

Server-side rendering (SSR)

• Pros:
  • Improves SEO (help people find our job posters)
  • Improve initial time to load because everything can be done with a single back-and-forth request one time.
  • Could make site map
  • Dynamic link previews
• Requires a backend JS (node) server. Can be done with NextJS, or Remix, but will require some work to transition.
• Remix wants to ONLY do SSR. They think its faster and easier than client-side SPA if you have good caching.
• Can allow for some HTTP caching (see below)
• Will require deployment pipeline changes and possible security review

Static Site Generation (SSG)

• Static pages will serve faster if they can be generated at build time
• Will make a bigger difference (compared to SSR) if the pages take a longer time to render on the server (can we time that?)
• Remix makes the point that SSR can be fast, and benefits from HTTP caching. Stale-while-revalidate strategy might be better than SSG for pages which update sometimes but not often
• Astro wants to lean into SSG, but allows for portions of the page (or all of it, if need be) to be rendered on client
• If not Astro, I think we'll always need some amount of client-side generation, ie hydration, due to things like the nav bar changing if you're logged in.
• NextJS claims to be able to automatically optimize if a page should be SSR or SSG.
• SSG will have the greatest benefit on our most static pages:
  • Home page
  • Manager and Exec home pages
  • IAP Homepage
  • Directory page
  • FAQ-type pages
  • Login and Register pages
• The Talent Search page is an interesting case, because it loads a lot of data before it can render: skills, streams, classifications ( and pools I think, though it might not have to). But all of that is data which changes relatively rarely. Very rarely, for the most part. It could usually be pre-rendered. But if an admin edits skills, for example, we need a way to invalidate that cached page. Remix's SSR+Cacheing strategy might be better here.

Client-side Caching

• We already use URQL to enable client-side caching of data. After it has fetched some piece of data, it should have it until it needs refreshing.
• This only helps AFTER the first load. The app always has to load all the constants (skills, departments, etc) the first time you come back to it.
  • Changing this would have a bigger impact if the loading of initial data is a long operation.
• We haven't put much effort into our caching strategy, so its possible we could be smarter about when to revalidate.
• We cover entire pages with spinners while requests are pending. Covering the minimal portions of the page would make a difference. There may also be places where we could use stale data while revalidating (if it has minimal effect on the page) or use [[#Optimistic Updates]].

HTTP Caching (of pages)

• use HTTP headers to tell the browser to cache pages for some amount of time.
• HTTP caching allows a browser, OR intermediaries, to cache a page and avoid hitting our server at all
• This requires server-side rendering
• Pages which don't change much can benefit most from this.
  • Home pages
  • Login and Register
  • Directory pages
• Pages which change sometimes but not often may benefit even more, because SSG isn't an alternative!
  • Search page
  • Individual Job Poster pages. We can't generate them at build time, but after publishing they basically never change. Just when they close.
• Even some often-updated pages could have improved initial load times if we use stale-while-revalidate cache control. If personal pages, like my profile or my experience, have a unique url, then they would be good candidates for this.

Caching GraphQL

• Some queries, like when we fetch constants on the Search page, are often repeated and rarely change.
• What is the bottleneck for those queries?
  • If it takes a lot of time to assemble the data on the backend, we can use cache directives to enable application level caching. See also [[#Backend Performance]]
  • If retrieving from DB and computing on server is trivial and actually transferring data to the user over the net is the bottleneck, then we'd need HTTP caching (and/or more aggressive urql caching?)

HTTP Caching for GraphQL

• Could combine Automatic Persisted Queries and using GET requests for graphql, to enable HTTP Caching for common queries
  • enabling in urql https://formidable.com/open-source/urql/docs/advanced/persistence-and-uploads/#automatic-persisted-queries
  • lighthouse cache-control https://lighthouse-php.com/master/performance/caching.html#http-cache-control-header
  • Is there a way in URQL to exempt a query from APQ, or opt in? For some queries, they may always cache-miss and attempting to hit the cache may be a waste of time.
• This will be most beneficial if the bottleneck for pages is the transfer time of loading data, especially on the first try.

Optimistic Updates

• Optimistic updates means assuming mutations will succeed. This will make interactions like submitting and editing data more snappy
• This will be useful for:
  • Editing profile
  • working on applications
  • working on job posters
• This will not help:
  • initial page loads
  • search page
  • backend search + filtering
• This requires:
  • good frontend validation, since we want server failures to be rare
  • Which mutations would this be most useful on? (and most reliable)
  • we know how to handle a failure for UX
• When do we show the success toast, in cases like this?
• This will be most useful when mutations take a long time (travel time + server execution time)

Backend Performance

If the time spent on the server (between receiving a response and returning it) is longer (or comparable) to the time in transit, we should focus on improving the performance of the server itself. Server execution time can be divided into DB and PHP code.

DB performance improvements

If we are spending a lot of time on DB calls, we likely just have too many queries. We can try using load methods to aggregate queries. We should look into whether there are any specific queries which take a long time - if there are, I'm sure there are other SQL-specific ways to improve performance but I'm less familiar with them.

An obvious place to look is whether we're indexing all the columns we're filtering on often.

If we are repeating the same queries a lot, we could also consider caching DB responses.

PHP performance improvements

We know that we've caused performance issues by running policy methods on each item returned by a list - particularly when the policy method itself triggered a DB query. This is the first place I'd look for improvements.

We could try caching the entire graphql response with cache directives. Especially makes sense for stuff like skills and departments which change very rarely.

Other

• Lighthouse recommends enabling OPcache for PHP. Are we doing that? https://lighthouse-php.com/master/performance/server-configuration.html#opcache

[vd1992]

OPcache should be enabled by default

[esizer]

Wow, this is a really great write up. Thanks for this! 🎉 🙏

React Router Data Loading

I think this is something we could start right away and would help us a lot! Though, this would mean transitioning to loaders. I was able to adapt the example they provide, which uses @tanstack/react-query to work with urql and it went fairly smoothly. It would be a fairly large undertaking but could be worth it while we wait and see about SSR/SSG.

Server-side rendering (SSR)

I think SPA has served us well in getting things up and running in absence of a nodejs server. However, now as we start to scale and promote the site publicly, we should seriously consider moving to SSR for all the reasons you stated. Again, this will be a massive project however, we are well positioned to do this behind the scenes since we are leveraging workspaces. I imagine we could slowly replicate existing functionality in a different workspace under apps/** while maintaining the existing application. Then, when we are ready we could just flip a switch.

Now, when it comes to choosing a framework, I am partial to remix having used it in the past. Thought, nextjs has a much larger community behind it. My reasons for preferring remix are:

• Everything they do is very close to how the web works, they simply write thin wrappers around existing web APIs where as nextjs is much higher level and feels more like magic to me (you can't even access the request object, only what vercel wants you to access)
• I like the idea of nested routes (which we use to some effect in react-router already). Nextjs has this now with the app router but I have heard mixed reviews on it
• I'm actually not sure about this for nextjs but in remix, if your page doesn't need javascript, no javascript ever gets sent to the browser reducing the amount of bandwidth required
• Remix, is owned by shopify🍁

Static Site Generation (SSG)

I'm wondering, if we find that SSG is better for mainly static pages could we do a mixed approach where we have a very tiny app that only contains static pages? 🤔 apps/web and apps/static. We could utilize Astro islands to have only say the auth links be dynamic and basically have everything else run by SSR.

Client-side Caching

We cover entire pages with spinners while requests are pending. Covering the minimal portions of the page would make a difference.

Our spinners are out of control 🤣 . I think if we implement the lazy routes, we can hopefully reduce them since we can start fetching data before we request the actual page component.

HTTP Caching (of pages)

This would be nice to get in!

Caching GraphQL

If it takes a lot of time to assemble the data on the backend, we can use cache directives to enable application level caching.

I would say we should go ahead and stick this on our more static resources (skills, skill families, departments, etc.) right away 😅

HTTP Caching for GraphQL

Could combine Automatic Persisted Queries and using GET requests for graphql, to enable HTTP Caching for common queries

Oh, this is neat! I was worried that lighthouse-php wouldn't support this but it looks like they do! 😄

Optimistic Updates

Hopefully at some point someone can look at getting the graphcache working with urql. Though if we do make the jump to SSR, it is pretty straight foreward with most frameworks (Remix, or NextJS

good frontend validation, since we want server failures to be rare

I think we should look into zod if we want to strengthen our validation. It does work with react-hook-form as well 👍

When do we show the success toast, in cases like this?

Hopefully our server isn't too far behind the client so popping a toast on a delay wouldn't be that noticeable 🤷‍♀️ .

[petertgiles]

I agree with everything written so far! A couple of words of caution though:

1. We have tried a bunch of optimization steps so far and sometimes they improve performance and sometimes they don't. We should ensure we're doing testing often and early verify that something is actually helping us.
2. Switching the server generated content will probably shoot up our Azure bill.
3. I know I keep banging this drum, but without good monitoring we can easily destabilize our stack and have no idea.

[tristan-orourke]

Questions:

• For any big improvements, can we implement progressive rollout?

Server Side Rendering

• What version of node would IMTD let us use? - 16, 18, 20
• Would urql queries on server side require a lot of changes?
• Do we need IMTD to create node appservices for us or can we do it ourselves?
• Does SSR actually improve experience (we'd expect faster initial page loads)
• Exactly how much of our existing app needs to be rewritten to work with Remix
• How would new app service affect deployment pipeline?
• What would we need to measure on a proof of concept to decide if its worth it?
  • SEO is hard to quantify
  • link previews are not possible currently
  • proper status codes
  • how do we measure performance?
    • time to initial contentful page load on a dev vertical

Optimistic Updates

• URQL optimistic updates blocked on graphecache
• graphecache blocked on polymorphic models
  • currently just experiences
• is there a middle path between setting up graphecache for everything, and manually doing some optimistic-type updates in high-value case
• How to handle failure in optimistic update cases?

Actual Performance vs Perceived Performance

• Its hard for devs to talk about perceived performance
• we can measure real performance on server

[tristan-orourke]

What do we know now?

• Slowest queries are on Pool Candidate table (especially when trying to download many candidates) and some filtered-candidates queries on the Talent Search page. Those same requests are slow on the admin side as well when viewing those requests.
• The pool candidate queries are so slow they often time out
• We have slow query monitoring set up but haven't looked at results

What are we missing?

• telemetry in our postgres database - need help from IMTD, and doesn't integrate easily into azure
• without that we can tell which queries take longest, but not necessarily why

[tristan-orourke]

Next steps

• Test a dev Node app service
• Build a minimal SSR proof of concept (capable of gradual rollout)
• Make experience models not polymorphic
• Survey of users - ask them what feels slowest
• Look at slow query monitoring logs

[esizer]

There was some discussion about how to handle our existing forms. Looks like there is a library to integrate it with react-hook-form.

[mnigh]

What about switching from webpack to vite for better performance? @esizer has already started work on a branch and there is already an issue for it, so what's holding it up?

[petertgiles]

One of the big things is that to use the features of this bundler we would have to be serving the page from its dev server, not from Nginx. That's why #9844 is listed as a blocker.

[petertgiles]

Additionally, there were questions about Hydrogen compatibility. That maybe have just been around HMR and not regular compiling, though.

[esizer]

Not necessarily HMR for h2 but generally watching files causing infinite rebuilds 😢

[esizer]

Was interested in this so I ran lighthouse again now that we have vite in prod. It came back with some interesting insights that might be easy wins for us.

Reduce Javascript

Seems like tiptap is loading all the time regardless of whether it is in use or not. Wondering if we can determine why. We could see about lazy loading it? 🤔

Render-blocking resources

There are two things of note in here:

1. We could avoid the CDN and serve the google fonts directly. Right now we need to make two blocking requests to the google api and this would remove one of those.
2. Our css file is very large

CSS size

This is one of the first blocking resoucrces (after html itself) so it is quite important to make reductions where possible. From what I can see, most of the bloat is coming from two main areas:

The following rules :root, [data-h2*=dark], [data-h2*=iap] and [data-h2*=dark][data-h2*=iap] rules account for close to 5kb and most of it is not being used at the time it is loaded.

The selectors themselves are causing alot of bloat. We have a selector that is 720bytes 🤯 and many others that are in the 200byte range.

// This is 720b of data to set a background color :grimacing: 
[data-h2*=dark] [data-h2-background-color*="base:all:focus-visible:children[>span:first-child](focus)"]:focus-visible>span:first-child:not([data-h2-background-color]),[data-h2*=dark] [data-h2-background-color*="base:all:focus-visible:children[.counter](focus)"]:focus-visible .counter:not([data-h2-background-color]),[data-h2*=dark] [data-h2-background-color*="base:all:focus-visible(focus)"]:focus-visible,[data-h2*=dark] [data-h2-background-color*="base:all:focus-visible:children[.Step__Icon](focus)"]:focus-visible .Step__Icon:not([data-h2-background-color]),[data-h2*=dark] [data-h2-background-color*="base:all:focus-visible:children[.Step__Flair](focus)"]:focus-visible .Step__Flair:not([data-h2-background-color])

REF: https://www.debugbear.com/css-size-analyzer?url=https%3A%2F%2Ftalent.canada.ca%2Fassets%2Findex-PT27_YV6.css

HTTP/2

We already have a ticket for this but just noting it.

[esizer]

Tiptap

Identified the issue, ticket created: #10897

Google fonts

It would be a trial for sure and could be slower. The main impact is we reduce the (blocking) request count by 1. So assuming we can make one trip to our server faster than 2 trips to googles CDN, it should help. Also in the highly unlikely chance that google fonts is down, we don;t lose our fonts.

I'd say it is super low priority and only necessary if we are really bored and want something to do 😛

CSS

Splitting is possible for sure, it all depends on how "smart" we want to be with it. I have done route splitting before so you only load the css needed for that page but I don't think this is even possible so maybe we could do theme splitting so we only load dark styles when dark mode is enabled (same with IAP)? 🤔

However, I think the main issues can't really be solved that easily 😢 :

• The rules mentioned contain a lot of unused css. Some of it, like css vars is never used on any page.
• The selectors are overly verbose so no matter what we have larger files regardless of when they are loaded coming down. Maybe they can be uglified more? I'm really not sure we can do anything about this 🤷‍♀️
  • Also, these selectors that are around 200b are duplicated since they need to exist in HTML and CSS simultaneously.

[petertgiles]

It's true that our CSS is very verbose with lots of repetition. Because of that it also compresses very well. 😁

dist/assets/index-B1TobbTt.css 324.16 kB │ gzip: 30.90 kB

Not to say we shouldn't try to improve it. But in the real world it's not 320 KB bad.

[esizer]

Is that from the build output? Cause when I run it through analyzer I get the full file size coming over the wire 🤔

[tristan-orourke]

Does that mean we're missing a compression step on production?

[mnigh]

@tristan-orourke see: #10935.