Skip to content

teaguestockwell/morebuilds-readme

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 

Repository files navigation

LinkedIn

Logo

More Builds

A social media platform for exploring and sharing buidable items
View Site · Report Bug

Table of Contents

  • About The Project
  • System Architecture
  • Roadmap
  • Contact
  • Acknowledgements
  • About The Project

    • share the build process of something that contains modular parts. for example a bike contains wheels, tires, a sprocket etc...
    • users can explore other buids by filtering based on a specific parts
    • more builds is a collection of topics that contain builds parts and their retialers.

    System Architecture

    • gcp oauth
    • vercel cdn
      • serve static acsess like html css and js to the browser
    • vercel sererless functions
      • stateless deployment of a backend service that exposes crud, rate limiting, authorization, rbac and presigned upload urls (sas urls)
    • aws cdn 1 + bucket 1
      • store and serve original imgs uploads
    • aws cdn 2 + bucket 2
      • store and serve size variants of the origianl blobs
    • aws functions
      • optimaize orignial imgs into many sizes with sharp
    • planetscale mysql
      • a managed rdbs for horizontally scaling a cluster of mysql databases
    • github next-api-mw
    • github price-scraper
    • github more-builds
      • a repo for a monolithic app service with nextjs
    • github scraper
      • a repo for a docker img built with express / nodejs / chromium for web scraping
    • azure container registry
      • a artifact feed for docker images waiting to be applied to k8s
    • azure k8s cert manager
      • rotates tls certs from lets encrpt
    • azure k8s control plane
      • scraping service orchestration
    • azure k8s ingress
      • https ingress and load balancer
    • azure k8s nodes
      • nodejs scraper instances

    UI

    • using nextjs and react most of the UI is served from a CDN as static html that gets updated with the latest data every so often
    • when the browser loads the app, the client shows that stale html while the js fetches the latest data from the server in tha background.
    • the ui used a mix of client and server side rendering to show contentfull content quickly
    • in either case the client is hard at work combining the server's content with rich client side functionality.

    API

    • deployed as serverless functions to vercel
    • these functions use an orm called prisma to query a managed mysql cluster hosted on planetscale
    • the api service is stateless and consumed using remote procedure call (RPC) and a middleware i wrote for serverless functions that makes api functionality composable in a similar way to react hooks

    Scraping service

    • allow users to add urls to parts they used in their post
    • the bulk of the scraping service is built as an open source library here.
    • it uses chromium to load a page then parses structured data embedded in the html similar to how google indexes products for rich results.
    • full writeup

    Serverless Connection Pooling

    • serverless functions quickly exhaust db connections: a few of the solutions I have tried include:
      • aws rds and rds proxy
      • pgbouncer on an ec2 instance
      • digital ocean with postgres and pgbouncer.
    • after working with these, I switched to planetscale to host a managed vitese cluster because it provides:
      • a free tier with up to 1000 connections
      • pull request based schema changes
      • excellent built in logs
      • multi AZ deployments
      • auto scaling to the moon because of vitess

    DB

    erDiagram
    
      User {
        String email PK 
        String userId  
        String name  
        DateTime createdAt  
        DateTime updatedAt  
        String profileSrc  "nullable"
        String profilePicS3Key  "nullable"
        String about  "nullable"
        Int profileViews  
        Int postViews  
        Int numFollowers  
        }
      
    
      Post {
        String postId PK 
        String email  
        String topicId  
        String topicName  
        String userId  
        DateTime createdAt  
        DateTime updatedAt  
        Int numLikes  
        String html  
        Int views  
        Json storePartsQty  "nullable"
        }
      
    
      Comment {
        String commentId PK 
        String email  
        String postId  
        String parentCommentId  
        DateTime createdAt  
        DateTime updatedAt  
        Int numLikes  
        String text  
        }
      
    
      Pic {
        String picId PK 
        String email  
        String postId  "nullable"
        String partId  "nullable"
        String topicId  "nullable"
        String s3Key  
        DateTime createdAt  
        }
      
    
      PicJob {
        String s3Key PK 
        DateTime createdAt  
        }
      
    
      RateLimit {
        String rateLimitId PK 
        String email  
        DateTime createdAt  
        Int resource  
        }
      
    
      Topic {
        String topicId PK 
        String name  
        String about  
        DateTime createdAt  
        DateTime updatedAt  
        Int followerCount  
        Int views  
        }
      
    
      PartCategory {
        String partCategoryId PK 
        String topicId  
        String topicName  
        String name  
        String about  
        }
      
    
      Part {
        String partId PK 
        String topicId  
        String partCategoryId  
        Int numPosts  
        String name  
        String description  "nullable"
        String src  "nullable"
        }
      
    
      Store {
        String storeId PK 
        String name  
        }
      
    
      StorePart {
        String storePartId PK 
        String partId  
        String storeId  
        String price  
        String url  
        DateTime updatedAt  
        }
      
        User o{--}o Post : ""
        User o{--}o Post : ""
        User o{--}o Topic : ""
        User o{--}o User : ""
        User o{--}o User : ""
        User o{--}o Topic : ""
        Post o{--|| User : "user"
        Post o{--|| Topic : "topic"
        Post o{--}o User : ""
        Post o{--}o User : ""
        Post o{--}o StorePart : ""
        Comment o{--|| User : "user"
        Comment o{--|| Post : "post"
        Comment o{--}o User : "liked"
        Comment o{--}o User : "saved"
        Pic o{--|o Post : "post"
        Pic o{--|| User : "user"
        Pic o|--|o Topic : "topic"
        RateLimit o{--|| User : "user"
        Topic o{--}o User : ""
        Topic o{--}o User : ""
        PartCategory o{--|| Topic : "topic"
        Part o{--|| Topic : "topic"
        Part o{--|| PartCategory : "partCategory"
        StorePart o{--|| Store : "store"
        StorePart o{--|| Part : "part"
        StorePart o{--}o Post : ""
    
    Loading

    Image Uploads

    sequenceDiagram
    participant bucket
    participant api2
    participant cdn
    participant service worker
    participant browser
    participant api1
    participant db
    
    note over bucket, db: cleanup imgs never commited
    opt chron job
    api1 ->> db: get jobs where expired
    db -->> api1: jobs
    api1 ->> bucket: delete imgs
    bucket -->> api1: ok
    api1 ->> db: delete jobs
    db -->> api1: ok
    end
    
    note over bucket, db: resized img cache miss
    opt cache miss
    cdn->>api2: resize img {key} {size}
    api2->> bucket: get {key}
    bucket -->> api2: img {key}
    api2 ->> api2: resize
    api2 ->> bucket: upload resized
    bucket -->> api2: ok
    api2 -->> cdn: cache hit
    end
    
    note over bucket, db: service worker cache miss
    opt cache miss
    service worker -->> cdn: get img {key} {size}
    cdn --> service worker: img {key} {size}
    end
    
    note over bucket, db: img query
    browser ->> service worker: get img {key} {size}
    service worker -->> browser: img {key} {size}
    
    note over bucket, db: img upload
    browser ->> api1: get presigned url
    api1 ->> db: rate limits where user, resourse, time
    db -->> api1: rows
    api1 ->> api1: rate limit using a sliding log algorithm
    api1 ->> api1: sign an upload for size and url with private key
    api1 ->> db: create job
    db -->> api1: ok
    api1 -->> browser: presigned url
    browser ->> browser: select and crop an img
    browser ->> service worker: resize img
    service worker ->> service worker: compress
    service worker -->> browser: compressed img
    browser ->> bucket: upload img with progress
    bucket -->> browser: ok
    browser ->> api1: commit upload {key}
    api1 ->> bucket: verify {key}
    bucket -->> api1: ok
    api1 -->> db: delete job
    db -->> api1: ok
    api1 -->> browser: commited
    
    Loading

    Incremental Static Regeneration & React Query

    sequenceDiagram
    participant client
    participant cdn
    participant api
    
    note over client, db: incremental static generation
    client ->> cdn: get html
    opt cache miss
    cdn ->> api: get html
    api ->> db: get data
    db -->> api: data
    api ->> api: render html
    api -->> cdn: html
    end
    cdn -->> client: html with stale data
    client ->> client: first contentful paint
    client -->> client: react hydrates
    client -->> client: query cache populated with stale data
    
    note over client, db: stale while revalidate
    client -->> api: get fresh data
    api -->> client: fresh data
    client -->> client: diff data
    opt is diffrent
    client ->> client: render  data
    end
    
    note over client, db: client side rendering
    client ->> api: get data
    api -->> client: data
    client ->> client: render data
    
    note over client, db: server side rendering
    client ->> api: get page
    api ->> db: get data
    db -->> api: data
    api ->> api: render html - server side rendering
    api -->> client: html with fresh data
    client ->> client: first contentful paint
    client --> client: react hydrates
    
    Loading

    Threaded comments

    • full writeup
    • statically cached html
    • client side stale while revalidate
    • optimistic updates

    Search

    a simple database query with a client cache

    Roadmap

    See the open issues for a list of proposed features (and known issues).

    Contact

    Teague Stockwell - LinkedIn

    Acknowledgements

    About

    A social media platform for exploring and sharing buidable items

    Resources

    Stars

    Watchers

    Forks

    Releases

    No releases published

    Packages

    No packages published