feature(global): auth?!?! USER AUTHENTICATION?!?!?!?! (#1156)

* wip: setup redirecting and dummy token validation

* wip: i believe i can do the full auth flow now

* wip: playing around with oidc

* wip: still working on authentication, oidc engulfs my mind

this is fun but i have spent hours on this

* wip: working on integrating all parts together

* wip: started working on session middleware

* wip: refactored requests helpers a little bit

* wip: got some semblence of sessions working

* feat: setup better errors for oidc and session

* wip: playing with cookies

* wip: ahhhhhhh

* wip: started getting somewhere with token pair flow

* wip: hooked up oidc checking/refreshing to the identity and userinfo

* wip: worked on identity fetching and more route protecting

* wip: made route to generate auth url

* wip: login url is now generated by backend

* wip: now verifying at_hash

* wip: setup some dummy setup checking

* fix: remove levenstein

* wip: fixup after merge

* wip: started refactor for new auth flow

* wip: redesigned the session interface

* wip: brainstorming new mongo and redis structure

* wip: changed sids to uuids

* wip: refactored session interface to check replay attack earlier

* wip: started rewriting auth routes - identity

* wip: wrote login route

* wip: rewrote rest of the main auth routes

* wip: the rewrite base flow works omg

* fix: now deletes state cookie if invalid auth code given

* wip: better error handling for logout

* wip: address most todos in oidc/requests.py

* wip: setup redis docker

* wip: hooked up the session token part to redis

* wip: started to refactor users db

* wip: added session related collections

* chore: fix up packages after rebase

* wip: converted over refresh token storage to mongo

* wip: finished converting all to mongo (first step)

* fix: didnt realise pagination happened on ft search ugh i hate this

* feat: ttl on redis session tokens wooo

* feat: indexes on session collections mongo

* feat: converted sids back to uuids

* feat: fixed up validation for new user storage

* wip: started integrating guest sessions

* wip: started to refactor storage interface

* wip: culled some interface functions

* feat: GUEST SESSIONS (not in user layer yet)

* wip: fixed up model props in anticipation for user overhall

* wip: added user validation schema and collection typing

* feat: created new user database helpers

* wip: remove uid from models temporarily and fix pylint

* wip: added new user storage setup on session creation

* wip: db refactor - refactored out the redis connection

* wip: db refactor - deleted storage.py and created all da helper files

* wip: db refactor - remove cringe db helper prefixes

* wip: db refactor - moved out mongo conns into new file

* wip: db refactor - no more database.py

* wip: db refactor - started combining init-mongo and init-sessionsdb

* wip: db refactor - combined init sessionsdb

* fix: remove nanoid and bring back old package-lock

* wip: user routes refactor - underwrote old set and get

* wip: user routes refactor - hooked up guest login and started with fixing edge cases of underwrite

* wip: user routes refactor - fixed bugs with degree wizard and db helpers update success checks

* wip: user routes refactor - fix for None marks

* wip: frontend token refactor - re-setup guest login and id providing and refreshing

* wip: frontend token refactor - created new redux slice for identity

* wip: frontend token refactor - fixed any straggling old redux usage

* wip: frontend token refactor - token param - addToUnplanned & removeCourse

* wip: frontend token refactor - token param - setPlanned & setUnplanned & unschedule & unscheduleAll & ignoreFromProg

* wip: frontend token refactor - token param - removeAll & validateTermPlanner

* wip: frontend token refactor - token param - setupDegree & resetDegree

* wip: frontend token refactor - token param - search & updateMark & validateCTF & toggleLocked

* wip: frontend token refactor - token param - getUser

* wip: frontend token refactor - token param - getUserDegree

* wip: frontend token refactor - token param - getUserPlanner & getUserCourses & setIsComplete

* wip: frontend token refactor - completely remove getToken

* wip: frontend token refactor - setup identity provider and playing around with different refresh methods

* wip: frontend token refactor - reworked token state checking and RequireToken

* wip: frontend token refactor - Wrapped routes in RequiredToken and PreventToken

* wip: frontend token refactor - fixed setup and reset on frontend

* wip: frontend token refactor - fixed PreventToken temporarily

* wip: frontend token refactor - remove Auth.tsx

* wip: frontend token refactor - add back pagetemplate to Login.tsx

* wip: frontend token refactor - create Logout page and fix up some query client invalidation

* wip: frontend token refactor - fix bug with refreshing again

* wip: fix up models after pydantic upgrade

* fix: unplanned courses have null for unplanned

* wip: moved all tokens to be passed via headers

* feat: create connect function for mongodb conn

* feat: replaced nodemon with uvicorn reload for better control of what reloads

* feat: effectively removed init-mongo

* fix: removed old collections

* feat: new straight to uid route dependency

* wip: minor route rename

* fix: get rid of  field since this is now inferred by the shape of the user

* fix: added WWW-Authenticate headers to 401 errors

* feat: delete all guest data on logout

* feat: added dev flag to runserver.py so production dont reload

* chore: relabeled all my todos lol

* chore: forgor some

* feat: setup secure cookies

* fix: search bar quick add buttons not updating

* feat: redis password

* fix: fixed up login success page (i think idk auth is down)

* chore: relabelled todos so i actually know when i need to do what

* fix: dict() -> model_dump()

* fix: removed the token state route in place for isSetup

* wip: changed the isSetup error handling to use throwOnError

* wip: added userinfo validation p1

* wip: fixed logout redirect and pondered on things

* wip: playing around with diff refresh methods

* wip: removed whacky thunks for refreshing

* wip: change secure cookie prefix and trying out samesite strict

* deps: fix after merge

* fix: cookies now get deleted on secure mode

* fix: rename session errors

* wip: investigated sid and logout again

* wip: more todo culling and cleanup

* wip: converting over oidc config to be dynamic

* fix: summer term off by default

* fix: move oidc config into helper (kinda mid sol rn)

* fix: changed logout to be post

* wip: started working out the redirect behaviour of isSetup 401

* fix: useToken now can actually hit error boundary

* fix: temporarily solved the redirection 401 conundrum

* fix: delete init-mongo from docker-compose

* fix: removed some debug token help

* feat: made logout a bit more robust i guess

* fix: forgot to remove a debug error in landingpage im dumb as hell

* fix: oop forgot some more debug comments in IdentityProvider

* fix: made the environment capture more explicit for run_app.py

* fix: cleanup auth.py logout route a lil

* fix: bug with double initial refresh and csesoc logins

* fix: changed logout to use hard navigation for data clearing

* fix: removed token playground

* fix: deleted some unneeded user db helpers

* feat: rewrote mongo setup a bit so that it doesnt drop users by default

* feat: moved collection names to constants and renamed the dyn cols

* feat: redis now gets cleared on startup

* fix: made some mongo setup funcs exportable for testing purposes

* fix: added redis reset to test clear helper

* fix: tests can run locally now given you have env files

* feat: backend can now run without fedauth credentials

* update CI for testing with new stuff

* update documentation

* fix readme

* fix: cleaned up various return values and files after PR review feedback

- fixed return values of delete session and refresh token helpers
- removed ideas.md file
- changed oidc error string method to use tabs instead of spaces
- renamed state cookie and its ttl constant

* fix: droppped 'New' naming from collection variables

* bump random thingo to remove high vuln

* fix: explored better marker typing for models, found out pydantic is broken

* fix: major issue of spelling raised by zax-xyz

Co-authored-by: Michael Vo <zax@zaxu.xyz>

* fix: cleaned up identity slice reducers and thrown errors on frontend

As per Michael feedback!

* fix: removed all the spreads on the withAuthorization, as per Michael feedback :)

* fix: removed suppressed and mostRecentPastTerm to align with #1158

* feat: added new cli arg for deleting user data aswell, updated ci backend container to mirror prod but use this

Also finally deleted init-mongo.dockerfile and init-database.py

* fix: updated ci to use env script

* fix: mypy issues with unsupported typing

.github/workflows/ci.yml

@@ -31,16 +31,19 @@ jobs:
  run: |
  python -m pip install --upgrade pip
  pip install -r backend/requirements.txt
- - name: Set up docker compose
- run: |
- mkdir env
- echo "MONGODB_USERNAME=github-ci" >> env/backend.env
- echo "MONGODB_PASSWORD=github-ci" >> env/backend.env
- echo "MONGODB_SERVICE_HOSTNAME=mongodb" >> env/backend.env
- echo "MONGO_INITDB_ROOT_USERNAME=github-ci" >> env/mongodb.env
- echo "MONGO_INITDB_ROOT_PASSWORD=github-ci" >> env/mongodb.env
- echo "VITE_BACKEND_API_BASE_URL=http://localhost:8000/" >> env/frontend.env
- docker compose run ci-backend
+ - name: Set up env
+ run: >
+ python3
+ setup_env.py
+ --default
+ --sessionsdb_username="github_ci"
+ --sessionsdb_password="github_ci"
+ --sessionsdb_service_hostname="sessionsdb"
+ --mongodb_username="github_ci"
+ --mongodb_password="github_ci"
+ --mongodb_service_hostname="mongodb"
+ - name: Run the backend container
+ run: docker compose up -d ci-backend
  - name: Run mypy checks
  run: cd backend && python -m mypy . && cd ..
  - name: Test algorithm with pytest
@@ -73,13 +76,16 @@ jobs:
  with:
  name: fe-test-results
  path: frontend/junit/fe-test-results.xml
+ - name: Set up env
+ run: >
+ python3
+ setup_env.py
+ --default
+ --sessionsdb_username="github_ci"
+ --sessionsdb_password="github_ci"
+ --sessionsdb_service_hostname="sessionsdb"
+ --mongodb_username="github_ci"
+ --mongodb_password="github_ci"
+ --mongodb_service_hostname="mongodb"
  - name: Test the frontend successfully builds
- run: |
- mkdir env
- echo "MONGODB_USERNAME=github-ci" >> env/backend.env
- echo "MONGODB_PASSWORD=github-ci" >> env/backend.env
- echo "MONGODB_SERVICE_HOSTNAME=mongodb" >> env/backend.env
- echo "MONGO_INITDB_ROOT_USERNAME=github-ci" >> env/mongodb.env
- echo "MONGO_INITDB_ROOT_PASSWORD=github-ci" >> env/mongodb.env
- echo "VITE_BACKEND_API_BASE_URL=http://localhost:8000/" >> env/frontend.env
- docker compose up -d frontend-prod
+ run: docker compose up -d frontend-prod

.vscode/settings.json

@@ -4,7 +4,6 @@
  ],
  "VitestRunner.executionArg": "npx --prefix frontend",
  "python.testing.unittestEnabled": false,
- "python.testing.pytestEnabled": true,
  "python.testing.cwd": "${workspaceFolder}/backend",
  "files.exclude": {
  "**/__pycache__": true,

backend/README.md

@@ -4,48 +4,68 @@
 
 Circles uses `docker` for development. This means you do not need to install anything locally on your machine and it avoids the hassle of configuring MongoDB locally. You will only need to install `docker` on your local system: `https://www.docker.com/get-started`.
 
-We use docker to build 'images' for the backend, frontend, and mongodb. These `images` can then be used to run `containers`. An image is a blueprint for a container and a container contains everything we need to run an application. All dependencies for code exist within these containers, and via `docker-compose`, these containers can talk to each other.
+We use docker to build 'images' for the backend, frontend, redis, and mongodb. These `images` can then be used to run `containers`. An image is a blueprint for a container and a container contains everything we need to run an application. All dependencies for code exist within these containers, and via `docker-compose`, these containers can talk to each other.
 
 ### Creating Environment Variables
 
-MongoDB and the backend require a few environment variables to get started. In the root folder, create a folder called `env` and add three files: `backend.env`, `mongodb.env` and `frontend.env`. 
+#### NEW: Short way
+
+You can now run the `setup_env.py` script to setup your env files automatically (it will prompt you every step of the way)!
+
+#### Long way
+
+MongoDB, redis and the backend require a few environment variables to get started. In the root folder, create a folder called `env` and add three files: `backend.env`, `sessionsdb.env`, `mongodb.env` and `frontend.env`. 
 
 In `backend.env`, add the environment variables:
 
 - `MONGODB_USERNAME=name`
 - `MONGODB_PASSWORD=name`
 - `MONGODB_SERVICE_HOSTNAME=mongodb`
+- `SESSIONSDB_USERNAME=name`
+- `SESSIONSDB_PASSWORD=name`
+
+wherever you see "name", the value is not actually important.
 
 FOR **PRODUCTION**, also add:
+- `AUTH_CSE_CLIENT_SECRET=********` (redacted, contact CSE or one of the faculty societies for this secret)
+- `AUTH_CSE_CLIENT_ID=********` (redacted, contact CSE or one of the faculty societies for this secret)
 - `FORWARDED_ALLOW_IPS=*`
 
 In `mongodb.env`, add:
 
-- `MONGO_INITDB_ROOT_USERNAME=name`
-- `MONGO_INITDB_ROOT_PASSWORD=name`
+- `MONGO_INITDB_ROOT_USERNAME=name` (must match `MONGODB_USERNAME` in `backend.env`)
+- `MONGO_INITDB_ROOT_PASSWORD=name` (must match `MONGODB_PASSWORD` in `backend.env`)
+
+In `sessionsdb.env`, add:
+
+- `REDIS_USERNAME=peedee` (must match `SESSIONSDB_USERNAME` in `backend.env`)
+- `REDIS_PASSWORD=peedee` (must match `SESSIONSDB_PASSWORD` in `backend.env`)
+- `REDIS_ARGS="--user ${REDIS_USERNAME} on >${REDIS_PASSWORD} ~* allcommands"` (use this exactly, it uses your other two vars to make an admin user)
 
 In `frontend.env`, add:
 
 - `VITE_BACKEND_API_BASE_URL=http://localhost:8000/`
 
 > NOTE: The `VITE_BACKEND_API_BASE_URL` environment variable is the base url endpoint that the backend is running on. If the environment variable is not specified, the react application will default to using `http://localhost:8000/` as the base url when calling the API endpoint.
 
-You can use any random username and password. The username and password in `backend.env` must match the values in `mongodb.env`. The `env` folder has been added to `.gitignore` and will not be committed to the repo.
+
+You can use any random username and password wherever `name` has been used. The username and password in `backend.env` must match the values in `mongodb.env`, as indicated. The `env` folder has been added to `.gitignore` and will not be committed to the repo.
 
 ### Removing Docker Containers
 
-To remove all containers and the docker network, run `docker-compose down`. Add the option `-v` to also remove persistent volumes: that is, the mongoDB data. I tend to run this before I rebuild if I have made changes to the backend code to keep everything clean on my system.
+To remove all containers and the docker network, run `docker-compose down`. Add the option `-v` to also remove persistent volumes: that is, the mongoDB and redis data. I tend to run this before I rebuild if I have made changes to the backend code to keep everything clean on my system.
 
 To stop a docker containers that is running, run `docker-compose stop <containerName>`. This will not remove the container.
 
 Every once in a while, you will want to clean up docker. To do this, use `docker compose prune`.
 ### Running Circles
-#### with docker
-assuming that you correctly followed the steps above, you can run all of circles by using `docker compose up frontend` in a dev environment which will reflect your changes. If you change anything related to `database.py`, you will need to rebuild it to refect the changes.
-#### without Docker
-It is possible to run Circles without docker, to conserve system resources if it is taking too much of a toll. To do this, ensure nodemon is installed on your linux distribution by running `npm i -g nodemon`. Nodemon is a node package which automatically restarts an app when it detects code changes. 
-You can then run `python run_app.py` to run all of circles locally (assuming that you have installed all dependencies).
-Ensure your python3 version is set to 3.10. All parts of the app should now be running and talking to eachother. 
+#### Recommended: (kinda) without Docker
+It is possible to run Circles without docker, to conserve system resources if it is taking too much of a toll. To do this, ensure you have the correct dependencies installed. This will require you to install the frontend dependencies using `npm i` and the backend dependencies using a **python virtual environment** (using venv or conda) and `python -m pip install -r backend/requirements.txt`. 
+You can then run `python run_app.py` to run all of circles locally. MongoDB and Redis will still run under Docker, because they are external databases that you won't edit.
+Ensure your python3 version is set toa t least 3.12. All parts of the app should now be running and talking to eachother. 
 
 If you have having trouble with `python not found`, manually choose what python version is being run by adding `PYTHON_VERSION=python3` or, any version of your choosing in `backend.env`.
-If you run into too many issues, you should use docker.
+If you run into too many issues, you should use a complete docker solution, as below. This will be more power intensive to run, but is more likely to work without configuring your computer too much.
+
+#### with docker
+assuming that you correctly followed the steps above, you can run all of circles by using `docker compose up frontend` in a dev environment which will reflect your changes. If you change anything related to `database.py`, you will need to rebuild it to refect the changes.

backend/dev.dockerfile

@@ -2,16 +2,6 @@
 # Python image
 FROM python:3.12.3-slim
 
-# install nodemon
-
-RUN apt-get update \
- && apt-get install nodejs npm -y \
- && npm install -g nodemon
-
-# gcc required for python-Levenshtein
-RUN apt-get install gcc -y \
- && apt-get clean
-
 # Set current working directory inside container to /backend
 WORKDIR /backend
 
@@ -25,5 +15,11 @@ COPY . .
 # Expose port 8000 to the outside world
 EXPOSE 8000
 
+# At time of writing this, uvicorn reload wouldn't pass through stdout
+# https://testdriven.io/blog/fastapi-docker-traefik/
+# ENV PYTHONDONTWRITEBYTECODE 1
+ENV PYTHONUNBUFFERED 1
+
 # Run the server
-ENTRYPOINT nodemon --exec python3 -u runserver.py
+# https://forums.docker.com/t/docker-run-cannot-be-killed-with-ctrl-c/13108/2
+ENTRYPOINT ["python3", "-u", "runserver.py", "--dev"]

backend/production.dockerfile

@@ -22,4 +22,4 @@ COPY . .
 EXPOSE 8000
 
 # Run the server
-ENTRYPOINT ["python3", "-u", "runserver.py", "--overwrite"]
+ENTRYPOINT ["python3", "-u", "runserver.py", "--overwrite-course-data"]

backend/redis.conf

@@ -0,0 +1,7 @@
+# https://raw.githubusercontent.com/redis/redis/7.2/redis.conf
+# https://stackoverflow.com/questions/28785383/how-to-disable-persistence-with-redis
+appendonly no
+save ""
+# ensures that the only access is via env user
+user default off
+# protected-mode yes # this doesnt work, but its ok, as long as port is not exposed without password

backend/requirements.txt

@@ -1,18 +1,55 @@
+absl-py==2.1.0
+annotated-types==0.6.0
+anyio==4.3.0
+attrs==23.2.0
+bcrypt==4.1.3
+certifi==2024.2.2
+cffi==1.16.0
+charset-normalizer==3.3.2
+click==8.1.7
+cryptography==42.0.7
+dnspython==2.6.1
 fastapi==0.110.2
 fuzzywuzzy==0.18.0
+h11==0.14.0
 hypothesis==6.61.0
-mypy-extensions==1.0.0
+idna==3.7
+immutabledict==4.2.0
+iniconfig==2.0.0
+Levenshtein==0.25.1
 mypy==1.10.0
+mypy-extensions==1.0.0
+numpy==1.26.4
 ortools==9.10.4067
+packaging==24.1
+pandas==2.2.2
 paramiko==3.4.0
+pluggy==1.5.0
+protobuf==5.26.1
+pycparser==2.22
 pydantic==2.7.1
+pydantic_core==2.18.2
+PyJWT==2.8.0
 pymongo==4.7.0
+PyNaCl==1.5.0
 pypdf==4.2.0
 pytest==7.4.4
+python-dateutil==2.9.0.post0
 python-dotenv==1.0.1
 python-Levenshtein==0.25.1
 python-multipart==0.0.9
+pytz==2024.1
+rapidfuzz==3.9.0
+redis==5.0.3
 requests==2.31.0
+six==1.16.0
+sniffio==1.3.1
+sortedcontainers==2.4.0
+starlette==0.37.2
 types-paramiko==3.4.0.20240423
 types-requests==2.31.0.20240406
+typing_extensions==4.11.0
+tzdata==2024.1
+urllib3==2.2.1
 uvicorn==0.29.0
+watchfiles==0.21.0

backend/runserver.py

@@ -4,25 +4,54 @@
 import argparse
 import sys
 
-# https://github.com/encode/uvicorn/issues/998
-import uvicorn # type: ignore
-from server.database import overwrite_all
-from server.server import app
+import uvicorn
 
 if __name__ == "__main__":
  parser = argparse.ArgumentParser()
  parser.add_argument(
- "--overwrite",
+ "--overwrite-course-data",
  action="store_true",
- help="Inclusion of option will overwrite the database",
+ help="Inclusion of option will force overwrite the courses database",
  )
+ parser.add_argument(
+ "--overwrite-all-data",
+ action="store_true",
+ help="Inclusion of option will force overwrite the entire database (including users)",
+ )
+ parser.add_argument(
+ "--dev",
+ action="store_true",
+ help="Inclusion of option will enable hot-reloading and smart-overwrite"
+ )
+
  try:
  args = parser.parse_args()
  except argparse.ArgumentError:
  parser.print_help()
  sys.exit(0)
 
- if args.overwrite:
- overwrite_all()
+ print(
+ "-- Running server with CLI options:",
+ ", ".join(f"{k}={v}" for k, v in vars(args).items())
+ )
+ if args.overwrite_course_data or args.overwrite_all_data or args.dev:
+ # TODO-OLLI(pm): abstract this, and remove these local imports once we have proper connection handling
+ # TODO-OLLI(pm): also do overwrite checks for dev mode using a timestamp
+ # TODO-OLLI: dont we always want to atleast setup the dbs???
+ from server.db.mongo.setup import setup_mongo_collections
+ from server.db.redis.setup import setup_redis_sessionsdb
+
+ setup_mongo_collections(clear_users=args.overwrite_all_data)
+ print(f"-- Finished Mongo Setup (drop users={args.overwrite_all_data})")
 
- uvicorn.run(app, host='0.0.0.0')
+ setup_redis_sessionsdb()
+ print("-- Finished Redis Setup")
+
+ print(f"-- Starting uvicorn(reload={args.dev})")
+ uvicorn.run(
+ "server.server:app",
+ host='0.0.0.0',
+ lifespan="on",
+ reload=args.dev,
+ reload_excludes="*.json" if args.dev else None,
+ )

backend/server/config.py

@@ -2,10 +2,10 @@
 
 URI = "mongodb://mongodb:27017/?readPreference=primary&appname=MongoDB%20Compass&directConnection=true&ssl=false"
 
-CLIENT_ID = "1017197944285-i4ov50aak72667j31tuieffd8o2vd5md.apps.googleusercontent.com"
-
 FINAL_DATA_PATH = "./data/final_data/"
 
 ARCHIVED_DATA_PATH = "./data/final_data/archive/processed/"
 
-DUMMY_TOKEN = "token"
+DUMMY_TOKEN = "token" # TODO: get rid of this
+
+SECURE_COOKIES = True