TheIndex

The frontend, editor panel, and API of the TheIndex.

Getting started

The easiest way is to get started is by using Docker Compose. You need to copy the docker-compose.yml and example.env file from GitHub. Rename example.env to .env and adjust the environment variables as you need to. with the following command:

docker-compose up -d

You'll need to change <host-port> to your port of choice. The web-server is not secured via SSL/TLS, it is in your responsibility to put a reverse proxy in front of this container. When you run the image for the first time, don't forget to set your own discord-id in the env SETUP_WHITELIST_DISCORD_ID to be able to login and edit. Once your container has set up itself once, you can remove the env variable from your setup.

Database

We use mongodb as our database server. You can deploy your own mongo setup as HA service or just a simple single docker container via e.g.:

Note: The database will start empty, you have to fill the data yourself.

docker run -d \
    --name mongo \
    -v ./db:/data/db \
    mongo

For development or testing purposes it is highly recommended to use mongo-express for accessing, viewing and editing the current state of the database. If you make it publicly accessible, don't forget to secure it with login credentials.

To simply spin up a mongo-express docker container, run:

docker run -d \
    --name mongo-express \
    -p 8081:8081 \
    mongo-express

You can also take a look at our provided docker-compose file on how to set it up.

Cache-DB

To increase performance we use redis to cache results from the mongoDB. The cache is being autopopulated on cache-misses.

You can create a new instance with docker by running:

docker run -d \
    --name redis \
    redis

The redis db is already included in the example docker-compose file

Updating container image

Warning: be aware, that we do not offer any kind of official support and every update may be with breaking changes. Be sure to make backups before you update

To get the newest version of the container image, you will need to run:

docker pull ghcr.io/snaacky/theindex:latest

Afterwards, you will need to stop and remove your current running instance and start it again.

Parameters

Here is a collection of the possible environment variables with their default values you should set in your .env file:

Parameter	Function	Default
NEXT_PUBLIC_SITE_NAME	The name of your site	"The Index"
NEXT_PUBLIC_DOMAIN	Your domain or IP, remove trailing slash	"https://theindex.moe"
NEXTAUTH_URL	Your domain or IP, remove trailing slash	$NEXT_PUBLIC_DOMAIN
DATABASE_URL	Take a look at mongodb docs	"mongodb://mongo:27017/index"
CACHE_URL	Connection string for the redis cache database	"redis://redis:6379"
CHROME_URL	WebSocket URL to a running chrome instance	"ws://chrome:3300"
AUDIT_WEBHOOK	WebHook-URL for audit-log, leave empty to disable support	""
DISCORD_CLIENT_ID	Discord OAuth2 client ID	"your_discord_oauth_client_id"
DISCORD_CLIENT_SECRET	Discord OAuth2 client secret	"your_discord_oauth_client_secret"
DISCORD_BOT_TOKEN	Required to access BOT resources	"your_discord_bot_token"
SETUP_WHITELIST_DISCORD_ID	If you need help getting your id, check out this guide	"your_discord_id"

And the following env variables are only needed when you are in dev mode and debugging the db

Parameter	Function	Default
ME_CONFIG_BASICAUTH_USERNAME	mongo-express username	"admin"
ME_CONFIG_BASICAUTH_PASSWORD	mongo-express password	"SUPER_SECRET"

If you want to verify how the docker-compose file fills the envs in, use docker-compose config

Getting started to code

Setup around the web app

1. Getting started isn't that straight forward. You will need to have installed the latest version of docker with docker-compose on your machine.

2. Start by cloning the repo via a graphical git client (highly recommended) or use the cli via

git clone https://github.com/snaacky/theindex

3. Copy the example.env file to .env.

How to fill the .env

• Replace NEXT_PUBLIC_DOMAIN and NEXTAUTH_URL with http://localhost:3000
• Generate a random strong string of at least 32 characters and use it for NEXTAUTH_SECRET. You can use generators e.g. 1Password or create it yourself.
• Change DATABASE_URL, CACHE_URL and CHROME_URL to use localhost instead of mongo, redis and chrome for example: mongodb://mongo:27017 -> becomes mongodb://localhost:27017
• Go to https://discord.com/developers -> Create a new Application -> Go into Auth2 inside your Application panel -> Copy the CLIENT ID and CLIENT SECRET into the .env file.
• In Redirects in Auth2 copy and paste the following URL needed to verify your Discord login http://localhost:3000/api/auth/callback/discord.
• Fill SETUP_WHITELIST_DISCORD_ID with your Discord ID to have an admin account when you login.
• Generate again another random strong string of at least 32 characters and use it for MEILI_MASTER_KEY. You can use generators e.g. 1Password or create it yourself.
• Lastly if you want to change the password for mongo-express, you can do so with the last two variables, usually though none of us do while developing locally.

4. Add the following ports to the images in the docker-compose file:

service	port-mapping
mongo	27017:27017
redis	6379:6379
meili	7700:7700
chrome	3300:3000

As an example, the setup for mongo should look similar to this:

mongo:
  image: mongo
  container_name: index-db
  restart: unless-stopped
  ports:
    - '27017:27017'
  volumes:
    - ./db:/data/db

5. Now run the command to start all the needed backend processes

docker-compose up -d mongo redis meili chrome mongo-express

Alternatively you can also just comment or remove the index service and run the command

docker-compose up -d

Web service

To start coding on the frontend, you will need to make sure, you have the latest version of node.js correctly installed. To install all the required dependencies run once:

bun install

Note: We decided to stick with npm instead of yarn to manage dependencies.

You should now have a folder called node_modules, which contains all the dependencies we need. We use Next.js as framework for our React web service. To test the web service you will have to run a db server in the background and start the frontend via:

bun run dev

After compiling you can open http://localhost:3000 in your browser of choice and see the running web application.

As we use Next.js, the frontend supports hot reloading, so you can just leave the page open, while you modify the code and see the changes on the fly in your browser.

Docker image

To create a ready-made docker image, just run:

docker build . -t index

You now have a local image with tag index that contains a distributable version of the code which can now be run.

Auto-formatter

We use prettier to ensure a consistent code style across all participants. You can simply auto-format everything with e.g. running the command

bunx prettier --write .

Design of the web app

ISR/SSR

Where possible we use ISR to pre generate all publicly accessible pages for caching by CDNs or proxies while validating and fetching new data with SWR requesting our own api.

API

We serve every api request over the endpoint /api, the corresponding code can be viewed at pages/api.

• /api/auth is reserved for NextAuth.js.
• /api/edit/... requires a logged-in user and usually (at least) editor rights and is for modifying or creating new content. The _id keyword _new is reserved for creating new content.
• /api/delete/... requires a logged-in user and usually (at least) editor rights and is for deleting content.
• /api/[content]s are public endpoints for fetching a list of all items of a certain content.
• /api/[content]/... are public endpoints for fetching information about a specific content.

Page restrictions

Every page request first has to go through _app.ts, where a basic layout is being applied and if a page has an auth property, it also validates whether the user can access the given page. Valid auth attributes are:

• auth itself is null or typeof auth === "undefined", no restrictions. This seems to be a public page.
• requireLogin, not really needed, but set it for logic reasons. User must be logged-in.
• requireAdmin, only a user with admin rights can access this page.
• requireEditor, only editors can view this page.

Contribution

Pull-requests are always welcome, but may not be always merged as it has to be in align with our idea of the index. If you want a certain feature or have an idea, you can always open a feature request in Issues or report it on our Discord in #index to be discussed. If it is not bad, in align with our ideas, and we find some time, we will certainly implement your requested feature (sometime...).

Things we use

• next.js as Webserver
• React as JS framework
• Styles from bootstrap
• Icons from Font Awesome
• We bake everything into a docker image
• Running on node js
• User-authentication via NextAuth.js
• mongodb as database backend
• redis as cache
• puppeteer to create screenshots of websites
• browserless/chrome to run our chrome instance
• react-in-viewport for checking if components are visible or not
• react-draggable for dragging components by the end user
• react-toastify for handling notifications
• iso-639-3 for the languages

And most importantly:

The help of our awesome community :3

Technical debts

• We use JS instead of TS (currently in migration)
• Unify Editor-views
• Unify db insert and updates to the format of func(_id, dataObject) and update only as needed, GraphQL would be nice...
• Move to add, remove api instead of having to send whole arrays to update lists -> GraphQL?