Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐳 Update Docker image build & push process to use dynamic repository name #107

Closed
wants to merge 12 commits into from
30 changes: 30 additions & 0 deletions .github/workflows/ghcr-image-github-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: Deploy Images to GHCR

on:
push:
branches:
- main
workflow_dispatch:

jobs:
push-store-image:
runs-on: ubuntu-latest
steps:
- name: 'Checkout GitHub Action'
uses: actions/checkout@main

- name: Extract version
id: extract-version
run: echo "version=$(cat package.json | jq -r '.version')" >> "$GITHUB_OUTPUT"

- name: 'Login to GitHub Container Registry'
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{github.actor}}
password: ${{secrets.Git_TOKEN}}

- name: Build inventory image
run: |
docker build . -t ghcr.io/forscht/ddrive:${{ steps.extract-version.outputs.version }} -f docker/Dockerfile
docker push ghcr.io/forscht/ddrive:${{ steps.extract-version.outputs.version }}
2 changes: 1 addition & 1 deletion LICENSE → LICENSE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2021 Lazy14K
Copyright (c) 2024 [fullname]

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
194 changes: 90 additions & 104 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,131 +1,112 @@
<h1 align="center" style="font-size: 60px"> DDRIVE </h1>

<p align="center"><strong> Turn Discord into a datastore that can manage and store your files. </strong></p>
<p align="center">
<a href="https://discord.gg/3TCZRYafhW">
<img src="https://img.shields.io/discord/1020806104881561754?color=5865F2&logo=discord&logoColor=white" alt="Discord server" /></a>
<a href="https://github.com/forscht/ddrive/actions/workflows/lint.yml">
<img src="https://github.com/forscht/ddrive/actions/workflows/lint.yml/badge.svg">
</a>
<a href="https://hub.docker.com/r/forscht/ddrive">
<img src="https://img.shields.io/docker/v/forscht/ddrive?logo=docker">
</a>
<a href="https://hub.docker.com/r/forscht/ddrive">
<img src="https://img.shields.io/docker/pulls/forscht/ddrive.svg?logo=docker">
</a>
<a href="https://github.com/forscht/ddrive/actions/workflows/codeql-analysis.yml">
<img src="https://github.com/forscht/ddrive/actions/workflows/codeql-analysis.yml/badge.svg">
</a>
<a href="https://github.com/forscht/ddrive/blob/v2/LICENSE">
<img src="https://img.shields.io/badge/License-MIT-yellow.svg">
</a>

</p>
<br>

##### **DDrive** A lightweight cloud storage system using discord as storage device written in nodejs. Supports an unlimited file size and unlimited storage, Implemented using node js streams with multi-part up & download.

https://user-images.githubusercontent.com/59018146/167635903-48cdace0-c383-4e7d-a037-4a32eaa4ab69.mp4

#### Current stable branch `4.x`

### Live demo at [ddrive.forscht.dev](https://ddrive.forscht.dev/)

### Features
- Theoretically unlimited file size, thanks to splitting the file in 24mb chunks using nodejs streams API.
- Simple yet robust HTTP front end
- Rest API with OpenAPI 3.1 specifications.
- Tested with storing 4000 GB of data on single discord channel (With max file size of 16GB).
- Supports basic auth with read only public access to panel.
- Easily deployable on heroku/replit and use as private cloud storage.
README.md
=======
# DDrive

## New Version 4.0
**Turn Discord into a datastore that can manage and store your files.**

[![Discord](https://img.shields.io/discord/1020806104881561754?color=5865F2&logo=discord&logoColor=white)](https://discord.gg/3TCZRYafhW)
[![Lint Workflow](https://github.com/forscht/ddrive/actions/workflows/lint.yml/badge.svg)](https://github.com/forscht/ddrive/actions/workflows/lint.yml)
[![Docker Version](https://img.shields.io/docker/v/forscht/ddrive?logo=docker)](https://hub.docker.com/r/forscht/ddrive)
[![Docker Pulls](https://img.shields.io/docker/pulls/forscht/ddrive.svg?logo=docker)](https://hub.docker.com/r/forscht/ddrive)
[![CodeQL Analysis](https://github.com/forscht/ddrive/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/forscht/ddrive/actions/workflows/codeql-analysis.yml)
[![License](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/forscht/ddrive/blob/v2/LICENSE)

This next major version release 4.0 is ddrive written from scratch. It comes with most requested features and several improvements.
DDrive is a lightweight cloud storage system that utilizes Discord as a storage device, written in Node.js. It supports an unlimited file size and storage capacity, implemented using Node.js streams with multipart upload and download.

- Now uses `postgres` to store files metadata. Why?
- Once you have huge amount of data stored on ddrive it makes ddrive significantly slow to start since ddrive have to fetch all the metadata from discord channel (For 3 TB of data it takes me 30+ minutes.)
- With postgres, deleting file is extremely faster because now ddrive don't have to delete files on discord channel and just need to remove from metadata only.
- With postgres now it's possible to move or rename files/folders which was impossible with older version.
- Added support for `rename` files/folders.
- Added support to `move` file/folder (Only via API, Not sure how to do it with frontend, PR welcomes.)
- Now uses `webhooks` instead of `bot/user tokens` to bypass the discord rate limit
- DDrive now uploads file chunks in parallel with limit. Which significantly increase the upload speed. I was able to upload file with `5GB of size in just 85 seconds`.
- Public access mode - It is now possible to provide users read-only access with just one config var
- Batch upload files - Now you can upload multiple files at once from panel. (DClone support has been removed from this version)
- Bug fix - `download reset` for few mobile devices
- Added support for optional encryption to files uploaded to discord
- DDrive now has proper rest API with OpenAPI 3.1 standards
- Added support for dark/light mode on panel
![DDrive Demo](https://user-images.githubusercontent.com/59018146/167635903-48cdace0-c383-4e7d-a037-4a32eaa4ab69.mp4)

I spent several weeks finalizing this new version. Any support is highly appreciated - [Buy me a coffee](https://www.buymeacoffee.com/forscht)
**Current stable branch: `4.x`**

### Requirements
- NodeJS v16.x or Docker
- Postgres Database, Discord Webhook URLs
- Avg technical knowledge

## Setup Guide
1. Clone this project
2. Create few webhook urls. For better performance and to avoid rate limit at least create 5 with 1 webhook / text channel. ([How to create webhook url](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks))
3. Setup postgres using docker, if you already don't have it running
- `cd .devcontainer`
- `docker-compose up -d`
4. Copy `config/.env_sample` to `config/.env` and make necessary changes
5. Optional - If you have lots of webhookURLs you can put those in `webhook.txt` with `\n` seperated.
6. Run - `npm run migration:up`
7. Run - `node bin/ddrive`
8. Navigate to `http://localhost:3000` in your browser.

### How to keep it running forever
1. Install pm2 with `npm install -g pm2`
2. Run - `pm2 start bin/ddrive`
3. Run - `pm2 list` to check status of ddrive
4. Run - `pm2 logs` to check ddrive logs

### Config variables explanation
```shell
# config/.env
**Live demo:** [ddrive.forscht.dev](https://ddrive.forscht.dev/)

# Required params
DATABASE_URL= # Database URL of postgres with valid postgres uri
### Features

WEBHOOKS={url1},{url2} # Webhook urls seperated by ","
- Unlimited file size through 24MB file chunks using Node.js streams API.
- Simple yet robust HTTP front end.
- REST API with OpenAPI 3.1 specifications.
- Tested with storing 4TB of data on a single Discord channel (max file size of 16GB).
- Basic auth with read-only public access to the panel.
- Easily deployable on Heroku/Replit for private cloud storage.

# Optional params
PORT=3000 # HTTP Port where ddrive panel will start running
## New Version 4.0

REQUEST_TIMEOUT=60000 # Time in ms after which ddrive will abort request to discord api server. Set it high if you have very slow internet
Version 4.0 is a complete rewrite of DDrive, incorporating the most requested features and several improvements:

CHUNK_SIZE=25165824 # ChunkSize in bytes. You should probably never touch this and if you do don't set it to more than 25MB, with discord webhooks you can't upload file bigger than 25MB
- Now uses `postgres` to store file metadata for improved performance and functionality.
- Supports file and folder `rename` operations.
- Enables `move` operations for files and folders via API.
- Utilizes `webhooks` instead of `bot/user tokens` to bypass Discord rate limits.
- Parallel file chunk uploads for increased speed (5GB file uploaded in 85 seconds).
- Public access mode for read-only access.
- Batch file uploads directly from the panel.
- Bug fixes, including improved `download reset` for mobile devices.
- Optional file encryption for enhanced security.
- Proper REST API adhering to OpenAPI 3.1 standards.
- Dark/light mode support on the panel.

SECRET=someverysecuresecret # If you set this every files on discord will be stored using strong encryption, but it will cause significantly high cpu usage, so don't use it unless you're storing important stuff
Your support is highly appreciated - [Buy me a coffee](https://www.buymeacoffee.com/forscht)

AUTH=admin:admin # Username password seperated by ":". If you set this panel will ask for username password before access
### Requirements

PUBLIC_ACCESS=READ_ONLY_FILE # If you want to give read only access to panel or file use this option. Check below for valid options.
# READ_ONLY_FILE - User will be only access download links of file and not panel
# READ_ONLY_PANEL - User will be able to browse the panel for files/directories but won't be able to upload/delete/rename any file/folder.
- Node.js v16.x or Docker
- PostgreSQL Database, Discord Webhook URLs
- Average technical knowledge

UPLOAD_CONCURRENCY=3 # ddrive will upload this many chunks in parallel to discord. If you have fast internet increasing it will significantly increase performance at cost of cpu/disk usage
## Setup Guide

```
1. Clone this project.
2. Create several webhook URLs (at least 5 for optimal performance).
3. Set up PostgreSQL using Docker:
- Navigate to `.devcontainer` directory.
- Run `docker-compose up -d`.
4. Copy `config/.env_sample` to `config/.env` and update with your details.
5. (Optional) If you have many webhook URLs, list them in `webhook.txt`, separated by newlines.
6. Run `npm run migration:up`.
7. Start the server with `node bin/ddrive`.
8. Open `http://localhost:3000` in your browser.

### Keeping DDrive Running

1. Install pm2: `npm install -g pm2`.
2. Start DDrive with pm2: `pm2 start bin/ddrive`.
3. Check the status: `pm2 list`.
4. View logs: `pm2 logs`.

### Configuration Variables

| Variable | Description |
|---------------------|-------------|
| `DATABASE_URL` | PostgreSQL database URL |
| `WEBHOOKS` | Discord webhook URLs, comma-separated |
| `PORT` | HTTP port for the DDrive panel |
| `REQUEST_TIMEOUT` | Timeout for Discord API requests (in milliseconds) |
| `CHUNK_SIZE` | Size of file chunks (in bytes) |
| `SECRET` | Secret for file encryption |
| `AUTH` | Basic auth credentials (`username:password`) |
| `PUBLIC_ACCESS` | Read-only access level (`READ_ONLY_FILE` or `READ_ONLY_PANEL`) |
| `UPLOAD_CONCURRENCY`| Number of parallel uploads to Discord |

### Docker Deployment

### Run using docker
```shell
docker run -rm -it -p 8080:8080 \
docker run --rm -it -p 8080:8080 \
-e PORT=8080 \
-e WEBHOOKS={url1},{url2} \
-e DATABASE_URL={database url} \
--name ddrive forscht/ddrive
```
### One Click Deploy with Railway

### One-Click Deploy with Railway

[![Deploy on Railway](https://railway.app/button.svg)](https://railway.app/new/template/tL53xa)

### Setup tutorials
- Setup under 4 minutes in local/cloud server using `neon.tech` postgres - [Youtube](https://youtu.be/Zvr1BHjrYC0)
## API Usage
### Setup Tutorials

- [Setup in under 4 minutes using `neon.tech` PostgreSQL](https://youtu.be/Zvr1BHjrYC0)

### API Usage

`npm install @forscht/ddrive`
```javascript
const { DFs, HttpServer } = require('@forscht/ddrive')
Expand Down Expand Up @@ -174,6 +155,11 @@ Migrating ddrive v3 to v4 is one way process once you migrate ddrive to v4 and a
8. Run `node bin/migrate old_data.json`
9. After few seconds once process is done you should see the message `Migration is done`

Feel free to create [new issue](https://github.com/forscht/ddrive/issues/new) if it's not working for you or need any help.
### Support and Contact

For support, join our [Discord server](https://discord.gg/3TCZRYafhW) or create a [new issue](https://github.com/forscht/ddrive/issues/new).

### License

DDrive is [MIT licensed](LICENSE).

[Discord Support server](https://discord.gg/3TCZRYafhW)
24 changes: 24 additions & 0 deletions ToDO.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
- To DO
1. Change the file extensions:
- Rename all `.js` files to `.ts`.
- If a file contains JSX, rename it from `.js` to `.tsx`.

2. Update imports:
- Modify the import statements in all files to reflect the new `.ts` or `.tsx` file extensions.

3. Install TypeScript:
- Run `npm install --save-dev typescript` to add TypeScript as a dev dependency.

4. TypeScript configuration:
- Initialize a TypeScript configuration file by running `npx tsc --init`.
- Configure the `tsconfig.json` file according to the project's needs.

5. Add type annotations:
- Add type annotations to variables, function parameters, and return types throughout the codebase.

6. Refactor the code:
- Refactor any code that TypeScript identifies as type-unsafe or that does not adhere to strict typing.

7. Compile TypeScript:
- Use the TypeScript compiler by running `npx tsc` to compile the `.ts` and `.tsx` files to JavaScript.
- Ensure that the compiled JavaScript code runs correctly
Loading
Loading