Skip to content

Commit

Permalink
Merge branch 'develop' of github.com:supabase/postgres
Browse files Browse the repository at this point in the history
  • Loading branch information
dragarcia committed Jul 26, 2021
2 parents 49f2e7e + a6545cd commit 3d5ec92
Show file tree
Hide file tree
Showing 71 changed files with 3,542 additions and 711 deletions.
Binary file removed .DS_Store
Binary file not shown.
59 changes: 0 additions & 59 deletions .github/workflows/provision.yml

This file was deleted.

3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
.DS_Store
.python-version
venv/
87 changes: 71 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,44 @@

Unmodified Postgres with some useful plugins. Our goal with this repo is not to modify Postgres, but to provide some of the most common extensions with a one-click install.

## Features

- ✅ Postgres [12](https://www.postgresql.org/about/news/1976/). Includes [generated columns](https://www.postgresql.org/docs/12/ddl-generated-columns.html) and [JSON path](https://www.postgresql.org/docs/12/functions-json.html#FUNCTIONS-SQLJSON-PATH) support.
- ✅ Ubuntu 18.04 (Bionic).
-[pg-contrib-12](https://www.postgresql.org/docs/12/contrib.html). Because everyone should enable `pg_stat_statements`.
## Primary Features
- ✅ Postgres [13](https://www.postgresql.org/about/news/postgresql-13-released-2077/).
- ✅ Ubuntu 20.04 (Focal Fossa).
-[wal_level](https://www.postgresql.org/docs/current/runtime-config-wal.html) = logical and [max_replication_slots](https://www.postgresql.org/docs/current/runtime-config-replication.html) = 5. Ready for replication.
-[PostGIS](https://postgis.net/). Postgres' most popular extension - support for geographic objects.
-[pgTAP](https://pgtap.org/). Unit Testing for Postgres.
-[pgAudit](https://www.pgaudit.org/). Generate highly compliant audit logs.
-[pgjwt](https://github.com/michelp/pgjwt). Generate JSON Web Tokens (JWT) in Postgres.
-[pgsql-http](https://github.com/pramsey/pgsql-http). HTTP client for Postgres.
-[plpgsql_check](https://github.com/okbob/plpgsql_check). Linter tool for PL/pgSQL.
-[plv8](https://github.com/plv8/plv8). Write in Javascript functions in Postgres.
-[plpython3u](https://www.postgresql.org/docs/current/plpython-python23.html). Python3 enabled by default. Write in Python functions in Postgres.
-[PL/Java](https://github.com/tada/pljaval). Write in Java functions in Postgres.
-[Large Systems Extensions](https://github.com/aws/aws-graviton-getting-started#building-for-graviton-and-graviton2). Enabled for ARM images.

## Extensions
| Extension | Description |
| ------------- | ------------- |
| [Postgres contrib modules](https://www.postgresql.org/docs/current/contrib.html) | Because everyone should enable `pg_stat_statements`. |
| [PostGIS](https://postgis.net/) | Postgres' most popular extension - support for geographic objects. |
| [pgRouting](https://pgrouting.org/) | Extension of PostGIS - provides geospatial routing functionalities. |
| [pgTAP](https://pgtap.org/) | Unit Testing for Postgres. |
| [pg_cron](https://github.com/citusdata/pg_cron) | Run CRON jobs inside Postgres. |
| [pgAudit](https://www.pgaudit.org/) | Generate highly compliant audit logs. |
| [pgjwt](https://github.com/michelp/pgjwt) | Generate JSON Web Tokens (JWT) in Postgres. |
| [pgsql-http](https://github.com/pramsey/pgsql-http) | HTTP client for Postgres. |
| [plpgsql_check](https://github.com/okbob/plpgsql_check) | Linter tool for PL/pgSQL. |
| [pg-safeupdate](https://github.com/eradman/pg-safeupdate) | Protect your data from accidental updates or deletes. |
| [wal2json](https://github.com/eulerto/wal2json) | JSON output plugin for logical replication decoding. |
| [PL/Java](https://github.com/tada/pljava) | Write in Java functions in Postgres. |
| [plv8](https://github.com/plv8/plv8) | Write in Javascript functions in Postgres. |

Can't find your favorite extension? Suggest for it to be added into future versions [here](https://github.com/supabase/supabase/discussions/679)!

## Enhanced Security
Aside from having [ufw](https://help.ubuntu.com/community/UFW),[fail2ban](https://www.fail2ban.org/wiki/index.php/Main_Page), and [unattended-upgrades](https://wiki.debian.org/UnattendedUpgrades) installed, we also have the following enhancements in place:
| Enhancement | Description |
| ------------- | ------------- |
| [fail2ban filter](https://github.com/supabase/postgres/blob/develop/ansible/files/fail2ban_config/filter-postgresql.conf.j2) for PostgreSQL access | Monitors for brute force attempts over at port `5432`. |
| [fail2ban filter](https://github.com/supabase/postgres/blob/develop/ansible/files/fail2ban_config/filter-pgbouncer.conf.j2) for PgBouncer access | Monitors for brute force attempts over at port `6543`. |

## Additional Goodies
| Goodie | Description |
| ------------- | ------------- |
| [PgBouncer](https://postgis.net/) | Set up Connection Pooling. |
| [PostgREST](https://postgrest.org/en/stable/) | Instantly transform your database into an RESTful API. |
| [WAL-G](https://github.com/wal-g/wal-g#wal-g) | Tool for physical database backup and recovery. |

## Install

Expand All @@ -26,13 +49,39 @@ See all installation instructions in the [repo wiki](https://github.com/supabase
[![Digital Ocean](https://github.com/supabase/postgres/blob/master/docs/img/digital-ocean.png)](https://github.com/supabase/postgres/wiki/Digital-Ocean)
[![AWS](https://github.com/supabase/postgres/blob/master/docs/img/aws.png)](https://github.com/supabase/postgres/wiki/AWS-EC2)

### Marketplace Images
| | Postgres & Extensions | PgBouncer | PostgREST | WAL-G |
|---|:---:|:---:|:---:|:---:|
| Supabase Postgres | ✔️ ||| ✔️ |
| Supabase Postgres: PgBouncer Bundle | ✔️ | ✔️ || ✔️ |
| Supabase Postgres: PostgREST Bundle | ✔️ || ✔️ | ✔️ |
| Supabase Postgres: Complete Bundle | ✔️ | ✔️ | ✔️ | ✔️ |

#### Availability
| | AWS ARM | AWS x86 | Digital Ocean x86 |
|---|:---:|:---:|:---:|
| Supabase Postgres | Coming Soon | Coming Soon | Coming Soon |
| Supabase Postgres: PgBouncer Bundle | Coming Soon | Coming Soon | Coming Soon |
| Supabase Postgres: PostgREST Bundle | Coming Soon | Coming Soon | Coming Soon |
| Supabase Postgres: Complete Bundle | Coming Soon | Coming Soon | Coming Soon |

### Quick Build

```bash
$ time packer build -timestamp-ui \
--var "aws_access_key=<insert aws access key>" \
--var "aws_secret_key=<insert aws secret key>" \
--var "ami_regions=<insert desired regions>" \
amazon-arm.json
```

## Motivation

After talking to a lot of techies, we've found that most believe Postgres is the best (operational) database but they *still* choose other databases. This is overwhelmingly because "the other one was quicker/easier". Our goal is to make it fast and simple to get started with Postgres, so that we never hear that excuse again.
After talking to a lot of techies, we've found that most believe Postgres is the best (operational) database but they _still_ choose other databases. This is overwhelmingly because "the other one was quicker/easier". Our goal is to make it fast and simple to get started with Postgres, so that we never hear that excuse again.

Our secondary goal is to show off a few of Postgres' most exciting features. This is to convince new developers to choose it over other database (a decision we hope they'll appreciate once they start scaling).

Finally, this is the same build we offer at [Supabase](https://supabase.io), and everything we do is opensource. This repo makes it easy to *install* Postgres, Supabase makes it easy to *use* Postgres.
Finally, this is the same build we offer at [Supabase](https://supabase.io), and everything we do is opensource. This repo makes it easy to _install_ Postgres, Supabase makes it easy to _use_ Postgres.

## Roadmap

Expand All @@ -43,3 +92,9 @@ Finally, this is the same build we offer at [Supabase](https://supabase.io), and
## License

[The PostgreSQL License](https://opensource.org/licenses/postgresql). We realize that licensing is tricky since we are bundling all the various plugins. If we have infringed on any license, let us know and we will make the necessary changes (or remove that extension from this repo).

## Sponsors

We are building the features of Firebase using enterprise-grade, open source products. We support existing communities wherever possible, and if the products don’t exist we build them and open source them ourselves.

[![New Sponsor](https://user-images.githubusercontent.com/10214025/90518111-e74bbb00-e198-11ea-8f88-c9e3c1aa4b5b.png)](https://github.com/sponsors/supabase)
60 changes: 60 additions & 0 deletions amazon-arm.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
{
"variables": {
"aws_access_key": "",
"aws_secret_key": "",
"region": "ap-northeast-1",
"ami_regions": "eu-central-1,eu-west-1,eu-west-2,ap-south-1,ap-southeast-1,ap-southeast-2,us-west-1,us-east-1,ca-central-1,sa-east-1,ap-northeast-1",
"ami": "ami-076d8ebdd0e1ec091",
"ami_name": "supabase-postgres-13.3.0",
"environment": "prod",
"ansible_arguments": "--skip-tags,update-only,--skip-tags,install-postgrest,--skip-tags,install-pgbouncer,--skip-tags,install-supabase-internal"
},
"builders": [
{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "{{user `region`}}",
"ami_regions": "{{user `ami_regions`}}",
"source_ami": "{{user `ami`}}",
"instance_type": "r6g.2xlarge",
"ssh_username": "ubuntu",
"ami_name": "{{user `ami_name`}}",
"tags": {
"environment": "{{user `environment`}}",
"appType": "postgres"
},
"launch_block_device_mappings": [
{
"device_name": "/dev/sda1",
"volume_size": 16,
"volume_type": "gp2",
"delete_on_termination": true
}
]
}
],
"provisioners": [
{
"type": "shell",
"inline": [
"while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done"
]
},
{
"type": "ansible",
"user": "ubuntu",
"playbook_file": "ansible/playbook.yml",
"extra_arguments": "{{user `ansible_arguments`}}"
},
{
"execute_command": "echo 'packer' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'",
"type": "shell",
"scripts": [
"scripts/02-credentials_cleanup.sh",
"scripts/90-cleanup.sh",
"scripts/91-log_cleanup.sh"
]
}
]
}
64 changes: 38 additions & 26 deletions amazon.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,31 +2,45 @@
"variables": {
"aws_access_key": "",
"aws_secret_key": "",
"region": "ap-southeast-1",
"ami_regions": "ap-southeast-1",
"ami": "ami-0f7719e8b7ba25c61",
"ansible_arguments": "--skip-tags,update-only"
"region": "af-south-1",
"ami_regions": "af-south-1",
"ami": "ami-08a4b40f2fe1e4b35",
"ami_name": "supabase-postgres-13.3.0.4",
"environment": "prod",
"ansible_arguments": "--skip-tags,update-only,--skip-tags,install-postgrest,--skip-tags,install-pgbouncer,--skip-tags,install-supabase-internal"
},
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "{{user `region`}}",
"ami_regions": "{{user `ami_regions`}}",
"source_ami": "{{user `ami`}}",
"instance_type": "m5.2xlarge",
"ssh_username": "ubuntu",
"ami_name": "supabase-postgres-0.13.0",
"launch_block_device_mappings": [
{
"device_name": "/dev/sda1",
"volume_size": 16,
"volume_type": "gp2",
"delete_on_termination": true
}
]
}],
"builders": [
{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "{{user `region`}}",
"ami_regions": "{{user `ami_regions`}}",
"source_ami": "{{user `ami`}}",
"instance_type": "m5.2xlarge",
"ssh_username": "ubuntu",
"ami_name": "{{user `ami_name`}}",
"tags": {
"environment": "{{user `environment`}}",
"appType": "postgres"
},
"launch_block_device_mappings": [
{
"device_name": "/dev/sda1",
"volume_size": 16,
"volume_type": "gp2",
"delete_on_termination": true
}
]
}
],
"provisioners": [
{
"type": "shell",
"inline": [
"while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done"
]
},
{
"type": "ansible",
"user": "ubuntu",
Expand All @@ -37,11 +51,9 @@
"execute_command": "echo 'packer' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'",
"type": "shell",
"scripts": [
"scripts/01-test",
"scripts/02-credentials_cleanup.sh",
"scripts/90-cleanup.sh",
"scripts/91-log_cleanup.sh",
"scripts/99-img_check.sh"
"scripts/91-log_cleanup.sh"
]
}
]
Expand Down
3 changes: 3 additions & 0 deletions ansible/files/docker_mnt/init.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
cat /etc/postgresql/postgresql.conf > $PGDATA/postgresql.conf
echo "host replication $POSTGRES_USER 0.0.0.0/0 trust" >> $PGDATA/pg_hba.conf
echo "host all all 127.0.0.1/32 trust" >> $PGDATA/pg_hba.conf
2 changes: 2 additions & 0 deletions ansible/files/fail2ban_config/filter-pgbouncer.conf.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[Definition]
failregex = ^.+@<HOST>:.+error: password authentication failed$
3 changes: 3 additions & 0 deletions ansible/files/fail2ban_config/filter-postgresql.conf.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[Definition]
failregex = ^.*,.*,.*,.*,"<HOST>:.*password authentication failed for user.*$
ignoreregex = ^.*,.*,.*,.*,"127\.0\.0\.1.*password authentication failed for user.*$
7 changes: 7 additions & 0 deletions ansible/files/fail2ban_config/jail-pgbouncer.conf.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[pgbouncer]
enabled = true
port = 6543
protocol = tcp
filter = pgbouncer
logpath = /var/log/pgbouncer.log
maxretry = 3
7 changes: 7 additions & 0 deletions ansible/files/fail2ban_config/jail-postgresql.conf.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[postgresql]
enabled = true
port = 5432
protocol = tcp
filter = postgresql
logpath = /var/lib/postgresql/data/pg_log/postgresql.csv
maxretry = 3
9 changes: 9 additions & 0 deletions ansible/files/logrotate-postgres
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
/var/lib/postgresql/data/pg_log/postgresql.log {
size 50M
rotate 3
copytruncate
delaycompress
compress
notifempty
missingok
}
16 changes: 16 additions & 0 deletions ansible/files/node_exporter.service.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
[Unit]
Description=Node Exporter
After=network-online.target

[Service]
Type=simple
ExecStart=/opt/node_exporter/node_exporter --web.disable-exporter-metrics --collector.disable-defaults {% for collector in collectors %} --collector.{{ collector }} {% endfor %}

User=root
StandardOutput=file:/var/log/node_exporter.stdout
StandardError=file:/var/log/node_exporter.error
Restart=on-failure
RestartSec=3

[Install]
WantedBy=multi-user.target
Loading

0 comments on commit 3d5ec92

Please sign in to comment.