Deployment questions

[lpatmo]

Logging some deployment questions for @billglover here! My guess is that it would be most efficient to walk through this in a hangout/Twitch stream soon, but here are some documented questions in advance.

#Context
Goal: deploy an api-staging.codebuddies.org URL partly to document the process, and partly so that folks who don't have the wifi bandwidth to install Docker (or can't install Docker for Windows properly because their computer does not have memory) can have a staging URL to make requests against from frontend app.

Resources I followed:

• Deployment docs at https://github.com/codebuddies/backend/wiki/Production-Deployment (written by Bill)
• This workflow: https://github.com/codebuddies/backend/actions/runs/46981824/workflow
• Diff of early PR by Bill: https://github.com/codebuddies/backend/compare/issue-65-production-deploy
• Tutorial for doctl, the command line interface for DigitalOcean: https://www.digitalocean.com/community/tutorials/how-to-use-doctl-the-official-digitalocean-command-line-client

What I did:

• Created a DigitalOcean managed Postgres DB at https://cloud.digitalocean.com/databases/db-postgresql-sfo2-66602?i=0c2df0
• Created a droplet in DigitalOcean by following the deployment docs above and running the following:

doctl auth init --context=cb

doctl --context=cb compute droplet create backend --enable-backups --enable-monitoring --image=docker-18-04 --region=sfo2 --size=s-1vcpu-1gb --ssh-keys=a8:a8:a8:a8:59:f3:ca:e4:8e:9b:6c:35:c7:a6:3f:e3 --tag-names=v3

Note: I found the fingerprint via this link: https://cloud.digitalocean.com/account/security?i=0c2df0

• SSHed in and added user and opened up firewalls

ssh root@IP_ADDRESS
adduser cb
usermod -aG docker cb
ufw allow http
ufw allow https

• Created a new workflow called Staging at https://github.com/codebuddies/backend/actions?query=workflow%3AStaging. It's tied to a new branch called staging-deployment-experiment.
  Workflow file: https://github.com/codebuddies/backend/actions/runs/241969451/workflow

I changed the work directory to be workdir: project instead of workdir: cbv3_django_prototype, since we changed out those names.

• I went to https://github.com/codebuddies/backend/settings/secrets and updated the value for DO_STAGING_DB_URL to be the new database URL I saw that was given as host in the PostGres instance -- e.g. something like db-postgresql-sfo2-XXXXX-do-user-XXXXXXX-0.db.ondigitalocean.com

• In DigitalOcean under the Networking tab, I updated the DNS recording for api-staging.codebuddies.org to point to the IP address of the new droplet I created in a previous step instead of the IP it was pointing to before, which looked like a Ghost blog.
  

• Created a docker-compose-staging.yaml file here: https://github.com/codebuddies/backend/blob/046a1c4486ce15e5041f23bde104d6b5fece4d47/project/docker-compose-staging.yaml, based off of the early PR diff.

• Updated the production.py file to follow the original diff:

  backend/project/config/settings/production.py

  Line 101 in 0aae81b

  "handlers": ["console"],

This isn't working yet, so questions as new threads below!

[lpatmo]

1) Where are the following commands run?

I assume this is a user added to DigitalOcean:

adduser cb
usermod -aG docker cb

Source: https://github.com/codebuddies/backend/wiki/Production-Deployment

[lpatmo]

Oh never mind, I just figured out this is after I ssh root@SERVER_IP_ADDRESS. Am reading https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-14-04.

[lpatmo]

(Resolved)

[billglover]

Digital Ocean droplet only created a root user. I didn't want to use the root user to do deployments so these commands do two things:

• create a new user called cb
• add the cb user to the docker group in addition to the groups it is already a member of

These should be run as root soon after droplet creation. The deployment pipeline (assume GitHub Actions) should then use the cb user and not the root user.

[lpatmo]

Got it, thanks! Followup questions:

1. Is there a place on DO where I can see what users belong to the docker group on DO itself? 🤔 I don't see a place, so I assume there's another CLI command I can run... update: found https://linuxize.com/post/how-to-list-groups-in-linux/ so I ran less groups and I see a lot of groups in there, including root and docker :)

And running the following let me know that only one user (cb) is in the docker group right now:

root@backend:~# getent group docker
docker:x:999:cb

Okay, nevermind!

2. It it okay to still ssh into the droplet from root? e.g. ssh root@IP_ADDRESS.

Update: from reading https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys-on-ubuntu-1804, it looks like I should start ssh'ing in as ssh username@IP_ADDRESS... but at the moment trying to figure out what my username is.

[lpatmo]

2) Any tips for how to start debugging?

PR so far: https://github.com/codebuddies/backend/pull/176/files

If I go to api-staging.codebuddies.org right now, I see:

I am guessing there may be some secrets (as documented in https://github.com/codebuddies/backend/wiki/Production-Deployment) that I need to update.

So far the only secret I've updated is DO_DB_STAGING_URL.

[billglover]

I'd start with the log of the GitHub Actions run. Depending on the steps used, you may need to enable some debugging flags. If the action ran successfully then I'd probably start looking at what was running on the droplet after the action completed. Is the Django app running (inside a Docker container)? Is the reverse proxy (assume Nginx) running? Do either of these have any logs?

[billglover]

It looks as though the copy files step has failed. If this is a new droplet, you will need to update the secrets for the SSH keys. The old keys will no longer be valid.

[lpatmo]

I'm a bit confused about what to fill in for the DO_SSH_KEY... is it supposed to be the fingerprint that corresponds to the SSH key for my local machine? i.e. something I can find on https://cloud.digitalocean.com/account/security

[billglover]

The fingerprint doesn't feel right as the action will need the full key in order to be able to authenticate against the DO droplet. The fingerprint would identify the key but not act as the key itself. I haven't tested this, but the documentation indicates this should probably contain the full SSH key.

key - content of ssh private key. ex raw content of ~/.ssh/id_rsa

Recommended debugging steps:

• create a new key pair (i.e. don't use your personal key)
• try using the SSH key to log in as the cb user from your laptop
• when that works, you'll know the key can be used for authentication
• add the key to the Actions secrets
• trigger the Action
• take a look at system or security logs on the droplet to identify authentication issues

[billglover]

FYI - I've suggested a slot for pairing on things identified in this thread.

[lpatmo]

Thank you! That time looks good, and I've scheduled it here: https://codebuddies.org/hangout/sQ9tTbM9YBMZiejfQ

Jotted down a rough agenda in the hangout description, but let me know if you had other ideas and we can adjust if necessary (e.g. if I figure this out before we meet XD which may not be likely but I'll finish attempting it at least...)

[lpatmo]

Recording: https://www.youtube.com/watch?v=ZfEUFMCUveQ&feature=youtu.be (video still processing right now but should be up soon)

[lpatmo]

Learnings from pairing on GitHub Actions and Docker deployment w/ @billglover

September 20th, 2020 stream (2 hours)

Recording: https://www.youtube.com/watch?v=ZfEUFMCUveQ&feature=youtu.be

The branch: staging-deployment-experiment

How do I create a new ssh key to link up the DO droplet and GitHub Actions?

1. cd ~/.ssh
2. ssh-keygen -t rsa and create a passphrase. (aka follow the instructions at https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2) and give the key a name -- e.g. cb--staging-rssa
3. ssh root@IP_ADDRESS and su cb to change the user
4. Put the public SSH key into authorized_keys inside ~/.ssh/authorized_keys inside the droplet

$ cat cb--staging-rssa.pub > pbcopy

5. Copy and paste the private cb--staging-rssa key into GitHub secrets as DO_SSH_KEY

6. Update the DO_SSH_PASSPHRASE with the passphrase you created when you generated the key.

DNS?

(Punting on DNS for now)

Reverse proxy notes

Best practice is to stick a reverse proxy (e.g. nginx, traefik) in front of Django when you deploy it.

In development, you don't have a reverse proxy.

The production setup uses the reverse proxy to serve the staticfiles as well as the proxy request that comes into django.
Had those in two separate containers...one was django and one was the proxy (the other docker-compose definition)and in order to get them to talk to each other (the two separate docker-compose projects)created a fixed network

... two separate stacks... common network which is externalHere, the error message is that we haven't created the network stack
Wasnt' using nginx, was using something else which required the auth token
Quickest way to make this work to make sure the container is startingand the django project is running is to delete the ref to the external network
Was using one droplet for both staging and prod.
Until we need to do that... let's delete the external network

---

Do we need reverse proxy? How to do it?

One answer: put it inside docker-compose stackSSL, TLS, http piece makes it difficult to do different stacks per release version
For first version, we may not need version

nginx
One reverse proxy that sits in front of everything and do things like take a PR number (PR.staigng.codebuddies.com) and route that to a deployment of PR123.staging.codebuddies.com. But that's fairly complex and lots of things to set up.
Simple: just deploy staging from the main branch whenever something gets merged.
And put the reverse proxy inside the same deployment. (In the first scenario, the reverse proxy isn't part of the deployment)
Simplest: staging is a reflection of what's in main

DO_AUTH_KEY

We deleted this during the stream.

The DO_AUTH_KEY was there to help support using a reverse proxy in front of Django that would createdthe TLS keys with letsencrypt.

See: https://github.com/codebuddies/backend/blob/54cb3fe1a91a8ab0af6e1dc4446439d0204e0025/docker-compose-prod.yaml

DJANGO_ALLOWED_HOSTS

Note: if there's a bug associated with it moving foward, set it to * (for now) or `api-staging.codebuddies.org)

Trigger the action on a tag which will give the version number

Note: can look into later.

We have tag_names=true in api-staging.yml, which comes from https://github.com/marketplace/actions/publish-docker

Would tag packages with the version number.That tag gets passed through an environment variablethrough docker-compose which pulls the latest versionof the container image.

Why that matters: what happens if you redeploy the sameimage and the tag hasn't changed

Stop the currently running one and re-pull the imagebut lose track of it without the tagging

DO_STAGING_SSH_PORT

No port exposed from that container, so not publicly accessible
If we deploy reverse proxy, reverse proxy will listen on 443 and will forward the requestto django backend which will be something

uwsgi.ini

TODO: a couple of warnings if we run cb@backend: docker-compose -f docker-compose-staging.yaml ps inside the DO droplet

cb@backend:~/project$ docker-compose -f docker-compose-staging.yaml ps
WARNING: The CB_IMAGE_TAG variable is not set. Defaulting to a blank string.WARNING: The DB_URL variable is not set. Defaulting to a blank string.WARNING: The DJANGO_SECRET_KEY variable is not set. Defaulting to a blank string.WARNING: The DJANGO_ALLOWED_HOSTS variable is not set. Defaulting to a blank string.Name   Command   State   Ports------------------------------cb@backend:~/project$ docker ps

Local tool to look into docker image contents

https://github.com/wagoodman/dive

brew install dive on a mac

What are the steps for checking on the docker-compose-staging.yaml inside the DigitalOcean droplet?

1. ssh root@IP_ADDRESS
2. su cb
3. cd /home/cb/project
4. docker ps
5. docker logs {CONTAINER_ID}

Where do I see the docker image that was published?

Answer: github.com/codebuddies -> packages
TODO: figure out permissions here

https://github.com/codebuddies/backend/packages/134362

Where do I see the GitHub Action workflow for the staging-deployment-experiment branch-- and check whether it's run successfully?

https://github.com/codebuddies/backend/actions?query=workflow%3AStaging

What are the next steps?

[ ] Linda to figure out the correct Database URL string to update DO_STAGING_DB_URL; then can run docker logs {container_id} again in the DO droplet
[ ] DNS work to do so that api-staging.codebuddies.org can point to the droplet
[ ] Reverse proxy question -- do we use nginx or traefik? Do we even need a reverse proxy when we deploy if the only static assets we're serving up are related to the /admin pages?

[lpatmo]

Update: found the connection string! :) Noting for future reference that it'll look something like: postgresql://username:password@db-postgresql-sfo2-00000-do-user-000000-0.db.ondigitalocean.com:00000/defaultdb?sslmode=require

Update: Made sure the username in the connection string was doadmin to match with the owner in the Postgres tables: ```defaultdb=> SELECT * FROM pg_tables ORDER BY tableowner;
schemaname | tablename | tableowner | tablespace | hasindexes | hasrules | hastriggers | rowsecurity
--------------------+-------------------------------------+------------+------------+------------+----------+-------------+-------------
public | django_celery_beat_crontabschedule | doadmin | | t | f | t | f
public | django_celery_beat_clockedschedule | doadmin | | t | f | t | f
public | resources_resource | doadmin | | t | f | t | f
public | auth_group_permissions | doadmin | | t | f | t | f
public | django_content_type | doadmin | | t | f | t | f
public | users_user | doadmin | | t | f | t | f
public | users_user_groups | doadmin | | t | f | t | f
public | users_user_user_permissions | doadmin | | t | f | t | f
public | account_emailaddress | doadmin | | t | f | t | f
public | account_emailconfirmation | doadmin | | t | f | t | f
public | auth_permission | doadmin | | t | f | t | f
public | auth_group | doadmin | | t | f | t | f
public | django_admin_log | doadmin | | t | f | t | f
public | django_celery_beat_periodictasks | doadmin | | t | f | f | f
public | django_migrations | doadmin | | t | f | f | f
public | django_celery_beat_intervalschedule | doadmin | | t | f | t | f
public | django_celery_beat_solarschedule | doadmin | | t | f | t | f
public | django_celery_beat_periodictask | doadmin | | t | f | t | f

[lpatmo]

Deployment workflow proposal discussion

Strawman proposal #1:

1. Branch is merged into staging and updates api-staging.codebuddies.org
2. The FE is still talking to api-production.codebuddies.org, but FE developers have the option to talk to api-staging.codebuddies.org to develop if they need to test something outside of the local DB (or can't spin up docker), and the data is saved in the staging database.
3. At some point, staging gets merged into main which triggers a release to api-production.codebuddies.org that the FE production talks to.

Strawman proposal #2:

1. Branch is merged into main, which updates api-production.codebuddies.org and ALSO triggers a release to the staging branch and api-staging.codebuddies.org.
2. This way, api-staging.codebuddies.org and api-production.codebuddies.org are in sync.
3. FE production talks to prod, but FE developers talk to api-staging.codebuddies.org if they need to test something outside of the local DB (or can't spin up docker).

[lpatmo]

(Am leaning towards #2)

[angelocordon]

+1

[BethanyG]

I need to think on this more, but I have a strong preference for Number 1.

Having the BE be in an environment where we can clone production data into a staging DB and test things like data migrations without fear of munging user data is one example that comes to mind.

Another is integration testing for things like passwordless links, password resets, token security testing and expiration, and different levels of authorization. Also a good environment for those contributors who've expressed interest in integration testing using BDD, or PEN testing, or other scenarios not covered by running the thing locally only. Finally, it is a good environment for testing things like GraphQL setup, alternate clients, or other apps that might consume the API data - but that we don't really trust (yet) to hit production.

I personally do not in any way feel comfortable with releasing the BE into "production" with only the (haphazard) unit tests we've written thus far - we really need to have a env where we conduct integration tests both on the BE itself, and in conjunction with the FE code BEFORE that code goes live to users. We still have potentially breaking changes that might be made.

I also think we need an env where we can interact with the code as users, but not be worried about the data being "precious" -- so we can dump it and start over if needed - and adjust the "next" version of the code before we as for feedback from the larger community.

Again - need to think on it a little more, but that's where I'm at currently. 😄

[lpatmo]

fwiw proposal #1 is how it currently works for github.com/codebuddies/codebuddies, and also what I'm used to. :) Proposal #2 struck me as possibly simpler (if we're okay with merging directly to prod in a sense), although as Bethany pointed out, if we want to be more rigorous about it and minimize the risk of breaking changes, #1 is safer.

I wanted to raise these two proposals because either approach will have implications for how the deployment is set up.

[lpatmo]

❓ Separate question -- can we get away with the api-staging.codebuddies.org Postgres DB_not_ being a managed DB in DigitalOcean ($15/mo)? Or would it be better if both staging and prod were each managed DBs in DigitalOcean?

(For example, for github.com/codebuddies/codebuddies, staging spoke to a free-plan DB on mlab while production spoke to a MongoDB Atlas DB that was not free)

[BethanyG]

IMHO, we don't need a dedicated and managed postgres staging DB. Sure - it would be super-convienient, but at an extra $200/yr - no.

🤔 Don't know if it is possible under the DO managed DB rules -- but if those allow us a second schema (think Django-style namespacing -- its effectively a collection of differently named tables that functions as a separate DB) on the managed instance, we could do prod-cb and staging-cb...but a quick look at the DO docs didn't give me an answer to that. It is also a risk, since it would only be one setting change - and boom! The app could write to production.

[lpatmo]

New error! I think this is because I need to updated the ALLOWED_HOSTS secret value, so am playing with that right now. (Pasting error here for reference)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/codebuddies/manage.py", line 30, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 87, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/loader.py", line 49, in __init__
    self.build_graph()
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/loader.py", line 212, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/recorder.py", line 74, in applied_migrations
    return {tuple(x) for x in self.migration_qs.values_list('app', 'name')}
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 274, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 144, in __iter__
    return compiler.results_iter(tuple_expected=True, chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1052, in results_iter
    results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch, chunk_size=chunk_size)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: permission denied for table django_migrations

[lpatmo]

👀 Actually instead of ALLOWED_HOSTS, looks like it's a postgres permissions error: https://stackoverflow.com/questions/38944551/steps-to-troubleshoot-django-db-utils-programmingerror-permission-denied-for-r

[lpatmo]

psql "postgresql://your_username:your_password@cluster-do-user-1234567-0.db.ondigitalocean.com:25060/defaultdb?sslmode=require"

Reference doc for how to connect to the DigitalOcean managed postgres DB: https://www.digitalocean.com/docs/databases/postgresql/how-to/connect/

Discovered that the django_migrations table is owned by "doadmin":

defaultdb=> SELECT * FROM pg_tables ORDER BY tableowner;
     schemaname     |              tablename              | tableowner | tablespace | hasindexes | hasrules | hastriggers | rowsecurity 
--------------------+-------------------------------------+------------+------------+------------+----------+-------------+-------------
 public             | django_celery_beat_crontabschedule  | doadmin    |            | t          | f        | t           | f
 public             | django_celery_beat_clockedschedule  | doadmin    |            | t          | f        | t           | f
 public             | resources_resource                  | doadmin    |            | t          | f        | t           | f
 public             | auth_group_permissions              | doadmin    |            | t          | f        | t           | f
 public             | django_content_type                 | doadmin    |            | t          | f        | t           | f
 public             | users_user                          | doadmin    |            | t          | f        | t           | f
 public             | users_user_groups                   | doadmin    |            | t          | f        | t           | f
 public             | users_user_user_permissions         | doadmin    |            | t          | f        | t           | f
 public             | account_emailaddress                | doadmin    |            | t          | f        | t           | f
 public             | account_emailconfirmation           | doadmin    |            | t          | f        | t           | f
 public             | auth_permission                     | doadmin    |            | t          | f        | t           | f
 public             | auth_group                          | doadmin    |            | t          | f        | t           | f
 public             | django_admin_log                    | doadmin    |            | t          | f        | t           | f
 public             | django_celery_beat_periodictasks    | doadmin    |            | t          | f        | f           | f
 public             | django_migrations                   | doadmin    |            | t          | f        | f           | f
 public             | django_celery_beat_intervalschedule | doadmin    |            | t          | f        | t           | f
 public             | django_celery_beat_solarschedule    | doadmin    |            | t          | f        | t           | f
 public             | django_celery_beat_periodictask     | doadmin    |            | t          | f        | t           | f

Maybe I need to change the owner from doadmin to something else? 🤔

UPDATE: Decided to update the DO_STAGING_DB_URL string in secrets to refer to the username doadmin

[lpatmo]

Yay, updating the DO_STAGING_DB_URL secret worked to reference doadmin as the username worked!

New error from docker logs <container_id>:

  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 234, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 247, in apply_migration
    migration_recorded = True
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 110, in __exit__
    self.execute(sql)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 137, in execute
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: foreign key constraint "resources_taggedreso_content_object_id_a169ab98_fk_resources" cannot be implemented
DETAIL:  Key columns "content_object_id" and "id" are of incompatible types: integer and uuid.
```

[lpatmo]

Slack thread: https://codebuddies.slack.com/archives/C04CA3Z8P/p1603785022035700

Ken:
So taggit is a standalone 3rd party library. I wonder if at some point the Taggit library was removed, and as a result, the resources_taggedresource_items table got deleted? This is one of those tricky problems that would probably require going through the git history and migration files.

Linda:
Ah thanks, that’s a good hypothesis! We’re still using taggit as a library, but I wonder now if it would be okay to delete the relevant migration file without consequence. Again, the strange thing is that I wasn’t seeing the issue on local docker-compose logs… so I suppose I can try it out on my branch, make sure local is working fine without this migration file, and then see what happens when I push this to prod.

@chris48s:
What do you get if you run ./manage.py showmigrations against the prod environment?

so probably docker-compose -f /path/to/your/docker-compose-staging.yaml run --rm app_staging python ./manage.py showmigrations

Ken:
Looking at resources/models.py, I’m seeing: tags = TaggableManager(through=TaggedItems, manager=CustomTaggableManager, blank=True)
The through= was added at a later date, which would explain why the old automatically generated through table existed, and then later disappeared.

Linda tried:
hmm, looks like getting the container ID from docker ps works, but the container is always restarting:
cb@backend:/project$ docker exec -it app_staging ./manage.py showmigrations
Error: No such container: app_staging
cb@backend:/project$ docker exec -it f25143721f4a ./manage.py showmigrations
Error response from daemon: Container f25143721f4aee34a0adbf8ae1a238884379f9780a1f30f97db2bb96e015c106 is restarting, wait until the container is running

cb@backend:~/project$ docker-compose -f docker-compose-staging.yaml run --rm app_staging python ./manage.py showmigrations
WARNING: The CB_IMAGE_TAG variable is not set. Defaulting to a blank string.
WARNING: The DB_URL variable is not set. Defaulting to a blank string.
WARNING: The DJANGO_SECRET_KEY variable is not set. Defaulting to a blank string.
WARNING: The DJANGO_ALLOWED_HOSTS variable is not set. Defaulting to a blank string.
ERROR: no such image: docker.pkg.github.com/codebuddies/backend/cb-backend:: invalid reference format

@chris48s:
Chris:
Yeah I suspect the container is always rebooting because your command basically says "apply the migrations then start uwsgi" and its failing to apply the migrations (but blind debugging this is really hard). Assuming you're running on DO Postgres, one option at this point is to dump the remote DB, load it into a local postgres and try migrating against that - you should be able to at least reproduce the problem locally.

Linda:
Ah, cool! So the tldr is that migrations (somehow) likely caused differences in the postgres DB between local and prod

Linda:
Do you happen to think the ERROR: no such image: docker.pkg.github.com/codebuddies/backend/cb-backend:: invalid reference format error is related to the container continuously rebooting?

Chris:
I guess so. I suspect you're probably not going to be able to run showmigrations cos the container keeps rebooting. Another possible option is to change your command to just uwsgi --ini /opt/codebuddies/uwsgi.ini - then it will just start uwsgi without applying the migrations. That won't fix the problem, but it might get you to a stage where the container is stable so you can at least run stuff against it to try and get some info. Another possible option is if you have no useful info in your DB at this point, you could nuke the DB, start again and see if the migrations will apply from scratch. Obviously if you have useful data, that's not a solution but if you're trying to biring a staging deploy up for the first time, no harm no foul...

[lpatmo]

👀 👀 Created a new Postgres managed DB based on @chris48s' suggestion (and updated the secret), made a new commit, and docker logs tells me it succeeded 🎉

cb@backend:~/project$ docker logs 649f6e837744
Operations to perform:
Apply all migrations: account, admin, auth, authtoken, contenttypes, django_celery_beat, osprojects, resources, sessions, sites, socialaccount, tagging, taggit, users
Running migrations:
Applying contenttypes.0001_initial... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0001_initial... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying users.0001_initial... OK
Applying account.0001_initial... OK
Applying account.0002_email_max_length... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying authtoken.0001_initial... OK
Applying authtoken.0002_auto_20160226_1747... OK
Applying django_celery_beat.0001_initial... OK
Applying django_celery_beat.0002_auto_20161118_0346... OK
Applying django_celery_beat.0003_auto_20161209_0049... OK
Applying django_celery_beat.0004_auto_20170221_0000... OK
Applying django_celery_beat.0005_add_solarschedule_events_choices... OK
Applying django_celery_beat.0006_auto_20180322_0932... OK
Applying django_celery_beat.0007_auto_20180521_0826... OK
Applying django_celery_beat.0008_auto_20180914_1922... OK
Applying django_celery_beat.0006_auto_20180210_1226... OK
Applying django_celery_beat.0006_periodictask_priority... OK
Applying django_celery_beat.0009_periodictask_headers... OK
Applying django_celery_beat.0010_auto_20190429_0326... OK
Applying django_celery_beat.0011_auto_20190508_0153... OK
Applying tagging.0001_initial... OK
Applying tagging.0002_auto_20200405_0312... OK
Applying tagging.0003_auto_20200508_1230... OK
Applying osprojects.0001_initial... OK
Applying taggit.0001_initial... OK
Applying taggit.0002_auto_20150616_2121... OK
Applying taggit.0003_taggeditem_add_unique_index... OK
Applying resources.0001_initial... OK
Applying resources.0002_auto_20200116_1610... OK
Applying resources.0003_auto_20200116_1619... OK
Applying resources.0004_auto_20200118_1014... OK
Applying resources.0005_auto_20200118_1036... OK
Applying resources.0006_auto_20200303_1257... OK
Applying resources.0007_auto_20200303_1258... OK
Applying sessions.0001_initial... OK
Applying sites.0001_initial... OK
Applying sites.0002_alter_domain_unique... OK
Applying sites.0003_set_site_domain_and_name... OK
Applying socialaccount.0001_initial... OK
Applying socialaccount.0002_token_max_lengths... OK
Applying socialaccount.0003_extra_data_default_dict... OK
[uWSGI] getting INI configuration from /opt/codebuddies/uwsgi.ini
*** Starting uWSGI 2.0.19.1 (64bit) on [Tue Oct 27 23:38:41 2020] ***
compiled with version: 8.3.0 on 27 October 2020 23:36:08
os: Linux-4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019
nodename: 649f6e837744
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 1
current working directory: /
detected binary path: /usr/local/bin/uwsgi
setgid() to 999
setuid() to 999
chdir() to /opt/codebuddies
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address :8000 fd 3
Python version: 3.7.9 (default, Oct 13 2020, 20:59:51) [GCC 8.3.0]
Python main interpreter initialized at 0x55c729ffbe10
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 250128 bytes (244 KB) for 4 cores
*** Operational MODE: preforking+threaded ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x55c729ffbe10 pid: 9 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 9)
spawned uWSGI worker 1 (pid: 12, cores: 2)
spawned uWSGI worker 2 (pid: 13, cores: 2)
Python auto-reloader enabled

[billglover]

I was going to chime in and say those missing environment variables might be problematic, but it looks as though things are now working.

[lpatmo]

Yay! :)

we haven't set-up certs

Ah... certs as in for example LetsEncrypt, right? I just found this article (https://www.digitalocean.com/community/tutorials/how-to-scale-and-secure-a-django-application-with-docker-nginx-and-let-s-encrypt#step-3-%E2%80%94-configuring-the-nginx-docker-container), and wondering if you think it is similar to what we intend with our setup.

One thing I'm confused about at the moment:

According to this article, we should be able to run the app by doing something like docker run --env-file env django-polls:v0 sh -c "python manage.py makemigrations && python manage.py migrate" , and then eventually be able to log into http://IP_ADDRESS/admin.

But in our droplet, we only have this file:

root@backend:/home/cb/project# ls
docker-compose-staging.yaml

... wondering if there's a way to run the app that I'm missing. Or maybe I still need to fixing environment variables?

root@backend:/home/cb/project# docker-compose -f docker-compose-staging.yaml run --rm app_staging python ./manage.py
WARNING: The CB_IMAGE_TAG variable is not set. Defaulting to a blank string.
WARNING: The DB_URL variable is not set. Defaulting to a blank string.
WARNING: The DJANGO_SECRET_KEY variable is not set. Defaulting to a blank string.
WARNING: The DJANGO_ALLOWED_HOSTS variable is not set. Defaulting to a blank string.
ERROR: no such image: docker.pkg.github.com/codebuddies/backend/cb-backend:: invalid reference format

Or when you said you were able to run the Django app, did you mean you used another command / were able to see something at IP_ADDRESS? 😅

[lpatmo]

@billglover (when you get a free moment) Do you know if we need to clone the application into the droplet, like the article at https://www.digitalocean.com/community/tutorials/how-to-scale-and-secure-a-django-application-with-docker-nginx-and-let-s-encrypt#step-3-%E2%80%94-configuring-the-nginx-docker-container mentions?

I vaguely remember that you had succeeded getting the app deployed at api-staging.codebuddies.org earlier this year with only one droplet, but I'm confused by that tutorial mentioning the presence of two application servers -- it can't be two different droplets, right? Sigh.

[billglover]

I don't think we need to clone the project into the droplet. In that article they build the Docker image on the target droplet. We don't need to do this as our Docker image is being built by the GitHub Actions Workflow. Other things to avoid in that article would be the second application server (for now).

The Nginx set-up looks quite sensible as a set of instructions so we could work through this manually and get it up and running before deciding what we want to do with automation.

The question we need to answer is... what do we do about static files?

• generate them in the GitHub Actions Workflow and SCP them to droplet?
• generate them on the Droplet when starting the app?
• just ignore them because they are not used in production deployment?
• other

Let me know if you'd like me to spend a little time on this over the next couple of days.

[lpatmo]

In that article they build the Docker image on the target droplet. We don't need to do this as our Docker image is being built by the GitHub Actions Workflow.

@billglover Thanks for reminding me/explaining again! I feel so out of my depth here XD. So GitHub Actions is building the Docker image, and publishing it at https://github.com/codebuddies/backend/packages/134362.

And inside our DO droplet, we run (?) that image, but we also need to set up nginx inside it and somehow (?) point it to our droplet IP address.

The question we need to answer is... what do we do about static files?

Either option 1 or 2 sounds fine (we're talking the python manage.py collecticstatic command, right?), but not sure which one is easier to implement.

I assume static files are necessary for CSS, and although api-staging.codebuddies.org/admin is not likely to get any important traffic (outside of admin traffic), it would still be nice to the deployed api-staging.codebuddies.org look clean instead of messy. Otherwise, however, it's technically not important -- we care more that the API works.

Let me know if you'd like me to spend a little time on this over the next couple of days.

That would be much appreciated, thank you 🙏 I'd like to learn / document as much of this as possible, but am stuck (aside from trying out those nginx commands you mentioned from the tutorial).

[billglover]

Your description of what is going on looks spot on to me. Going to have a crack at getting Nginx set-up now. I'm going to use the guide above and see how I get on. I'll record my screen, and try and take notes where I deviate. If anyone wants to pair on this, drop me a message on Slack.

[billglover]

Some deployment notes:

• Followed the tutorial for getting Nginx running without too much deviation
• SSL set-up with LetsEncrypt worked exactly as described
• We mistakenly used 127.0.0.1 as the IP address for our Django app (this ended up giving us a redirect loop)
• We realised that the Nginx and Django containers are deployed on two different networks
• Redeploying on the same network (and remembering to use port 8000) gets them talking to each other
• Django app is now accessible on https://api-staging.codebuddies.org/ with a valid certificate

Still to do:

• Auto renewal for SSL certificate
• Add Nginx definition to the docker-compose-staging.yaml file
• Use hostname instead of container network IP address to reference Django container in Nginx config
• Add mounts for static files
• Collect static files as a pipeline step

[lpatmo]

I'm VERY excited to learn from the video recording (if you still have it) -- was an excellent idea to record it.

And thanks again 🙏🙏🏅🏅🎖️

[billglover]

Recording uploaded here (currently a private link).

[lpatmo]

Thanks Bill! I think you'll have to make it unlisted for me to see it (don't think it's possible to share private videos with other emails). (Or you can email me an unlisted link)

…

On Thu, Nov 19, 2020, 5:26 AM Bill Glover ***@***.***> wrote: Recording uploaded here <https://youtu.be/ZgsRd3_0gSY> (currently a private link). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#177 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCNXO7GAQYIY7FM4I2JRNLSQUMJZANCNFSM4Q43RRNA> .

[billglover]

Done, changed the link to be unlisted. It's available in the CodeBuddies channels.

[lpatmo]

LOL I didn't realize you'd uploaded to the CodeBuddies YT account 😂 <https://emojipedia.org/face-with-tears-of-joy/> I <https://emojipedia.org/face-with-tears-of-joy/> couldn't see it earlier because I was logged in to a different account (and thought it was under a personal account), but got it now. I'll make it private again unless someone else requests access, and watch/take notes later. Thanks again!

…

On Fri, Nov 20, 2020 at 6:42 AM Bill Glover ***@***.***> wrote: Done, changed the link to be unlisted. It's available <https://youtu.be/ZgsRd3_0gSY> in the CodeBuddies channels. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#177 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCNXO77H234UNF7UKXG44DSQZ54ZANCNFSM4Q43RRNA> .

[lpatmo]

Oops sorry, replying via mobile caused a new post instead of a threaded reply.

[lpatmo]

Notes from Bill's nginx deployment recording (work in progress)

GH Actions workflow builds a container image
That image deploys a DO droplet using a docker-compose stack
The stack specification currently only includes the Django container itself, and doesn't include the reverse proxy.

The article at https://www.digitalocean.com/community/tutorials/how-to-scale-and-secure-a-django-application-with-docker-nginx-and-let-s-encrypt#step-3-%E2%80%94-configuring-the-nginx-docker-container talks about building a container image on the droplet itself, and has multiple servers. We've already built our container image using GH Actions (and hosted on GH).

Goal: get nginx deployed

Known issue: environment variables not set in the session

Potentially, we can put these environment variables into a file.

(Note from Linda: I actually saved values for those secret keys into GitHub secrets, but docker didn't seem to register the change)

Bill: Docker logs look good!

Trying:
curl -X GET http://localhost:8000
Problem: Port is not exposed outside of the stack; it's only accessible on the container network

docker port [CONTAINER_ID]

Expose the port on a running container

We've found our application!

Explanation: docker container is running, but what is serving the django app?

The Django app is deployed in the production configuration which sets up the redirect from http to https.

We want to see if a request made to port 8000 was actually being responded to. When we were running this outside the container, the port wasn't exposed.

Rather than fix exposing the port outside the container, we created a terminal session inside the container and used CURL to validate that the expected behavior is happening -- i.e. when we do curl -i -X GET http://localhost:8000 we're getting a response.

Now we're seeing a redirect:

So now we know the app server is serving correctly!

Now, back into getting nginx deployed correctly.

(13:26)

We make a new directory conf and vim nginx.conf

444 -> error code for ngnix

(18:31)

(TO BE CONTINUED)

[lpatmo]

(16:09)

IPV6, IPV4 definition

Use nginx to do the redirect.

Next, we specify a server that listens on port 443. And specify the SSL certificates.

(Perhaps these fullchain.pem files will be created later on.)

(22:01)

How we tell nginx to process the request that comes in:

Letsencrypt places a key in this location:

Looks like 1.19.0 is the latest nginx image:

(28:29)

Failures:

Let's use full paths, not relative paths.

Yay, different error now:

Reason: we missed a couple of semicolons in the file.

(32:37)

Now we get the "cannot load certificate" error as expected:

Checking that the domain name maps to the IP address:

Note:
-it means interactive session

So now we're using the container to run the certbot process to store certificates on the hostfile system.

It worked!

So now we switch back to the cb user

Hmm... we had accidentally created a conf folder (now removed)

Now:

(42:25)
"Cannot load certificate" warning, for some reason:

For clarity, let's use full paths:

OK great:

Interesting... letsencrypt staging certificates are not trusted by browsers

We're on https, which is good:

(47:38)
To recap:

• we're using a staging SSL certificate
• we told nginx to redirect from http to https
• We get "too many redirects" error occurring, but we don't know why it's happening!

When we run docker ps:

When we run docker logs CONTAINER_ID:

... we see these requests are coming from localhost (?), which is great

Tutorial is telling us to re-run the certs.

So now we've re-run the staging command... (50:36)

PS @billglover I'm so sorry I didn't have codebuddiesdotorg@gmail.com clearly visible on the site anywhere 😭 but glad you found it in #ask-an-admin 😂

(52:20)
So now we're on our own, past the tutorial, which is assuming that the app works.

We need to run certbot in renew mode:

Run nginx in detached mode by adding the flag -d:

docker ps shows that two containers are running: nginx and the Django backend

docker exec -it CONTAINER_ID /bin/bash

<-- now we're in our Django container
(56:40)

OK we can run python manage.py commands here now :)

Backing up a bit...

Looks like the redirect is coming from nginx

ifconfig (?)

Why is the backend doing a redirect?

(1:09:00)

[lpatmo]

(1:10:04) Digging into app files...

Docs:

Testing out a /bill path:

Researching nginx log format:

... specifically, what is the last parameter?

Looking for where the default config is, specifically:

/etc/nginx/conf.d/...

OK found it:

Lightbulb moment

(1:29:34)

We get an empty reply here because we're using localhost.

What's happening: the host is expecting us to use the

On the droplet, Django is listening to :443 and :80 inside a docker container

The way we defined the config for that Django host is...
cat conf/nginx.conf conf/
upstream django {
server 127.0.0.1
}

Our django app is running inside a container, and our nginx app is also running inside a container.

And so this will point to the container's IP address. This is why we're getting that infinite redirect loop.

We're passing the request to http django...

... and listening... and bouncing around again

... The problem is that 127.0.0.1 on the droplet means the droplet back interface

On the container, it means the container loop back interface.

So we never send the upstream django { server 127.0.0.1 } request to the Django container.

Two ways of fixing this:

Put all of nginx config into docker-compose stack... deployed on the same container network

(Second way?)

docker network list

(1:33:06)

[lpatmo]

Goal: find out what network the nginx logs are attached to, and what network the Django app is attached to.

Me: 😭

Bill: Something about bridge??

Ah cool we are using network stage_default:

We were telling nginx to redirect back to itself.

So the solution would be to get the apps to be on the same network.

OK:

Ahhh we're going to kill nginx

... and then run the nginx command again

Ok yay if we do docker inspect on the nginx container now we see the network is stage_default

Now we change nginx conf again:

Run this command according to the tutorial to restart nginx:

docker logs nginx looks good:

Hmm not quite:

(1:41:31)

Hmm... 502 now??

Why did upstream refuse the connection?

Wait, we're listening on port 8000....

Huh wow docker kill f7 worked?

OK WE'RE UP AND RUNNING

🔥 🔥

We've hardcoded the IP address

Bill's notes here: #177 (comment)

[lpatmo]

TIL about https://www.ssllabs.com/ssltest/analyze.html?d=api-staging.codebuddies.org:

[lpatmo]

Auto-renewing certificates

🤔 What's the difference between the certbot package mentioned in https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04, which auto-renews, compared to the certbot in https://www.digitalocean.com/community/tutorials/how-to-scale-and-secure-a-django-application-with-docker-nginx-and-let-s-encrypt#step-3-%E2%80%94-configuring-the-nginx-docker-container which does not auto-renew?

[lpatmo]

Hmm looks like the first tutorial (the one we followed for api-staging.codebuddies.org) is using https://hub.docker.com/r/certbot/certbot/ whereas the other tutorial installed it via:

sudo apt-get install python-certbot-nginx
sudo apt-get update
python-certbot-nginx
```

[lpatmo]

Update: Step 6 of https://www.digitalocean.com/community/tutorials/how-to-secure-a-containerized-node-js-application-with-nginx-let-s-encrypt-and-docker-compose#step-6-%E2%80%94-renewing-certificates%5D(https://www.digitalocean.com/community/tutorials/how-to-secure-a-containerized-node-js-application-with-nginx-let-s-encrypt-and-docker-compose#step-6-%E2%80%94-renewing-certificates) looks promising!