Skip to content

Latest commit

 

History

History
105 lines (71 loc) · 3.94 KB

create-initial-data-migrations.md

File metadata and controls

105 lines (71 loc) · 3.94 KB

Create initial data migrations

Overview

The goal is to convert our initial data into migration files so that they can be loaded into the database automatically

  1. Export the data into JSON
  2. Call convert.py to generate a script for the data
  3. Create a blank migration and call the initial data script from there
  4. Test the migration

Convert the data into JSON

  1. Export the data from the spreadsheet
    1. Find the sheet in the document for the data to export. Let's use the SOC Major data as our example
    2. Make sure that the first row (column descriptions) is frozen. Otherwise, freeze it by selecting the first row in the sheet, Menu > View > Freeze > Up to row 1
    3. Export to JSON. Export JSON > Export JSON for this sheet
  2. Save the JSON into a file
    1. Select and copy all the JSON text
    2. Paste it into a new file and save it as [ModelNameInPascalCase]_export.json under app/core/initial_data/
    3. The Pascal case is important in the next step to generate a python script to insert the data
    4. Potential problem: There was a problem with the JSON exporter where it omitted the underscore in occ_code. It should be fixed now but it's good to pay attention to other column name problems and fix them in the Google Apps script in the spreadsheet.

Convert JSON into python

  1. Go to the project root and run this command

    docker-compose exec web python scripts/convert.py core/initial_data/SOCMajor_export.json

  2. Check that there's a new file called app/scripts/socmajor_seed.py and that it looks correct

    1. You can run it to verify, but will need to remove that data if you care about restoring the database state

      1. Run this command to run the script

        docker-compose exec web python manage.py runscript socmajor_seed

      2. To remove the data, go into the database and delete all from core_socmajor

        docker-compose exec web python manage.py dbshell

# see if all the seed data got inserted
select count(*) from core_socmajor;
# shows 22 rows

# now we're inside dbshell
delete from core_socmajor;
# DELETE 22

# ctrl-d to exit dbshell

Create the migration

  1. Create a blank migration file (for the core app, because all our models are in there)

    docker-compose exec web python manage.py makemigrations --empty core --name socmajor_initial_data

  2. Call our script from the migration file

    from django.db import migrations


def add_data(apps, schema_editor):
    from ..scripts import socmajor_seed

    socmajor_seed.run()


def delete_data(apps, schema_editor):
    SOCMajor = apps.get_model("core", "SOCMajor")
    SOCMajor.objects.all().delete()


class Migration(migrations.Migration):

    dependencies = [
        ("core", "0007_socmajor"),
    ]

    operations = [migrations.RunPython(add_data, delete_data)]
    1. We pass 2 arguments to RunPython: functions for forward and reverse migrations
    2. add_data calls the seed script
    3. delete_data empties the table

  3. Verify the migration works

    # apply the new migration
docker-compose exec web python manage.py migrate core

# reversing to a previous migration (best to go back just 1 from the current count)
docker-compose exec web python manage.py migrate core 0004

# forwarding to the latest migration
docker-compose exec web python manage.py migrate core