This doesn't entirely explain how to get it all running, but it may help future debugging...
Global pip install, the venv is specified in the uwsgi "home" below
pip install uwsgi
location / {
include uwsgi_params;
uwsgi_pass unix:/home/spx5ich/wiserd3.5.0/wiserd3/wiserd3.sock;
}
location /static {
autoindex on;
alias /home/spx5ich/wiserd3.5.0/wiserd3/dataportal3/static/;
}
client_max_body_size 10m;
client_header_buffer_size 32k;
large_client_header_buffers 4 32k;
##
# `gzip` Settings
#
#
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_http_version 1.1;
gzip_min_length 256;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript application/vnd.ms-fontobject application/x-font-ttf font/opentype image/svg+xml image/x-icon;
[Unit]
Description=uWSGI Emperor service
After=syslog.target
[Service]
ExecStart=/usr/bin/uwsgi --emperor /etc/uwsgi/sites
Restart=always
KillSignal=SIGQUIT
Type=notify
StandardError=syslog
NotifyAccess=all
[Install]
WantedBy=multi-user.target
[uwsgi]
project = wiserd3
base = /home/spx5ich/wiserd3.5.0
chdir = %(base)/%(project)
home = /home/spx5ich/venv_3.5/
module = %(project).wsgi:application
buffer-size = 65536
master = true
processes = 8
socket = %(base)/%(project)/%(project).sock
chmod-socket = 666
vacuum = true
sudo service nginx restart
sudo systemctl daemon-reload
sudo systemctl restart uwsgi
sudo systemctl enable uwsgi
sudo service redis restart
Install iptables and fail2ban
firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="<webserver_ip>" port port=<postgresql_port> protocol="tcp" accept'
sudo firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="131.251.239.154" port port=5432 protocol="tcp" accept'
BUT for 16.04 we use systemd, same as CentOS7 above
http://serverfault.com/questions/775965/wiring-uwsgi-to-work-with-django-and-nginx-on-ubuntu-16-04
apt-get install npm nodejs-legacy
npm install -g mapshaper
ls *.zip|awk -F'.zip' '{print "unzip "$0" -d "$1}'|sh
11, 13, 14
Convert shapefile from whatever projection it currently is, to 4326 for WGS 84 (web mercator, but ovoid, not sphere)
ogr2ogr -f 'ESRI Shapefile' -t_srs EPSG:4326 input-fixed.shp Wales_lsoa_2011.shp
This will be massive, and probably unusable
mapshaper -i input-fixed.shp snap -simplify dp 100% keep-shapes -o output-fixed.geojson format=geojson
Topojson, much better. 1% is around 500k
mapshaper -i input-fixed.shp snap -simplify dp 100% keep-shapes -o output-fixed.json format=topojson
mapshaper -i input-fixed.shp snap -simplify dp 1% keep-shapes -o output-fixed-1.json format=topojson
The topojson file is printed all on a single line in the file, which breaks editors
node -e "console.log(JSON.stringify(JSON.parse(require('fs') \
.readFileSync(process.argv[1])), null, 4));" output-fixed-1.json > pretty.json
create user dataportal_username; alter user dataportal_username with encrypted password "supersecretpassword"; GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO dataportal_username; GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO dataportal_username;
sudo -u <postgres_user> pg_dump new2 --file /tmp/new_survey_jun_16_dir --format=d
DB VM
sudo -u <postgres_user> psql < build_sql.sql
Django VM
source ~/venv/bin/activate
python manage.py makemigrations dataportal3
Centos7 is a pain with shp2pgsql:
sudo yum install pgdg-centos94-9.4-2.noarch.rpm
yum install postgis2_94 postgis2_94-client postgis2_94-utils
may also need the full postgresql9.4 packages
note alternate shp2pgsql binaries location due to messed up install not updating alternatives left alone this time as previous postgresql version expects postgis-2.0.7 at this point, 2 versions of each are installed... shp2pgsql from 9.4 appears to work fine with 9.5 so far.
/usr/pgsql-9.4/bin/shp2pgsql -I spatialdata.parl/spatialdata.parl.shp spatialdata_parl | sudo -u postgres psql -d "NewSurvey"
Dump shp file to DB for spatial search :
shp2pgsql -W LATIN1 -I ~/shp/x_sid_liw2007_pcode_/x_sid_liw2007_pcode_.shp pcode | sudo -u postgres psql -d "NewSurvey"
do this for one of each type of boundary
Check the SRID is set, if it's '0', you need to set it, probably to 4326:
select code, ST_SRID("table"."geom") from "table";
(public is the schema)
select UpdateGeometrySRID('public', 'table', 'geom_column', 4326) ;
Existing WISERD localities boundaries need to be 27700, not 4326:
select UpdateGeometrySRID('public', 'heads_of_the_valleys', 'geom', 27700) ;
select UpdateGeometrySRID('public', 'aberystwyth_locality_dissolved', 'geom', 27700) ;
select UpdateGeometrySRID('public', 'bangor_locality_dissolved', 'geom', 27700) ;
The new shpfile tables need the right permissions
grant select, insert, update on all tables in schema public to dataportal;
In dataportal3/utils/spatial_search/spatial_search.py The "geometry_columns" array needs a new dict for this models data.
{
'table_name': 'spatialdata_aefa',
'geometry_column': 'geom',
'label': 'label',
'table_model': models.SpatialdataAEFA
},
Obviously a new model needs to be made which describes the new table, in models.py Inspect the table in the database with
sudo -u postgres psql
d+ new_table_name
Similar to this, but with the correct field/ column names:
class SpatialdataAEFA(models.Model):
gid = models.IntegerField(primary_key=True)
name = models.CharField(max_length=254, blank=True, null=True)
label = models.CharField(max_length=254, blank=True, null=True)
geom = models.GeometryField(blank=True, null=True) # This field type is a guess.
objects = models.GeoManager()
class Meta:
managed = False
db_table = 'spatialdata_aefa'
These have become pretty overloaded over the years, and need tidying up. At the moment we have:
LocalResearch:PostGIS database data
CKAN_datahub:DataHub, powered by CKAN
StatsWales:StatsWales
User:Local user data not stored on the server
Qual:DataPortal Qualitative data
Survey:DataPortal Survey data
Nomis:Data retrieved from the NomisAPI
Also, LocalResearch searches need the extra "remote_value_key" key in the "Display Attributes" dict, pointing at a column name for that model.
Shapefile import process, celery needs explicit export of settings module location
sudo yum install redis
sudo systemctl start redis.service
export DJANGO_SETTINGS_MODULE='wiserd3.settings'
celery -A dataportal3.utils.ShapeFileImport worker --loglevel=info
Assume a search has given us a UUID
Home › Dataportal3 › Nomis searchs › winning_party_11:4bcb579a-e507-404b-9412-c6a89065ca4a:constituency
A "NomisSearch" has this UUID The NomisSearch object has a geography (ie ua, constituency) The NomisSearch has a search attributes dictionary, with key:value data_name:< an identifier for some data > If the NomisSearch type is "Survey" then the Dataset_ID is the an "identifier" of a Survey object
Home › Dataportal3 › Spatial survey links › wisid_AssemblyConstituency_Data_57057b24ed6a1:nawer:aecMaj11
Find "a" SpatialSurveyLink where the Survey_ID, the boundary name and the data_name match the above. This SpatialSurveyLink has a RegionalData dictionary, with key:value < region id > : Here is any identifier which matches the ID in the topojson/geojson (created from a shapefile), defined during the shapefile importing process
Once we have that SpatialSurveyLink
###Constituencies
python dataportal3/utils/ShapeFileImport.py -i path/NAW_3_constituencies.zip -s wisid_AssemblyConstituency_Data_57057b24ed6a1 -u a_user -n constituency_test -g nawer -b "National Assembly Constituency"
python dataportal3/utils/rubbish/update_survey_region_metadata.py -i path/NAW_3/NAWConstituencyDataNamesLookup.csv -s wisid_AssemblyConstituency_Data_57057b24ed6a1
python dataportal3/utils/ShapeFileImport.py -i path/NAW_3_constituencies.zip -s wisid_AssemblyConstituency_Data_57057b24ed6a1 -u a_user -n constituency_test -g nawer -b "National Assembly Constituency" -x True -r constituency
python dataportal3/utils/ShapeFileImport.py -i path/NAW_3/NAW_3_regions.zip -s wisid_AssemblyRegions_Data_5705394bb0ec5 -u ubuntu -n region_test -g nawer -b "National Assembly Region"
python dataportal3/utils/rubbish/update_survey_region_metadata.py -i path/NAW_3/NAWRegionDataNamesLookup.csv -s wisid_AssemblyRegions_Data_5705394bb0ec5
python dataportal3/utils/ShapeFileImport.py -i path/NAW_3/NAW_3_regions.zip -s wisid_AssemblyRegions_Data_5705394bb0ec5 -u ubuntu -n region_test -g nawer -b "National Assembly Region" -x True -r region
python manage.py compilemessages --locale=cy
python manage.py makemessages --locale=cy
tail --bytes=+4 UTF8WithBom.txt > UTF8WithoutBom.txt
A UserGroup has a name and a collection of users The UserGroupSurveyCollection is a collection of surveys which the user group has a read/access claim over A SurveyVisibilityMetadata defines the visibility of a single survey as seen as part of a single UserGroupSurveyCollection
This means: A survey can have multiple visibilities, as multiple UserGroupSurveyCollection's can "own" it e.g. Surveys A, B, C, D, E UserGroups X, Y, Z
Members of group X can view Surveys A, B, C Members of group Y can view C, D, E Members of group Z can view A, E
A single user can be in any/ all groups without any contradictions. If a user both Can and Cannot view a survey due to visibilities specifically denying access, access is still granted.
Xvfb :99 -ac -screen 0 1280x1024x24 &
http://blog.webkid.io/maps-with-leaflet-and-topojson/
https://github.com/gka/chroma.js
http://gka.github.io/chroma.js/#chroma-contrast
http://sgillies.net/blog/1159/topojson-with-python
https://github.com/mbloch/mapshaper/wiki/Command-Reference
http://bootsnipp.com/snippets/featured/quotwaiting-forquot-modal-dialog
http://jrue.github.io/coding/2014/exercises/basicbubblepackchart/
http://stackoverflow.com/a/13332300/2943238
https://github.com/calvinmetcalf/topojson.py
https://github.com/komoot/staticmap
https://dzone.com/articles/taking-browser-screenshots-no
http://selenium-python.readthedocs.io/waits.html
http://www.slideshare.net/syerram/postgres-search-in-django-34452700
https://github.com/javve/list.js/
http://mc.bbbike.org/mc/?num=2&mt0=mapnik&mt1=mapnik-bw
http://www.mapcruzin.com/free-wales-arcgis-maps-shapefiles.htm
http://tuxtrix.com/2014/12/how-to-fix-psql-version-84-server.html
https://www.postgresql.org/docs/9.2/static/upgrading.html
https://www.postgresql.org/docs/9.5/static/functions-json.html
https://www.postgresql.org/docs/current/static/datatype-json.html
https://blog.chaps.io/2016/02/08/upgrading-postgresql-from-9-4-to-9-5-on-ubuntu-15-10.html
https://docs.djangoproject.com/en/1.9/ref/contrib/postgres/fields/#jsonfield
http://docs.celeryproject.org/en/latest/getting-started/brokers/redis.html#broker-redis
http://michal.karzynski.pl/blog/2013/07/14/using-redis-as-django-session-store-and-cache-backend/
http://django-rosetta.readthedocs.io/en/latest/settings.html
CACHES = {
'default': {
'BACKEND': 'redis_cache.RedisCache',
'LOCATION': '/tmp/redis.sock',
},
}
also make sure redis unixsocket is enabled and pointing to the location listed in settings.py
sudo nano /etc/redis.conf
unixsocket /var/run/redis/redis.sock
unixsocketperm 700
ALTER TABLE django_content_type ADD COLUMN name character varying(50) NOT NULL DEFAULT 'someName';