jason.7

.\" Automatically generated by Pandoc 2.6
.\"
.TH "jason" "7" "" "" "cloud and database-wrangler - Linux man page"
.hy
.PP
NAME
.PP
SYNOPSIS
.PP
DESCRIPTION
.PP
OPTIONS
.PP
HISTORY
.PP
SEE ALSO
.PP
BUGS
.PP
AUTHOR
.SH NAME
.PP
jason tevnan - cloud and database-wrangler with a penchant for
automation and sous vide cooking
.SH SYNOPSIS
.PP
jason [--store SQL|NOSQL] [--administrate] [--automate] [--monitor]
[--virtualize] [--containerize] [--script] [--think]
.SH DESCRIPTION
.PP
I worked in a variety of environments, sometimes outside of my
comfort-zone where I have been forced to learn and run technologies in
production at the same time.
I pursue issues with a passion (as any good DBRE/SRE would) and
don\[cq]t settle for it just works.
I love learning new technologies and wrapping my head around new
paradigms, ideally while enjoying dark-roast coffee at my standing desk.
.SH OPTIONS
.SS --store=\f[I]SQL\f[R]
.SS MySQL is my true love.
.IP \[bu] 2
Verifying database workloads, identifying schema bottlenecks, tuning
innodb settings, while providing an HA database can be a lot of fun
.IP \[bu] 2
Scaled Runtastic\[cq]s core database starting at 1M users to 99M users
and the ability to handle 100k QPS
.IP \[bu] 2
Migrated 90+ microservice DBs from Cloudsql to RDS with 7 seconds of
downtime
.IP \[bu] 2
Partitioning a table or verifying the innodb engine status are things i
consider fun
.IP \[bu] 2
Experience finding and debugging DBaaS provider based issues while
gathering all the metrics possible
.IP \[bu] 2
Designed and implemented upgrade system for 200+ mysql 5.7 databases to
8.0 with 3 seconds of downtime
.SS PostgreSQL is different but fun
.IP \[bu] 2
Used it with all its geo-spatial features for OpenStreetMap
.IP \[bu] 2
Automated replicating to another database with slony before 9.0 came out
with log shipping
.IP \[bu] 2
Automated replication and fail-over with repmgrd/patroni
.IP \[bu] 2
Had lots of fun automating pgbouncer with consul as a HA write endpoint
.IP \[bu] 2
Have come to embrace MVCC and the pros and cons that it brings
.IP \[bu] 2
Have solidified my understanding of the pg_% tables and the plethora of
information they provide
.IP \[bu] 2
Love (and sometimes hate) the pg_stat_Statements extension
.SS Oracle is what i cut my teeth on.
.IP \[bu] 2
Had to learn on highly performant, always-on database systems
.IP \[bu] 2
Learned what a missing index can do, which queries to avoid and how to
find the meaning of all those v$ views
.IP \[bu] 2
The experience I gathered understanding B-Tree index look-ups, helped me
to understand look-ups across all database systems
.SS MSSQL
.IP \[bu] 2
Enjoyed a love/hate relationship with Azure SQL server as a core
database
.IP \[bu] 2
Love the depth of the query planner and the insight into live statistics
.IP \[bu] 2
Hate the fact that indexes are built around fragmentable structures and
inline statistics calculation is enabled by default
.SS --store=\f[I]NOSQL\f[R]
.SS MongoDB is webscale... Schema-less?!
.IP \[bu] 2
Attended admin training courses to understand: schema-less can also be a
curse since it is incredibly challenging when defining a scalable,
future proof indexing concept
.IP \[bu] 2
Enjoy shard key definitions for collections to ensure the fewest
scatter-gather queries, Optimizing storage read ahead settings and
playing with Wiredtiger cache options
.IP \[bu] 2
Have lots of experience running the robust HA setup which is very easy
to run from an operations standpoint
.SS Cassandra forgives a lot.
.IP \[bu] 2
I find the replication approach and ring setup amazingly robust with
anisotropic read repairs and the ability to stream and receive data
while accepting queries
.IP \[bu] 2
Scaled Runtastic\[cq]s cluster from 4 to 8 to 32 and finally to 64 nodes
with almost no downtime
.SS Redis and Redis cluster are my nemesis.
.IP \[bu] 2
Run instances under normal conditions as well as high load condition
.IP \[bu] 2
Have found the right balance between RAM and IOs to ensure that they can
flush to disk
.IP \[bu] 2
Since Redis is the core of many ruby queueing solutions, I have been
forced to face my fears, tuning BGSAVE cycles and finding the optimal
fsync to AOF ratios
.SS CouchDB eventually consistent fun
.IP \[bu] 2
Was thrown into cold water with multiple clusters running at high load
.IP \[bu] 2
When building the prometheus exporter for a CouchDBaaS provider, learned
about many of the internal stats and their meanings.
.IP \[bu] 2
Translated that knowledge into practical experience by building
Kubernetes powered scale-out deployments to load test, while starting an
ever growing comprehensive cluster administration documentation.
.SS --administrate
.PP
(run, play, break, repeate)
.SS LAMP
.IP \[bu] 2
Traditional stack (e.g.
apache2, nginx)
.IP \[bu] 2
Extended web servers (e.g.
trinidad, passenger)
.SS Loadbalancers
.IP \[bu] 2
Experience debugging performance bottlenecks
.IP \[bu] 2
Setup instances which handle > 80k rpm
.IP \[bu] 2
HaProxy, nginx
.SS Golden Oldies
.IP \[bu] 2
LOTS of experience with the normal Linux stack (e.g.
bind, dhcpd, ldap, openvpn, ssh, memcached ...)
.IP \[bu] 2
After > 15 years experience with all aspects of the os, i still think
Linux is the best for servers
.SS Distributed fun
.IP \[bu] 2
zookeeper
.IP \[bu] 2
rabbitmq
.IP \[bu] 2
activemq
.IP \[bu] 2
nats (with jetstream)
.IP \[bu] 2
consul (for service discovery)
.IP \[bu] 2
All in clusters running at least 3 nodes
.IP \[bu] 2
Very interesting (i.E.
challenging) to scale
.SS GitLab
.IP \[bu] 2
Run at scale (gitlab.com) as well as company wide implementations.
.IP \[bu] 2
Experience with the joys and pains of CI implementation and
administration.
.SS --automate
.PP
(Automation, testing and auditing is inevitable in today\[cq]s world of
highly fluctuant infrastructure)
.SS Chef
.IP \[bu] 2
Wrote and deployed cookbooks for every aspect of Runtastic\[aq]s
infrastructure
.IP \[bu] 2
Try to ensure that all infrastructure code has full test coverage
.IP \[bu] 2
Test-Kitchen, inspec and chefspec are my friends
.SS Ansible
.IP \[bu] 2
Wrote and deployed roles to automate cache layer deployments
.IP \[bu] 2
Discovered the love/hate relationship in the python\[aq]s jinja2
.SS Terraform
.IP \[bu] 2
Compiled modules to simplify complex deployments
.IP \[bu] 2
Wrote a provider to interface with OpenNebula
.IP \[bu] 2
Used to deploy all aspects of Cabify and Fonoa\[aq]s non application
layer infrastructure
.SS Kubernetes
.IP \[bu] 2
flux - to run large and small infrastructure
.IP \[bu] 2
ArgoCD - and the app-of-app-of-apps
.IP \[bu] 2
Helm - the joys and the follies
.SS --monitor
.PP
(No observability, means not knowing anything)
.SS Nagios3
.IP \[bu] 2
Wrote and deployed numerous checks
.IP \[bu] 2
Running an nrpe based deployment with full automation
.IP \[bu] 2
> 10k checks distributed across 1k servers
.SS Collectd
.IP \[bu] 2
Wrote and integrated checks for nfs-iostat and mongodb
.IP \[bu] 2
Running and fully automated with a graphite front end
.IP \[bu] 2
Collecting > 100k metrics an hour
.SS Cacti
.IP \[bu] 2
Implemented Percona\[cq]s graphing suite for mysql
.IP \[bu] 2
Collect all core database metrics from connections to innodb flush times
.SS Prometheus
.IP \[bu] 2
Alert-manager, recording-rule, exporter - oh my.
Very powerful solution with an ever growing community?
Count me in.
.IP \[bu] 2
Wrote recording/alerting rules with unit tests
.IP \[bu] 2
Experience with some storage engine and memory shenanigans
.IP \[bu] 2
Visualization with graphana
.IP \[bu] 2
Wrote exporters for databases and weather stations
.SS Cloud Services
.IP \[bu] 2
Very familiar with New Relic, Pingdom, Dynatrace, PagerDuty, VictorOps
.SS --virtualize
.PP
(control your destiny - as much as you can)
.SS OpenNebula
.IP \[bu] 2
Experienced every phase of growth from 8 hypervisors to 60
.IP \[bu] 2
Have run opennebula as an EC2 replacement as a native cloud (extensive
API) and as a simple server manager
.IP \[bu] 2
In the process of automating setup and configuration via teraform
.SS KVM
.IP \[bu] 2
Qemu based
.IP \[bu] 2
NFS and Ceph storage backend
.IP \[bu] 2
Currently use it as a minikube virtualizer
.SS Virtualbox
.IP \[bu] 2
Runtastic\[cq]s pre-production system ran on vbox for a long time (hard
to imagine)
.IP \[bu] 2
Mainly running older cookbook tests with vbox
.SS Cloud Services
.IP \[bu] 2
Automate Google Cloud Platform (GCP) and Azure instance deploys with
terraform
.IP \[bu] 2
Experience the joys (its so easy) and pains (why is the db rebooting?)
of not controlling your hypervisors
.SS --containerize
.PP
(run it like
mike (https://en.wikipedia.org/wiki/Mike_the_Headless_Chicken))
.SS Docker
.IP \[bu] 2
Write dockerfiles to encapsulate many applications
.IP \[bu] 2
Build typical applications as well as X based, multi-arch, multi-stage
ones
.IP \[bu] 2
Automated container builds with GitLab CI and BATs
.SS Kubernetes
.IP \[bu] 2
Wrote many manifests for different applications, ranging from banal to
complex
.IP \[bu] 2
Run my own cluster on RaspberryPis for all my home needs
.IP \[bu] 2
Gave a talk at SFSCon about using Flux to automate manifest deployments:
link (https://www.sfscon.it/talks/gitops-with-gitlab-terraform/)
.SS Nomad
.IP \[bu] 2
Experience running complex and simple jobs
.IP \[bu] 2
Integrated with other HashiCorp products (Consul, Hashiui)
.SS LXC
.IP \[bu] 2
As a plugin for new test-kitchen deployments
.IP \[bu] 2
Played around a bit LXD
.SS --script
.SS Bash
.IP \[bu] 2
wrote extensive bash scripts for automation with unit tests (bats)
.IP \[bu] 2
found out that bash has its limits :)
.SS Golang
.IP \[bu] 2
Enjoy writing and maintaining a go backend for a research project
.IP \[bu] 2
Wrote gobench (https://github.com/tnosaj/gobench) to benchmark schemas
in mysql and postgres for high throughput
.IP \[bu] 2
Learned about api design the hardway while using grpc
.SS --think
.SS imho
.IP \[bu] 2
vim > emacs
.IP \[bu] 2
zsh > bash
.IP \[bu] 2
tmux > screen
.SH HISTORY
.SS Cabify (05.2023 - .)
.SS Database Reliability Engineer
.IP \[bu] 2
Built automation for seemless mysql 5.7 to 8.0 live migrations
.IP \[bu] 2
Took on a more staff engineering role focused on mentoring and teaching
.IP \[bu] 2
Drove redefinition of higher level engineers role through out the
company
.IP \[bu] 2
Re-imagine how database related support is handled with data and chatgpt
.IP \[bu] 2
Fully remote
.SS Fonoa (01.2022 - 04.2023)
.SS Database/Site Reliability Engineer
.IP \[bu] 2
Automate, run, manage all database related technologies: MySQL/Postgres
(Google Cloudsql), MsSQL (Azure)
.IP \[bu] 2
Ensured that monitoring and alerting was availible with end2end testing
.IP \[bu] 2
Build exporter for missing SQL
server (https://github.com/tnosaj/mssql_exporter/) metrics
.IP \[bu] 2
Create runbooks for oncall team members with little database context to
ensure service continuity
.IP \[bu] 2
Write design guides to help developers understand their schema and
engine decisions
.IP \[bu] 2
Fully remote
.IP \[bu] 2
Downsized out of a job :/
.SS Cabify (05.2018 - 11.2021)
.SS Database Reliability Engineer
.IP \[bu] 2
Tasked with automating, managing, running all database related
technologies: MySQL (Google Cloudsql), Couchdb (Cloudant), Redis,
Memcached, Elasticsearch
.IP \[bu] 2
Made fully monitored, highly available database creation self service
.IP \[bu] 2
Build exporters for missing observability in DBaaS platform
.IP \[bu] 2
Automate no-downtime sql based CI powered schema changes
.IP \[bu] 2
Continually document and assist developers in making persistence
decisions
.IP \[bu] 2
Support developers in identifying design bottlenecks in query pattern,
database design.
.IP \[bu] 2
Fully remote
.SS GitLab (04.2017 - 04.2018)
.SS Senior Production Engineer
.IP \[bu] 2
Memeber of a small fully remote team
.IP \[bu] 2
Scale gitlab.com (millions of users) using GitLab (typically built for
thousands of users) in a cloud environment
.IP \[bu] 2
Collaborate on developing HA solution for PostgreSQL in the GitLab
omnibus package
.IP \[bu] 2
Strove to fully automate environments from terraform to multi-tiered HA
stack
.IP \[bu] 2
Build a back-end agnostic solution for secrets in chef
.IP \[bu] 2
Use chef to automate all-the-things
.IP \[bu] 2
Fully remote
.SS Runtastic GmbH (04.2012 - 03.2017)
.SS Infrastructure Architect
.IP \[bu] 2
Define setup and strategy for each upcoming stack
.IP \[bu] 2
Ensure scalability of technologies and concepts
.IP \[bu] 2
Setup workflows for automation and deployments
.SS Head of Operations
.IP \[bu] 2
Organize small team while fighting to stay ahead of growth
.IP \[bu] 2
Very challenging for me to lead a team of inexperienced ops and shaping
our infrastructure
.SS Operations Engineer
.IP \[bu] 2
Nested under the web development team
.IP \[bu] 2
Start automation
.IP \[bu] 2
Improve uptime through monitoring and derive future actions
.IP \[bu] 2
Conceptualize private cloud based on opennebula
.SS APEX Gaming (04.2010 - 03.2012)
.SS Head of Customer Care
.IP \[bu] 2
Setup ticketing workflow based on ITIL best practices
.IP \[bu] 2
Created automated master/slave setup with slony for PostgreSQL 8.3/8.4
.IP \[bu] 2
Spent time training staff in the casino headquarters to be first level
support techs
.SS Knapp Systems Integration (09.2007 - 03.2010)
.SS Technical Project Lead
.IP \[bu] 2
Introduce metric collection to visualize hardware utilization for the
customer
.IP \[bu] 2
Manage customer care projects
.IP \[bu] 2
Responsible for everything from planning to doing
.IP \[bu] 2
Largest project was complete warehouse upgrade to a medium sized 24x7
cosmetic distributed
.IP \[bu] 2
Organized and held numerous on-site training courses around the world
.SS Second Level Support Engineer
.IP \[bu] 2
Field production problems in a 24x7 environment
.IP \[bu] 2
Handle issues ranging from PLC (Siemens S7) to tablespace cleanups on a
core Oracle instance
.SS FH Joanneum - University of Applied Science
.SS Bachelor of Science in Software Engineering
.IP \[bu] 2
Extra-occupational program
.SH SEE ALSO
.IP \[bu] 2
Email: jason.tevnan\[at]gmail.com
.IP \[bu] 2
Phone: +43.650.2167444
.IP \[bu] 2
LinkedIn: https://at.linkedin.com/in/jason-tevnan-5390b4a8
.SH BUGS
.PP
Prone to flu if left in rain.
.SH AUTHOR
.PP
Jason Tevnan (jason.tevnan\[at]gmail.com)