Skip to content

Integrate with Teletraan

Jump to bottom
Tyler Ouyang edited this page Feb 22, 2023 · 18 revisions

To plugin Teletraan into your existing release workflow, you would need to

Run Teletraan Services and DeployBoard In Production

The Installation section explains how to setup Teletraan for development. The out-of-box Teletraan service Start Script & Config and the DeployBoard Start Script & manage.py are for development purpose. You are recommended to create new config files and start scripts for production. Refer to the comments in config yaml file and manage.py for details on how to customize these configuration files.

We use Teletraan to deploy Teletraan at Pinterest. Basically we use the Production Teletraan service to deploy Teletraan code changes to a staging Teletraan service (the integ deploy agents talk to prod teletraan service). We then make sure the staging Teletraan service works as expected, and deploy the code changes to the Production Teletraan service (the prod deploy agents talk to integ teletraan service). To be able to deploy Teletraan using Teletraan, we made the Teletraan code itself Deployable, by creating a teletraan directory under the Teletraan source tree, and added the production start scripts and config files under it. We did not share the teletraan directory and configs since it is very Pinterest specific. We did the setup for DeployBoard as well.

We have the following tips we'd like to share:

  • We run Teletraan service and Teletraan Agent service in separate clusters
  • We run multiple deployboard process on the same box behind an nginx HTTP server
  • Both host-info file and DeployAgent are installed through provisioning pipeline. The DeployAgents are configured to run every min as a cron job
  • To make it easier to create Deploy Scripts, we created a python package, deploy-utils, to share some common routines, such as create directories and files, restart monit or supervisor managed services, register with Service Discovery etc. We do plan to open source deploy-utils in the near future.
  • We created a publish-builds jenkins job, and make it a downstream job for all our service builds to call in the end.

Make application deployable

Add Deploy Scripts

Teletraan can deploy services written in any programming language, as long as any of the following deploy scripts packaged into your service build artifacts:

  • PRE_DOWNLOAD
    Before deploy agent download the build. If build download might affect your service in anyway, you might want to use this opportunity to shutdown your service first. In general this script is seldom used.
  • POST_DOWNLOAD
    After deploy agent download the build. This script is also seldom used.
  • PRE_RESTART
    Before deploy agent restart the service. Usually this is the time to unregister with load-balancer or service discovery, drain traffic, basically anything needs to be done before shutdown your service.
  • RESTARTING
    Deploy agent is restarting the service. Could be just start, if service has already been stopped previously
  • POST_RESTART
    After deploy agent restart the service. Usually this is the time to warm up cache, run smoke tests or any sanity check, register back load-balancer or service discovery, and start to receive traffic etc.
  • STOPPING
    Deploy Agent is shutting down the service. This script is executed when the host is being terminated and not when the running service is being stopped or restarted. Thus it is the last chance to manage the service.

Although being referred as Deploy Scripts, they can be written in any programming language, as long as they are executable. Also, even though we provided some recommendations above regard what each script should do, but you are free to use them to do just about anything you need. Sky is the limit when it comes to these Deploy Scripts.

  • It is important that these scripts use the exact same names as above, and they are put under a specific directory teletraan directly. The teletraan directory will have to be top-level directory in the artifact tar ball. For example:
>tar tvf servicex-adb87dc.tar.gz
lib/servicex.jar
teletraan/RESTARTING
teletraan/POST_RESTART

  • It is important to know that during deploy, deploy agent executes the PRE_DOWNLOAD and POST_DOWNLOAD scripts from the current serving build, and will execute the PRE_RESTART, RESTARTING and POST_RESTART scripts from the newer version of build (to be deployed). Since the very first deploy does not have "current serving build", so no PRE_DOWNLOAD and POST_DOWNLOAD will be executed. It is also important to know during rollback, the current PRE_DOWNLOAD and POST_DOWNLOAD are used, and the previous version of PRE_RESTART, RESTARTING and POST_RESTART will then be used.

  • Sometimes the same build artifact could be used to run different services based on different configurations. In this case, create different directories under the top-level teletraan with the deploy environment names, and put the corresponding deploy scripts under the proper environment directories separately. For example:

>tar tvf service-adb87dc.tar.gz
lib/service.jar
teletraan/serverx/RESTARTING
teletraan/serverx/POST_RESTART
teletraan/servery/RESTARTING

Host level deploy cycle

UNKNOWN->PRE_DOWNLOAD->[DOWNLOADING]->POST_DOWNLOAD->[STAGING]->PRE_RESTART->RESTARTING->POST_RESTART->SERVING_BUILD

As you might have already guessed, deploy agent executes the corresponding deploy scripts (exact match) in each of the steps except DOWNLOADING and STAGING, which are reserved by Teletraan to download the build artifacts, untar, create symbolic links and change permissions etc. It is generally not recommended to mess with these build artifacts, directories, links manually on host.

Pass Environment Variables to Deploy Scripts

Deploy agent passes the following environment variables to Deploy Scripts when invoking them:

  • Environment Related
    • ENV_NAME
    • STAGE_NAME
  • Deploy Related
    • DEPLOY_ID
    • DEPLOY_STEP
    • OPCODE
    • DEPLOY_TYPE
  • Build Related
    • BUILD_COMMIT
    • BUILD_NAME
    • BUILD_REPO
    • BUILD_BRANCH
    • BUILD_ID
    • BUILD_URL
  • Deploy Agent Config Related
    • TARGET
    • BUILDS_DIR

Customized Environment Variables

You can also define a set of Environment Variables through Script Config in the deployboard UI. They will be passed to all the Deploy Scripts when executing them. It gives Teletraan server extra power to control deploy agents and the services. For example, one can define LOG_LEVEL in the Script Config section on deployboard. Without creating a new build and deploy, one can simply change the LOG_LEVEL on deployboard, restart the current deploy, and the service will start to generate different level logging.

Use Script Templates

Any scripts or config files with .tmpl suffix under teletraan directory will be examined by Deploy Agent in STAGING step. Any variables in these files in the form of {$TELETRAAN_VALUE1:default} will either be replaced by Script Config in the deployboard UI, or if missing in the UI it will use the default value. This feature is important when these .tmpl files will be used outside Deploy Agent scripts scope. Once transformed, the .tmpl suffix will be dropped.

Run Deploy Agents

Deploy Agent is released as a pip package and can be installed by "pip install deploy-agent". Deploy Agent can be run either in a long-running daemon mode or a command mode, which will exit for each run. It is up to you to decide which way to use to best fit your infrastructure. At Pinterest every machine created with a deploy-agent cronjob entry to run every minute:

* * * * * /mnt/deployd_venv/bin/deploy-agent -f /etc/deployagent.conf >/var/log/deployd/deploy-start.log 2>&1

Customize Deploy Agent

You can customize Deploy Agent with a config file. A sample config file could be found at here.

Provide Host Info File (optional)

Deploy Agent prefers to run with a host_info file on host. Without host_info file, Deploy Agent will still be able to communicate with Teletraan server. But in this case, Deploy Agent will only provide Teletraan server with its hostname. One can certainly associate individual host name with a particular Deploy Environment as target host to deploy to, but the real power of Teletraan comes when associating a cluster of, or a group of, hosts with a particular environment. Whenever new hosts are provisioned to be part of certain group, Teletraan will make sure these new hosts running on the same version of code as the rest of the existing members in that group. See Capacity Config for more details on how to associate hosts and groups with certain deploy environment.

We could've made Teletraan to have the ability managing cluster or group membership, but we decided to leave it to dedicated host provisioning process instead. Not only we think this is something the provisioning process should provide or should easily provide (Puppet group?), but also it simplifies Teletraan. That said, we are open to any future development of managing group or cluster membership directly through Teletraan.

A host_info file could look like:

>cat /mnt/deployd/host_info 
groups=prod,servicex
id=i-dcbcee23
host=service-x-001

Notice that there could be multiple groups for a host, which basically makes a group hierarchy. Teletraan supports deploying multiple services onto the same host. Combining with the group hierarchy, Teletraan provides a powerful way to deploy services to different scope of hosts. In the above example, a more basic service could be deployed to all prod hosts, including all the hosts belong to servicex.

The location of this host_info File can be customized through Deploy Agent config file.

Publish Builds

Add Publish Build as the last step of your build pipeline. Once published, the build is available for deploys.

Clone this wiki locally