Playbook: Dealing with API abuse

Every so often we get a rouge user hammering the API and causing heavy load on the system. Here are some general steps to diagnose and block.

Often the first sign of trouble is the "Postgres High Rows Returned" metric. (https://app.datadoghq.com/dash/integration/postgresql?live=true&tpl_var_scope=env%3Aproduction)

So now we need to figure out what is causing the extra traffic. On the load balancer server, we can use the ngxtop tool to quickly analyze the incoming traffic.

We first run sudo ngxtop top remote_addr to view the top list of incoming IPs. If there is a rouge user we will usually see it jump to the top after a few seconds.

running for 16 seconds, 8287 records processed: 514.70 req/sec

top remote_addr
| remote_addr    |   count |
|----------------+---------|
| 1.2.3.4        |    1536 |
| 5.6.7.8        |     334 |
| 5.6.7.1        |     258 |
| 5.6.7.2        |     139 |

Then we run sudo ngxtop -i 'remote_addr=="1.2.3.4"' (filling in the IP we found) to see what requests this user is hitting. It's usually pretty clear that they are either hitting the same url repeatedly or a series of urls to scrape data.

running for 72 seconds, 2695 records processed: 37.38 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|    2695 |          162.000 |     0 |     0 |  2695 |     0 |

Detailed:
| request_path                                  |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|-----------------------------------------------+---------+------------------+-------+-------+-------+-------|
| /api/v1/activity/just_updated.json            |    1339 |          162.000 |     0 |     0 |  1339 |     0 |
| /api/v1/activity/latest.json                  |    1691 |          162.000 |     0 |     0 |  1691 |     0 |

Now we need to block the offending IP. Usually we don't need to block long-term but we need to get the platform stable again.

Add a line to /etc/nginx/blacklist.conf like this: deny 1.2.3.4;. Then reload nginx: sudo service nginx reload.