Protecting from spammy requests

[swombat]

As soon as a site goes live properly, it's going to be hit by an endless stream of spammy requests trying to hack into the server. These end up causing some annoyance and filling up the error logs (and using up the error logging capacity on nothing).

One simple solution is to install rack-attack, and implement a simple blocklist:

class Rack::Attack
  spammy_path_includes = [
    'geoserver',
    '/vpn',
    '/webui',
    '/PSIA',
    '/HNAP1',
    '/wp-',
    '/xmlrpc',
    '.php',
    'wordpress',
    'cgi-bin',
    '.env',
    'merchants',
    'readme.html',
    'license.txt',
    'mailman',
    '/remote',
    '/owa',
    '/ecp',
    '_ignition',
    'hetong.js',
    'check.js'
  ]
  Rack::Attack.blocklist('block hacking requests') do |req|
    spammy_path_includes.select { |spam| req.path.downcase.include?(spam.downcase) }.any?
  end
end

This blocks anyone accessing a URL that contains one of those strings somewhere in it. Possibly some of those should not be blocked, application-dependent, but for example just blocking all requests with "wp-" in them will remove 90% of the hack attempts which try to break into an insecure Wordpress blog. Worth noting the block is per-request. If a user mistakenly requests one of those they are not blocked permanently.

While at it, it may be worth adding the other two recommended rack-attack throttles:

  ### Throttle Spammy Clients ###

  # If any single client IP is making tons of requests, then they're
  # probably malicious or a poorly-configured scraper. Either way, they
  # don't deserve to hog all of the app server's CPU. Cut them off!
  #
  # Note: If you're serving assets through rack, those requests may be
  # counted by rack-attack and this throttle may be activated too
  # quickly. If so, enable the condition to exclude them from tracking.

  # Throttle all requests by IP (60rpm)
  #
  # Key: "rack::attack:#{Time.now.to_i/:period}:req/ip:#{req.ip}"
  throttle('req/ip', limit: 300, period: 5.minutes) do |req|
    req.ip unless req.path.start_with?('/assets')
  end

  ### Prevent Brute-Force Login Attacks ###

  # The most common brute-force login attack is a brute-force password
  # attack where an attacker simply tries a large number of emails and
  # passwords to see if any credentials match.
  #
  # Another common method of attack is to use a swarm of computers with
  # different IPs to try brute-forcing a password for a specific account.

  # Throttle POST requests to /login by IP address
  #
  # Key: "rack::attack:#{Time.now.to_i/:period}:logins/ip:#{req.ip}"
  throttle('logins/ip', limit: 5, period: 20.seconds) do |req|
    if req.path == '/users/sign_in' && req.post?
      req.ip
    end
  end

Thoughts?

[swombat]

Fwiw, I've grown tired of updating the list of banned URLs already, and so instead I've started whitelisting URLs...

This makes it really helpful that most of the functionality in a BulletTrain app is under /account :-)

  routes_paths = [
    '/users',
    '/account',
    '/webhooks',
    '/oauth',
    '/testing',
    '/invitation',
    '/rails',
    '/recede_historical_location',
    '/resume_historical_location',
    '/refresh_historical_location',
    '/assets',
    '/public/assets',
    '/cable'
  ]

  Rack::Attack.blocklist('block paths that are not in routes') do |req|
    !routes_paths.any? { |path| req.path.downcase.start_with?(path) } &&
      !(req.path == "/" && req.get?)
  end

[jagthedrummer]

This is an interesting idea. I definitely like things that point in the direction of "secure by default". @gazayas @kaspth @pascallaliberte @andrewculver, what do y'all think?

I wonder if we could generate that allow list dynamically by inspecting/walking the actual routes in the application.

[swombat]

Fwiw, blocklisting became a Sisyphean task so I've ended up going for whitelisting instead:

  routes_paths = [
    '/users',
    '/account',
    '/webhooks',
    '/oauth',
    '/testing',
    '/invitation',
    '/rails',
    '/recede_historical_location',
    '/resume_historical_location',
    '/refresh_historical_location',
    '/assets',
    '/public/assets',
    '/cable'
  ]

  Rack::Attack.blocklist('block paths that are not in routes') do |req|
    !routes_paths.any? { |path| req.path.downcase.start_with?(path) } &&
      !(req.path == "/" && req.get?)
  end

Since I put this in, my rollbar logs are no longer filling up with garbage requests, and the app seems to work just fine!

[kaspth]

Looks like there's a routes.reognize_path method that assert_recognizes uses internally, which be usable for this.

https://github.com/rails/rails/blob/747a03ba7722b6f0a7ce42e86cea83cf07a2e6ef/actionpack/lib/action_dispatch/testing/assertions/routing.rb#L260

Something like this maybe?

  Rack::Attack.blocklist('block paths that are not in routes') do |request|
   Rails.application.routes.recognize_path(request.path, method: request.method, extras:)
  end

I have no idea what the extras are supposed to be.

[kaspth]

It may be costly to run through the routing tree twice — Rails already does that to serve the request, so I think there'd have to be somewhere internally where Rails raises on an unknown route.

Because by default, Rails rescues some exceptions and converts them into 404 responses. These seem pertinent: https://github.com/rails/rails/blob/692f25a9254c58c64b138770e6b604e73de38620/actionpack/lib/action_dispatch/middleware/exception_wrapper.rb#L13-L14

You may be able to go further based on these exceptions and ignore them specifically in Rollbar. I'm not sure if there's cases where you'd want to have 404 exceptions in Rollbar in general.