Skip to content
This repository has been archived by the owner on Feb 15, 2024. It is now read-only.

Latest commit

 

History

History
461 lines (397 loc) · 17.4 KB

demo.md

File metadata and controls

461 lines (397 loc) · 17.4 KB
Raw

brick: Demo

brick project logo

Table of contents

Why brick was created

This demo is intended to answer that question by covering:

  1. brief history of dealing with account abuse
  2. current functionality provided by this application
  3. future improvements, pending approval of time and resources

Preparation

Environment

  1. Boot throwaway Ubuntu Linux 18.04 LTS desktop VM older or newer versions may work, but have not received as much testing
  2. Install git
    • sudo apt-get update && sudo apt-get install -y git
  3. Clone remote repo
    1. cd $HOME/Desktop
    2. git clone https://github.com/atc0005/brick
      • Note: Depending on project development timing, this might actually be a different repo path
  4. Run demo scripts 1-4
    1. contrib/demo/scripts/01-install-docker.sh
    2. contrib/demo/scripts/02-install-dependencies.sh
    3. contrib/demo/scripts/03-install-demo-tooling.sh
    4. contrib/demo/scripts/04-setup-env.sh
  5. Shutdown VM
  6. Take snapshot
  7. Boot VM
  8. Run go version to ensure that go is properly registered in PATH
  9. Run demo script 5
    • contrib/demo/scripts/05-deploy.sh
  10. Reset demo environment
    • run contrib/demo/scripts/06-reset-env.sh
    • this is even for the first part of the demo to ensure we are working from a consistent state throughout the demo
    • this does not modify existing copies of the config files in the /usr/local/etc/brick/ directory

Web browser / MailDev

  1. Open a web browser directly within the demo VM or on the VM host
  2. Navigate to the IP Address of the VM + :1080
    • e.g., http://192.168.92.136:1080/ or http://localhost:1080/
  3. Enable automatic refresh of mailbox contents

Visual Studio Code

  1. Add brick repo path to workspace

  2. Add /var/log/brick path to workspace

  3. Add /usr/local/etc/brick path to workspace

  4. Add /var/cache/brick path to workspace

  5. Add /usr/local/ezproxy path to workspace

  6. Install extensions

    • Even Better TOML (tamasfe.even-better-toml)

  7. Configure settings

    {
   "workbench.startupEditor": "newUntitledFile",
   "telemetry.enableTelemetry": false,
   "telemetry.enableCrashReporter": false,
   "workbench.activityBar.visible": true,
   "editor.minimap.enabled": false
}

  8. Open /usr/local/etc/brick/config.toml

  9. Open /usr/local/etc/brick/users.brick-ignored.txt

  10. Open /usr/local/etc/brick/ips.brick-ignored.txt

We're going to make changes to these latter files in a bit and having them open already during the demo should help speed things up.

brick

Configure brick:

  1. Open /usr/local/etc/brick/config.toml
  2. Set ignore_lookup_errors to true
  3. Retrieve webhook URL from Microsoft Teams Connector
    • assumption: We will use an existing "testing" channel
  4. Set msteams.webhook_url to the webhook URL retrieved from the test Microsoft Teams channel
  5. Comment out msteams.webhook_url line
    • this disables Microsoft Teams notifications for now
  6. Set ezproxy.terminate_sessions to true
    • TODO: Decide if this will be enabled initially, or as a follow-up item
  7. Configure email settings
    1. email.server to localhost
    2. email.port to 25
    3. email.client_identity to brick
      • a production system should set this to the fully-qualified hostname of the sending system
      • a production system should also have a Forward-confirmed reverse DNS (FCrDNS configuration to enable the most reliable delivery possible
    4. email.sender_address to brick@example.org
    5. email.recipient_addresses to ["help@example.org"]
    6. email.rate_limit to 3
    7. email.retries to 2
    8. email.retry_delay to 2
  8. Comment out email.server
    • this disables email notifications for now

Terminal

Configure guake for demo:

  1. create labeled tabs
    1. Deploy
    2. Splunk (Payload)
    3. syslog entries
    4. disabled user entries
    5. reported user entries
    • aka, "action" entries
    1. fail2ban log
  2. Configure 100% height/width
  3. Disable transparency
  4. Enable 9000 lines scroll-back (hopefully not needed)
  5. Clear screen in each guake tab, tail log files, pipe through ccze -A
    1. Deploy
      • none
    2. Splunk (Payload)
      • No log file; stage curl command
        1. cd $HOME/Desktop/brick/contrib/tests
        2. curl -X POST -H "Content-Type: application/json" -d @splunk-sanitized-payload-formatted.json http://localhost:8000/api/v1/users/disable
    3. syslog entries
      • clear && tail -f /var/log/brick/syslog.log | ccze -A
    4. disabled user entries
      • clear && tail -f /var/cache/brick/users.brick-disabled.txt | ccze -A
    5. reported user entries
      • clear && tail -f /var/log/brick/users.brick-reported.log | ccze -A
    6. fail2ban log
      • clear && tail -f /var/log/fail2ban.log | ccze -A
    7. email log
      1. clear && tail -f /var/log/mail.log | ccze -A

Presentation

Slideshow

  1. Start slideshow
  2. Step through each slide, checking for notes in the presenter's Notes box
  3. Make sure to pause momentarily for questions
  4. On the Demonstration slide, switch back to this document (offscreen)

First payload with as-is demo settings

  1. Start with payload tab
    • Mention that this emulates receiving a Splunk alert
    • We have permission from NetSec to run the search every 5 minutes
    • Splunk alerts will continue to trigger for every Username/IP pair associated with the thresholds we have defined for the Splunk alerts.
    • We may have to adjust those thresholds to get the desired behavior over time.
  2. Switch to syslog entries tab
    1. syslog entries are automatically forwarded to sawmill1 and on to Graylog where they are searchable
    2. Mention the periodic stats output showing how many notification messages were generated vs what was actually set
      • At this point Teams should have all zeros; we haven't enabled Teams notifications yet
  3. Switch to disabled user entries tab
    • Emphasize format and fields recorded
      • comment
        1. Username
        2. Source IP (aka, User IP)
        3. Alert name (1:1 with name shown in Splunk Alerts panel)
        4. Alert sender IP (aka, Distributed IT Splunk search head)
          • Useful if we have someone intentionally (or otherwise) attempt to spoof a Splunk alert payload
          • Note: Splunk does not appear to support providing any sort of authentication token/key to prove its identity, so we'll need to use host-level firewall rules to help control abuse
        5. SearchID (should be 1:1 with value recorded in Splunk's logs if we ever need to dig that deep)
      • username::deny format
        • matches existing users.disabled.txt file format which we maintain by hand
        • plan for disabled user files
          • keep one file for manual entries
          • add another to be automatically maintained by brick
  4. Switch to reported user entries
    1. Mention that the entries come in pairs
      1. an alert was received
      2. this is what we did about it
    2. Mention that there is no guarantee that they'll be in order (though they often will be)
      • use SearchID to match up report/action pairs
    3. Mention that we'll come back to this tab later to show additional entry types
  5. Switch to fail2ban log tab
    1. Note bantime value
    2. Note found line
    3. Note Ban line
    4. Mention that we'll come back in a moment
  6. Open web browser
    1. Navigate to http://localhost:1080/
    2. Show the fail2ban alert email
      1. mention that GeoIP functionality would be available for non-private IPs
      2. mention that we can customize the alerts to use Redmine "include" pages to allow a standardized "what do I do with this alert?" guide

Simulate a repeat Splunk alert

  1. Switch back to Splunk (Payload) tab
    1. Submit another payload for the same username
    2. Mention that this simulates a repeat alert in case the Splunk Agent/Forwarder on the EZproxy server gets "stuck" after the initial alert and finally unsticks sending in queued entries after the "cooldown" timer on the Splunk "search head" expires, which from brick's perspective is a duplicate disable request
  2. Switch to the syslog entries tab
    • Note "already disabled" wording
  3. Switch to the disabled user entries tab
    • Note that there is still only one entry
      • this is as expected
  4. Switch to the reported user entries tab
    • Note the new pair of entries
      1. Report: same format as before
      2. Action: same "DISABLE" prefix as before, but …
        1. The log entries explicitly notes that the user account has already been disabled, but had it not, it would be again due to XYZ alert (should be a different SearchID than the original disable action)
        2. The IP Address could be different and often would be different for cases where a compromised account is shared. We want to disable ALL IPs associated with a compromised account (note that we'll return to that shortly; (we're going to demo the "ignored users/ips" support, but don't mention that just yet)
  5. Switch to the fail2ban log tab
    1. Note that it recognizes the "duplicate" report
      • The current configuration doesn't do this, but fail2ban offers the ability to reset unban timers for each additional match
      • We don't do that on purpose; we want to ban the IP of the original disable request only long enough to terminate the matching user session as other legitimate users may be coming from that IP.
    2. Note that once the fail2ban timer expires for the original IP, other users from that IP Address will be allowed to connect to EZproxy again. The disabled account will stay disabled.

Enable Teams and email notifications

Use the same test username as before.

  1. Enable Teams webhook in /usr/local/etc/brick/config.toml
    • Uncomment webhook URL entry added earlier
  2. Reset demo environment
  3. Submit two payloads
  4. Switch to Teams channel
    • First message indicates payload receipt
    • Second message indicates errors (if any), actions taken
  5. Switch to the syslog entries tab
    • point out Teams notifications details
  6. Switch to the disabled user entries tab
    • show that there is still only one entry
  7. Switch to the reported user entries tab
    • Show familiar entries
  8. Switch to the fail2ban log tab
    • Show familiar entries
  9. Switch to the MailDev container
    • Accessible within demo VM or externally on NAT network
    • Step through all notifications

Same test username, different IP Address

Do not reset demo environment; we need to have the prior entries/settings as-is.

  1. Modify the contrib/tests/splunk-sanitized-payload-formatted.json file
    • Replace test account IP Address with 123.123.123.123
  2. Switch to Splunk (Payload) tab
    • Submit payload
  3. Switch to syslog entries tab
  4. Switch to disabled user entries tab
    • Note that the user account won't be disabled twice
  5. Switch to the reported user entries tab
    • Note that the "already disabled" message is present as before, but now we see a different IP Address than when the user was first banned
  6. Switch to the fail2ban log tab
    • Mention that the new IP associated with the disabled user account has been banned
      • This will cause the associated session to timeout

Ignored username, different IP Address

  1. Modify the contrib/tests/splunk-sanitized-payload-formatted.json file
    1. Replace earlier test account with mine
    2. Replace test IP Address with 4.4.4.4
  2. Add my user account to the ignored user file
    • /usr/local/etc/brick/users.brick-ignored.txt
    • Should already be open in VS Code from earlier prep step
  3. Submit the updated payload
  4. Switch to the syslog entries tab
    • Note the Ignored entry there
  5. Switch to the disabled user entries tab
    • show that no new entries were added
  6. Switch to the reported user entries tab
    • show that an IGNORED entry was added along with sufficient details explaining why
    • "ignored" user accounts are "protected" from disable actions (see next)
  7. Switch to the fail2ban log tab
    • show that no new bans are present
    • Note: Other requests from Splunk tied to the same IP for other usernames won't disable this account, but they will be temporarily banned due to the shared IP Address. Once the IP is unbanned, non-disabled accounts can use EZproxy as before
  8. Switch to Microsoft Teams a. Show the notification pair

Ignored IP Address, different username

  1. Modify the contrib/tests/splunk-sanitized-payload-formatted.json file
    • Replace my account with another team member's
  2. Add IP Address to the ignored IP Address file
    • IP: 10.10.10.10
      • arbitrary, just writing it out here for reference
    • File: /usr/local/etc/brick/ips.brick-ignored.txt
      • Note: should already be open in VS Code from earlier prep step
  3. Switch to the payload tab
    • Submit payload
  4. Switch to the syslog entries tab
    • Note ignored entry there
  5. Switch to the disabled user entries tab
    • show that no new entries were added
  6. Switch to the reported user entries tab
    • show that an IGNORED entry was added along with sufficient details explaining why
  7. Switch to the fail2ban log tab
    • show that no new bans are present
    • Note: all non-disabled user accounts associated with the ignored IP entry are "protected" or ignored
  8. Switch to Microsoft Teams
    • Show the notification pair

High-level overview

See Overview doc for additional details.

Improvements

Recent

  • Email notifications directly from brick

    • analogue to Teams notifications

  • Support for automatic sessions termination

    • using official ezproxy binary
    • using unsupported kill subcommand of official ezproxy binary

Planned

  • Additional endpoints
    • list disabled user accounts
      • user accounts disabled by this application (users.brick-disabled.txt)
      • user accounts disabled manually (users.disabled.txt)
    • list log messages associated with disabled user accounts

Potential

Many of these improvements are likely on hold pending investment of time/support from leadership.

  • "Warning-only" behavior based on specific fields or alert name prefixes
    • Disable for high-confidence alerts
    • Warning for anything else
  • LDAP
    • Use LDAP package to add disabled users to a Library-specific AD group
    • Update EZproxy (test instance first) to deny login access to users in this AD group
  • Refactoring to allow brick to take general actions from Splunk, Graylog or another alert
    • restart a service
    • force a user account to logout & ban them
    • submit API request to Service Now to indicate problems with a system
    • submit API requests to Redmine to add, create or update tickets
  • Refactoring to allow brick to receive payloads from other services (and take specific actions, perhaps branching paths based on the notification source or endpoint used (most likely)
    • GitHub
    • Office 365
    • Our own services/tooling
      • e.g., RSS or Atom feed updates monitoring by another could result in a payload submission to brick which kicks off a pipeline

Upstream feature requests

The following enhancement requests would help regardless of whether we use brick:

  • Add support for dynamically denying access to specified IPs
    • without restarting EZproxy
  • Add (official) support to terminate active user sessions via API or other external control
    • without restarting EZproxy
    • Note: This support is available as of right now (learned this after the May 2020 demo), but OCLC Support has indicated it isn't official and could go away without notice.

References

These are references intended for the audience to review just after a demo wraps up.