Vi 607 implement identity load testing

[anjolovic]

Summary

• Added a new load testing module to enable automated performance testing of authentication flows and API endpoints. This includes test session management, k6 integration, and proper error handling for Login.gov authentication.
• Working for the Platform team, which will maintain this component as part of our core infrastructure.

Related issue(s)

• [Link to Platform team Jira ticket to be added]

Testing done

• New code is covered by unit tests (UPCOMING)
  • Tests will be added for:
    • TestSessionsController
    • TestSession model
    • LoadTestJob
    • Engine configuration
    • Authentication flows
• Previously, load testing was done manually without proper session management or error handling
• Steps to verify changes:
  1. Install dependencies and run migrations:

     bundle install
     bundle exec rails db:migrate

  2. Configure test settings in config/load_testing.yml
  3. Run test scenario:

     k6 run modules/load_testing/scripts/scenarios/full_auth_flow.js \
       --env API_BASE_URL=http://localhost:3000 \
       --env LOGIN_EMAIL=test@example.com \
       --env LOGIN_PASSWORD=password123

  4. Verify test session is created and results are stored
  5. Check logs for proper error handling and rate limiting

Known testing gaps:

• Unit tests need to be implemented for all components
• Integration tests for full authentication flow needed
• Load test scenarios need validation
• Error handling scenarios need coverage

What areas of the site does it impact?

• Authentication flow testing
• API endpoint load testing
• Test session management
• No impact on production traffic as this is isolated to the load testing module

Acceptance criteria

• Add comprehensive unit tests for controllers, models, and jobs (UPCOMING)
• No error nor warning in the console during normal operation
• Events are being logged to Rails logger with appropriate detail level
• No sensitive information is logged (passwords and tokens are redacted)
• Authentication flows are properly tested with both success and failure scenarios (UPCOMING)
• Added screenshots of test execution and results

Requested Feedback

Specific areas for review:

1. Login.gov form handling approach - ensuring we're following best practices
2. Rate limiting implementation - confirming it's sufficient for our needs
3. Test session management - verifying proper cleanup and resource management
4. Security considerations - ensuring we're not exposing sensitive information
5. Test coverage plan - reviewing proposed test structure and coverage goals

Core Module Files

Controllers

• modules/load_testing/app/controllers/load_testing/v0/test_sessions_controller.rb
  • Manages test session lifecycle (create, show, update, destroy)
  • Handles test session configuration and results
  • API endpoints for k6 test scripts to interact with

Models

• modules/load_testing/app/models/load_testing/test_session.rb
  • Represents a load test session
  • Stores configuration, status, and results
  • Validates test parameters and manages state

Jobs

• modules/load_testing/app/jobs/load_testing/load_test_job.rb
  • Background job for executing load tests
  • Manages test execution lifecycle
  • Handles result collection and storage

Database Setup

Migrations

• modules/load_testing/db/migrate/20241122090000_create_load_testing_tables.rb

  • Creates initial test_sessions table
  • Sets up basic schema for test management

• modules/load_testing/db/migrate/20241122090001_create_load_testing_client_config.rb

  • Creates client_config table for storing OAuth client configurations
  • Manages test client credentials

• modules/load_testing/db/migrate/20241122090002_add_results_to_test_sessions.rb

  • Adds results column to store test execution data
  • Enables storing metrics and performance data

Schema

• db/schema.rb
  • Contains complete database schema including load testing tables
  • Shows relationship between different tables

Configuration

Engine Setup

• modules/load_testing/lib/load_testing/engine.rb
  • Rails engine configuration
  • Sets up autoloading, middleware, and initializers
  • Configures authentication service registration

Routes

• modules/load_testing/config/routes.rb
  • Defines API endpoints for test session management
  • Maps URLs to controller actions

Configuration Files

• modules/load_testing/config/load_testing.yml

  • Module-specific configuration
  • Test parameters and limits
  • Environment-specific settings

• config/settings.yml

  • Application-wide settings
  • Integration with main application

Puma Configuration

• config/puma.rb
  • Web server configuration
  • Handles concurrent test requests

Test Scripts

k6 Scripts

• modules/load_testing/scripts/scenarios/full_auth_flow.js
  • Main load test script
  • Implements Login.gov authentication flow
  • Manages test scenarios and metrics

Documentation

Setup Guide

• .developer-setup
  • Developer setup instructions
  • Environment configuration
  • Test execution guide

API Documentation

• app/controllers/v0/apidocs_controller.rb

  • Swagger documentation controller
  • API endpoint documentation

• spec/requests/swagger_spec.rb

  • Swagger specification tests
  • API documentation validation

Integration

Sign In Controller

• app/controllers/v0/sign_in_controller.rb
  • Main authentication controller
  • Integrates with load testing module

Dependencies

• Gemfile.lock
  • Module dependencies
  • Version locks

Module Specification

• modules/load_testing/load_testing.gemspec
  • Module metadata
  • Dependencies
  • Version information

Load Testing Setup Guide

1. Initial Setup

Install Dependencies

bundle install

Run Migrations

bundle exec rails db:migrate

2. Configure Test Client

Create Client Configuration

Open a Rails console:

bundle exec rails c

Create the test client:

SignIn::ClientConfig.create!(
  client_id: 'load_test_client',
  authentication: 'api',
  anti_csrf: true,
  redirect_uri: 'http://localhost:3000/load_testing/callback',
  access_token_duration: 'PT30M',
  refresh_token_duration: 'P45D',
  description: 'Load Testing Client',
  pkce: true,
  shared_sessions: false,
  service_levels: ['min', 'loa1', 'loa3', 'ial1', 'ial2'],
  access_token_audience: 'load_test_client',
  access_token_attributes: ['email'],
  credential_service_providers: ['logingov', 'idme', 'dslogon', 'mhv']
)

3. Configure Load Testing Settings

Update Configuration File

In modules/load_testing/config/load_testing.yml:

default: &default
  allowed_teams:
    - identity
  max_concurrent_users: 1000
  token_lifetime: 1800 # 30 minutes
  rate_limit:
    requests_per_minute: 60
    burst: 10
  login_credentials:
    logingov:
      email: "your-test-user@example.com"
      password: "your-test-password"

development:
  <<: *default

test:
  <<: *default

staging:
  <<: *default
  max_concurrent_users: 500

4. Run Load Tests

Basic Test Run

k6 run modules/load_testing/scripts/scenarios/full_auth_flow.js \
  --env SESSION_ID=1 \
  --env API_BASE_URL=http://localhost:3000 \
  --env LOGIN_EMAIL=your-test-user@example.com \
  --env LOGIN_PASSWORD=your-test-password \
  --env STAGES='[{"target":1,"duration":"1m"},{"target":1,"duration":"3m"},{"target":0,"duration":"1m"}]' \
  --vus 1

Test with Different Scenarios

Ramp-up Test

k6 run modules/load_testing/scripts/scenarios/full_auth_flow.js \
  --env SESSION_ID=2 \
  --env API_BASE_URL=http://localhost:3000 \
  --env LOGIN_EMAIL=your-test-user@example.com \
  --env LOGIN_PASSWORD=your-test-password \
  --env STAGES='[
    {"target":10,"duration":"1m"},
    {"target":50,"duration":"3m"},
    {"target":0,"duration":"1m"}
  ]' \
  --vus 50

Constant Load Test

k6 run modules/load_testing/scripts/scenarios/full_auth_flow.js \
  --env SESSION_ID=3 \
  --env API_BASE_URL=http://localhost:3000 \
  --env LOGIN_EMAIL=your-test-user@example.com \
  --env LOGIN_PASSWORD=your-test-password \
  --env STAGES='[
    {"target":20,"duration":"5m"}
  ]' \
  --vus 20

5. Monitor Results

View Test Session Results

curl http://localhost:3000/v0/load_testing/test_sessions/{SESSION_ID}

Check Rails Logs

tail -f log/development.log

Common Issues and Solutions

Rate Limiting

If you encounter rate limiting:

• Reduce the number of VUs
• Increase the stage duration
• Add more sleep time between requests

Authentication Failures

If authentication fails:

• Verify client configuration
• Check test user credentials
• Ensure Login.gov sandbox is accessible

Database Issues

If you encounter database errors:

• Verify migrations are up to date
• Check database connections
• Ensure proper permissions

Additional Configuration

Puma Settings

In config/puma.rb:

workers ENV.fetch("WEB_CONCURRENCY") { 2 }
threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
threads threads_count, threads_count

Environment Variables

Required environment variables:

export RAILS_ENV=development
export DATABASE_URL=postgresql://localhost/vets-api_development
export WEB_CONCURRENCY=2
export RAILS_MAX_THREADS=5

Load Testing Setup Guide

1. Initial Setup

Install Dependencies

bundle install

Run Migrations

bundle exec rails db:migrate

2. Configure Test Client

Create Client Configuration

Open a Rails console:

bundle exec rails c

Create the test client:

SignIn::ClientConfig.create!(
  client_id: 'load_test_client',
  authentication: 'api',
  anti_csrf: true,
  redirect_uri: 'http://localhost:3000/load_testing/callback',
  access_token_duration: 'PT30M',
  refresh_token_duration: 'P45D',
  description: 'Load Testing Client',
  pkce: true,
  shared_sessions: false,
  service_levels: ['min', 'loa1', 'loa3', 'ial1', 'ial2'],
  access_token_audience: 'load_test_client',
  access_token_attributes: ['email'],
  credential_service_providers: ['logingov', 'idme', 'dslogon', 'mhv']
)

3. Configure Load Testing Settings

Update Configuration File

In modules/load_testing/config/load_testing.yml:

default: &default
  allowed_teams:
    - identity
  max_concurrent_users: 1000
  token_lifetime: 1800 # 30 minutes
  rate_limit:
    requests_per_minute: 60
    burst: 10
  login_credentials:
    logingov:
      email: "your-test-user@example.com"
      password: "your-test-password"

development:
  <<: *default

test:
  <<: *default

staging:
  <<: *default
  max_concurrent_users: 500

4. Run Load Tests

Basic Test Run

k6 run modules/load_testing/scripts/scenarios/full_auth_flow.js \
  --env SESSION_ID=1 \
  --env API_BASE_URL=http://localhost:3000 \
  --env LOGIN_EMAIL=your-test-user@example.com \
  --env LOGIN_PASSWORD=your-test-password \
  --env STAGES='[{"target":1,"duration":"1m"},{"target":1,"duration":"3m"},{"target":0,"duration":"1m"}]' \
  --vus 1

Test with Different Scenarios

Ramp-up Test

k6 run modules/load_testing/scripts/scenarios/full_auth_flow.js \
  --env SESSION_ID=2 \
  --env API_BASE_URL=http://localhost:3000 \
  --env LOGIN_EMAIL=your-test-user@example.com \
  --env LOGIN_PASSWORD=your-test-password \
  --env STAGES='[
    {"target":10,"duration":"1m"},
    {"target":50,"duration":"3m"},
    {"target":0,"duration":"1m"}
  ]' \
  --vus 50

Constant Load Test

k6 run modules/load_testing/scripts/scenarios/full_auth_flow.js \
  --env SESSION_ID=3 \
  --env API_BASE_URL=http://localhost:3000 \
  --env LOGIN_EMAIL=your-test-user@example.com \
  --env LOGIN_PASSWORD=your-test-password \
  --env STAGES='[
    {"target":20,"duration":"5m"}
  ]' \
  --vus 20

5. Monitor Results

View Test Session Results

curl http://localhost:3000/v0/load_testing/test_sessions/{SESSION_ID}

Check Rails Logs

tail -f log/development.log

Common Issues and Solutions

Rate Limiting

If you encounter rate limiting:

• Reduce the number of VUs
• Increase the stage duration
• Add more sleep time between requests

Authentication Failures

If authentication fails:

• Verify client configuration
• Check test user credentials
• Ensure Login.gov sandbox is accessible

Database Issues

If you encounter database errors:

• Verify migrations are up to date
• Check database connections
• Ensure proper permissions

Additional Configuration

Puma Settings

In config/puma.rb:

workers ENV.fetch("WEB_CONCURRENCY") { 2 }
threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
threads threads_count, threads_count

Environment Variables

Required environment variables:

export RAILS_ENV=development
export DATABASE_URL=postgresql://localhost/vets-api_development
export WEB_CONCURRENCY=2
export RAILS_MAX_THREADS=5