# Performance Testing Guide

This guide provides comprehensive performance testing strategies for the Payment Service to ensure it can handle millions of requests per minute.

## 🎯 Performance Targets

- **Throughput**: 1,000,000+ requests per minute
- **Latency**: < 100ms for 95th percentile
- **Availability**: 99.9% uptime
- **Concurrent Users**: 10,000+ simultaneous connections

## 🛠️ Testing Tools

### 1. Hey (HTTP Load Testing)
```bash
# Install hey
go install github.com/rakyll/hey@latest

# Basic load test
hey -n 10000 -c 100 http://localhost:8080/health

# Payment initiation test
hey -n 1000 -c 50 -m POST \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"phone_number":"254712345678","amount":100,"description":"Test payment"}' \
  http://localhost:8080/api/v1/payments/initiate
```

### 2. Apache Bench (ab)
```bash
# Install ab (usually comes with Apache)
# Ubuntu/Debian: sudo apt-get install apache2-utils
# CentOS/RHEL: sudo yum install httpd-tools

# Basic test
ab -n 10000 -c 100 http://localhost:8080/health

# POST request test
ab -n 1000 -c 50 -p payment.json -T application/json \
  -H "Authorization: Bearer YOUR_TOKEN" \
  http://localhost:8080/api/v1/payments/initiate
```

### 3. Artillery
```bash
# Install Artillery
npm install -g artillery

# Run configuration
artillery run artillery-config.yml
```

### 4. K6
```bash
# Install k6
# Ubuntu/Debian: sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
# echo "deb https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
# sudo apt-get update && sudo apt-get install k6

# Run test
k6 run performance-test.js
```

## 📊 Test Scenarios

### 1. Baseline Performance Test
```yaml
# artillery-config.yml
config:
  target: 'http://localhost:8080'
  phases:
    - duration: 60
      arrivalRate: 10
    - duration: 120
      arrivalRate: 50
    - duration: 60
      arrivalRate: 100

scenarios:
  - name: "Health Check"
    weight: 30
    flow:
      - get:
          url: "/health"

  - name: "Payment Initiation"
    weight: 70
    flow:
      - post:
          url: "/api/v1/payments/initiate"
          headers:
            Authorization: "Bearer {{ token }}"
            Content-Type: "application/json"
          json:
            phone_number: "254712345678"
            amount: 100
            description: "Performance test payment"
```

### 2. Stress Test
```javascript
// k6-stress-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
  stages: [
    { duration: '2m', target: 100 }, // Ramp up
    { duration: '5m', target: 100 }, // Stay at 100 users
    { duration: '2m', target: 200 }, // Ramp up to 200 users
    { duration: '5m', target: 200 }, // Stay at 200 users
    { duration: '2m', target: 0 },   // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<100'], // 95% of requests under 100ms
    http_req_failed: ['rate<0.1'],   // Error rate under 10%
  },
};

export default function() {
  let response = http.get('http://localhost:8080/health');
  check(response, {
    'status is 200': (r) => r.status === 200,
    'response time < 100ms': (r) => r.timings.duration < 100,
  });
  sleep(1);
}
```

### 3. Spike Test
```javascript
// k6-spike-test.js
import http from 'k6/http';
import { check } from 'k6';

export let options = {
  stages: [
    { duration: '1m', target: 10 },   // Normal load
    { duration: '30s', target: 1000 }, // Spike
    { duration: '1m', target: 10 },   // Normal load
  ],
};

export default function() {
  let response = http.get('http://localhost:8080/health');
  check(response, {
    'status is 200': (r) => r.status === 200,
  });
}
```

## 🔧 Performance Optimization

### 1. Database Optimization
```sql
-- Create indexes for better performance
CREATE INDEX CONCURRENTLY idx_transactions_client_id ON transactions(client_id);
CREATE INDEX CONCURRENTLY idx_transactions_checkout_id ON transactions(checkout_id);
CREATE INDEX CONCURRENTLY idx_transactions_status ON transactions(payment_status);
CREATE INDEX CONCURRENTLY idx_transactions_created_at ON transactions(created_at);

-- Partition large tables by date
CREATE TABLE transactions_2024 PARTITION OF transactions
FOR VALUES FROM ('2024-01-01') TO ('2025-01-01');
```

### 2. Redis Optimization
```bash
# Redis configuration for high performance
maxmemory 2gb
maxmemory-policy allkeys-lru
tcp-keepalive 60
timeout 300
```

### 3. Application Optimization
```go
// Connection pool settings
db.SetMaxOpenConns(100)
db.SetMaxIdleConns(10)
db.SetConnMaxLifetime(time.Hour)

// Redis connection pool
redis.NewClient(&redis.Options{
    PoolSize:     100,
    MinIdleConns: 10,
    MaxRetries:   3,
})
```

## 📈 Monitoring During Tests

### 1. Application Metrics
```bash
# Monitor application metrics
curl http://localhost:9090/metrics | grep http_requests_total
curl http://localhost:9090/metrics | grep http_request_duration_seconds
```

### 2. System Metrics
```bash
# Monitor system resources
htop
iostat -x 1
netstat -i
```

### 3. Database Metrics
```sql
-- Monitor database performance
SELECT * FROM pg_stat_activity;
SELECT * FROM pg_stat_database;
```

## 🎯 Load Testing Results Analysis

### Key Metrics to Monitor

1. **Response Time**
   - Average response time
   - 95th percentile response time
   - 99th percentile response time

2. **Throughput**
   - Requests per second
   - Transactions per minute

3. **Error Rate**
   - HTTP error rate
   - Application error rate

4. **Resource Utilization**
   - CPU usage
   - Memory usage
   - Database connections
   - Redis connections

### Performance Benchmarks

| Metric | Target | Acceptable | Poor |
|--------|--------|------------|------|
| Response Time (95th) | < 50ms | < 100ms | > 200ms |
| Throughput | > 10,000 RPS | > 5,000 RPS | < 1,000 RPS |
| Error Rate | < 0.1% | < 1% | > 5% |
| CPU Usage | < 70% | < 85% | > 95% |
| Memory Usage | < 80% | < 90% | > 95% |

## 🚀 Continuous Performance Testing

### 1. CI/CD Integration
```yaml
# .github/workflows/performance.yml
name: Performance Tests
on:
  schedule:
    - cron: '0 2 * * *'  # Daily at 2 AM
  push:
    branches: [main]

jobs:
  performance-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Setup Go
        uses: actions/setup-go@v2
        with:
          go-version: 1.21
      - name: Run performance tests
        run: |
          go test -bench=. -benchmem ./...
          artillery run artillery-config.yml
```

### 2. Automated Performance Regression Detection
```bash
#!/bin/bash
# performance-regression-check.sh

# Run baseline test
artillery run baseline-config.yml --output baseline.json

# Run current test
artillery run current-config.yml --output current.json

# Compare results
node compare-results.js baseline.json current.json
```

## 📋 Performance Testing Checklist

- [ ] Baseline performance established
- [ ] Load testing completed
- [ ] Stress testing completed
- [ ] Spike testing completed
- [ ] Endurance testing completed
- [ ] Database performance optimized
- [ ] Redis performance optimized
- [ ] Application performance optimized
- [ ] Monitoring and alerting configured
- [ ] Performance regression tests automated
- [ ] Documentation updated with results

## 🔍 Troubleshooting Performance Issues

### Common Issues and Solutions

1. **High Response Times**
   - Check database query performance
   - Verify Redis connection pool
   - Review application logs for bottlenecks

2. **High Error Rates**
   - Check external API availability
   - Verify rate limiting settings
   - Review circuit breaker configuration

3. **Memory Leaks**
   - Monitor memory usage over time
   - Check for goroutine leaks
   - Review garbage collection metrics

4. **Database Connection Issues**
   - Check connection pool settings
   - Monitor database performance
   - Verify connection timeout settings

## 📊 Sample Performance Test Results

```
Hey v0.14.0

Summary:
  Total:        10.0000 secs
  Slowest:      0.0456 secs
  Fastest:      0.0001 secs
  Average:      0.0023 secs
  Requests/sec: 10000.0000

Response time histogram:
  0.000 [1]     |
  0.005 [9999]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.010 [0]     |
  0.015 [0]     |
  0.020 [0]     |
  0.025 [0]     |
  0.030 [0]     |
  0.035 [0]     |
  0.040 [0]     |
  0.045 [0]     |

Latency distribution:
  10% in 0.0002 secs
  25% in 0.0003 secs
  50% in 0.0004 secs
  75% in 0.0005 secs
  90% in 0.0006 secs
  95% in 0.0007 secs
  99% in 0.0010 secs

Status code distribution:
  [200] 10000 responses
```