# API Normalization - Failover Module

Provides automatic provider failover and circuit breaker protection for high availability API operations.

## Features

### Automatic Failover
- Automatically switches to backup providers when the primary provider fails
- Configurable failover chains per provider
- Smart provider selection based on availability and circuit breaker state

### Circuit Breaker Pattern
- Implements Martin Fowler's Circuit Breaker pattern
- Three states: CLOSED (normal), OPEN (failing fast), HALF_OPEN (testing recovery)
- Prevents cascading failures across distributed systems
- Automatic recovery detection and state transitions

### Retry Logic
- Exponential backoff with configurable parameters
- Per-provider retry configuration
- Maximum retry delay limits
- Prevents overwhelming failed services

### Event System
- Provider failover events for monitoring and alerting
- Circuit breaker state change events
- Comprehensive context information in events

## Why Use This Module?

Production systems need resilience. When an AI provider has an outage or rate limits your requests, your application shouldn't fail. This module provides:

1. **High Availability**: Automatic failover to backup providers
2. **Fault Tolerance**: Circuit breaker prevents cascading failures
3. **Cost Optimization**: Fallback to cheaper providers (e.g., Ollama) when cloud providers fail
4. **Operational Insight**: Events for monitoring and alerting

## Installation

```bash
composer require drupal/api_normalization_failover
drush en api_normalization_failover
```

## Configuration

### Via UI

Navigate to: Configuration > API Normalization > Failover Settings

Configure:
- Failover chains per provider
- Retry attempts and delays
- Circuit breaker thresholds
- Provider priorities

### Via Config File

Edit `config/install/api_normalization_failover.settings.yml`:

```yaml
providers:
  anthropic:
    enabled: true
    priority: 10
    fallback_to:
      - openai
      - ollama
    retry_attempts: 3
    retry_delay: 1000
    retry_multiplier: 2
    max_retry_delay: 30000

circuit_breaker:
  failure_threshold: 5
  recovery_timeout: 60
  success_threshold: 2
```

## Usage

### Automatic Failover

```php
use Drupal\api_normalization_failover\Service\ProviderFailoverService;

// Inject the service
$failover = \Drupal::service('api_normalization_failover.provider_failover');

// Execute with automatic failover
$result = $failover->executeWithFailover('anthropic', function($provider_id) {
  // Your API call here
  return \Drupal::service('ai.provider')->chat($messages, $provider_id);
});
```

**What happens:**
1. Tries Anthropic (primary)
2. If Anthropic fails, checks circuit breaker
3. If circuit breaker is closed, tries OpenAI (fallback)
4. If OpenAI fails, tries Ollama (last resort)
5. Throws `AllProvidersFailedException` if all fail

### Circuit Breaker

```php
use Drupal\api_normalization_failover\Service\CircuitBreakerService;

$circuitBreaker = \Drupal::service('api_normalization_failover.circuit_breaker');

// Check if provider is available
if ($circuitBreaker->isAvailable('anthropic')) {
  // Make request
  $circuitBreaker->recordSuccess('anthropic');
} else {
  // Circuit is open, skip this provider
  $logger->warning('Anthropic circuit breaker is open');
}

// Get circuit breaker statistics
$stats = $circuitBreaker->getStatistics('anthropic');
// Returns: state, failure_count, last_failure_time, etc.
```

### Event Subscribers

Subscribe to failover events for monitoring:

```php
namespace Drupal\mymodule\EventSubscriber;

use Drupal\api_normalization_failover\Event\ProviderFailoverEvent;
use Drupal\api_normalization_failover\Event\CircuitBreakerStateChangeEvent;
use Symfony\Component\EventDispatcher\EventSubscriberInterface;

class FailoverMonitoringSubscriber implements EventSubscriberInterface {

  public static function getSubscribedEvents() {
    return [
      ProviderFailoverEvent::EVENT_NAME => 'onProviderFailover',
      CircuitBreakerStateChangeEvent::EVENT_NAME => 'onCircuitBreakerStateChange',
    ];
  }

  public function onProviderFailover(ProviderFailoverEvent $event) {
    // Alert ops team
    $this->logger->critical('Provider failover: {from} -> {to}', [
      'from' => $event->getFromProvider(),
      'to' => $event->getToProvider(),
    ]);
  }

  public function onCircuitBreakerStateChange(CircuitBreakerStateChangeEvent $event) {
    if ($event->getNewState() === CircuitBreakerService::STATE_OPEN) {
      // Circuit opened - provider is down
      $this->alerting->sendAlert('Circuit breaker opened for ' . $event->getEndpoint());
    }
  }
}
```

## Configuration Examples

### Production Setup (Cloud Providers)

```yaml
providers:
  anthropic:
    enabled: true
    priority: 10
    fallback_to: [openai]
    retry_attempts: 3

  openai:
    enabled: true
    priority: 20
    fallback_to: [anthropic]
    retry_attempts: 3
```

**Use case**: Primary cloud provider with peer failover.

### Cost-Optimized Setup

```yaml
providers:
  anthropic:
    enabled: true
    priority: 10
    fallback_to: [openai, ollama]
    retry_attempts: 2

  openai:
    enabled: true
    priority: 20
    fallback_to: [ollama]
    retry_attempts: 2

  ollama:
    enabled: true
    priority: 30
    fallback_to: []
    retry_attempts: 5
```

**Use case**: Fallback to local Ollama when cloud providers fail.

### High Availability Setup

```yaml
providers:
  anthropic:
    enabled: true
    priority: 10
    fallback_to: [openai, cohere, ollama]
    retry_attempts: 2

circuit_breaker:
  failure_threshold: 3  # Fail fast
  recovery_timeout: 30  # Try recovery sooner
  success_threshold: 1  # Single success to close
```

**Use case**: Maximum uptime with multiple fallback options.

## Monitoring and Metrics

### Get Failover Statistics

```php
$failover = \Drupal::service('api_normalization_failover.provider_failover');
$stats = $failover->getFailoverStatistics();

// Returns:
// [
//   'circuit_breaker_states' => [
//     'anthropic' => [...],
//     'openai' => [...],
//   ]
// ]
```

### Get Circuit Breaker Health

```php
$circuitBreaker = \Drupal::service('api_normalization_failover.circuit_breaker');
$health = $circuitBreaker->getHealthMetrics();

foreach ($health as $provider_id => $metrics) {
  echo "{$provider_id}: {$metrics['state']} (failures: {$metrics['failure_count']})\n";
}
```

### Manual Circuit Reset

```php
// Reset a specific provider's circuit breaker
$circuitBreaker = \Drupal::service('api_normalization_failover.circuit_breaker');
$circuitBreaker->reset('anthropic');
```

## Architecture

### Circuit Breaker States

1. **CLOSED** (Normal Operation)
   - All requests pass through
   - Failures increment counter
   - Transitions to OPEN after threshold exceeded

2. **OPEN** (Failing Fast)
   - Requests fail immediately without trying
   - Prevents overwhelming failing service
   - Transitions to HALF_OPEN after timeout

3. **HALF_OPEN** (Testing Recovery)
   - Limited test requests allowed
   - Success transitions back to CLOSED
   - Failure transitions back to OPEN

### Retry Strategy

Exponential backoff with configurable parameters:

```
Attempt 1: delay = 1000ms
Attempt 2: delay = 2000ms (1000 * 2)
Attempt 3: delay = 4000ms (2000 * 2)
Max delay: 30000ms (configurable cap)
```

## Troubleshooting

### All Providers Failing

```php
try {
  $result = $failover->executeWithFailover('anthropic', $callback);
} catch (\Drupal\api_normalization_failover\Exception\AllProvidersFailedException $e) {
  // Get detailed error information
  $errors = $e->getProviderErrors();
  foreach ($errors as $provider_id => $error) {
    $logger->error("Provider {$provider_id} failed: {$error}");
  }

  // Get error summary
  echo $e->getErrorSummary();
}
```

### Circuit Breaker Stuck Open

```bash
# Manually reset via Drush
drush eval "\Drupal::service('api_normalization_failover.circuit_breaker')->reset('anthropic');"

# Or via code
$circuitBreaker->reset('anthropic');
```

### Debugging Failover

```php
// Subscribe to events for debugging
use Drupal\api_normalization_failover\Event\ProviderFailoverEvent;

\Drupal::service('event_dispatcher')->addListener(
  ProviderFailoverEvent::EVENT_NAME,
  function(ProviderFailoverEvent $event) {
    \Drupal::logger('mymodule')->debug('Failover: {from} -> {to}', [
      'from' => $event->getFromProvider(),
      'to' => $event->getToProvider(),
      'context' => $event->getContext(),
    ]);
  }
);
```

## Dependencies

- `api_normalization:api_normalization` (core module)
- `drupal:ai` (AI provider integration)

## Related Modules

- **api_normalization_tamper**: Format transformation
- **api_normalization_audit**: Metrics and logging
- **api_normalization_gateway**: API routing and transformation

## Performance Considerations

- Circuit breaker state is cached for fast lookups
- Metadata stored in State API (persistent across clears)
- Event dispatching adds minimal overhead
- Retry logic respects max delay caps to prevent long waits

## Security Notes

- Circuit breaker prevents denial-of-service via failing providers
- Exponential backoff prevents overwhelming services
- Events may contain sensitive context - sanitize before logging
- Provider credentials managed by parent module (api_normalization)

## Contributing

See main module README for contribution guidelines.

## License

GPL-2.0-or-later

## Credits

Implements Martin Fowler's Circuit Breaker pattern:
https://martinfowler.com/bliki/CircuitBreaker.html
