# FFmpeg Media

[![Drupal 11.3+ | 12](https://img.shields.io/badge/Drupal-11.3%2B%20%7C%2012-blue.svg)](https://www.drupal.org)
[![PHP 8.5+](https://img.shields.io/badge/PHP-8.5%2B-purple.svg)](https://www.php.net)
[![FFmpeg 8.x](https://img.shields.io/badge/FFmpeg-8.x-green.svg)](https://ffmpeg.org)
[![License: GPL-2.0-or-later](https://img.shields.io/badge/License-GPL--2.0--or--later-blue.svg)](https://www.gnu.org/licenses/gpl-2.0.html)

FFmpeg Media is a professional video processing suite for Drupal 11.3+/12 that provides queue-based transcoding, adaptive streaming, and a modern admin UI on top of FFmpeg 8.x.

## Highlights

- Queue-based conversion with dual queues (standard/large), retry/backoff, and stuck-process recovery
- Hardware acceleration with auto-detection and circuit breaker (NVENC, CUDA, VAAPI, QSV, VideoToolbox, AMF, Vulkan, OpenHarmony)
- Plugin architecture for codecs, outputs, and processors
- Adaptive HLS/DASH streaming with per-title bitrate ladders and GOP alignment
- Segment thumbnails/GIFs, metadata extraction, and optional manifest generation
- External subtitle tracks + multi-language audio tracks for HLS/DASH
- Synchronous conversion and path/URL API (no Media entity required)
- Media-first upload workflow (Drupal Media Library / media fields, no custom upload transport)

## Supported Codecs (Core)

| Codec | Notes | Hardware Accelerated | License |
|-------|------|----------------------|---------|
| AV1 | Next-gen codec (recommended) | NVENC, CUDA, VAAPI, QSV, AMF, Vulkan | Royalty-free (AOM) |
| VP9 | WebM standard | VAAPI, QSV | Royalty-free |
| FFV1 | Lossless archival | Software only | Open standard (IETF) |
| H.264/AVC | Broad compatibility | NVENC, CUDA, VAAPI, QSV, VideoToolbox, AMF, Vulkan, OpenHarmony | Patent-encumbered |
| HEVC/H.265 | High-efficiency | NVENC, CUDA, VAAPI, QSV, VideoToolbox, AMF, Vulkan, OpenHarmony | Patent-encumbered |
| VVC/H.266 | Experimental | Software only | Patent-encumbered |
| ProRes | Professional mezzanine | Software only | Proprietary (Apple) |

## Output Formats

**Core outputs:** MP4, WebM, MKV, FLV, JPEG XL (image output plugin)

**Streaming (submodule):** HLS (.m3u8), DASH (.mpd), WHIP, RTP

### Audio Codec Compatibility

| Container | Supported Audio Codecs | Notes |
|-----------|----------------------|-------|
| MP4 | AAC, Opus | AAC for broadest compatibility; Opus for royalty-free |
| WebM | Opus, Vorbis | Opus recommended for quality |
| MKV | All | Accepts any audio codec |
| FLV | AAC, MP3 | AAC recommended; MP3 for legacy |
| JPEG XL | None | Image-only format, no audio |
| HLS | AAC, Opus | AAC required for Apple devices; Opus for fMP4 HLS |
| DASH | AAC, Opus, Vorbis | AAC for broadest support |
| WHIP | Opus | WebRTC standard audio codec |
| RTP | AAC, Opus | Opus preferred for low latency |

`copy` (stream copy without re-encoding) is allowed for all containers that support audio. The ProfileForm validates audio codec compatibility at save time.

## Requirements

- Drupal 11.3+ or 12
- PHP 8.5+
- FFmpeg 8.x (ffmpeg + ffprobe)
- Core modules: Media, File

**FFmpeg and ffprobe must be installed on the host system; they are not bundled with this module.**

Optional FFmpeg components (only if you use these features):
- [whisper.cpp](https://github.com/ggml-org/whisper.cpp) CLI for AI transcription (`whisper-cli` binary + GGML model)
- libsvtav1 for best AV1 quality

## Whisper Setup (Optional, for `ffmpeg_media_whisper`)

The `ffmpeg_media_whisper` runtime in this module uses the `whisper-cli` binary
from `whisper.cpp` (audio is extracted via FFmpeg, then transcribed by
`whisper-cli`).

### 1) Install `whisper.cpp` + CLI

```bash
git clone https://github.com/ggml-org/whisper.cpp.git
cd whisper.cpp
cmake -B build
cmake --build build --config Release
sudo cmake --install build --config Release
```

### 2) Download a GGML model

```bash
./models/download-ggml-model.sh base.en
```

### 3) Verify installation

```bash
whisper-cli --help
```

Then configure CLI path + model in Drupal at:
`/admin/config/media/ffmpeg-media/whisper`

### Optional: FFmpeg 8 `whisper` audio filter (direct FFmpeg usage)

If you want to run FFmpeg's own `whisper` filter directly, your FFmpeg build
must include whisper support (`--enable-whisper`).

```bash
# Example (when building FFmpeg from source):
./configure --enable-whisper [other options...]
make -j"$(nproc)"
sudo make install

# Verify the filter is present:
ffmpeg -filters | grep whisper
```

Example command (from FFmpeg filter documentation):

```bash
ffmpeg -i input.wav -vn -af "whisper=model=../whisper.cpp/models/ggml-base.en.bin:language=en" -f null -
```

## Installation

```bash
composer require drupal/ffmpeg_media
# or enable an existing local module

drush en ffmpeg_media
```

## Configuration

### Getting Started

After installation, configure conversion by adding the `ffmpeg_media_converted_video` or `ffmpeg_media_streaming_video` formatter to your video field under Manage Display. FFmpeg paths are auto-detected on install. Check the status report at `/admin/reports/status` for FFmpeg availability.

### Core Settings

- **Settings form:** `/admin/config/media/ffmpeg-media`
- **Profiles:** `/admin/config/media/ffmpeg-media/profiles`
- **Conversion setup:** Manage Display on your media type (`/admin/structure/media/manage/{bundle}/display`)

Key settings include:
- `ffmpeg_path`, `ffprobe_path`, `ffmpeg_timeout`
- `output_directory`, `max_file_size`, `preserve_original`, `auto_cleanup_failed`
- `queue_claim_timeout`, `large_file_threshold`, `max_concurrent`, `lock_timeout`
- `retry.*` (enabled, max_retries, base_delay, max_delay)
- `hardware_acceleration` (auto/disabled/nvenc/vaapi/qsv/...)

### Conversion Profiles

Profiles define output formats, codecs, and quality settings for video conversion. Manage profiles at `/admin/config/media/ffmpeg-media/profiles`.

- **List view** with explanatory guide, settings reference, and profile cards
- **Full CRUD:** add, edit, and delete profiles (default profile is protected)
- **AJAX output management:** dynamically add/remove outputs per profile
- Profiles are stored as config objects (`ffmpeg_media.profile.{id}`)
- Each profile defines one or more output configurations (container + codec + quality)
- Formatter settings reference profiles via a select dropdown

### Formatter-Based Configuration

Conversion configuration is managed through Drupal's native field formatter system. Instead of a separate field mapping admin form, you configure conversion by adding the `ffmpeg_media_converted_video` or `ffmpeg_media_streaming_video` formatter to a field under Manage Display (`/admin/structure/media/manage/{bundle}/display`).

- **Native Drupal workflow:** configuration lives in EntityViewDisplay formatter settings (like Image Styles)
- **No separate admin form:** add formatter, set `auto_convert` and `conversion_profile` options
- **Profile integration:** select from available conversion profiles in formatter settings
- **Per-field configuration:** each field can have its own conversion profile and auto-convert setting

### Streaming Settings (Submodule)

- **Streaming settings:** `/admin/config/media/ffmpeg-media/streaming`
- Configure HLS/DASH segment duration, playlist type, representation ladder, and naming patterns.

### UI Settings (Submodule)

- **UI settings:** `/admin/config/media/ffmpeg-media/ui`
- **Dashboard:** `/admin/content/ffmpeg-media`
- Dashboard/actions only: uploads remain in Drupal core media workflows

### Player Settings (Submodule)

- **Player settings:** `/admin/config/media/ffmpeg-media/player`
- Configure Vidstack player library source (local or CDN), default theme, aspect ratio, lazy loading, preload strategy, and keyboard navigation.

### CDN Configuration

- **CDN settings:** `/admin/config/media/ffmpeg-media/cdn`
- S3-compatible storage integration (AWS S3, CloudFront, MinIO, DigitalOcean Spaces, Backblaze B2, Wasabi)
- Automatic upload of converted files to CDN after conversion
- CloudFront cache invalidation, presigned URLs for private files, URL rewriting
- Credentials can be set via environment variables (`FFMPEG_MEDIA_S3_ACCESS_KEY`, `FFMPEG_MEDIA_S3_SECRET_KEY`)
- Optional: store credential references via Key module (`s3.access_key_key`, `s3.secret_key_key`)

## Output Storage

Converted outputs are stored in the `ffmpeg_media_outputs` database table, managed by `OutputStorageService`. Common output types include:

- `mp4`, `webm`, `mkv`, `flv`, `thumbnail`
- `gif`, `segment_thumbnails`, `segment_gifs`
- `metadata` (JSON), `transcript` (VTT)
- `hdr_output`, `sdr_output`
- `hls`, `dash`
- `subtitle_tracks`
- `manifest` (JSON/YAML)

## Upload Workflow (Drupal Best Practice)

FFmpeg Media does not replace Drupal file upload handling. Source videos are
uploaded through standard Drupal workflows:

- Media Library
- Media add/edit forms with file fields
- Entity forms that embed media/file fields

After the media entity is saved, FFmpeg Media handles conversion, processing,
queueing, and output storage.

## Submodules

### ffmpeg_media_streaming
Adds adaptive streaming (HLS/DASH) with representation ladders and streaming settings UI.

```bash
drush en ffmpeg_media_streaming
```

### ffmpeg_media_whisper
AI-powered subtitle generation (VTT/SRT/ASS) using `whisper.cpp` CLI
(`whisper-cli`) in a two-step pipeline (FFmpeg audio extract + transcription).

**Requires:** [whisper.cpp](https://github.com/ggml-org/whisper.cpp) CLI binary (`whisper-cli`) and a GGML model file. See [FFMPEG8_FEATURES.md](docs/FFMPEG8_FEATURES.md#4-whisper-transcription).

```bash
drush en ffmpeg_media_whisper
```

Activation steps after enabling:
1. Verify whisper-cli is installed: `whisper-cli --help`
2. Configure CLI path and model at `/admin/config/media/ffmpeg-media/whisper`
3. Enable the Whisper processor in your conversion profile at `/admin/config/media/ffmpeg-media/profiles`
4. Transcripts are stored automatically via `OutputStorageService` when the Whisper processor runs
5. Use the "Generate Transcript" button on any media status page, or run `drush ffmpeg:transcribe <id>`

### ffmpeg_media_advanced
HDR processing, two-pass encoding, watermark/subtitle/normalization helpers.

```bash
drush en ffmpeg_media_advanced
```

### ffmpeg_media_image
Thumbnail generation, GIF creation, segment thumbnails, and segment GIF processors.
Provides ThumbnailProcessor, SegmentThumbnailProcessor, GifProcessor, SegmentGifProcessor, and ThumbnailGeneratorService.

```bash
drush en ffmpeg_media_image
```

### ffmpeg_media_ui
Admin dashboard, conversion actions, poster selector, progress UI.

```bash
drush en ffmpeg_media_ui
```

### ffmpeg_media_player
Vidstack-based video player for both progressive (MP4/WebM/MKV/FLV) and adaptive
streaming (HLS/DASH) playback. When enabled, `ConvertedVideoFormatter` automatically
delegates to Vidstack for progressive videos; `VideoPlayerFormatter` uses it for
HLS/DASH. Works standalone — `ffmpeg_media_streaming` is only needed for HLS/DASH.

The player supports a CDN mode and a fully vendored local mode. The local bundle
ships Vidstack core, HLS.js, and DASH.js from the module — no external runtime
requests are made (GDPR-compliant). Google Cast is intentionally disabled in
local mode.

```bash
drush en ffmpeg_media_player
```

## Third-Party Formatter Support

FFmpeg Media can inject converted video sources into third-party video
formatters via `hook_media_view_alter()`. The supported integration contract is
`VideoSourceResolverInterface`, which resolves converted outputs, poster
thumbnails, and subtitle tracks for formatter-specific rendering.

Verified out of the box:
- **Video.js** (`videojs` module) — sources and poster injected automatically
- **Plyr** (`plyr` module) — sources and poster injected automatically

Other formatters that emit native `<video>` / `<source>` markup may also work,
but they should be treated as compatibility candidates rather than universally
guaranteed integrations.

Custom formatters can inject `VideoSourceResolverInterface` to access converted
outputs directly:

```php
use Drupal\ffmpeg_media\Resolver\VideoSourceResolverInterface;

$resolver = \Drupal::service(VideoSourceResolverInterface::class);
$resolved = $resolver->resolveSources(
  mediaId: (int) $media->id(),
  fieldName: $fieldName,
  profileId: $formatterSettings['conversion_profile'] ?? NULL,
  formatPriority: array_filter($formatterSettings['format_priority'] ?? []),
);

// $resolved->sources — array of source definitions (src, type, label)
// $resolved->poster — poster URL or null
// $resolved->subtitles — array of subtitle track definitions
// $resolved->isConverted — whether converted outputs exist
```

Supported extension points and BC expectations are documented in
`docs/ARCHITECTURE.md` and `ffmpeg_media.api.php`. Most other public
services are runtime-public internals required by Drupal construction paths
(forms, plugins, route access, subscribers) and are not part of the supported
contrib extension API.

## Drush Commands

| Command | Alias | Description |
|---------|-------|-------------|
| `ffmpeg:convert <id>` | `ffc` | Convert a single video by media ID |
| `ffmpeg:info <id>` | `ffi` | Show video metadata using ffprobe |
| `ffmpeg:convert-all` | `ffca` | Queue all unconverted videos |
| `ffmpeg:reconvert` | `ffrc` | Reconvert already converted videos |
| `ffmpeg:requeue-missing` | `ffrm` | Requeue unconverted videos not in queue |
| `ffmpeg:process` | `ffp` | Process videos from conversion queue |
| `ffmpeg:queue-clear` | `ffqc` | Clear all conversion queues |
| `ffmpeg:status` | `ffs` | Show system status and queue counts |
| `ffmpeg:progress [id]` | `ffpg` | Show active conversion progress |
| `ffmpeg:processes` | `ffpr` | Show running FFmpeg processes |
| `ffmpeg:kill-stuck` | `ffks` | Kill stuck processes exceeding limits |
| `ffmpeg:cleanup` | `ffcl` | Clean up old progress records |
| `ffmpeg:generate-thumbnails` | `ffgt` | Generate thumbnails for videos |
| `ffmpeg:integrity-check` | `ffic` | Check/repair field definition integrity |
| `ffmpeg:transcribe <id>` | `fft` | Generate transcript for a media item (Whisper) |

## API Usage

### Queue Conversion

```php
use Drupal\ffmpeg_media\Service\QueueManagerInterface;

$queueManager = \Drupal::service(QueueManagerInterface::class);
$media = \Drupal::entityTypeManager()->getStorage('media')->load($mediaId);

$queueManager->queueForConversion($media);
```

### Synchronous Conversion

```php
use Drupal\ffmpeg_media\Service\VideoConverterInterface;

$converter = \Drupal::service(VideoConverterInterface::class);
$media = \Drupal::entityTypeManager()->getStorage('media')->load($mediaId);

$result = $converter->convert($media, profileId: 'web_optimized', synchronous: TRUE);
if ($result->isSuccess()) {
  // $result->outputs contains output paths
}
```

## Troubleshooting

**FFmpeg not found**

```bash
which ffmpeg
```

**Hardware acceleration issues**

```bash
ffmpeg -encoders | grep -E "nvenc|vaapi|qsv|videotoolbox|amf|vulkan"
```

**Queue not processing**

```bash
drush ffmpeg:status
drush ffmpeg:process --limit=5
```

## Development & Testing

```bash
# Embedded checkout inside a Drupal project
cd /var/www/html/web/modules/contrib/ffmpeg_media
composer run qa:ci

# Standalone checkout on the host
cd /path/to/ffmpeg_media
export DRUPAL_PROJECT_ROOT=/path/to/drupal

# Canonical QA commands
composer run qa:ci
composer run qa:phpcs
composer run qa:phpstan
composer run qa:phpunit
composer run qa:phpunit:unit
composer run qa:phpunit:integration
composer run qa:phpunit:kernel
composer run qa:phpunit:functional

# Equivalent direct runner commands
php scripts/qa-runner.php phpcs --standard=Drupal,DrupalPractice --extensions=php,module,install,inc,theme src/ modules/ tests/
php scripts/qa-runner.php phpstan analyse -c phpstan.neon --memory-limit=512M
php -d opcache.enable_cli=0 scripts/qa-runner.php phpunit -c phpunit.xml.dist

# E2E tests (Playwright)
npm ci
DRUPAL_PROJECT_ROOT=/path/to/drupal PROJECT_DOMAIN=example.local npm run test:e2e:prepare
DRUPAL_PROJECT_ROOT=/path/to/drupal PROJECT_DOMAIN=example.local npx playwright test
```

Notes:
- The QA commands resolve the Drupal root automatically when the module lives
  inside a Drupal checkout. For standalone clones, set `DRUPAL_PROJECT_ROOT`.
- Kernel and functional tests require `SIMPLETEST_BASE_URL` and `SIMPLETEST_DB` environment variables.
- `qa:phpunit` disables CLI OPcache (`opcache.enable_cli=0`) to avoid stale bytecode in long-running CI containers.
- Ensure `/tmp/browser_output` is writable for BrowserTest output.
- Playwright provisions a dedicated local E2E admin account before authentication via `scripts/prepare-e2e-user.sh`.
- The E2E prepare step rebuilds Drupal caches by default and writes runtime metadata to `tests/e2e/.auth/runtime.json` so browser tests can respect the current probe-token configuration.
- Default E2E credentials are `ffmpeg_media_e2e` / `ffmpeg_media_e2e`; override with `DRUPAL_ADMIN_USER`, `DRUPAL_ADMIN_PASS`, and `DRUPAL_ADMIN_MAIL` if needed.
- For standalone module checkouts, set `DRUPAL_PROJECT_ROOT=/path/to/drupal`; `PROJECT_DOMAIN` or `DRUPAL_BASE_URL` selects the browser target.
- `npm run test:e2e:prepare` can be run explicitly to verify the Drupal-side E2E account setup before browser tests.
- For player builds: run `npm ci && npm run build` in `modules/ffmpeg_media_player`.

## Codec Licensing Notice

This module invokes FFmpeg as an external binary and **does not bundle or distribute any codec implementations**. The server operator is responsible for ensuring their FFmpeg installation complies with applicable patent and licensing requirements in their jurisdiction.

### Codec Patent Status

| Category | Codecs | Notes |
|----------|--------|-------|
| **Royalty-Free** | AV1, VP9, Opus, Vorbis, FLAC, FFV1 | No licensing required. |
| **Patents Expired** | MP3 | All patents expired worldwide (2017). |
| **Patent-Encumbered** | H.264, HEVC/H.265, VVC/H.266, AAC, ProRes | May require licensing depending on jurisdiction and use case. |

**Key considerations:**

- **H.264/AVC:** Core patents have expired in the EU (Jan 2025) and are expiring in the US (~2027). Risk is low and decreasing.
- **AAC-LC:** Core US patents have expired. Higher-efficiency profiles (HE-AAC) may still be encumbered.
- **HEVC/H.265:** Complex, fragmented patent ownership across multiple pools with active litigation. Highest risk among supported codecs.
- **VVC/H.266:** Licensing pools are still forming. Similar concerns as HEVC.
- **ProRes:** Proprietary to Apple. FFmpeg's encoder is a compatible implementation without an official Apple license.

Default installation profiles use royalty-free codecs (AV1, VP9, Opus) wherever possible. Patent-encumbered codecs remain available for users who require them and have appropriate licensing.

For more information, see the [FFmpeg Legal page](https://ffmpeg.org/legal.html).

## License

GPL-2.0-or-later. See `LICENSE`.
