# Native Observability

**Native Observability brings structured, request-level observability directly inside Drupal.**

No external agents.
No heavy APM stack.
No vendor lock-in.

This project introduces a native observability layer built specifically for **Drupal 11**, enabling request tracing,
execution monitoring, metrics aggregation, reporting, telemetry export, and event-driven integrations directly from
the application runtime.

Native Observability 1.0.7 extends that base with stronger Grafana/Prometheus support, the first ECA bridge, default
exclusion of self-observability routes, and a long-form report that helps less experienced users understand what is
happening.

Instead of relying on external APM systems, **Native Observability integrates observability capabilities directly into
Drupal's architecture**.

---

# 🎯 Vision

Modern Drupal applications require deep runtime introspection in order to:

- understand request lifecycle behaviour
- measure performance bottlenecks
- inspect internal service execution
- trace external API calls
- correlate AJAX and sub-requests
- analyze performance trends
- generate structured technical reports
- export telemetry to external monitoring platforms

Native Observability provides all of this **natively inside Drupal**, without requiring external agents or invasive
instrumentation.

---

# 🚀 Global Features

Native Observability provides a complete observability stack composed of:

## Request Tracing
- ULID based Request IDs
- Kernel request lifecycle tracing
- Parent / child request correlation
- AJAX request correlation
- Correlation ID injected in response headers

## Execution Instrumentation
- Execution Span Recorder API
- Service execution tracking
- HTTP client tracing
- External API instrumentation
- Span categorization

## Trace Visualization
- Trace list UI
- Trace detail inspection
- Execution timeline visualization

## Performance Metrics
- Route aggregation
- Throughput analysis
- Percentile calculation (P95 / P99)

## Observability Reporting
- Report builder UI
- Printable HTML reports
- Long-form report output for a broader technical snapshot
- KPI summaries
- Technical appendices

## Telemetry Export
- OpenTelemetry exporter
- Prometheus endpoint

## Event-Driven Automation
- ECA bridge for runtime observability events
- Optional demo rules for alerts and log-based workflows

## Request Log Exclusion
- Core request blacklist with pattern-based matching
- Dedicated settings UI for exclusion-rule management
- Query-condition operators for noisy or synthetic traffic
- Dashboard shortcuts to exclude or reactivate route/path logging
- Full-pipeline suppression across traces, spans, metrics, and exports

---

# 🧱 Architecture Overview

Core contracts:

- TraceRecorderInterface
- TraceStorageInterface
- RequestIdGeneratorInterface
- ExecutionSpanRecorder

All services follow Drupal 11 best practices.

---

# 🧩 Module Architecture

## Core Module

### native_observability
Core tracing infrastructure.

---

## Extension Modules

### native_observability_execution
Execution tracing.

### native_observability_spans
Span management.

### native_observability_metrics
Performance metrics.

### native_observability_otel
Telemetry export.

### native_observability_eca_bridge
Exposes observability events to ECA.

### native_observability_eca_bridge_demo
Ready-made ECA demo models.

### native_observability_status_block
Admin diagnostics.

### native_observability_dashboard
Provides the main observability dashboard UI.

Features:
- Aggregated metrics visualization
- Route performance tables
- AJAX and request analysis
- Cache and database insights
- Technical insight panels

This module acts as the **central visualization layer** of the system.

### native_observability_report
Structured reporting and printable HTML export.

---

# 📦 Installation

## Requirements

- Drupal 11
- PHP 8.2+
- Database (MySQL / MariaDB / PostgreSQL)

## Install via Composer

composer require drupal/native_observability

## Enable modules

drush en native_observability

Recommended:

drush en native_observability_execution
drush en native_observability_spans
drush en native_observability_metrics
drush en native_observability_dashboard
drush en native_observability_report

Optional:

drush en native_observability_otel
drush en native_observability_status_block
drush en native_observability_eca_bridge
drush en native_observability_eca_bridge_demo

## Cache rebuild

drush cr

---

# 🚫 Request Log Exclusion

Request Log Exclusion is a **core operational feature** of Native Observability.

Its purpose is simple:

- keep dashboards readable
- keep latency and error metrics honest
- remove noisy traffic that adds no analytical value
- suppress known synthetic or external probe traffic before it pollutes traces, spans, metrics, or exports

This is **not** a cosmetic UI filter.
When a request matches an exclusion rule, Native Observability treats that request as if it does not exist for observability purposes.

That means:

- no trace row is stored
- no request-linked spans are persisted or linked
- no aggregated metrics include that request
- no live metrics expose that request
- no downstream exporter forwards that request flow

In practice, exclusion is a **hard suppression rule**, not a search filter.

## Why this feature exists

Real projects accumulate noisy traffic:

- legacy endpoints still called by external systems
- uptime checks and synthetic tools
- old monitoring scrapers hitting paths that no longer exist
- repeated 404 probes
- endpoints whose relevance depends on GET parameters
- report/export routes called with rotating date ranges

Without exclusion rules, that traffic can:

- inflate request counters
- skew percentile calculations
- make slow-endpoint tables misleading
- create false noise in Technical Insights
- confuse less experienced operators who cannot distinguish real user traffic from external probes

## Where to manage exclusions

The authoritative settings page is:

`/admin/config/development/native-observability/exclusions`

This page lives under the Native Observability settings area and is the canonical place to manage request-log exclusion rules.

From there you can:

- list existing rules
- add a new rule
- edit an existing rule
- delete a rule
- disable or re-enable a rule through its form state

The dashboard action links are only shortcuts into this same authority. They do **not** create a second rule system.

## How rule matching works

Each exclusion rule can define:

- HTTP method
- path matching mode
- path pattern
- optional query conditions

Multiple query conditions inside the same rule are evaluated with **AND** semantics.

That means a request is excluded only if:

- method matches
- path matches
- every configured query condition also matches

## Path matching modes

### Exact

Matches only one exact path.

Example:

- `/native-observability/prometheus`
- `/foo/bar`

Use exact match when you want to suppress one known route or dead endpoint precisely.

### Prefix

Matches everything beginning with a path prefix.

Example:

- `/reports/`
- `/internal/monitoring/`

Use prefix match when a whole path family is noisy.

### Regex

Matches paths using a regular expression.

Example:

- `@^/reports/[0-9]{4}/.*$@`
- `@^/health(check)?$@`

Use regex only when exact or prefix matching is not expressive enough.

## Query conditions and operators

Rules can optionally include query-condition lines.

The form accepts one condition per line in this format:

`key|operator|value`

For `between`, the format is:

`key|between|min|max`

### Supported operators

#### `exists`

Matches when the query key is present.

Example:

`preview|exists`

#### `missing`

Matches when the query key is absent.

Example:

`token|missing`

#### `equals`

Matches an exact query-string value.

Example:

`preview|equals|1`

#### `regex`

Matches the query value using a regular expression.

Example:

`source|regex|@^(cron|monitoring|synthetic)$@`

#### `gte`

Matches when the value is greater than or equal to a comparable threshold.

Comparable values may be:

- integers
- floats
- parseable dates
- parseable datetimes

Example:

`page|gte|100`

#### `lte`

Matches when the value is lower than or equal to a comparable threshold.

Example:

`limit|lte|5`

#### `between`

Matches when the value falls between two comparable values.

Example:

`from|between|2026-01-01|2026-01-31`

This is especially useful for report-like traffic that changes by date range but should still be suppressed from observability.

## Practical examples

### Example 1: suppress a dead endpoint hit by an external tool

- Method: `GET`
- Path type: `Exact`
- Path pattern: `/native-observability/prometheus`
- Query conditions: none

Use this when an old external tool still calls a path that no longer exists and keeps generating 404 noise.

### Example 2: suppress synthetic report preview traffic

- Method: `GET`
- Path type: `Exact`
- Path pattern: `/reports/export`
- Query conditions:
  - `preview|equals|1`

Use this when preview traffic is operationally different from real user exports and should not distort metrics.

### Example 3: suppress a date-qualified report window

- Method: `GET`
- Path type: `Exact`
- Path pattern: `/reports/custom`
- Query conditions:
  - `from|between|2026-01-01|2026-01-31`
  - `to|between|2026-01-01|2026-01-31`

Use this when a scheduled or synthetic reporting tool always hits a known date range.

### Example 4: suppress a family of monitoring paths

- Method: `ANY`
- Path type: `Prefix`
- Path pattern: `/internal/monitoring/`
- Query conditions: none

Use this when a whole group of endpoints is guaranteed observability noise.

## Dashboard workflow

The fastest workflow usually starts from the dashboard, not from settings.

On the main dashboard page:

`/admin/reports/native-observability/dashboard`

the Technical Insights explorer now exposes an `Actions` column in the relevant route/path tables.

Per row, you can see:

- `Exclude from logs`
- `Re-enable logs`

### What happens when you click `Exclude from logs`

Native Observability opens the exclusion-rule form with prefilled values based on the selected row, typically:

- method
- path match type
- path pattern
- query condition payload when available
- destination/backlink to the dashboard

After saving, the form returns to the dashboard if the destination/backlink is present.

### What happens when you click `Re-enable logs`

If the current row already corresponds to an existing exclusion rule, the action deep-links back to that rule so logging can be reactivated through the same workflow.

## External requests in the dashboard

When a request has no resolved Drupal route, the dashboard no longer treats it as a generic anonymous route only.

Instead, it can present it as an external request, for example:

- `External Request`
- `External Request From IP "172.23.0.11"`
- masked form when privacy mode requires masking

This matters because unresolved traffic is often:

- external probing
- legacy clients
- synthetic tools
- dead-path traffic

and should be visually distinguishable from normal Drupal route traffic.

## IP display and privacy behavior

Route-area labeling for external requests follows the configured IP capture mode.

Expected behavior:

- full IP when full display is allowed
- masked IP when anonymized display is required
- neutral external-request label when IP should not be exposed

The goal is to make the traffic understandable without violating the privacy intent of the capture settings.

## Runtime performance and caching

The exclusion system is designed to be cheap at runtime.

Rules are:

- stored in the database as the authoritative source
- normalized into a runtime-friendly structure
- cached in Drupal persistent cache
- reused in memory during the current request lifecycle

Normal request processing must **not** repeatedly query the database to decide whether a request should be excluded.

When a rule is:

- created
- updated
- deleted
- enabled
- disabled

the dedicated exclusion cache is invalidated so the next request rebuilds the runtime rule set.

## Safe usage guidance

Use exclusion rules when traffic is clearly operational noise.

Good candidates:

- dead endpoints
- repeated monitoring probes
- synthetic preview traffic
- known machine-to-machine calls that distort real-user analysis

Avoid excluding traffic when:

- it represents real user behavior
- it is still useful for latency/error investigation
- you are uncertain whether it should appear in business-facing dashboards

If in doubt, start with the narrowest possible rule:

- exact path before prefix
- prefix before regex
- one query condition before broad pattern sets

## Summary

Request Log Exclusion exists to keep Native Observability honest.

It gives operators a precise way to say:

“This request flow is not analytically useful. Do not let it pollute the observability dataset.”

Used carefully, it keeps traces cleaner, metrics more representative, dashboards easier to read, and external noise from being mistaken for application behavior.

## Performance cost

From a runtime perspective, the feature is designed to be low-cost.

The database is intentionally used as the **master source of truth** for exclusion rules because operators need a durable and authoritative place to create, edit, disable, and delete rules from the settings UI.

However, Native Observability does **not** query the database continuously on every request just to decide whether a request must be excluded.

Instead, the system works like this:

- the database remains the authoritative source
- rules are normalized and saved into Drupal cache
- during normal traffic, exclusion decisions are read from cache
- inside the current request lifecycle, the normalized rule set is reused in memory

This means the feature avoids repeated DB lookups during normal request processing and does not keep hitting the database for every incoming request.

The database is consulted again only when needed, mainly after rule changes invalidate the dedicated cache and the runtime rule set must be rebuilt.

So the chosen model is:

- **DB as master**
- **cache as runtime source**
- **in-memory reuse per request**

This was selected specifically to keep the exclusion system operationally safe without introducing continuous database overhead.

---

# 🖨️ Report Export

Reports are generated as printable HTML.

Workflow:
1. Generate preview
2. Open printable HTML
3. Use browser print → Save as PDF

---

# 📊 Metrics & Observability (Prometheus + Grafana)

The `native_observability_otel` module exposes a Prometheus-compatible metrics endpoint at `/native-observability/prometheus`.

When `native_observability_otel` is disabled, that route is not provided. When it is enabled, the route remains excluded from
Native Observability tracing by default so the module does not observe its own Prometheus scrape traffic.

## What is scraped

The endpoint reads **live data** from source tables at scrape time (no cron dependency) with a configurable rolling window (default: 5 minutes). Metrics follow the `drupal_*` naming convention standard in the Drupal community.

Key metrics exposed:

- `drupal_http_requests_total` / `drupal_http_requests_window` — HTTP traffic by route, method, status code
- `drupal_http_request_duration_ms{quantile}` — P50/P95/P99 latency per route
- `drupal_http_error_rate` — percentage of 5xx responses per route
- `drupal_cache_operations_total` — cache events by operation (HIT/MISS/…) and bin
- `drupal_span_duration_avg_ms` / `drupal_span_count` — execution span metrics by category

## Local stack (DDEV)

```bash
ddev add-on get MurzNN/ddev-grafana
ddev restart
```

This starts Grafana + Mimir. If you also enable the optional Tempo/Loki/Alloy add-on, the local stack can export traces and logs as well. Three dashboards are pre-provisioned automatically:

- **Drupal — HTTP Overview** — request rate, error rate, P95 latency, top slow routes, AJAX split
- **Drupal — Cache** — operations rate per bin, HIT/MISS breakdown
- **Drupal — Spans Breakdown** — avg duration per category, donut distribution

See full documentation in [`modules/native_observability_otel/README.md`](modules/native_observability_otel/README.md).

If you also enable the DDEV Tempo/Loki/Alloy add-on, expect some runtime noise in the container logs while traces are being written and blocks are being closed. Messages such as Tempo WAL block completion or Grafana dashboard-service bootstrap logs are normal.

The Tempo backend worker can also emit `no jobs found` warnings in a local single-node setup. In this project that is usually operational noise, not a module failure. If you want quieter local logs, the DDEV Tempo config in [`.ddev/grafana/tempo/tempo.yaml`](.ddev/grafana/tempo/tempo.yaml) uses `log_level: error`.

## Release Notes

### Native Observability 1.0.6

This release adds a first ECA bridge, strengthens Prometheus/Grafana support, and improves the built-in dashboard and reporting layer.

- First Native Observability to ECA integration with a demo module for ready-made rules.
- Better Prometheus export and Grafana integration for external observability setups.
- Safer default exclusion of module-own routes from tracing, including the Prometheus scrape endpoint.
- Expanded dashboard coverage, including technical insights and a long-form report for less experienced users.
- Updated OpenTelemetry guidance for secure production configuration.

## Bearer token (production)

Configure the bearer token in **Admin → Configuration → Development → Native Observability OpenTelemetry** (`/admin/config/development/native-observability/otel`). When set, the endpoint requires `Authorization: Bearer <token>`. The settings page shows a ready-to-copy `prometheus.yml` scrape config snippet.

---

# 🧠 Philosophy

Observability should be part of the application architecture itself.
