# Data Processing Nodes

Nodes for transforming, extracting, and reshaping data.

---

## Data Shaper

Reshapes data between different structures using field mappings and optional JSON Schema validation. This is the primary node for converting data between formats when connecting nodes with different data shapes.

**Category:** Data Processing

### Input Ports

| Port | Type | Required | Description |
|------|------|----------|-------------|
| **data** | Mixed | Yes | The source data to reshape |

### Configuration

| Parameter | Type | Description |
|-----------|------|-------------|
| **mapping** | Object | Field mapping configuration — maps source paths to target paths |
| **schema** | Object | Optional JSON Schema for validating the output |
| **validationMode** | String | `strict` (fail on invalid) or `soft` (warn but continue) |

### Output Ports

| Port | Type | Description |
|------|------|-------------|
| **data** | Mixed | The reshaped data |
| **valid** | Boolean | Whether the output passed schema validation |
| **errors** | Array | Validation errors (if any) |
| **mappedFields** | Number | Number of fields successfully mapped |

### Usage

Use Data Shaper when you need to transform data from one structure to another — for example, extracting specific fields from a complex object, renaming fields, or restructuring nested data.

---

## Data Extractor

Extracts specific values from complex data structures using JSONPath (RFC 9535) or dot-notation property paths.

**Category:** Data Processing

### Input Ports

| Port | Type | Required | Description |
|------|------|----------|-------------|
| **data** | Mixed | Yes | The data to extract from |

### Configuration

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| **path** | String | Yes | JSONPath expression (e.g., `$.users[0].name`) or dot notation (e.g., `users.0.name`) |
| **default** | Mixed | No | Default value if the path doesn't match |
| **extract_all** | Boolean | No | Extract all matches (true) or just the first (false) |

### Output Ports

| Port | Type | Description |
|------|------|-------------|
| **data** | Mixed | The extracted value(s) |
| **path** | String | The extraction path used |
| **success** | Boolean | Whether the extraction found a match |
| **match_count** | Number | Number of matches found |

### Common JSONPath Expressions

| Expression | Description |
|------------|-------------|
| `$.name` | Top-level field |
| `$.users[0].email` | First user's email |
| `$.users[*].email` | All users' emails |
| `$.items[?(@.active)]` | Items where `active` is true |
| `$..name` | All `name` fields at any depth |

---

## Data Operations

Performs common array operations on data: filtering, sorting, grouping, and more.

**Category:** Data Processing

### Input Ports

| Port | Type | Required | Description |
|------|------|----------|-------------|
| **data** | Array | Yes | The array to operate on |

### Configuration

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| **operation** | String | Yes | Operation: `filter`, `sort`, `group`, `map`, `reduce`, `unique`, `slice`, `merge` |
| **key** | String | Varies | The field/key to operate on |
| **value** | Mixed | Varies | The value for comparison operations |
| **condition** | Mixed | Varies | Condition for filter operations |

### Output Ports

| Port | Type | Description |
|------|------|-------------|
| **result** | Array | The operation result |
| **operation** | String | The operation performed |
| **input_count** | Number | Input array size |
| **output_count** | Number | Output array size |

---

## Split Text

Splits text into chunks using various methods.

**Category:** Data Processing

### Input Ports

| Port | Type | Required | Description |
|------|------|----------|-------------|
| **text** | String | Yes | The text to split |

### Configuration

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| **method** | String | `words` | Split method: `words`, `characters`, `sentences`, `paragraphs` |
| **chunkSize** | Number | (varies) | Number of units per chunk |
| **separator** | String | (auto) | Custom separator string |

### Output Ports

| Port | Type | Description |
|------|------|-------------|
| **chunks** | Array | The resulting text chunks |
| **method** | String | The split method used |
| **chunk_size** | Number | The chunk size used |
| **total_chunks** | Number | Total number of chunks |

---

## Regex Extractor

Extracts data using regular expressions with security protections against ReDoS (catastrophic backtracking).

**Category:** Data Processing

### Input Ports

| Port | Type | Required | Description |
|------|------|----------|-------------|
| **text** | String | Yes | The text to search |

### Configuration

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| **pattern** | String | Yes | The regular expression pattern |
| **flags** | String | No | Regex flags (e.g., `i` for case-insensitive) |
| **matchAll** | Boolean | No | Find all matches (true) or just the first (false) |

### Output Ports

| Port | Type | Description |
|------|------|-------------|
| **matches** | Array | Matched values |
| **total_matches** | Number | Number of matches found |

---

## Data to JSON

Converts structured data (objects, arrays) to a JSON string.

**Category:** Data Processing

### Input Ports

| Port | Type | Required | Description |
|------|------|----------|-------------|
| **data** | Mixed | Yes | The data to serialize |

### Output Ports

| Port | Type | Description |
|------|------|-------------|
| **json** | String | The JSON string |
| **success** | Boolean | Whether serialization succeeded |

---

## JSON to Data

Parses a JSON string into structured data (objects, arrays).

**Category:** Data Processing

### Input Ports

| Port | Type | Required | Description |
|------|------|----------|-------------|
| **json** | String | Yes | The JSON string to parse |

### Output Ports

| Port | Type | Description |
|------|------|-------------|
| **data** | Mixed | The parsed data structure |
| **success** | Boolean | Whether parsing succeeded |

---

## Data to Dataframe

Converts data into dataframe format with multiple output orientations.

**Category:** Data Processing

### Input Ports

| Port | Type | Required | Description |
|------|------|----------|-------------|
| **data** | Array | Yes | The input data |

### Configuration

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| **format** | String | `json` | Output format: `json`, `csv`, `parquet` |
| **includeIndex** | Boolean | false | Include row index |
| **orient** | String | `records` | Orientation: `records`, `index`, `columns`, `split` |

### Output Ports

| Port | Type | Description |
|------|------|-------------|
| **dataframe** | Mixed | The dataframe output |
| **format** | String | The format used |
| **rows_count** | Number | Number of rows |
| **columns_count** | Number | Number of columns |

---

## Message to Data

Converts text messages to structured data by parsing JSON, CSV, XML, YAML, or key-value format.

**Category:** Data Processing

### Input Ports

| Port | Type | Required | Description |
|------|------|----------|-------------|
| **message** | String | Yes | The message text to parse |

### Configuration

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| **format** | String | `json` | Expected format: `json`, `csv`, `xml`, `yaml`, `key_value` |
| **extractFields** | Array | (none) | Specific fields to extract from the parsed data |

### Output Ports

| Port | Type | Description |
|------|------|-------------|
| **data** | Object | The parsed structured data |
| **format** | String | The format used |
| **original_message** | String | The original message text |
