# Taxonomy Specification

The ability tree is generated from `../data/task-dimension.csv` using one
classification logic at each layer:

`static/dynamic -> passive/active -> capability stage -> capability family -> normalized task -> paper-task`

`paper-task` preserves the task wording used by the source paper. `normalized
task` is the manually reviewed canonical task name used to compare tasks across
papers. The generator reads both fields directly from the CSV and does not infer
task names, stages, or families.

## Structural Layers

### Temporal Setting

- `static`: the spatial state relevant to the question is fixed. Multiple images
  of one unchanged state are still static.
- `dynamic`: the relevant scene, object, observer, video, or world state changes
  over time.

### Agency Setting

- `passive`: the model receives observations under a fixed evaluation protocol.
  Producing an action description or plan does not by itself make a task active.
- `active`: model-selected actions change a later observation or environment
  state in the benchmark protocol.

### Capability Stage

The stage order follows the approximate order in which an agent forms and acts
on a response:

`perception -> memory -> reasoning -> planning -> execution`

- `perception`: directly extracts, recognizes, grounds, or estimates information
  available in the current observation.
- `memory`: retains, updates, or retrieves information from earlier observations.
- `reasoning`: combines evidence, relations, transformations, rules, or predicted
  consequences to derive an answer that is not a direct readout.
- `planning`: selects or organizes future actions, routes, procedures, or views.
- `execution`: actually changes an object, the agent position, the viewpoint, or
  the environment/interface state. Execution is valid only under `active`.

Generation is not a capability stage. A generated output is placed according to
the capability being evaluated, such as physical reasoning or route planning.

## Capability Families

Capability families describe the operation evaluated within a stage. They do
not repeat input protocols or surface properties such as single-view,
multi-view, distance, or direction.

### Perception

- `Grounding`: associate a reference with an entity or spatial region.
- `Recognition`: identify a scene, object, attribute, event, or visible state.
- `Metric Estimation`: estimate or compare depth, distance, size, or path length.
- `Pose Estimation`: estimate orientation, pose, or camera geometry.
- `Motion Tracking`: observe motion state, displacement, trajectory, speed, or
  acceleration.
- `Visibility Detection`: determine whether an entity is currently visible or
  exists in the current observation.
- `Counting`: directly enumerate currently available entities.
- `Scene Parsing`: recover a structured representation of the current scene or
  board state.
- `Correspondence Detection`: directly match equivalent current observations or
  representations.

### Memory

- `Episodic Memory`: retain and retrieve events or episode-level information.
- `State Tracking`: maintain changing object, relation, or scene state.
- `Sequence Memory`: retain an observed order or sequence.
- `Route Memory`: retain a previously observed or traversed route.
- `Count Memory`: accumulate counts over time or revisits.
- `Spatial Working Memory`: temporarily maintain and manipulate spatial state.

### Reasoning

- `Spatial Relation Inference`: derive spatial relations between entities.
- `Perspective Transformation`: convert or reason between viewpoints or
  coordinate frames.
- `Geometric Reasoning`: perform geometric construction, rotation, projection,
  folding, or assembly reasoning.
- `Cross-view Correspondence`: associate entities, actions, people, or states
  across viewpoints.
- `Occlusion Inference`: infer hidden, partially visible, or out-of-view content.
- `Physical Reasoning`: infer motion, causality, stability, feasibility, or future
  physical state.
- `Affordance Reasoning`: infer possible uses, requirements, or interactions.
- `Counterfactual Reasoning`: infer consequences under a hypothetical spatial
  change.
- `Order Reasoning`: derive spatial, temporal, or procedural order.
- `Consistency Reasoning`: judge agreement or compatibility across observations
  or representations.
- `Quantitative Reasoning`: derive a count or quantity through multi-step
  reasoning rather than direct enumeration.
- `Topology Reasoning`: infer connectivity, containment, adjacency, or reachability.
- `Pattern Reasoning`: infer a spatial pattern or identify an outlier.

### Planning

- `Route Planning`: choose or evaluate a route through space.
- `Procedure Planning`: construct, revise, or evaluate an action procedure.
- `View Planning`: select a useful observation or exploration viewpoint.
- `Motion Planning`: plan a future trajectory or motion sequence.
- `Affordance Planning`: choose a feasible interaction or grasp location.

### Execution

Execution families are defined by what the model actually affects:

- `Manipulation`: changes an object or object configuration.
- `Motion`: changes the agent's own position.
- `Viewpoint Control`: changes camera position, orientation, or observation
  direction.
- `Environment Interaction`: changes environment or interface state not covered
  by the other three families.

## Normalized Task Rules

1. A normalized task has exactly one `(capability stage, capability family)`
   path throughout the CSV.
2. A paper-task may appear on more than one path only when the source explicitly
   evaluates more than one task or capability in that subcategory.
3. Different source terms that denote the same task share one normalized task.
4. A broad capability, dataset split, metric, input protocol, or paper-specific
   label is not promoted to a normalized task unless it denotes an evaluated task.
5. Source subtask tables and definitions take priority over task names inferred
   from examples or downstream applications.
6. `Visual Grounding` is used for the established language-conditioned visual
   grounding task; `Spatial Localization` is reserved for localization tasks that
   do not themselves define a grounding protocol.

## Tag Dimensions

Tags describe evaluation conditions and never create duplicate ability branches.
Multiple values are separated by semicolons.

| Dimension | Values | Definition |
| --- | --- | --- |
| `input media` | `image`; `video`; `audio`; `text` | Media channels required by the task. |
| `input source configuration` | `single-source`; `double-source-aligned`; `double-source-unaligned`; `multi-source-aligned`; `multi-source-unaligned` | Number of independently supplied sources plus their cross-source alignment. One video is one source. For images, aligned sources depict the same unchanged scene or state. For videos, aligned sources share an event timeline. |
| `temporal protocol` | `offline`; `prefix`; `online`; `full duplex` | `offline`: complete input is available before response. `prefix`: a fixed causal prefix ending at the query time is assembled into one request. `online`: chunks incrementally update one inference session, with input paused during output. `full duplex`: new input continues to arrive and be processed during output. |
| `reference frame` | `egocentric`; `allocentric` | Coordinate frame required by the requested judgment, not merely the camera viewpoint. Leave blank when the output has no coordinate-frame semantics or the source is insufficient. |

No current row is coded as `full duplex`; the value is retained because it is a
valid protocol category for future benchmarks.

## Generator Invariants

The generator must fail rather than silently infer or repair data when any of
the following is violated:

- required fields are blank;
- a tag contains an unsupported value;
- one normalized task maps to multiple stage/family paths;
- an execution row is passive;
- a capability family is not valid for its stage.

The rendered tree keeps the static/dynamic and passive/active skeleton visible.
Passive branches contain perception, memory, reasoning, and planning. Active
branches additionally contain execution. Empty stages remain visible without
inventing placeholder tasks. Paper-task leaves are collapsed by default and
display the four tag dimensions when expanded.