The Complete Production SOP: How to Deploy and Run Arize Phoenix OpenTelemetry RAG Evaluator in Autonomous AI Agents

---
name: "arize-phoenix-rag-evaluator"
description: "Visualize RAG retrieval trajectories and evaluate relevance using Arize Phoenix."
metadata:
  author: "Arize AI"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "trace rag with arize phoenix"
  - "evaluate retrieval precision phoenix"
  - "visualize embedding space phoenix"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# Arize Phoenix OpenTelemetry RAG Evaluator

Instruments LLM pipelines with Phoenix OpenTelemetry tracing: inspecting embedding vector projections, evaluating retrieval precision, diagnosing latency bottlenecks, and export to OpenInference.

## When to Invoke
Invoke this skill whenever you need to:
- trace rag with arize phoenix
- evaluate retrieval precision phoenix
- visualize embedding space phoenix

## Execution SOP & Rules
1. Instrument applications with openinference-instrumentation-langchain or llamaindex.
2. Launch local visualization server with phoenix.launch_app() during development.
3. Run benchmark evaluations using phoenix.evals: evaluate QA correctness and hallucination.
4. Export OpenTelemetry traces to enterprise monitoring collectors in production.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.

import phoenix as px
from openinference.instrumentation.langchain import LangChainInstrumentor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from openinference.instrumentation import OTLPSpanExporter

def setup_phoenix_tracing(local_ui: bool = True):
    """Initializes OpenInference tracing and mounts Arize Phoenix collector."""
    if local_ui:
        session = px.launch_app(port=6006)
        print(f"Phoenix UI accessible at {session.url}")
    
    # Register LangChain auto-instrumentation
    LangChainInstrumentor().instrument()
    return True

if __name__ == '__main__':
    setup_phoenix_tracing()

import pandas as pd
import phoenix as px
from phoenix.evals import (RAG_RELEVANCY_PROMPT_TEMPLATE, llm_classify, OpenAIModel)

def evaluate_retrieval(query_df: pd.DataFrame):
    """Evaluates retrieval precision against OpenInference trace exports."""
    eval_model = OpenAIModel(model_name="gpt-4o-mini")
    relevance_eval = llm_classify(
        dataframe=query_df,
        template=RAG_RELEVANCY_PROMPT_TEMPLATE,
        model=eval_model,
        rails=["relevant", "irrelevant"],
        provide_explanation=True
    )
    px.log_evaluations(relevance_eval)
    return relevance_eval

#!/usr/bin/env bash
set -euo pipefail

# Run unit tests validating trace span generation
python -m unittest discover -s tests

# Check if Phoenix application endpoint is active
STATUS_CODE=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:6006 || echo "500")
if [ "$STATUS_CODE" != "200" ]; then
    echo "Warning: Phoenix server not responding on port 6006"
fi

echo "Verification completed successfully."