🦜×🔭 Instrumenting a Multi-Provider AI App with Arize Phoenix

1. Integration Overview

The Pattern

This case study instruments Poly-Glot AI Workspace, a production app with 1,022 prompt templates across 38 languages, with Arize Phoenix observability. The pipeline is template-driven and multi-provider: it assembles prompts from structured templates and dispatches to 9 AI providers (ChatGPT, Claude, Gemini, Grok, Perplexity, Copilot, Mistral, DuckDuckGo AI, HuggingChat). All 6 test cases are automated with Playwright, producing real OpenTelemetry traces with parent/child span hierarchies and structured evaluations flowing into Phoenix.

What Makes This Integration Different

Most Phoenix examples trace direct LLM API calls. This app doesn't make server-side LLM calls at all. It builds prompts and routes them to provider web UIs. Phoenix still captures the full pipeline because the instrumentation targets the application logic, not the model call:

• Non-LLM-call tracing: Every stage from template selection through prompt assembly to provider dispatch is a traced span, without wrapping an LLM SDK.
• Multi-provider dispatch: 6 providers in one test run, each with its own traced launch span and delivery method (auto-fill vs. clipboard paste).
• Template-driven pipelines: A growing pattern where AI logic lives in prompt engineering, not model calls. Phoenix instruments this cleanly.
• Evaluation-as-observability: Structured pass/fail evaluations attached to spans (18/18 passed), not just latency metrics.

2. Application Architecture & Instrumentation

Instrumentation Architecture

Playwright Test Runner (Node.js ESM)
├─ @opentelemetry/sdk-trace-base 1.30.1 (BasicTracerProvider)
├─ @opentelemetry/exporter-trace-otlp-proto 0.57.2 (protobuf — required by Phoenix)
├─ @opentelemetry/resources 1.30.1
│   └─ Resource: openinference.project.name = "polyglot-free-templates"
▼
Arize Phoenix v20.11.0 (localhost:6006)
├─ OTLP: /v1/traces (protobuf only — HTTP 415 for JSON)
├─ GraphQL: /graphql
└─ Web UI: traces, spans, evaluations

Span Hierarchy (Per Test Case)

3. Test Results: 6 Free Templates × 6 Providers

Evaluation Details (18/18 Passed)

Provider Coverage (6 of 9)

4. Phoenix Dashboard: Live Trace Data

Arize Phoenix v20.11.0 running locally during the test run. All traces routed to polyglot-free-templates project via protobuf OTLP.

📊 Projects Dashboard

The polyglot-free-templates project: 8 traces (6 template tests + 1 compare check + 1 setup), P50 latency 13.2s. The default project shows 0 traces, all telemetry routed correctly via openinference.project.name.

📋 Trace List

Each row is one template test. Root span polyglot.free_template contains 7 child spans (navigate, open, fill, build, send, launch, evaluate). Durations 11-14s. All OK status.

🔍 Trace Detail / Waterfall

Parent-child span hierarchy with timing. fill_fields and build_prompt complete in milliseconds; provider.*.launch dominates latency. Each span carries structured attributes (template name, plan, goal, prompt length, provider key) queryable via GraphQL.

5. Application Test Flow Screenshots

Playwright captures of the template, fill, build, dispatch flow exercised during each traced test.

Free templates filtered (25)

Template picker showing 25 free templates available

Summarization Tool (6 fields)

Template with 6 dynamic input fields populated by test

Code Review (Python input)

Code Review Assistant with Python code pasted as input

Interview Coach (Grok auto-fill)

Interview Coach dispatched to Grok via ?q= auto-fill

LinkedIn Post (Gemini paste)

LinkedIn Post Writer dispatched to Gemini via clipboard paste

Recipe Generator (Claude paste)

Recipe Generator dispatched to Claude via clipboard paste

Compare Mode Verification

Compare Mode verified as available (trial active). Locks behind Pro paywall after 3-day trial. Confirmed via DOM inspection of _appTrialExpired and _isPro. Generated its own trace in Phoenix.

Compare Mode UI — verified unlocked during trial period

6. Node.js Setup Guide: Protobuf OTLP + ESM

npm install @opentelemetry/exporter-trace-otlp-http

npm install @opentelemetry/exporter-trace-otlp-proto

📦 ESM Compatibility — The Second Gotcha

OpenTelemetry v2 packages have ESM/CJS module conflicts that crash NodeTracerProvider in ESM projects. The fix: pin to the 1.x SDK line and use BasicTracerProvider:

@opentelemetry/api@1.9.0
@opentelemetry/sdk-trace-base@1.30.1          ← BasicTracerProvider, not NodeTracerProvider
@opentelemetry/exporter-trace-otlp-proto@0.57.2    ← protobuf, not JSON
@opentelemetry/resources@1.30.1

🏷️ Project Routing via Resource Attribute

Phoenix project assignment uses openinference.project.name as a resource attribute — not an HTTP header, query param, or endpoint path. Must be set at TracerProvider creation time:

const resource = new Resource({
  "openinference.project.name": "my-project",
});
const provider = new BasicTracerProvider({ resource });

Recommendations for Phoenix Docs

• Quickstart: Add a callout that /v1/traces requires protobuf, not JSON. The HTTP 415 response body should include "Expected Content-Type: application/x-protobuf".
• Node.js guide: Show exporter-trace-otlp-proto as the default package, not exporter-trace-otlp-http.
• ESM compatibility: Document the 1.x SDK pin requirement for ESM projects.

Known Instrumentation Gaps

• Response detection: Cross-origin prevents reading provider page content.
• Token counts: Not available — Poly-Glot dispatches to web UIs, not APIs.
• Compare Mode multi-provider: Not tested (consumes trial sends). Verified unlocked.
• Provider-side latency: TTFT not measurable through web UI dispatch.

7. Span Reference & Production Context

Span Counts Summary

Account State During Test

No production behavior modified. Real trial/entitlement system exercised. First free send triggered trial automatically.

Production Reliability (Post-Remediation)

The observability work surfaced 6 production error classes in the MCP server logs. All were remediated and deployed as v1.9.2.

Errors Remediated (22 events → 0)

8. Production MCP Server & Tool Surface

15 Canonical MCP Tools