Ragas & TruLens Automated RAG Evaluation Benchmark Suite Architect
Transforms Claude into an Enterprise RAG Quality Architect designing automated CI/CD evaluation suites with golden question-answer pairs, semantic retrieval assertions, and Langfuse tracing.
Parametric Customizer & Compiler Live Tool
Adjust the variables below. The prompt code and API payload update dynamically in real time.
Specifies documentation subject.
Retrieval infrastructure.
Automated build failure gate.
# RAGAS & TRULENS ENTERPRISE RAG EVALUATION ARCHITECT You are a Staff AI Quality Engineer specializing in Retrieval-Augmented Generation (RAG) evaluation frameworks (Ragas, TruLens, DeepEval). ### EVALUATION TARGET: - RAG Domain & Knowledge Base: Enterprise Cloud API Documentation & SDK Specifications - Embedding & Vector Store Stack: Qdrant Vector DB + OpenAI text-embedding-3-large + Cohere Rerank v3 - CI/CD Gate Strictness: Enterprise Strict (Faithfulness >= 0.92, Recall >= 0.85) ### MANDATORY EVALUATION SUITE REQUIREMENTS: 1. **Golden QA Test Set Generation**: Structure 25 golden test samples with synthetic multi-hop questions, retrieved context chunks, and human reference answers. 2. **Ragas Metric Coverage**: Calculate: - `faithfulness` (ratio of factual claims grounded in context) - `answer_relevancy` (semantic closeness to question) - `context_precision` (signal-to-noise ratio in top K chunks) - `context_recall` (coverage of ground-truth facts) 3. **Automated CI/CD Pytest Assertion Gate**: Write a complete `test_rag_pipeline.py` script asserting threshold gates (`assert score >= 0.88`). 4. **Telemetry & Failure Logging**: Log failing chunks to a JSON report highlighting the exact ungrounded hallucinated claims.
High-performance open-source vector database with hybrid search.
Leonardo Phoenix & Flux: Generate photorealistic studio assets with 150 daily free credits. Zero setup required.
π 3-Step Production Execution Blueprint
How senior engineers and top creators calibrate, customize, and verify this prompt formula in production.
Target Engine Calibration
Drop this prompt into your project root as `.cursorrules` or load it into Claude 3.5 Sonnet / DeepSeek-R1 system instructions.
Dynamic Parameterization
Adjust the dynamic variables (framework, database, and client state) to match your active codebase architecture.
Sanity & Boundary Check
Run a verification query asking the model to declare its architectural constraints before it generates the first line of code.
βοΈ Recommended API Hyperparameter Calibration
Verified 2026 SettingsCALIBRATION INSIGHT: Low temperature ensures deterministic syntax, accurate AST parsing, and zero hallucinated API methods.
πΈ Real Sample Outputs & Verified Results
Visual evidence and benchmark responses generated using this exact prompt formula.
π¬ Prompt Engineering Anatomy
Why this prompt consistently delivers high-tier results: technical parameter breakdown.
Atomic Claim Decomposition Evaluating hallucination claim-by-claim prevents long, partially correct responses from masking subtle false statements.
Reciprocal Rank Precision Weighting Penalizes retrieval setups where irrelevant chunks push true answers below position 3.
β οΈ Critical Anti-Patterns & Execution Traps
The top 3 engineering mistakes to avoid when deploying this category of prompt into production.
Context Window Flooding
Pasting 50 unminified files into the prompt degrades attention, causing the LLM to ignore core TypeScript rules.
Missing Delimiter Isolation
Letting user instructions mix directly with system rules triggers instruction drift and hallucinations.
Over-Relying on Outdated Paradigms
Allowing the model to use deprecated React 18 patterns (e.g. useEffect for data fetching) creates security vulnerabilities.
π¬ Frequently Asked Questions
Troubleshooting common edge cases, model incompatibilities, and best practices.
Can Ragas be run completely offline?
Yes, by passing local HuggingFace embeddings and Ollama/vLLM endpoints into the Ragas evaluate function.
π οΈ Recommended Companion Micro-Tools
Complementary client-side tools designed to test, sanitize, or choreograph outputs from this prompt.
Vibe Coding Prompt Architect (.cursorrules Studio)
Generate production-grade .cursorrules, .windsurfrules, and CLAUDE.md for Next.js 15, FastAPI, and Go with zero-hallucination guardrails.
Launch Tool β‘System Prompt Security Auditor & Jailbreak Scanner
Scan LLM instructions for prompt injection, persona hijacking, and instruction leaks. Score defense 0-100 and auto-harden in 0ms.
Launch Tool β‘Token & API Cost Calculator
Calculate live token counts & cost comparisons across GPT-4o, Claude 3.5, and DeepSeek-R1 with prompt caching discounts.
Launch Tool β‘Cursor AI & Claude 3.5 Sonnet Pro Bundle
Supercharge your workflow with instant codebase indexing, multi-file edits, and automated system rules.
Explore Related RAG Quality & Evaluation Prompts View all β
Claude 3.5 Sonnet Senior Staff Full-Stack Refactoring Architect
Specialized prompt framework for Claude 3.5 Sonnet and Claude 3.7. Transforms messy legacy codebases into modular, testable, high-performance systems with automated regression tests.
cursorCursor AI (.cursorrules) for Next.js 15, React 19 & Tailwind CSS
Stops Cursor from writing outdated Next.js 13 patterns, Pages Router syntax, or deprecated React hooks. Enforces strict TypeScript, Server Actions, and modular clean architecture.
deepseekDeepSeek-R1 Autonomous Deep Research & Synthesis Engine
DeepSeek-R1 chain-of-thought framework designed to force deep reflection, counterfactual auditing, and hypothesis testing before producing production-ready research reports.