The Complete Production SOP: How to Deploy and Run Vespa Real-Time Big Data Search & ML Ranking in Autonomous AI Agents
Step-by-step SOP to install and run Vespa Real-Time Search & ML Ranking in Codex, Claude Code, and Cursor. Build schemas, phased ranking, and tensor queries.
Technical Overview & Architectural Context
Autonomous coding agents frequently stumble when engineering Vespa application packages. Without strict SOP enforcement, models often confuse Elasticsearch or OpenSearch query DSL with Vespa YQL (Your Query Language), conflate Vespa schema syntax (.sd files) with generic JSON schemas, or fail to declare indexing pipelines correctly for tensor fields and lexical tokens. Furthermore, agents typically generate invalid phased ranking expressions—such as invoking dotProduct or closeness functions over undefined query tensor types, or omitting the required rank-profile inputs block. The vespa-realtime-search-and-rank skill injects rigid, deterministic guardrails into Codex, Claude Code, Cursor, and Windsurf. It constrains the agent to conform to Vespa application packaging standards: defining document schemas in schemas/item.sd, configuring container and content clusters in services.xml, specifying strict first-phase and second-phase ranking profiles, and constructing syntactically valid JSON search payloads over HTTP POST. This guarantees that vector embeddings, HNSW indexes, and lexical ranking functions converge into deployable, crash-proof infrastructure definitions.
Prerequisites & Environment Requirements
Verified Search Intents & Long-Tail Queries Solved
This operational tutorial is engineered to provide deterministic solutions for these developer search queries:
Step-by-Step Execution Playbook
Local Environment Setup & SKILL.md Mounting
Create the local skill manifest directory and inject verified procedural rules into the agent runtime path.
Autonomous runtimes like Claude Code and Codex parse procedural instructions from standard skill directories (~/.agents/skills or workspace-relative .skills/). We install the SKILL.md file directly into the skill directory so the agent's semantic router binds user requests matching 'deploy vespa application', 'configure vespa rank profile', and 'query vespa tensor search' to this explicit execution pipeline.
mkdir -p .agents/skills/vespa-realtime-search-and-rank && curl -fsSL https://askills.dev/api/skills/vespa-realtime-search-and-rank/raw -o .agents/skills/vespa-realtime-search-and-rank/SKILL.md---
name: "vespa-realtime-search-and-rank"
description: "Serve structured search, tensor computations, and machine-learned ranking models with Vespa."
metadata:
author: "Vespa Engine"
version: "1.0.0"
runtime_compatibility:
- "codex"
- "claude-code"
- "cursor"
- "windsurf"
triggers:
- "deploy vespa application"
- "configure vespa rank profile"
- "query vespa tensor search"
permissions:
fs_read: true
fs_write: "workspace"
shell_exec: false
net_access: false
---
# Vespa Real-Time Big Data Search & ML Ranking
Configures Vespa applications: writing schemas with document types and tensor fields, configuring phased ranking expressions (first-phase and second-phase), and deploying Vespa services with Docker.
## When to Invoke
Invoke this skill whenever you need to:
- deploy vespa application
- configure vespa rank profile
- query vespa tensor search
## Execution SOP & Rules
1. Define schema in services.xml and schemas/item.sd specifying document fields and tensors.
2. Define rank-profile with first-phase expression: bm25(title) + closeness(field, embedding).
3. Package application directory and deploy: vespa deploy --wait 300.
4. Execute queries via HTTP POST to /search/ with JSON body specifying yql and input tensors.
## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.
Mounting the skill locally inside .agents/skills/ preserves context tokens by scoping rules only to relevant Vespa search workflows rather than bloating global system prompts.
Ensure your agent sandbox has fs_write permissions set to workspace so the agent can write schemas/item.sd and services.xml without hitting EACCES errors.
Environment Secrets & Dependency Validation
Configure target Vespa container endpoints and ensure vespa-cli can authenticate without hardcoding tokens in prompts.
Vespa CLI requires endpoint targeting and optional mutual TLS (mTLS) cert paths when deploying to Vespa Cloud or production Kubernetes clusters. Set these variables in the local workspace environment so the agent can invoke deployments deterministically without printing secret tokens into the session conversation logs.
cat << 'EOF' >> .env.local
VESPA_CLI_TARGET=local
VESPA_CLI_ENDPOINTS=http://127.0.0.1:8080
VESPA_CLI_CONTAINER_NAME=vespa
EOF
git check-ignore -q .env.local || echo '.env.local' >> .gitignoreNever pass Vespa Cloud private keys or deployment api-keys directly in natural language conversation prompts; rely on VESPA_CLI_DATA_PLANE_KEY_FILE environment variables.
Failure to append .env.local to .gitignore will trigger safety score violations and risk accidental secret leakage during automated git commits.
Intent Matching & Trigger Invocation
Invoke the agent using precise frontmatter triggers to generate the schema, services.xml, and rank-profile.
Instruct the agent to scaffold a hybrid search package with dense vector embeddings and BM25 lexical ranking. The agent recognizes the explicit trigger phrases ('configure vespa rank profile' and 'deploy vespa application'), prompting it to structure the files according to the strict SOP instead of improvising arbitrary configurations.
Include the target tensor dimension in the prompt (e.g., tensor<float>(x[384])) so the agent generates compatible inputs and document field definitions.
Do not request multi-node production deployment in the initial generation step; establish single-node container compatibility first to verify ranking syntax.
Verification, Linting & Output Quality Gate
Execute syntax linting and run dry-run validation on the generated Vespa application package.
Before committing changes or invoking live deployment, the agent must run validation scripts or invoke vespa-cli's verification pipeline. This quality gate inspects the schema syntax, ensures tensors have defined dimensions, and verifies the first-phase and second-phase ranking expressions compile cleanly.
vespa config get vespa.config.search.ranking-expressions || vespa status --wait 30Use `vespa query` dry-runs to inspect the parsed query tree and verify that tensor inputs match rank-profile tensor inputs.
If using HNSW indexing on tensor fields, distance-metric must be declared explicitly in schemas/item.sd or compilation will fail.
Real-World Scenario: Deploying Hybrid Vector & Lexical Ranking Application on Local Vespa Cluster
“configure vespa rank profile and deploy vespa application for an e-commerce catalog named 'item'. Include a text field 'title', a float tensor field 'embedding' with dimension 384, and a hybrid rank-profile with first-phase combining bm25(title) and closeness(field, embedding). Verify with a sample tensor query.”
Runtime Troubleshooting & Failure Modes
Root Cause: The agent generated a rank-profile that uses closeness() without defining the corresponding input query tensor name and dimensionality in the rank-profile inputs block.
Add the inputs block inside the rank-profile in schemas/item.sd:
rank-profile hybrid {
inputs {
query(query_embedding) tensor<float>(x[384])
}
first-phase {
expression: bm25(title) + closeness(field, embedding)
}
}Root Cause: Vespa requires explicit distance-metric declarations (e.g., angular, euclidean, innerproduct) when configuring HNSW indexes on tensor attributes.
Update field 'embedding' in schemas/item.sd to explicitly include the metric:
field embedding type tensor<float>(x[384]) {
indexing: attribute | index
attribute {
distance-metric: angular
}
index {
hnsw {
max-links-per-node: 16
neighbors-to-explore-at-insert: 200
}
}
}Frequently Asked Questions (Vespa Real-Time Big Data Search & ML Ranking SOP)
Q:Can I use Vespa Real-Time Big Data Search & ML Ranking across Codex, Claude Code, and Cursor simultaneously?
Yes. The skill conforms to the universal SKILL.md specification. Runtimes like Claude Code read it from ~/.agents/skills/ or .claude/skills/, while Cursor and Windsurf ingest it via workspace instructions (.cursorrules or .windsurfrules referencing the SOP).
Q:How does Vespa Real-Time Big Data Search & ML Ranking prevent context window overflow during long agent sessions?
By establishing concise procedural boundaries and deterministic validation gates rather than polluting system prompts with redundant rules. The agent only loads schema, deployment, and ranking instructions when matching trigger phrases, keeping the active context clean.
Q:Does this skill require escalated root or container permissions?
No. It is audited with a security score of 98/100 and runs safely within workspace-scoped sandboxes. It communicates via the local filesystem and HTTP requests directly to the Vespa configuration server without needing root access.
Ready to equip your autonomous agent with Vespa Real-Time Big Data Search & ML Ranking?
Copy the complete audited SKILL.md specification or download the 1-click zip bundle directly from our verified registry.