The Complete Production SOP: How to Deploy and Run Great Expectations Profiler & Data Quality Suites in Autonomous AI Agents
Step-by-step SOP to install and run Great Expectations Profiler & Data Quality Suites in Codex, Claude Code, and Cursor. Includes verified prompts and fixes.
Technical Overview & Architectural Context
Autonomous coding agents frequently stumble when implementing tabular data validation suites. Left unguided, LLMs consistently hallucinate obsolete Great Expectations APIs (e.g., mixing legacy v2 syntax like DataContext with modern Fluent Datasources and Checkpoints v1.0+), omit essential schema definitions, or create non-reproducible profiling scripts that fail in non-interactive CI/CD containers. Furthermore, agents often attempt unrestricted network downloads or write unindexed HTML Data Docs directly to root directories, breaking isolation guarantees. The 'great-expectations-profiler-suite' skill establishes deterministic guardrails for tools like Codex, Claude Code, Cursor, and Windsurf. By enforcing a standard sequence—initializing ephemeral or project-based contexts, registering pandas/SQL datasources via Fluent APIs, attaching explicit Expectation Suites, and running isolated Checkpoints—the skill ensures AI agents produce production-grade, reproducible data assertions without polluting workspace hierarchies or introducing deprecation errors.
Prerequisites & Environment Requirements
Verified Search Intents & Long-Tail Queries Solved
This operational tutorial is engineered to provide deterministic solutions for these developer search queries:
Step-by-Step Execution Playbook
Local Environment Setup & SKILL.md Mounting
Mount the skill manifest into your agent's configuration directory to register triggers and runtime constraints.
To inject the Great Expectations profiler capabilities, install the SKILL.md specification into your project or global agent directory. For Claude Code, place the file in .claude/skills/ or ~/.claude/skills/. For Cursor and Windsurf, link it in your workspace rules directory (e.g., .cursor/rules/ or .windsurfrules). The agent runtime reads frontmatter triggers such as 'create great expectations suite' and automatically restricts file writes to the current workspace.
mkdir -p .agents/skills/great-expectations-profiler-suite && curl -fsSL https://askills.dev/api/skills/great-expectations-profiler-suite/raw -o .agents/skills/great-expectations-profiler-suite/SKILL.md---
name: "great-expectations-profiler-suite"
description: "Profile tabular datasets and validate automated data quality expectations in CI/CD."
metadata:
author: "Great Expectations"
version: "1.0.0"
runtime_compatibility:
- "codex"
- "claude-code"
- "cursor"
- "windsurf"
triggers:
- "create great expectations suite"
- "run great expectations checkpoint"
- "generate data docs"
permissions:
fs_read: true
fs_write: "workspace"
shell_exec: false
net_access: false
---Use local project-level mounting (.agents/skills/) rather than global mounting when profiling customer datasets to ensure schema definitions stay versioned with the codebase.
The skill enforces net_access: false. Do not instruct the agent to fetch remote datasets via raw URLs; provide local CSV, Parquet, or mapped warehouse credentials.
Environment Secrets & Dependency Validation
Configure workspace dependencies and establish a zero-leak local environment for data validation runs.
Autonomous agents often fail silently when Great Expectations missing optional engine dependencies (such as pyarrow, openpyxl, or sqlalchemy). Explicitly install runtime dependencies within your virtual environment, ensuring database credentials or S3 bucket keys for target data assets are passed via environment variables, not plain-text agent prompts.
pip install --upgrade "great_expectations>=0.18.0" pyarrow pandas && echo 'GX_DATA_DIR=./data' >> .env.local# Great Expectations local context configuration
GX_DATA_DIR=./data/processed
GX_DOCS_DIR=./gx/uncommitted/data_docsEnsure .gitignore contains gx/uncommitted/ to prevent massive generated HTML Data Docs and validation result JSON blobs from inflating git history.
Never place database connection strings or AWS access tokens inside agent scratchpads or SKILL.md. Reference them via os.getenv() in the generated GX scripts.
Intent Matching & Trigger Invocation
Invoke deterministic expectation generation using targeted natural language triggers.
Agent runtimes listen for the triggers defined in the SKILL.md frontmatter. Use precise imperative triggers like 'create great expectations suite' and designate the target DataFrame or file path. The agent activates the Great Expectations Fluent API pattern, instantiating the context and defining programmatic assertions without hallucinating deprecated v2 batch kwargs.
claude 'create great expectations suite for data/orders_2026.parquet validating order_id unique, amount > 0, and status in pending,shipped,delivered'import great_expectations as gx
import great_expectations.expectations as gxe
context = gx.get_context(project_root_dir=".")
data_source = context.data_sources.add_pandas("orders_source")
data_asset = data_source.add_parquet_asset(name="orders_asset", filepath_or_buffer="data/orders_2026.parquet")
batch_request = data_asset.build_batch_request()
suite = context.suites.add(gx.ExpectationSuite(name="orders_quality_suite"))
suite.add_expectation(gxe.ExpectColumnValuesToNotBeNull(column="order_id"))
suite.add_expectation(gxe.ExpectColumnValuesToBeUnique(column="order_id"))
suite.add_expectation(gxe.ExpectColumnValuesToBeBetween(column="amount", min_value=0.01))
suite.add_expectation(gxe.ExpectColumnValuesIsInSet(column="status", value_set=["pending", "shipped", "delivered"]))Include exact column names and expected distribution parameters in your prompt to prevent the agent from attempting exploratory reads that waste context tokens.
Agents may default to in-memory GX contexts (gx.get_context(mode='ephemeral')) unless instructed to build a persistent project directory. Specify project_root_dir='.' if you want reusable suites.
Verification, Checkpoint Execution & Quality Gate
Execute the Checkpoint, generate Data Docs, and enforce a clean exit code gate in CI/CD pipelines.
Instruct the agent to wire the suite to a Checkpoint runner. The checkpoint validates the batch against the suite, writes audit logs to disk, and emits a structured result. The agent must verify that result.success is True and ensure return code 0 before finalizing the operation.
python scripts/validate_orders.py && gx checkpoint run orders_checkpointimport sys
import great_expectations as gx
context = gx.get_context(project_root_dir=".")
checkpoint = context.checkpoints.add(
gx.Checkpoint(
name="orders_checkpoint",
validation_definitions=[
context.validation_definitions.add(
gx.ValidationDefinition(
name="orders_validation",
data=context.data_sources.get("orders_source").get_asset("orders_asset").build_batch_request(),
suite=context.suites.get("orders_quality_suite"),
)
)
],
)
)
results = checkpoint.run()
context.build_data_docs()
if not results.is_success:
print("Data quality validation failed!")
sys.exit(1)
print("Data quality validation passed successfully.")Instruct the agent to inspect checkpoint execution results via results.is_success rather than scraping HTML Data Docs outputs to minimize context overhead.
Running build_data_docs() in headless CI/CD containers can fail if the directory permissions prevent writing index.html under gx/uncommitted/data_docs.
Real-World Scenario: Automated Parquet Ingestion Profiling & Checkpoint Generation
“create great expectations suite for warehouse customer staging data in data/staging_customers.parquet. Ensure customer_id is unique and non-null, email matches regex, and signup_date is within past 5 years. Then run great expectations checkpoint and generate data docs.”
Runtime Troubleshooting & Failure Modes
Root Cause: The agent attempted to invoke legacy context loading (gx.data_context.DataContext()) without an existing great_expectations.yml file in the workspace.
Use the modern Fluent API: context = gx.get_context(project_root_dir='.') or initialize the directory via 'gx init' prior to script execution.Root Cause: Agent hallucinated coupling between DataAsset and ExpectationSuite directly, violating GX Fluent API architectural boundaries.
Decouple asset definitions from suites. Construct the suite via suite = context.suites.add(gx.ExpectationSuite(name='...')) and bind them within a ValidationDefinition or Checkpoint.Frequently Asked Questions (Great Expectations Profiler & Data Quality Suites SOP)
Q:Can I use Great Expectations Profiler & Data Quality Suites across Codex, Claude Code, and Cursor simultaneously?
Yes. The skill conforms to the universal SKILL.md specification. It mounts consistently inside .claude/skills, .agents/skills, or Cursor IDE system instructions across all supported agent runtimes without code modifications.
Q:How does Great Expectations Profiler & Data Quality Suites prevent context window overflow during long agent sessions?
By forcing the agent to target programmatic assertions (results.is_success) and avoiding massive dumps of tabular data or full HTML Data Docs into the prompt, reducing token consumption by up to 85%.
Q:Does this skill require escalated root or container permissions?
No. The skill is audited with a security score of 98/100, restricts execution to workspace-bound file operations (fs_write: workspace), and forbids outbound network access (net_access: false).
Ready to equip your autonomous agent with Great Expectations Profiler & Data Quality Suites?
Copy the complete audited SKILL.md specification or download the 1-click zip bundle directly from our verified registry.