The Complete Production SOP: How to Deploy and Run Argo Workflows DAG Pipelines & Artifact Passing in Autonomous AI Agents
Production SOP to deploy and run Argo Workflows DAG Pipelines in Codex, Claude Code, and Cursor. Includes artifact passing, DAG syntax, and troubleshooting.
Technical Overview & Architectural Context
Autonomous coding agents frequently generate invalid Argo Workflows definitions due to structural ambiguity in Kubernetes custom resource definitions (CRDs). Without deterministic procedural guardrails, LLMs routinely conflate Argo Steps with DAG templates, introduce circular dependencies across `dependencies` keys, fail to scope output artifact archive formats properly, and hallucinate invalid parameter passing schemas between upstream and downstream tasks. Furthermore, unconstrained agents often miss required S3/MinIO bucket secrets or configure brittle retry strategies that trigger infinite execution loops. The argo-workflows-dag-pipelines skill establishes rigid boundaries for the agent runtime. It enforces standard `Workflow` and `WorkflowTemplate` schemas, mandate-driven artifact passing protocols (`tasks.<task-name>.outputs.artifacts.<artifact-name>`), explicit memoization hashing using input parameter markers, and strict verification gates using `argo lint` or schema-validated dry runs before any manifest is committed to the workspace.
Prerequisites & Environment Requirements
Verified Search Intents & Long-Tail Queries Solved
This operational tutorial is engineered to provide deterministic solutions for these developer search queries:
Step-by-Step Execution Playbook
Local Environment Setup & SKILL.md Mounting
Mount the skill manifest into local agent configuration directories for instant runtime discovery.
Agent runtimes such as Claude Code, Codex, and Cursor scan specific skill trees (like `~/.agents/skills/` or `.cursor/skills/`) to parse triggers and execution SOPs. By placing `SKILL.md` into your local workspace or global config, the agent runtime extracts triggers like 'write argo workflow dag' and enforces DAG-specific CRD validation constraints before authoring YAML manifests.
mkdir -p ~/.agents/skills/argo-workflows-dag-pipelines && curl -fsSL https://askills.dev/api/skills/argo-workflows-dag-pipelines/raw -o ~/.agents/skills/argo-workflows-dag-pipelines/SKILL.md---
name: "argo-workflows-dag-pipelines"
description: "Run complex data processing and ML training DAG workflows natively on Kubernetes."
metadata:
author: "Argo Project"
version: "1.0.0"
runtime_compatibility:
- "codex"
- "claude-code"
- "cursor"
- "windsurf"
triggers:
- "write argo workflow dag"
- "pass artifacts argo workflows"
- "configure workflow retry strategy"
permissions:
fs_read: true
fs_write: "workspace"
shell_exec: false
net_access: false
---Keep the skill markdown focused on structural YAML constraints. Forcing the agent to adhere to standard task inputs/outputs schemas eliminates 90% of artifact routing errors without bloating the LLM context window.
Ensure directory permissions match your active user; mounting as root inside containerized dev environments can prevent Cursor or Codex from reloading the skill index dynamically.
Artifact Repository & Namespace Secret Binding
Ensure secret bindings and S3-compatible artifact storage configurations exist in the target namespace.
Argo Workflows requires access to an object store (MinIO, AWS S3, or GCS) to pass files between isolated container pods. The agent must reference an existing `artifactRepositoryRef` or configure an inline repository that pulls credentials from Kubernetes Secrets without emitting raw secret keys into manifest declarations.
kubectl create secret generic my-minio-cred --from-literal=accesskey=minioadmin --from-literal=secretkey=minioadmin -n argo --dry-run=client -o yaml | kubectl apply -f -apiVersion: v1
kind: ConfigMap
metadata:
name: default-artifact-repo
namespace: argo
data:
default-artifact-repo: |
s3:
endpoint: minio:9000
bucket: workflows-artifacts
insecure: true
accessKeySecret:
name: my-minio-cred
key: accesskey
secretKeySecret:
name: my-minio-cred
key: secretkeyConfigure default artifact repository references in the workflow-controller ConfigMap to avoid repeating object storage declarations in every generated workflow.
Never allow the AI agent to hardcode AWS_SECRET_ACCESS_KEY or secret keys directly inside workflow container environment variables; instruct it to use `secretKeyRef` exclusively.
Deterministic Trigger Invocation & DAG Generation
Invoke the agent using calibrated triggers to generate a validated DAG pipeline with artifact dependencies.
Provide the agent with explicit inputs and outputs requirements matching the frontmatter trigger 'write argo workflow dag' or 'pass artifacts argo workflows'. The agent executes SOP rules: assigning unique task names, chaining tasks using `dependencies`, declaring `artifacts` paths, and setting conditional retry parameters.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: ml-pipeline-dag-
namespace: argo
spec:
entrypoint: main-pipeline
templates:
- name: main-pipeline
dag:
tasks:
- name: ingest-data
template: data-ingest
- name: train-model
dependencies: [ingest-data]
template: model-train
arguments:
artifacts:
- name: dataset
from: "{{tasks.ingest-data.outputs.artifacts.raw-dataset}}"
- name: data-ingest
container:
image: python:3.11-slim
command: [sh, -c]
args: ["python -c \"import os; os.makedirs('/mnt/out', exist_ok=True); open('/mnt/out/data.csv', 'w').write('id,val\\n1,100\\n2,200')\""]
outputs:
artifacts:
- name: raw-dataset
path: /mnt/out/data.csv
- name: model-train
inputs:
artifacts:
- name: dataset
path: /mnt/in/data.csv
container:
image: python:3.11-slim
command: [sh, -c]
args: ["wc -l /mnt/in/data.csv"]Explicitly specify `archive: { none: {} }` on output artifacts when downstream containers expect flat files rather than tarballs (.tar.gz), avoiding uncompression step overhead.
When declaring task dependencies, using `dependencies: [task-a]` enforces sequential order, while `depends: "task-a.Succeeded || task-b.Failed"` requires expression evaluation syntax.
Verification, Linting & Dry-Run Quality Gate
Run local Argo CLI linting and schema validation checks before submitting the workflow to the cluster.
Before executing manifests on Kubernetes, the agent runs `argo lint` against target files. This ensures parameters, artifact templates, DAG dependency references, and volume mounts comply with the installed Argo CRD specification.
argo lint workflows/ml-pipeline-dag.yaml && kubectl apply --dry-run=client -f workflows/ml-pipeline-dag.yaml#!/usr/bin/env bash
set -euo pipefail
TARGET_WORKFLOW="${1:-workflows/ml-pipeline-dag.yaml}"
echo "==> Linting Argo Workflow manifest..."
argo lint "$TARGET_WORKFLOW"
echo "==> Executing dry-run submission..."
argo submit --dry-run "$TARGET_WORKFLOW" -n argo
echo "Validation passed successfully."Embed `argo lint` into pre-commit git hooks or your agent's test command execution phase to prevent invalid YAML from reaching production GitOps repositories.
Passing `--dry-run` to `argo submit` validates controller-side admissions and templates without provisioning pods or consuming cluster compute.
Real-World Scenario: Two-Stage Feature Extraction and Model Training DAG with Artifact Passing
“Write an Argo Workflow DAG named feature-extraction-pipeline in namespace argo. Task 1 runs a data preprocessor script that emits an artifact features.parquet at /tmp/features.parquet. Task 2 depends on Task 1, consumes features.parquet, has a retry strategy with 3 retries on NonZeroExitCode, and writes an evaluation metric output.”
Runtime Troubleshooting & Failure Modes
Root Cause: The upstream task failed to write the designated file to the exact path specified under `outputs.artifacts.path`, or the container terminated prematurely before the artifact collector sidecar could upload the file to S3/MinIO.
Ensure the container command creates parent directories and flushes file writes: `mkdir -p $(dirname /tmp/features.parquet) && python extract.py --out /tmp/features.parquet`. Additionally verify the output artifact declaration matches: `outputs: { artifacts: [{ name: 'data', path: '/tmp/features.parquet' }] }`.Root Cause: Typographical error or non-existent task name declared in `dependencies`. Argo Workflows requires every item listed in `dependencies` to exist identically within `tasks[].name` in the current DAG template.
Run `argo lint <manifest>.yaml` to detect unmatched dependency identifiers. Align `dependencies: [exact-task-name]` with the corresponding upstream `name: exact-task-name`.Frequently Asked Questions (Argo Workflows DAG Pipelines & Artifact Passing SOP)
Q:Can I use Argo Workflows DAG Pipelines & Artifact Passing across Codex, Claude Code, and Cursor simultaneously?
Yes. The skill adheres to the open SKILL.md standard. By placing the manifest in your shared repository or symlinking it across `~/.agents/skills/` and `.cursor/skills/`, all major LLM agents parse the triggers and adhere to identical DAG modeling SOPs.
Q:How does this skill handle large files passed between DAG nodes without exhausting pod memory?
The skill enforces external object storage artifact passing (MinIO/S3/GCS) with tar archive overrides (`archive: { none: {} }` when appropriate). Files are streamed directly to and from object storage buckets by Argo init and wait sidecars rather than residing in agent memory or container root overlay filesystems.
Q:Does this skill require escalated root or container permissions?
No. The skill operates with a security score of 98/100 (`shell_exec: false`, `net_access: false`, `fs_write: workspace`). It simply enforces deterministic generation and validation of declarative Kubernetes manifests within your local project files.
Ready to equip your autonomous agent with Argo Workflows DAG Pipelines & Artifact Passing?
Copy the complete audited SKILL.md specification or download the 1-click zip bundle directly from our verified registry.