Deploy Pipeline

Deploying AI-powered applications introduces challenges that traditional CI/CD pipelines were never designed for. Here are the key concerns:

Model Versioning: Unlike application code, ML models are large binary artifacts that change independently of source code. A robust pipeline tracks model versions alongside code versions, storing artifacts in a model registry (e.g., MLflow, Weights & Biases). Each deployment bundles a specific model checkpoint with the application code that serves it.

Data Drift Detection: Models degrade silently when production data diverges from training data. Modern pipelines include automated drift detection as a post-deploy verification step — monitoring feature distributions, prediction confidence scores, and output distributions. When drift exceeds a threshold, the pipeline can trigger retraining or alert the team.

A/B Testing & Shadow Deploys: Rolling out a new model to 100% of traffic is risky. AI-native pipelines support canary releases where a new model serves a small percentage of requests while metrics are compared against the baseline. Shadow mode runs the new model in parallel without serving its results, collecting performance data before any user impact.

A production GitHub Actions workflow for deploying a Django application. This YAML defines the full pipeline from checkout to production verification:

name: Deploy to Production

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Lint
        run: |
          flake8 . --count --select=E9,F63,F7,F82 --show-source
          flake8 . --count --max-line-length=120 --statistics

      - name: Run tests
        run: |
          python manage.py test --verbosity=2
        env:
          DATABASE_URL: sqlite:///db.sqlite3
          SECRET_KEY: $

      - name: Build Docker image
        run: |
          docker build -t app:$ .
          docker tag app:$ registry.example.com/app:latest
          docker push registry.example.com/app:latest

      - name: Deploy to production
        run: |
          ssh deploy@prod-1.example.com \
            "docker pull registry.example.com/app:latest && \
             docker service update --image app:latest web"

      - name: Verify deployment
        run: |
          sleep 10
          curl -sf https://example.com/health || exit 1
          echo "Health check passed"

Key details: the workflow uses pip caching for faster installs, runs lint before tests to fail fast, and includes a post-deploy health check that will mark the workflow as failed if the site is unreachable.

When a deployment goes wrong, speed of recovery matters more than root cause analysis. Here are three proven rollback strategies:

Blue-Green Deployment: Two identical production environments ("blue" and "green") run simultaneously. Only one serves live traffic at a time. A new release deploys to the idle environment. After health checks pass, the load balancer switches traffic. If anything goes wrong, flip traffic back instantly — the previous version is still running. Downside: requires double the infrastructure.

Canary Releases: Route a small percentage (1-5%) of traffic to the new version while the rest continues on the old one. Monitor error rates, latency, and business metrics. Gradually increase traffic if metrics hold steady — 5%, 25%, 50%, 100%. If any metric degrades, route all traffic back to the old version. This approach catches issues that only appear under real user load.

Feature Flags: Deploy code changes behind runtime toggles. The new code ships to production but is disabled by default. Enable it gradually, per-user, per-region, or per-percentage. If a feature causes issues, flip the flag off without any deployment. Tools like LaunchDarkly, Unleash, or even a simple database-backed toggle provide this capability. Feature flags decouple deployment from release — you can deploy daily but release weekly.