Live verification

How we verify a deployed change actually works in production. This is layer 4 in the test philosophy — the only layer that catches "the code shipped but the deployment didn't pick it up" and "the code is correct but prod data has a state we didn't anticipate."

When to run live verification

After every backend deploy that:

Adds or changes an API endpoint
Adds or changes server-side state transitions (cron, lazy-expire, scheduled jobs)
Touches authentication, RBAC, or rate limits

Skip it for:

Pure frontend changes (CSS, copy)
Test-only changes
Changes guarded by a feature flag that's off in prod

The pattern

We sign a JWT for a real user with the production JWT_SECRET (which we read from the Cloud Run env), then hit the live API with that JWT.

This works because:

The JWT secret in our Cloud Run service is what verifies the JWT in production
We can read it via gcloud run services describe
We can sign a token for any consumer / seller / doctor without their password

This is acceptable for verification by the engineer who owns the system. Do not use this pattern for anything that affects user data without explicit user permission.

Setup

# Read the prod JWT secret from Cloud Run
SECRET=$(gcloud run services describe ka26-marketplace --region us-central1 \
  --format=json | python3 -c "
import json, sys
d = json.load(sys.stdin)
env = {e['name']: e.get('value', '') for e in d['spec']['template']['spec']['containers'][0].get('env', [])}
print(env.get('JWT_SECRET', ''))
")

# Sign a JWT for any user type
BUYER_TOKEN=$(node -e "
  const jwt = require('/Users/.../marketplace/node_modules/jsonwebtoken');
  console.log(jwt.sign({consumerId: 6, type: 'consumer'}, '$SECRET', {expiresIn: '1h'}));
")
SELLER_TOKEN=$(node -e "
  const jwt = require('/Users/.../marketplace/node_modules/jsonwebtoken');
  console.log(jwt.sign({sellerId: 13, type: 'seller'}, '$SECRET', {expiresIn: '1h'}));
")

Auth header conventions per app

App	Auth header
Consumer	`Authorization: Bearer $BUYER_TOKEN`
Seller	`Cookie: seller_token=$SELLER_TOKEN`
Doctor	`Authorization: Bearer $DOCTOR_TOKEN`

Common queries

# Read seller's bid inbox
curl -s -H "Cookie: seller_token=$SELLER_TOKEN" \
  https://ka26.shop/api/seller/offers | jq

# Read seller's bids with optional sections
curl -s -H "Cookie: seller_token=$SELLER_TOKEN" \
  "https://ka26.shop/api/seller/offers?include=expired,completed" | jq

# Place a buyer bid (POST)
curl -s -H "Authorization: Bearer $BUYER_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST https://ka26.shop/api/products/126/offers \
  -d '{"offeredPrice": 85}' | jq

# Counter as seller (PATCH)
curl -s -H "Cookie: seller_token=$SELLER_TOKEN" \
  -H "Content-Type: application/json" \
  -X PATCH https://ka26.shop/api/offers/35 \
  -d '{"action":"counter","price":95,"expectedVersion":0}' | jq

# Delete an expired bid (DELETE)
curl -s -X DELETE -H "Cookie: seller_token=$SELLER_TOKEN" \
  https://ka26.shop/api/seller/offers/42

# Check live DB state
PGPASSWORD='Ka26Mkt2026' psql -h 34.123.40.64 -U ka26user -d ka26 -t -c "
  SELECT id, status, \"awaitingActionFrom\" FROM \"PriceOffer\" WHERE id=42;
"

Forcing edge-case state in the DB

When you need to test "what happens if a bid is stale," directly INSERT/UPDATE the right row, then hit the API:

# Force-create a stale ACCEPTED bid (lockedUntil 1h ago, no order)
PGPASSWORD='Ka26Mkt2026' psql -h 34.123.40.64 -U ka26user -d ka26 -c "
  INSERT INTO \"PriceOffer\" (
    \"productId\",\"storeId\",\"buyerId\",\"sellerId\",
    \"askingPriceSnapshot\",\"offeredPrice\",\"counterPrice\",\"lockedPrice\",
    status,\"attemptNumber\",\"expiresAt\",\"lockedUntil\",\"acceptedAt\",
    \"awaitingActionFrom\",\"createdAt\",\"updatedAt\"
  )
  VALUES (
    126,8,6,13,
    100,85,95,95,
    'accepted',1,
    NOW() + INTERVAL '1 day',NOW() - INTERVAL '1 hour',NOW() - INTERVAL '2 hour',
    'none',NOW() - INTERVAL '2 hour',NOW() - INTERVAL '2 hour'
  ) RETURNING id;
"

# Now hit the GET endpoint and check whether lazy-expire flipped it
curl -s -H "Cookie: seller_token=$SELLER_TOKEN" \
  https://ka26.shop/api/seller/offers | jq '.capacity.active, [.offers[].id]'

# Verify in DB
PGPASSWORD='Ka26Mkt2026' psql -h 34.123.40.64 -U ka26user -d ka26 -t -c "
  SELECT id, status FROM \"PriceOffer\" WHERE id=42;
"

Multi-step e2e (place → counter → accept)

This is what we run after every bidding-related deploy.

# Step 1: clean slate
PGPASSWORD='...' psql ... -c "
  UPDATE \"PriceOffer\" SET status='withdrawn'
  WHERE \"buyerId\"=6 AND \"productId\"=126 AND status IN ('pending','countered','accepted');
"

# Step 2: buyer places bid
RESP=$(curl -s -H "Authorization: Bearer $BUYER_TOKEN" -H "Content-Type: application/json" \
  -X POST https://ka26.shop/api/products/126/offers -d '{"offeredPrice":85}')
OID=$(echo "$RESP" | jq -r '.offer.id')

# Step 3: seller counters
VER=$(curl -s -H "Cookie: seller_token=$SELLER_TOKEN" https://ka26.shop/api/seller/offers \
  | jq ".offers[] | select(.id == $OID) | .optimisticVersion")
curl -s -H "Cookie: seller_token=$SELLER_TOKEN" -H "Content-Type: application/json" \
  -X PATCH https://ka26.shop/api/offers/$OID \
  -d "{\"action\":\"counter\",\"price\":95,\"expectedVersion\":$VER}"

# Step 4: buyer accepts
VER=$(curl -s -H "Authorization: Bearer $BUYER_TOKEN" https://ka26.shop/api/offers \
  | jq ".offers[] | select(.id == $OID) | .optimisticVersion")
curl -s -H "Authorization: Bearer $BUYER_TOKEN" -H "Content-Type: application/json" \
  -X PATCH https://ka26.shop/api/offers/$OID \
  -d "{\"action\":\"accept\",\"expectedVersion\":$VER}"

# Step 5: verify analytics fired (was the BUG-006e fix)
PGPASSWORD='...' psql ... -c "
  SELECT \"eventType\", \"createdAt\" FROM \"UserEvent\"
  WHERE \"entityId\"=$OID AND \"eventType\" LIKE 'bid%'
  ORDER BY \"createdAt\";
"
# Expected output: bid_placed, bid_countered, bid_accepted_buyer

Things to verify on every deploy

curl -sI https://ka26.shop/api/health returns 200
The Cloud Run revision name has changed: gcloud run services describe ka26-marketplace --region us-central1 --format="value(status.latestReadyRevisionName)"
The Cloud Build deploy succeeded: gh run list --workflow=deploy.yml --limit 1

What to do when live verification fails

Don't roll back yet. Cloud Run keeps the previous revision warm; new requests will fall back if the new revision throws on startup.
Check the deploy logs: gh run view <run-id> --log-failed
Check Cloud Run logs: gcloud run services logs read ka26-marketplace --region us-central1 --limit 100
If the bug is in your code → fix forward. Push a new commit; CI re-deploys.
If the bug is in deployed code from a teammate → revert via git revert, push, CI re-deploys the previous behavior.
Document in BUGS-FIXED.md with the bug ID + fix pattern so future-you doesn't re-introduce it.

Deploy "failure" but actual revision rolls correctly

Common with our setup. The post-deploy health-check probe sometimes fires before the Cloud Run revision is fully warm, so GitHub Actions marks the deploy as "failed" even though the actual code is live and serving traffic. To confirm:

gh run list --workflow=deploy.yml --limit 1 — note the status
gcloud run services describe ka26-marketplace --region us-central1 --format="value(status.latestReadyRevisionName)" — note the revision name; if it changed since your push, the new code IS live
curl -sI https://ka26.shop/api/health — confirms healthy
Live verify the actual change

If all 3 are green, the "failure" is the cosmetic post-deploy probe race; ignore it.

When to run live verification​

The pattern​

Setup​

Auth header conventions per app​

Common queries​

Forcing edge-case state in the DB​

Multi-step e2e (place → counter → accept)​

Things to verify on every deploy​

What to do when live verification fails​

Deploy "failure" but actual revision rolls correctly​