Problem Statement

Two smoke test engines are completely broken as of 2026-05-01T05:18 UTC. Both produce 0 agent turns and fail immediately in the agent job. Every PR that triggers these workflows will see red.

Affected Workflows and Run IDs

Root Causes

Gemini (Smoke Gemini — run 25203411975)

The GOOGLE_GEMINI_API_KEY secret (or equivalent) is invalid or expired. The Gemini CLI successfully reaches generativelanguage.googleapis.com but receives HTTP 400 INVALID_ARGUMENT:

ApiError: {"error":{"code":400,"message":"API key not valid. Please pass a valid API key.",
  "status":"INVALID_ARGUMENT","details":[{"reason":"API_KEY_INVALID",
  "domain":"googleapis.com","metadata":{"service":"generativelanguage.googleapis.com"}}]}}

Crush (Smoke Crush — run 25203411962)

The @charmland/crush npm package downloads crush_0.59.0_Linux_x86_64.tar.gz successfully but then tries to write the extracted binary to /opt/hostedtoolcache/node/24.14.1/x64/lib/node_modules/@charmland/crush/bin, which is a read-only filesystem:

Error: Installation failed: EROFS: read-only file system, mkdir
  '/opt/hostedtoolcache/node/24.14.1/x64/lib/node_modules/`@charmland/crush`/bin'
    at install (.../node_modules/`@charmland/crush`/lib.js:162:9)

Proposed Remediation

Gemini: Rotate or renew the GOOGLE_GEMINI_API_KEY repository secret. If the key is managed externally, verify it hasn't expired or been revoked. Re-run Smoke Gemini to confirm.

Crush: Change the Crush installation to extract the binary into a writable location. Options:

• Use the RUNNER_TOOL_CACHE environment variable (already writable) instead of the global node_modules
• Pre-install the binary in a workflow pre_activation step to /tmp/gh-aw/ or $RUNNER_TEMP
• Pin to a crush binary path that doesn't go through @charmland/crush's install() routine

Success Criteria

• Smoke Gemini completes at least 1 agent turn and produces a noop/safe output
• Smoke Crush completes at least 1 agent turn and produces a noop/safe output
• Both verify against the next PR that runs them

Parent Issue

Tracked in: #29458

Generated by [aw] Failure Investigator (6h) · ● 263.7K · ◷

• expires on May 8, 2026, 7:43 AM UTC