Skip to content

Commit 08cd698

Browse files
lpcoxCopilotCopilotCopilot
authored
feat: validate config files against published JSON Schema at runtime (#2384)
* feat: validate config files against published JSON Schema at runtime Replace the hand-written validateAwfFileConfig() implementation with ajv-based validation against the published awf-config.schema.json. This makes the JSON Schema the single source of truth for: 1. Runtime validation inside awf (this change) 2. External consumers like the gh-aw compiler (schema is downloadable as a release asset and from docs/awf-config.schema.json) 3. IDE autocomplete via $schema field Changes: - Move ajv from devDependencies to dependencies (runtime use) - Add src/schema-validator.ts: compiles schema once, formats ajv errors into human-readable messages matching the previous validator style - Add src/awf-config-schema.json: bundleable copy of the schema (imported at compile time, works with esbuild single-file bundle) - Update generate-schema.mjs to write both docs/ and src/ copies - Add sync test ensuring src/ and docs/ schemas stay aligned - Update one test assertion for new enum formatting The error messages are backward-compatible with the previous hand-written validator (all 113 config/schema tests pass unchanged except one enum format improvement: '"5m" or "1h"' → 'one of: 5m, 1h'). Closes #2374 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * feat: version the awf-config JSON Schema as v1 Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/b9f23f04-41af-4ebf-ba7f-52aa520cad1d * feat: version the awf-config JSON Schema as v1 Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/b9f23f04-41af-4ebf-ba7f-52aa520cad1d * docs: document JSON Schema versioning and release tag pinning in releasing.md Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/72cdaa37-37d8-4564-937b-e5f2ee8eafc1 Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com> * test: add schema-validator tests and simplify unreachable branches - Add 18 unit tests covering all error formatting paths - Remove dead code: hasMinimumOne() and determineArticle() functions (all integer fields have minimum:1; no vowel-starting types reach the article logic after prior case handling) - Simplify isArrayOfStringsField() to a single boolean expression - Branch coverage for schema-validator.ts: 97.77% Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(docker): make apt mirror selection resilient to DNS failures - Conditionally switch to azure.archive.ubuntu.com only if DNS resolves it - Falls back to default archive.ubuntu.com when BuildKit DNS can't reach the Azure mirror (common in Docker-in-Docker scenarios) - Improve apt_update_retry() to detect silent failures (apt returns 0 with 'Failed to fetch' warnings) and fall back to archive.ubuntu.com - Fixes persistent CI failures where BuildKit couldn't resolve azure.archive.ubuntu.com during container image builds Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
1 parent 0413d68 commit 08cd698

15 files changed

Lines changed: 1154 additions & 283 deletions

‎.github/workflows/release.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -523,9 +523,10 @@ jobs:
523523
- name: Generate versioned JSON Schema
524524
run: |
525525
mkdir -p release
526-
node scripts/generate-schema.mjs --version ${{ needs.bump-version.outputs.version }} --print > release/awf-config.schema.json
526+
node scripts/generate-schema.mjs --version ${{ needs.bump-version.outputs.version }} --print > release/awf-config.v1.schema.json
527+
cp release/awf-config.v1.schema.json release/awf-config.schema.json
527528
echo "=== Schema preview (first 10 lines) ==="
528-
head -10 release/awf-config.schema.json
529+
head -10 release/awf-config.v1.schema.json
529530
530531
- name: Generate checksums
531532
run: |
@@ -653,6 +654,7 @@ jobs:
653654
release/awf.tgz
654655
release/containers.txt
655656
release/awf-config.schema.json
657+
release/awf-config.v1.schema.json
656658
release/checksums.txt
657659
env:
658660
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎containers/agent/Dockerfile‎

Lines changed: 58 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -9,29 +9,45 @@ ARG BASE_IMAGE=ubuntu:22.04
99

1010
FROM ${BASE_IMAGE}
1111

12-
# Switch to Azure apt mirror for faster, more reliable package fetches in CI
13-
# GitHub Actions runners are Azure-hosted; azure.archive.ubuntu.com is geographically closer
14-
# Handles both traditional sources.list (jammy) and DEB822 format (noble+)
15-
RUN if [ -f /etc/apt/sources.list ]; then \
16-
sed -i 's|http://archive.ubuntu.com|http://azure.archive.ubuntu.com|g' /etc/apt/sources.list; \
17-
sed -i 's|http://security.ubuntu.com|http://azure.archive.ubuntu.com|g' /etc/apt/sources.list; \
18-
fi && \
19-
if [ -d /etc/apt/sources.list.d ]; then \
20-
find /etc/apt/sources.list.d -name '*.sources' -exec \
21-
sed -i 's|http://archive.ubuntu.com|http://azure.archive.ubuntu.com|g' {} + 2>/dev/null || true; \
22-
find /etc/apt/sources.list.d -name '*.sources' -exec \
23-
sed -i 's|http://security.ubuntu.com|http://azure.archive.ubuntu.com|g' {} + 2>/dev/null || true; \
12+
# Optionally switch to Azure apt mirror for faster package fetches in CI
13+
# Only rewrite if azure.archive.ubuntu.com is resolvable (BuildKit DNS can fail)
14+
# Falls back to default archive.ubuntu.com which is universally reachable
15+
RUN if getent hosts azure.archive.ubuntu.com >/dev/null 2>&1; then \
16+
echo "Using Azure apt mirror (DNS resolved successfully)"; \
17+
if [ -f /etc/apt/sources.list ]; then \
18+
sed -i 's|http://archive.ubuntu.com|http://azure.archive.ubuntu.com|g' /etc/apt/sources.list; \
19+
sed -i 's|http://security.ubuntu.com|http://azure.archive.ubuntu.com|g' /etc/apt/sources.list; \
20+
fi; \
21+
if [ -d /etc/apt/sources.list.d ]; then \
22+
find /etc/apt/sources.list.d -name '*.sources' -exec \
23+
sed -i 's|http://archive.ubuntu.com|http://azure.archive.ubuntu.com|g' {} + 2>/dev/null || true; \
24+
find /etc/apt/sources.list.d -name '*.sources' -exec \
25+
sed -i 's|http://security.ubuntu.com|http://azure.archive.ubuntu.com|g' {} + 2>/dev/null || true; \
26+
fi; \
27+
else \
28+
echo "Azure apt mirror not reachable, using default archive.ubuntu.com"; \
2429
fi
2530

2631
# Install required packages and Node.js 22
2732
# Note: Some packages may already exist in runner-like base images, apt handles this gracefully
28-
# apt_update_retry: retries up to 3 times with backoff to survive prolonged mirror syncs
33+
# apt_update_retry: retries up to 3 times with backoff; if all fail, reverts to archive.ubuntu.com
2934
RUN set -eux; \
3035
apt_update_retry() { \
3136
local i; for i in 1 2 3; do \
32-
rm -rf /var/lib/apt/lists/* && apt-get update && return 0; \
33-
echo "apt-get update attempt $i/3 failed, retrying in $((i*10))s..." >&2; sleep $((i*10)); \
34-
done; return 1; \
37+
rm -rf /var/lib/apt/lists/* && apt-get update 2>&1 | tee /tmp/apt-update.log && \
38+
if ! grep -q "Failed to fetch" /tmp/apt-update.log; then return 0; fi; \
39+
echo "apt-get update attempt $i/3 had fetch failures, retrying in $((i*10))s..." >&2; sleep $((i*10)); \
40+
done; \
41+
echo "All apt-get update retries failed, falling back to archive.ubuntu.com..." >&2; \
42+
if [ -f /etc/apt/sources.list ]; then \
43+
sed -i 's|http://azure.archive.ubuntu.com|http://archive.ubuntu.com|g' /etc/apt/sources.list; \
44+
sed -i 's|http://security.ubuntu.com|http://archive.ubuntu.com|g' /etc/apt/sources.list 2>/dev/null || true; \
45+
fi; \
46+
if [ -d /etc/apt/sources.list.d ]; then \
47+
find /etc/apt/sources.list.d -name '*.sources' -exec \
48+
sed -i 's|http://azure.archive.ubuntu.com|http://archive.ubuntu.com|g' {} + 2>/dev/null || true; \
49+
fi; \
50+
rm -rf /var/lib/apt/lists/* && apt-get update; \
3551
}; \
3652
PKGS="iptables curl ca-certificates git gh gnupg dnsutils net-tools netcat-openbsd gosu libcap2-bin"; \
3753
apt_update_retry && \
@@ -60,9 +76,19 @@ RUN set -eux; \
6076
RUN set -eux; \
6177
apt_update_retry() { \
6278
local i; for i in 1 2 3; do \
63-
rm -rf /var/lib/apt/lists/* && apt-get update && return 0; \
64-
echo "apt-get update attempt $i/3 failed, retrying in $((i*10))s..." >&2; sleep $((i*10)); \
65-
done; return 1; \
79+
rm -rf /var/lib/apt/lists/* && apt-get update 2>&1 | tee /tmp/apt-update.log && \
80+
if ! grep -q "Failed to fetch" /tmp/apt-update.log; then return 0; fi; \
81+
echo "apt-get update attempt $i/3 had fetch failures, retrying in $((i*10))s..." >&2; sleep $((i*10)); \
82+
done; \
83+
echo "All apt-get update retries failed, falling back to archive.ubuntu.com..." >&2; \
84+
if [ -f /etc/apt/sources.list ]; then \
85+
sed -i 's|http://azure.archive.ubuntu.com|http://archive.ubuntu.com|g' /etc/apt/sources.list; \
86+
fi; \
87+
if [ -d /etc/apt/sources.list.d ]; then \
88+
find /etc/apt/sources.list.d -name '*.sources' -exec \
89+
sed -i 's|http://azure.archive.ubuntu.com|http://archive.ubuntu.com|g' {} + 2>/dev/null || true; \
90+
fi; \
91+
rm -rf /var/lib/apt/lists/* && apt-get update; \
6692
}; \
6793
PARITY_PKGS="libgdiplus libev-dev libssl-dev php-intl php-gd"; \
6894
apt_update_retry && \
@@ -77,9 +103,19 @@ RUN set -eux; \
77103
# Retry logic handles transient mirror sync failures during apt-get update
78104
RUN apt_update_retry() { \
79105
local i; for i in 1 2 3; do \
80-
rm -rf /var/lib/apt/lists/* && apt-get update && return 0; \
81-
echo "apt-get update attempt $i/3 failed, retrying in $((i*10))s..." >&2; sleep $((i*10)); \
82-
done; return 1; \
106+
rm -rf /var/lib/apt/lists/* && apt-get update 2>&1 | tee /tmp/apt-update.log && \
107+
if ! grep -q "Failed to fetch" /tmp/apt-update.log; then return 0; fi; \
108+
echo "apt-get update attempt $i/3 had fetch failures, retrying in $((i*10))s..." >&2; sleep $((i*10)); \
109+
done; \
110+
echo "All apt-get update retries failed, falling back to archive.ubuntu.com..." >&2; \
111+
if [ -f /etc/apt/sources.list ]; then \
112+
sed -i 's|http://azure.archive.ubuntu.com|http://archive.ubuntu.com|g' /etc/apt/sources.list; \
113+
fi; \
114+
if [ -d /etc/apt/sources.list.d ]; then \
115+
find /etc/apt/sources.list.d -name '*.sources' -exec \
116+
sed -i 's|http://azure.archive.ubuntu.com|http://archive.ubuntu.com|g' {} + 2>/dev/null || true; \
117+
fi; \
118+
rm -rf /var/lib/apt/lists/* && apt-get update; \
83119
}; \
84120
apt_update_retry && \
85121
apt-get upgrade -y && rm -rf /var/lib/apt/lists/* || \

‎containers/squid/Dockerfile‎

Lines changed: 31 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,42 @@
11
FROM ubuntu/squid:latest
22

3-
# Switch to Azure apt mirror for faster, more reliable package fetches in CI
4-
# GitHub Actions runners are Azure-hosted; azure.archive.ubuntu.com is geographically closer
5-
# Handles both traditional sources.list (jammy) and DEB822 format (noble+)
6-
RUN if [ -f /etc/apt/sources.list ]; then \
7-
sed -i 's|http://archive.ubuntu.com|http://azure.archive.ubuntu.com|g' /etc/apt/sources.list; \
8-
sed -i 's|http://security.ubuntu.com|http://azure.archive.ubuntu.com|g' /etc/apt/sources.list; \
9-
fi && \
10-
if [ -d /etc/apt/sources.list.d ]; then \
11-
find /etc/apt/sources.list.d -name '*.sources' -exec \
12-
sed -i 's|http://archive.ubuntu.com|http://azure.archive.ubuntu.com|g' {} + 2>/dev/null || true; \
13-
find /etc/apt/sources.list.d -name '*.sources' -exec \
14-
sed -i 's|http://security.ubuntu.com|http://azure.archive.ubuntu.com|g' {} + 2>/dev/null || true; \
3+
# Optionally switch to Azure apt mirror for faster package fetches in CI
4+
# Only rewrite if azure.archive.ubuntu.com is resolvable (BuildKit DNS can fail)
5+
# Falls back to default archive.ubuntu.com which is universally reachable
6+
RUN if getent hosts azure.archive.ubuntu.com >/dev/null 2>&1; then \
7+
echo "Using Azure apt mirror (DNS resolved successfully)"; \
8+
if [ -f /etc/apt/sources.list ]; then \
9+
sed -i 's|http://archive.ubuntu.com|http://azure.archive.ubuntu.com|g' /etc/apt/sources.list; \
10+
sed -i 's|http://security.ubuntu.com|http://azure.archive.ubuntu.com|g' /etc/apt/sources.list; \
11+
fi; \
12+
if [ -d /etc/apt/sources.list.d ]; then \
13+
find /etc/apt/sources.list.d -name '*.sources' -exec \
14+
sed -i 's|http://archive.ubuntu.com|http://azure.archive.ubuntu.com|g' {} + 2>/dev/null || true; \
15+
find /etc/apt/sources.list.d -name '*.sources' -exec \
16+
sed -i 's|http://security.ubuntu.com|http://azure.archive.ubuntu.com|g' {} + 2>/dev/null || true; \
17+
fi; \
18+
else \
19+
echo "Azure apt mirror not reachable, using default archive.ubuntu.com"; \
1520
fi
1621

1722
# Install additional tools for debugging, healthcheck, and SSL Bump
18-
# apt_update_retry: retries up to 3 times with backoff to survive prolonged mirror syncs
23+
# apt_update_retry: retries up to 3 times with backoff; if all fail, reverts to archive.ubuntu.com
1924
RUN set -eux; \
2025
apt_update_retry() { \
2126
local i; for i in 1 2 3; do \
22-
rm -rf /var/lib/apt/lists/* && apt-get update && return 0; \
23-
echo "apt-get update attempt $i/3 failed, retrying in $((i*10))s..." >&2; sleep $((i*10)); \
24-
done; return 1; \
27+
rm -rf /var/lib/apt/lists/* && apt-get update 2>&1 | tee /tmp/apt-update.log && \
28+
if ! grep -q "Failed to fetch" /tmp/apt-update.log; then return 0; fi; \
29+
echo "apt-get update attempt $i/3 had fetch failures, retrying in $((i*10))s..." >&2; sleep $((i*10)); \
30+
done; \
31+
echo "All apt-get update retries failed, falling back to archive.ubuntu.com..." >&2; \
32+
if [ -f /etc/apt/sources.list ]; then \
33+
sed -i 's|http://azure.archive.ubuntu.com|http://archive.ubuntu.com|g' /etc/apt/sources.list; \
34+
fi; \
35+
if [ -d /etc/apt/sources.list.d ]; then \
36+
find /etc/apt/sources.list.d -name '*.sources' -exec \
37+
sed -i 's|http://azure.archive.ubuntu.com|http://archive.ubuntu.com|g' {} + 2>/dev/null || true; \
38+
fi; \
39+
rm -rf /var/lib/apt/lists/* && apt-get update; \
2540
}; \
2641
PKGS="curl dnsutils net-tools netcat-openbsd openssl squid-openssl"; \
2742
apt_update_retry && \

‎docs/awf-config.schema.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
"$schema": "https://json-schema.org/draft/2020-12/schema",
33
"$id": "https://github.com/__raw/github/gh-aw-firewall/main/docs/awf-config.schema.json",
44
"title": "AWF Configuration",
5+
"version": "1",
56
"description": "JSON/YAML configuration for awf CLI. CLI flags override config file values. See https://github.com/github/gh-aw-firewall for documentation.",
67
"type": "object",
78
"additionalProperties": false,

0 commit comments

Comments
 (0)