Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 105 additions & 0 deletions apps/starry/gateway/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# gateway -- angie full-directive gateway carpet (StarryOS, 4 arch)

`gateway` exercises [angie](https://angie.software) 1.11.5 -- an nginx-compatible HTTP server /
reverse-proxy fork -- as a carpet-grade API gateway on StarryOS across x86_64, aarch64, riscv64
and loongarch64, covering the full HTTP / TLS / cache / gzip / stream directive surface. It is the
`angie` half of the `gateway` sub-task. (The `higress` half is a multi-component Istio/Envoy
control-plane stack, not a headless single binary, and is tracked separately.)

## Topology (self-contained, one process, `wget` the only client)

A single foreground angie (`daemon off; master_process off`) hosts BOTH the gateway front-end
(127.0.0.1:18080) AND every back-end it proxies to -- plaintext HTTP back-ends, TLS vhosts, and a
`stream{}` four-layer tier. angie therefore supplies both ends of every TLS / stream /
proxy_protocol hop, so busybox `wget` (plaintext, to :18080) is the ONLY client needed and no
external tool (openssl CLI, grpcurl, nc, socat) has to be present on target. Self-signed TLS
fixtures (a trusted CA + a rogue CA + `backend.local` / `foo.test` / `bar.test` / client certs)
are generated by `prebuild.sh` into `/etc/angie/certs`.

## Carpets (`programs/run-gateway.sh`, gate = `GATEWAY_OK=109/109` + `TEST PASSED`)

Every carpet is a real assertion against an EXPECTED constant. Groups:

* **CLI surface** -- `angie -v` version red line (`Angie/1.11.5`), `angie -t` accepts the full config
AND rejects a config with an unknown directive (emerg + non-zero), `angie -h` usage lists
`-c filename`, `angie -V` reports `TLS SNI support enabled`, and `angie -m` lists every module the
carpet depends on by name (http_ssl / http_v2 / realip / sub_filter / gzip_static / gunzip /
stub_status / grpc / upstream_least_conn / upstream_ip_hash / upstream_keepalive /
stream_ssl_preread / stream_realip) -- this last group is the guard that keeps the riscv64 /
loongarch64 source build's module set in lock-step with the x86_64 / aarch64 apk.
* **location matching** -- exact / longest-prefix / regex / case-insensitive regex / `^~`
prefix-stop / named-fallback priority.
* **proxy request rewriting** (reflected by a back-end that echoes what it received) --
`proxy_pass` URI passthrough vs prefix-replace, `proxy_set_header` (Host / X-Custom /
`$proxy_add_x_forwarded_for` / X-Real-IP), `proxy_set_body`, `proxy_method`,
`proxy_http_version` 1.0 vs 1.1, `proxy_pass_request_body off`.
* **proxy response rewriting** -- `proxy_hide_header`, `proxy_pass_header` (X-Pad, a default-hidden
header), `proxy_redirect` (rewrite / off), `proxy_cookie_domain`, `proxy_cookie_path` (rewritten
against a real upstream `Set-Cookie` on a 200, so busybox `wget` sees the whole header block).
* **upstream / load balancing** -- round-robin, weighted RR (3:1 distribution), `least_conn`,
`ip_hash` (sticky per client IP), keyed consistent `hash`, `random`, `keepalive` pool,
`max_fails` ejection to a `backup`, `slow_start`, and `proxy_next_upstream` failover
(dead->alive and http_502->200) vs `off`.
* **timeouts / errors** -- `proxy_connect_timeout` to a blackhole (192.0.2.1), dead upstream,
`proxy_intercept_errors` on (error_page) vs off.
* **status / error paths** -- `return 403`, `return 302`, unmatched location -> 404, `error_page`
-> named location recode to 200, `internal` location unreachable from the client (404),
`return 204`, multiple `add_header` on one response.
* **proxy cache** -- `$upstream_cache_status` MISS then HIT, `proxy_ignore_headers Cache-Control`.
* **gzip / gunzip** -- gzip 1f8b magic + size shrink (via an internal HTTP/1.1 hop that sets
`Accept-Encoding`), `Content-Encoding: gzip` + `Vary` response headers, `gunzip` decompression
round-trip.
* **proxy_ssl (TLS to back-ends)** -- `proxy_ssl_verify on` success, verify failure against a
rogue-CA cert (502), no-verify, mTLS (`proxy_ssl_certificate` -> `$ssl_client_verify=SUCCESS`),
`proxy_ssl_protocols` TLSv1.3 success and TLSv1.2-vs-TLSv1.3 mismatch (502).
* **limit_req / limit_conn** -- concurrent bursts trigger 429 (rate limit + connection limit).
* **rewrite / map / split_clients / return** -- `rewrite ... last`, `return 301`, `map`
default/exact/regex-capture, `split_clients` bucket, `set`, `if`.
* **realip** -- `set_real_ip_from` + `real_ip_header X-Forwarded-For` rewrites `$remote_addr`.
* **sub_filter** (single + chained with `add_header` on one proxied hop), **stub_status**.
* **stream (four-layer)** -- TCP `proxy_pass`, `proxy_protocol` (self-proving chain that echoes
`$proxy_protocol_addr`), `ssl_preread` SNI routing (foo.test / bar.test / default resolved
from the peeked ClientHello and routed to the matching TLS vhost), and a UDP datagram tier
(`listen ... udp` / stream `return` / `proxy_responses`) bound live at startup.
* **multi-worker setuid** -- `master_process on; worker_processes 2; user nobody` forks two workers
that drop to uid 65534 and keep serving.

Scope notes: `headers-more` is a third-party module angie does not ship, so header manipulation is
covered by the built-in `add_header` / `proxy_set_header` / `proxy_hide_header` /
`proxy_pass_header` / `proxy_ignore_headers` instead. Two data paths are config-wired and validated
(`angie -t` on all four arches + bound live at startup) but not asserted end-to-end because the
required client is not on target: the gRPC upstream (`grpc_pass` / `grpc_set_header`) needs an
HTTP/2 gRPC client (grpcurl), and the UDP stream tier needs a UDP client (the on-target busybox
`nc` is the minimal TCP-only variant). Both are exercised host-side. NTLS / QUIC(HTTP3) are out of
scope (need TongSuo / a QUIC back-end).

## Provenance (`prebuild.sh`)

* **x86_64 / aarch64** -- angie's official Alpine (musl) apk, pinned `1.11.5-r0`, sha256-verified,
extracted with its full dependency closure (pcre2 / zlib / libssl3 / libcrypto3 / openssl). The
apk is the FULL upstream module set (`angie -V` shows http_ssl / http_v2 / http_v3 / gzip /
gzip_static / gunzip / grpc(default) / stream / stream_ssl / stream_ssl_preread / realip / sub /
... + proxy_cache / limit_req / map).
* **riscv64 / loongarch64** -- angie's official repo ships no apk for these (APKINDEX 404), so
angie 1.11.5 is cross-compiled from the official source tarball (`build-angie-full.sh` in the
gateway-bins asset store) **WITH the full gateway module set** -- `--with-http_ssl_module
--with-http_v2_module --with-http_gzip_static_module --with-http_gunzip_module
--with-http_realip_module --with-http_sub_module --with-http_stub_status_module
--with-stream --with-stream_ssl_module --with-stream_ssl_preread_module
--with-stream_realip_module` plus the default proxy / cache / grpc / gzip / map / limit modules --
linked against openssl 3.5.x + zlib + pcre2 taken from the SAME Alpine v3.23/main branch as the
base rootfs (byte-for-byte ABI match). Delivered as a `payload-full` tree (angie + libssl /
libcrypto / libz / libpcre2 .so closure). The `angie -m` module carpet asserts this set matches
the apk arches, so a missing module fails the gate rather than passing silently. Never a SKIP.

Binaries are never committed to the source tree: the apk arches fetch/verify pinned apks, the
source arches consume the reproducible cross-built `payload-full` artifact.

## Run

```
source tgoskits/.starry-env.sh # qemu 10, single core
cargo xtask starry app qemu -t gateway --arch x86_64 # aarch64 | riscv64 | loongarch64
```

LoongArch uses the dynamic platform (`axplat-dyn`, `to_bin=true`); see the build/qemu tomls.
7 changes: 7 additions & 0 deletions apps/starry/gateway/build-aarch64-unknown-none-softfloat.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
target = "aarch64-unknown-none-softfloat"
log = "Warn"
features = [
"ax-driver/serial",
"ax-driver/virtio-blk",
"ax-driver/virtio-net",
]
13 changes: 13 additions & 0 deletions apps/starry/gateway/build-loongarch64-unknown-none-softfloat.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# LoongArch64 MUST use the dynamic platform (axplat-dyn): keep ax-driver/serial + do NOT opt out
# of dynamic platform mode. The retired static LoongArch path lacked the serial-console binding
# ("/dev/console has no serial TTY binding" -> early exit). qemu-loongarch64.toml boots the raw
# binary (uefi=false / to_bin=true).
target = "loongarch64-unknown-none-softfloat"
env = { AX_IP = "10.0.2.15", AX_GW = "10.0.2.2" }
log = "Warn"
features = [
"axplat-dyn/efi",
"ax-driver/serial",
"ax-driver/virtio-blk",
"ax-driver/virtio-net",
]
7 changes: 7 additions & 0 deletions apps/starry/gateway/build-riscv64gc-unknown-none-elf.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
target = "riscv64gc-unknown-none-elf"
log = "Warn"
features = [
"ax-driver/serial",
"ax-driver/virtio-blk",
"ax-driver/virtio-net",
]
6 changes: 6 additions & 0 deletions apps/starry/gateway/build-x86_64-unknown-none.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
target = "x86_64-unknown-none"
log = "Warn"
features = [
"ax-driver/virtio-blk",
"ax-driver/virtio-net",
]
28 changes: 28 additions & 0 deletions apps/starry/gateway/conf/gateway-workers.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# gateway-workers.conf -- multi-worker angie for the StarryOS `gateway` carpet.
# master_process on + worker_processes 2 makes the master fork two worker processes;
# `user nobody` makes each worker drop privileges via setgid()/setgroups()/setuid() to the
# unprivileged nobody account (uid/gid 65534, present in the Alpine base /etc/passwd). The
# master (root) keeps the listening socket and re-parents the workers. error_log goes to a
# file so the carpet can assert both "start worker process" lines (the fork evidence).
user nobody;
daemon off;
master_process on;
worker_processes 2;
pid /run/angie-w.pid;
error_log /var/log/angie/workers.log info;

events {
worker_connections 128;
}

http {
access_log off;
default_type text/plain;
client_body_temp_path /var/lib/angie/tmp/w_client_body;

server {
listen 127.0.0.1:18090;
server_name localhost;
location / { return 200 "WORKER_OK"; }
}
}
Loading