Building a Two-Layer Logging Pipeline for Next.js on Kubernetes
In the previous post, I described the four-day debugging saga of getting pino to produce any output at all inside a Next.js standalone Docker container. That was Phase 1a — the server-side foundation.
This post covers what came after: extending the logging system from server-only to a full two-layer pipeline where browser-side errors, API failures, and React crashes all flow through the same pino JSON output — sanitized, batched, and protected against abuse.
Three phases, seven PRs, and about 600 lines of production code later, here is the system.
The Architecture
Browser Server (Next.js)
───────────────────────────────────────────────────────────────
clientLogger ──→ log-forwarder ──→ POST /service-api/log ──→ pino
(warn/error) (batch + debounce) (validate, rate-limit, (JSON → stdout → K8s)
(sendBeacon on unload) redact PII, re-emit)The core idea: the browser should never manage logs. It should report them. The server decides what to keep, what to redact, and where to send them.
Everything — server-side request logs, forwarded client errors, axios failures, React error boundary crashes — converges into pino’s JSON output on stdout, which Kubernetes collects automatically.
Phase 1b: Client Log Forwarding (PR #389)
The server logger from Phase 1a gave us structured JSON for API route handlers. But browser-side events — React crashes, auth failures, API timeout errors seen by the user — were invisible to the server. A user could hit a blank screen, close the tab, and we’d have no record of what happened.
The Client Logger
client-logger.ts is a thin facade over console.*:
debug— suppressed entirely in production (never forwarded)info,warn,error— always callconsole.*AND forward to the server
Every forwarded entry includes the page pathname (never query params — those can contain tokens), a client version stamp, and optional structured metadata: errorType, componentName, stack, and a free-form context record.
The Forwarder
log-forwarder.ts sits between the client logger and the server endpoint. Its job is to batch entries and deliver them reliably without creating error loops.
Three flush triggers:
- An
error-level entry arrives — flush immediately (cancel any pending debounce) - Buffer reaches 10 entries — flush immediately
- Otherwise — debounced flush after 5 seconds
Page unload safety: The forwarder registers visibilitychange and beforeunload listeners that call navigator.sendBeacon. Even if the user closes the tab mid-error, the batch gets delivered.
Circuit breaker: If 50+ errors accumulate within a 10-second window, the forwarder suspends itself. This prevents a cascading failure where a broken page generates errors, the forwarder tries to send them, the sends fail, generating more errors, which generate more sends. The circuit breaker cuts the loop.
Axios Interceptors
The existing tokenAxios.tsx already handled auth token injection and 401 redirects. I added logging to both request and response interceptors across all seven axios instances (main BRS, virtual account, messaging, incentive trip, auth, member, promotion):
→ GET /api/member/search (request start)
← GET /api/member/search 200 42ms (response success)For errors, the interceptor logs status, method, URL, error code, and duration — enough to diagnose whether a failure is a gateway timeout, a network error, or a server 500, without needing to reproduce it.
Error Boundaries
Four React error boundaries — global, root, dashboard, and users — each fire clientLogger.error in a useEffect([error]) with structured metadata:
clientLogger.error("Dashboard error boundary caught", {
errorType: error.name,
stack: error.stack,
componentName: "DashboardError",
});This means a React crash on any page produces a structured log entry that reaches the server — even if the error boundary’s fallback UI is all the user sees.
The Server Endpoint
POST /service-api/log receives batched client entries and re-emits them through pino. It’s the bridge between two worlds — untrusted browser input and the server’s structured log pipeline.
The initial implementation had Zod validation (1-20 entries per batch, field length limits, level restricted to info/warn/error), IP-based rate limiting (20 requests/minute per IP), and basic PII redaction on the context field.
It worked. Tests passed. 93% coverage. Ship it.
Phase 1c: Hardening (PR #406)
Then I ran a code review — a structured review using AI agents (code-reviewer + security-reviewer from the BMAD workflow). The review found real problems.
The PII Gaps
The initial redaction only covered the context field. But message and stack are free-text fields up to 2000 and 5000 characters respectively — both client-controlled. A message like "Failed to load user Bearer eyJhbG..." or a stack trace containing interpolated tokens would pass straight through to pino and into the K8s log pipeline.
The fix is a redactText() function that runs before any free-text field is logged:
function redactText(text: string): string {
let result = text.normalize('NFKC');
for (const createPattern of TEXT_REDACTION_PATTERN_FACTORIES) {
result = result.replace(createPattern(), '[REDACTED]');
}
return result;
}Three patterns: bearer tokens (bearer <token>), JWT-like strings (eyJ... with 10+ base64 characters), and Korean resident registration numbers (주민등록번호, format YYMMDD-G###### where G is a gender/century digit 1-4).
Two subtle details worth explaining:
NFKC normalization. Unicode has multiple ways to represent the same character. An attacker could use homoglyphs (visually identical characters from different scripts) or zero-width joiners to bypass regex patterns. text.normalize('NFKC') collapses these representations to their canonical form before the regex runs.
Factory pattern for regex. JavaScript regex objects with the /g flag maintain internal lastIndex state. If a regex is reused across calls, lastIndex carries over from the previous match, causing patterns to silently skip matches on alternating calls. The factory pattern — () => /pattern/gi — creates a fresh regex for each invocation, eliminating the shared state bug.
The Rate Limiter Leak
The in-memory rate limit map could grow to 10,000 entries (fail-closed at capacity to prevent memory exhaustion). But stale entries were only evicted when the same IP returned — meaning 10,000 unique IPs hitting once and never returning would permanently block all new IPs.
The fix: a periodic cleanup via setInterval every 60 seconds that sweeps entries older than the rate limit window. The interval is .unref()’d so Node.js can exit cleanly without waiting for it.
The Interceptor Stacking Bug
TokenAxiosIntercepter’s useEffect depended on [auth, router]. Every time auth or router changed (which happens on navigation), the effect would tear down and re-apply all interceptors. But between teardown and re-application, there’s a window where requests have no auth header. And if teardown is delayed, interceptors stack — each one firing clientLogger.info, multiplying log volume.
The fix: store auth in a useRef, remove all dependencies from the effect ([]), and access the current auth value via authRef.current inside the interceptor callbacks. Interceptors mount once and never re-stack.
Query Params in Axios Logs
The axios interceptors were logging config.url directly — which includes query parameters. A URL like /api/member/search?name=김철수&rrn=900101-1234567 would be logged with full PII in the query string.
Added a stripUrlQueryParams() helper applied to every URL reference in the request, response, and error interceptors.
Nested Object Redaction
The initial redactContext() used String(value) to convert context values for sensitivity checking. But String({token: "secret"}) produces "[object Object]" — which doesn’t match any sensitive pattern. The token passes through unredacted.
The fix: use JSON.stringify(value).slice(0, 500) for objects, which produces '{"token":"secret"}' — now the token pattern matches and the value is redacted.
The Final System
Seven files, roughly 600 lines of production code, 29 tests on the server endpoint alone:
| File | Lines | Purpose |
|---|---|---|
logger.ts | ~50 | Server-side pino instance |
request-logger.ts | ~40 | Per-request child logger factory |
client-logger.ts | ~70 | Browser-side logging facade |
log-forwarder.ts | ~120 | Client-side batching + circuit breaker |
route.ts (log endpoint) | ~310 | Server bridge: validate, rate-limit, redact, re-emit |
tokenAxios.tsx | ~300 | Axios interceptors with structured logging |
| Error boundaries (x4) | ~30 each | React crash reporting |
What Gets Redacted
| Layer | What | How |
|---|---|---|
| Client (log-forwarder) | Page URL query params | window.location.pathname only |
| Client (axios interceptors) | API URL query params | stripUrlQueryParams() helper |
| Server (pino base config) | Authorization headers, cookies, tokens, passwords | Pino’s built-in redact option |
| Server (log endpoint) | Bearer tokens in message/stack | Regex: bearer\s+[^\s,;]+ |
| Server (log endpoint) | JWTs in message/stack | Regex: eyJ[A-Za-z0-9_-]{10,} |
| Server (log endpoint) | Korean RRNs in message/stack | Regex: \d{6}-[1-4]\d{6} |
| Server (log endpoint) | Sensitive context keys | 8 patterns: token, password, secret, authorization, cookie, credential, session, 주민등록번호 |
| Server (log endpoint) | Unicode bypass attempts | NFKC normalization before all regex checks |
What Protects the Endpoint
| Protection | Detail |
|---|---|
| Rate limiting | 20 req/min per IP, sliding window, periodic cleanup |
| Payload size | 50KB max (checked via header AND actual byte count) |
| Batch size | 1-20 entries per request |
| Field limits | message: 2000 chars, stack: 5000 chars, context: 10 keys |
| Schema validation | Zod — level restricted to info/warn/error, datetime format enforced |
| Circuit breaker (client) | 50 errors in 10s → forwarding suspended |
| Fail-closed rate limiter | Map capped at 10K entries — new IPs rejected at capacity |
The PR Timeline
| PR | Title | Phase |
|---|---|---|
| #335 | Pino structured logging system | 1a — Foundation |
| #338 | Security hardening (8 patches) | 1a — Hardening |
| #339 | Add serverExternalPackages | 1a — Docker fix attempt 1 |
| #340 | Move pino-pretty to dependencies | 1a — Docker fix attempt 2 |
| #341 | Explicit Dockerfile COPY of pino deps | 1a — Docker fix (actual) |
| #389 | Client log forwarding + axios error logging | 1b — Client pipeline |
| #406 | PII redaction + rate limiter + interceptor fix | 1c — Hardening |
Deferred to Epic 8
Two items from the security review were intentionally deferred:
Endpoint authentication (CRITICAL). The log endpoint has no auth — only IP-based rate limiting. An external attacker can send arbitrary log entries. This is being addressed in Epic 8’s auth migration to Auth.js v5, where the endpoint will verify session cookies.
Rate limiter session binding (HIGH). Currently rate-limited per IP. After auth migration, it should be rate-limited per authenticated session to prevent abuse from shared IPs (office networks, VPNs).
What I Learned
Redaction is harder than it looks. The first pass seemed thorough — it caught context keys matching sensitive patterns. But free-text fields bypass key-based checks entirely. And String() coercion on nested objects hides the values you’re trying to match. Defense in depth means applying redaction at every layer, to every field, with every encoding considered.
JavaScript regex state is a trap. Global regexes (/g) maintaining lastIndex between calls is a well-known footgun, but it’s easy to forget when you’re focused on the pattern itself rather than its lifecycle. The factory pattern is a clean fix that makes the statelessness explicit.
Browser logging needs server protection. The client can send anything. Validation, rate limiting, and redaction on the server side aren’t optional hardening — they’re the minimum viable implementation. Without them, client log forwarding is a PII leak and a log poisoning vector.
Interceptor stacking is invisible. When React effects re-run and add duplicate interceptors, there’s no error — just silently doubled log volume. The useRef pattern for stable callback references is the standard fix, but the symptom (excess logs) doesn’t obviously point to the cause (effect dependencies).
Built across 7 PRs, hardened by AI code review, protected by 29 endpoint tests. The browser reports. The server decides.