System Design Reference

Anything originating a request - the source of traffic.

• Browser
  Runs your web app. Pushes more logic to the edge of the network than it used to - caches, workers, WASM.
• Mobile
  Native iOS/Android or RN/Flutter. Offline-tolerant, push-driven, latency-sensitive.
• CLI / SDK
  Scripts, internal tools, developer flows. Usually authenticated with a long-lived token or device flow.
• Agent
  An LLM (or program) acting on a user's behalf - calls your APIs the same as a human would, just faster and weirder.
• Other service
  Server-to-server. Typically mTLS or signed requests, no UI in the loop.

First thing your traffic actually touches.

• DNS
  Maps names to addresses. Also a routing tool via weighted, geo, and latency records.
• CDN
  Geo-distributed cache + TLS terminator. Often the first real performance win.
• WAF
  Web application firewall - blocks common attacks before they reach your app.
• DDoS protection
  Absorbs volumetric attacks at the edge so the origin never sees them.
• Edge functions
  Tiny compute at PoPs - auth, redirects, A/B, personalization without an origin hop.
• TLS termination
  Decrypts at the edge so internal hops can be simpler - re-encrypt for zero-trust.

Its own tier now - has infra of its own and reaches sideways into data.

• Model gateway
  Single endpoint in front of many models. Handles routing, fallback, key management, logging.
• Prompt / router
  Picks model + system prompt per request based on intent, cost, latency.
• RAG orchestration
  Retrieve from vector + structured stores, rerank, stuff context, call the model.
• Vector store
  ANN search on embeddings. Also lives in the data layer - same store, two consumers.
• Guardrails
  Input/output filters - PII, jailbreaks, policy, schema validation on tool calls.
• Eval
  Offline + online quality measurement. The thing that tells you a prompt change broke something.

Contract layer between the outside world and your services.

• Auth
  Validates tokens, exchanges sessions, injects identity downstream.
• Rate limiting
  Per-user, per-key, per-route. Protects the app and shapes pricing tiers.
• Schema / validation
  Rejects bad shapes before they hit your code. OpenAPI, gRPC, GraphQL schemas.
• Request shaping
  Rewrites, header injection, response transforms, request collapsing.
• Routing
  Path + method → upstream service. Often combined with canary and traffic splitting.

Where product behavior lives - stateless so it can scale.

• Services
  Domain-bounded units of logic. Independently deployable, owned by a team.
• Backend for frontend
  Per-client aggregation layer - trims and reshapes for web vs mobile vs agent.
• Business logic
  The actual rules of your product. Should be the boring, well-tested core.
• Background workers
  Lives here architecturally even if it's invoked from async - pulls jobs and runs them.

Where most real systems actually live.

• Message queue
  Decouples producers and consumers. Retries, dead-letter, backpressure for free.
• Event bus
  Pub/sub fan-out - one event, many subscribers. Service Bus topics, SNS, NATS.
• Event stream
  Durable, replayable log. Kafka, Event Hubs, Kinesis, Redpanda.
• Workers
  Consume from queues/streams. Slow, retryable, or off the request path.
• Scheduler
  Cron-ish triggers - periodic jobs, scheduled emails, batch rollups.

Owns durable state. Usually the hardest tier to scale.

• OLTP
  Transactional relational DB. Postgres, MySQL, SQL Server. Default until you have a reason.
• Cache
  In-memory in front of slower systems. Redis, Memcached. Pick a consistency story.
• Search
  Inverted indexes for text + facets. Elasticsearch, OpenSearch, Meilisearch.
• OLAP / warehouse
  Columnar, scans huge datasets. BigQuery, Snowflake, Synapse, ClickHouse.
• Object storage
  Cheap, durable blobs. S3, R2, ADLS/Blob. Backbone of data lakes and uploads.
• Vector
  Embedding search. pgvector, Pinecone, Weaviate. Bridges into the AI layer.

What actually runs the app and data tiers.

• Kubernetes
  Scheduler, networking, rollout for containers. AKS, EKS, GKE.
• Serverless
  Functions, Container Apps, Cloud Run. Scales to zero, pays per request.
• Containers
  The unit of deployment - image + runtime contract.
• Service mesh
  Sidecar proxies for service-to-service TLS, retries, observability. Istio, Linkerd.

The plumbing every layer above rides on.

• VNet / VPC
  Private network boundary in the cloud. Public, private, and database subnets is the usual split.
• Subnets
  Address space carved up by tier and AZ. Drives blast radius and routing.
• NSG / firewall
  Allow/deny rules at the subnet or NIC level. Default-deny, document exceptions.
• Private endpoints
  Reach PaaS over the private network instead of the public internet.
• Load balancers
  L4 routes by IP/port, L7 by URL/header/cookie. Pair with health checks.
• Peering
  Private connection between VNets/VPCs - same region or across.
• ExpressRoute / VPN
  Private link from on-prem into cloud. ExpressRoute is dedicated; VPN is over internet.
• Internal DNS
  Private name resolution inside the VNet. Often the source of mystery outages.

The actual machines under the platform.

• VMs
  General-purpose virtual machines. Most workloads still land here under the abstraction.
• Bare metal
  No hypervisor. Used when you need every cycle - HPC, large DBs, some AI training.
• GPUs / accelerators
  Training and inference for ML, plus rendering. Increasingly the bottleneck and the budget.
• Hypervisor
  Hyper-V, KVM, Xen. Mostly invisible - but it sets the noisy-neighbor and live-migration story.

Real concrete and copper.

• Region
  Geographic area. Drives latency, data residency, and pricing.
• Availability zone
  Independent datacenter(s) within a region. Failure domain for HA.
• Datacenter
  The building. Power, cooling, security, connectivity.
• Rack
  Servers + top-of-rack switch. A common failure unit.
• Power & cooling
  PSUs, UPS, generators, chillers. Capacity here caps everything above.
• Fiber
  Inter-DC and long-haul links. Cuts, congestion, and BGP issues live down here.

Who is calling, what can they do, how do we prove it.

• Entra / IdP
  Identity provider - humans, services, managed identities.
• OAuth / OIDC
  Delegated auth and federated login. Standard everywhere.
• mTLS
  Mutual TLS between services. Identity on the wire, not just in a token.
• Key Vault
  Stores secrets, certs, keys. Rotate often, never check in.
• Managed identity
  Workload identity issued by the platform - no secrets to manage.

If you can't see it, you can't fix it.

• Logs
  Structured events. Cheap to write, expensive to query at scale.
• Metrics
  Numbers over time. Cheap, aggregatable, what your dashboards run on.
• Traces
  Per-request span tree across services. The thing that actually explains latency.
• Profiling
  Continuous CPU/memory profiles. Finds the slow function, not just the slow service.

Applied at edge, gateway, app, and data - not a single layer.

• WAF rules
  Pattern-based blocks at the edge. Tune to your traffic, not the defaults.
• RBAC
  Role-based access. The simple model that scales further than people expect.
• Governance
  Org policy, naming, tagging, allowed regions. Enforced in IaC, not by hope.
• Compliance
  SOC2, ISO, HIPAA, PCI. Mostly evidence collection on top of good engineering.

Every layer has knobs that move the bill.

• Quotas
  Hard limits per subscription, region, SKU. Surprise here = outage.
• Autoscale
  Scale on the right signal - CPU rarely is one. Often queue depth or p95.
• FinOps
  Tag, attribute, forecast, and right-size. Make cost a product metric.

How code and config get to prod without taking it down.

• CI/CD
  Build, test, scan, deploy. Fast feedback is the whole game.
• IaC
  Terraform, Bicep, Pulumi. Infra in version control, reviewed like code.
• Progressive rollout
  Canary, blue/green, feature flags. Limits blast radius of every change.