Building a Lightweight Nginx Replacement in Go

Introduction

Nginx is a powerhouse, but there are many scenarios where you want something lean, tailored to your service, and easier to reason about. In this post, we’ll explore building a lightweight Nginx-like replacement in Go that focuses on a small feature set: reverse proxying, basic TLS termination, static file serving, and simple load balancing. We’ll keep the code approachable, demonstrate a basic round-robin load balancer, and highlight design choices and trade-offs.

Goals and constraints

• Provide a minimal, reliable reverse proxy with TLS termination.
• Support a small set of features you typically need for internal services: static assets, basic load balancing, and straightforward logging.
• Keep dependencies low and the code approachable for Go developers.
• Avoid feature bloat and focus on clarity, performance, and ease of deployment.

Architecture overview

• Listener: an HTTP(S) server that terminates TLS and routes requests.
• Proxy handler: implements a round-robin load balancer over backend backends and forwards requests using a reverse proxy.
• Static file server: serves a small static content directory to satisfy common hosting needs.
• Observability: simple request logging to surface basic latency and status information.
• Configuration: simple in-code wiring with the option to switch to environment-based config later.

Minimal Go implementation

Below is a compact, working starting point. It demonstrates:

• A round-robin proxy over multiple backends
• TLS termination (you’ll provide server.crt and server.key)
• Static file serving under /static
• Basic error handling

package main

import (
	"crypto/tls"
	"log"
	"net/http"
	"net/http/httputil"
	"net/url"
	"sync/atomic"
)

type Backends struct {
	urls []*url.URL
	// round-robin counter
	counter int64
}

func (b *Backends) Next() *url.URL {
	// simple atomic round-robin
	n := atomic.AddInt64(&b.counter, 1)
	return b.urls[int(n-1)%len(b.urls)]
}

type Proxy struct {
	backends *Backends
}

func (p *Proxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	// Choose a backend for this request
	dest := p.backends.Next()

	// Create a reverse proxy targeting the chosen backend
	proxy := httputil.NewSingleHostReverseProxy(dest)

	// It's often helpful to preserve the original Host header
	r.Host = dest.Host

	// Optional: customize error handling
	proxy.ErrorHandler = func(w http.ResponseWriter, r *http.Request, err error) {
		http.Error(w, "Bad Gateway", http.StatusBadGateway)
	}

	// Forward the request
	proxy.ServeHTTP(w, r)
}

func mustParseURL(raw string) *url.URL {
	u, err := url.Parse(raw)
	if err != nil {
		panic(err)
	}
	return u
}

func main() {
	// Define backends (adjust to your environment)
	backends := &Backends{
		urls: []*url.URL{
			mustParseURL("http://127.0.0.1:8081"),
			mustParseURL("http://127.0.0.1:8082"),
		},
	}

	proxy := &Proxy{backends: backends}

	// Static file server
	staticDir := "./static"
	staticFS := http.FileServer(http.Dir(staticDir))
	mux := http.NewServeMux()
	mux.Handle("/static/", http.StripPrefix("/static/", staticFS))

	// Fallback to the proxy for all other routes
	mux.Handle("/", proxy)

	// TLS configuration (replace with your certs)
	tlsConfig := &tls.Config{
		MinVersion: tls.VersionTLS12,
	}

	server := &http.Server{
		Addr:      ":443",
		Handler:   mux,
		TLSConfig: tlsConfig,
	}

	log.Println("Starting lightweight proxy on :443 with TLS termination")
	// In production, ensure you have server.crt and server.key available
	if err := server.ListenAndServeTLS("server.crt", "server.key"); err != nil {
		log.Fatalf("server failed: %v", err)
	}
}

Notes:

• This is intentionally minimal. It uses a simple atomic counter for round-robin load balancing. If you need more sophisticated load balancing (e.g., least connections, weightedBackends), you can extend the Backends type.
• The TLS termination is performed by the Go server. Your backend services should be accessible over HTTP in this example. If you want end-to-end TLS to backends, you can adjust the backend URLs to use https and configure TLS transport.

TLS termination and performance hints

• Use TLS termination close to the edge to reduce the backends’ TLS load. Go’s TLS implementation is fast and has built-in HTTP/2 when using TLS, which helps with multiplexing.
• Enable HTTP/2 in production by serving over TLS (as shown) and ensure your certs are valid and up to date.
• Consider tuning TLS sessions and ciphers only if you observe TLS-related performance issues.

Static content and caching

• Serving static assets from the same binary simplifies deployment. If your static content grows, consider serving it from a dedicated static server or a CDN.
• For small assets, you can add simple cache-control headers in the static file handler. For example:
  • mux.Handle(“/static/”, http.StripPrefix(“/static/”, http.FileServer(http.Dir(staticDir))))
  • You can wrap the file server with a custom handler that sets Cache-Control headers.

Observability and debugging

• Add request logging to capture latency, method, path, and status codes.
• Instrument metrics (e.g., request counts, error rates, latency histograms) for basic visibility.
• Keep an eye on error paths (e.g., 502 Bad Gateway) to identify backend health or misconfigurations.

Deployment notes

• Build a single binary with go build -o lightweight-proxy .
• Ensure you have TLS certificates available at server.crt and server.key, or use a certificate management system in production.
• If you plan to run multiple instances, consider a simple external load balancer in front of them or extend the internal load-balancing logic to support health checks and dynamic backends.

Security considerations

• Validate and sanitize all headers that cross the proxy boundary, especially if you enable header rewriting in the future.
• Keep TLS up to date and rotate certificates regularly.
• Implement basic health checks for backends to avoid routing to unhealthy instances.

Conclusion

This lightweight Go-based proxy demonstrates how to build a focused, maintainable alternative for scenarios where you don’t need the full feature set of a heavyweight proxy like Nginx. By starting with a simple round-robin reverse proxy, TLS termination, and static content serving, you gain a predictable, easily auditable path to scale, customize, and optimize for your services. As your needs grow, you can layer in additional features—observability, health checks, richer load balancing strategies, and more—without sacrificing the clarity that comes from keeping the design intentionally small.