Go, End to End

Concurrency Patterns

Goroutines and channels are primitives. This chapter is the handful of patterns built from those primitives that show up constantly in real Go code: running a fixed pool of workers, combining several producers into one stream, and cleanly cancelling work that's no longer needed.

Worker pools

A worker pool runs a fixed, bounded number of goroutines pulling jobs off a shared channel, instead of spawning one goroutine per job (which, under enough load, is exactly the unbounded-goroutine risk flagged in the previous chapter).

go · live, editable, runnable

package main import ( "fmt" "sync" ) func worker(id int, jobs <-chan int, results chan<- int, wg *sync.WaitGroup) { defer wg.Done() for j := range jobs { results <- j * j // pretend this is expensive work } _ = id } func main() { const numWorkers = 3 jobs := make(chan int, 10) results := make(chan int, 10) var wg sync.WaitGroup for w := 1; w <= numWorkers; w++ { wg.Add(1) go worker(w, jobs, results, &wg) } for j := 1; j <= 5; j++ { jobs <- j } close(jobs) // no more jobs coming, workers finish their range loop and exit wg.Wait() close(results) total := 0 for r := range results { total += r } fmt.Println("sum of squares:", total) }

Notice the channel directions in the signature: jobs <-chan int means worker can only receive from jobs, results chan<- int means it can only send to results. The compiler enforces this, a worker literally cannot accidentally send on jobs or receive from results, it's a real, checked contract, not just a naming convention.

Fan-in / fan-out

Fan-out is starting multiple goroutines to process work in parallel from a shared source. Fan-in is merging multiple result channels back into a single channel. The worker pool above is fan-out; here's the fan-in half, standalone, combining two independent producers into one stream.

go · live, editable, runnable

package main import ( "fmt" "sync" ) func producer(id int, out chan<- string, wg *sync.WaitGroup) { defer wg.Done() out <- fmt.Sprintf("message from producer %d", id) } func fanIn(sources ...chan string) chan string { merged := make(chan string) var wg sync.WaitGroup for _, s := range sources { wg.Add(1) go func(c chan string) { defer wg.Done() for msg := range c { merged <- msg } }(s) } go func() { wg.Wait() close(merged) }() return merged } func main() { c1 := make(chan string) c2 := make(chan string) var wg sync.WaitGroup wg.Add(2) go producer(1, c1, &wg) go producer(2, c2, &wg) go func() { wg.Wait(); close(c1); close(c2) }() for msg := range fanIn(c1, c2) { fmt.Println(msg) } }

context.Context for cancellation

`context.Context` is the idiomatic way to propagate "stop now" (or "stop after this deadline") through a call chain, especially one that spans goroutines. A function that does long-running or blocking work should accept a `Context` as its first parameter and check it periodically, so a caller further up the chain can cancel everything below it with one call.

go · live, editable, runnable

package main import ( "context" "fmt" "time" ) func doWork(ctx context.Context) { for i := 0; ; i++ { select { case <-ctx.Done(): fmt.Println("cancelled:", ctx.Err()) return default: time.Sleep(20 * time.Millisecond) fmt.Println("working, step", i) if i >= 10 { return // safety net for this demo } } } } func main() { ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond) defer cancel() doWork(ctx) }

`context.WithTimeout` (and its sibling `context.WithCancel`) returns both a context and a cancel function. Calling `cancel()`, or letting the timeout elapse, closes the context's internal Done channel, which every goroutine watching that context sees immediately via `<-ctx.Done()`. This is how you cancel an entire tree of goroutines with a single function call, rather than manually threading a stop signal through every layer yourself.

sync.WaitGroup

A `WaitGroup` is a counter: `Add(n)` increments it by `n`, `Done()` decrements it by one (almost always called via `defer` right at the top of the goroutine), and `Wait()` blocks until it hits zero. It's the standard way to wait for a known, fixed number of goroutines to finish, seen already in both examples above.

var wg sync.WaitGroup
wg.Add(1)          // one goroutine to wait for
go func() {
	defer wg.Done() // always defer this, right after starting the goroutine's real work
	// ... do work ...
}()
wg.Wait()          // blocks until Done() has been called once per Add()

Picking the right tool

Worker pools bound concurrency for a stream of independent jobs. Fan-in/fan-out combine or split streams of work. WaitGroup answers "wait for these N things to finish". Context answers "stop everything if the caller no longer wants the result", which frequently gets combined with a WaitGroup: context cancels the work, WaitGroup confirms every goroutine actually noticed and exited.