Using Nginx as a proxy to multiple Unix sockets

Using Nginx as a proxy to multiple Unix sockets

#nginx#go#unixsocket

TL;DR Listening port may be a contended resource on a busy shared machine, unix sockets are virtually unlimited. Nginx can expose them with a single port and prefixed URLs.

In some situations you may want to run many (instances of) applications on a single machine. Each instance may need to provide internal information (e.g. Prometheus /metrics, profiling/debug handlers) over restricted HTTP.

When number of instances grows it becomes a burden to provision listening ports without conflicts. In contrast, using Unix sockets allows for more transparency (readable filenames) and scalability (easy to come up with unique name).

Here is a small demo program written in Go that would serve trivial HTTP service with Unix socket.

package main

import (
    "context"
    "flag"
    "io/fs"
    "log"
    "net"
    "net/http"
    "os"
    "os/signal"
)

func main() {
    var socketPath string

    flag.StringVar(&socketPath, "socket", "./soc1", "Path to unix socket.")
    flag.Parse()

    if socketPath == "" {
        flag.Usage()
        return
    }

    listener, err := net.Listen("unix", socketPath)
    if err != nil {
        log.Println(err.Error())
        return
    }

    // By default, unix socket would only be available to same user.
    // If we want access it from Nginx, we need to loosen permissions.
    err = os.Chmod(socketPath, fs.ModePerm)
    if err != nil {
        log.Println(err)
        return
    }

    httpServer := http.Server{
        Handler: http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
            log.Println(request.URL.String())
            if _, err := writer.Write([]byte(request.URL.String())); err != nil {
                log.Println(err.Error())
            }
        }),
    }

    // Setting up graceful shutdown to clean up Unix socket.
    go func() {
        sigint := make(chan os.Signal, 1)
        signal.Notify(sigint, os.Interrupt)
        <-sigint
        if err := httpServer.Shutdown(context.Background()); err != nil {
            log.Printf("HTTP Server Shutdown Error: %v", err)
        }
    }()

    log.Printf("Service is listening on socket file %s", socketPath)
    err = httpServer.Serve(listener)
    if err != nil {
        log.Println(err.Error())
        return
    }
}

Now let's run a couple of instances in separate shells.

./soc -socket /home/ubuntu/soc1

./soc -socket /home/ubuntu/soc2

Here is a minimal Nginx config to serve those instances with URL prefixes. It would receive http://my-host/soc1/foo/bar, strip path prefix /soc1 and pass /foo/bar to soc1.

to soc1.

server {
    listen 80 default;

    location /soc1/ {
        proxy_pass http://soc1/;
    }
    location /soc2/ {
        proxy_pass http://soc2/;
    }
}

upstream soc1 {
    server unix:/home/ubuntu/soc1;
}

upstream soc2 {
    server unix:/home/ubuntu/soc2;
}

Every Unix socket is defined as upstream and has /location statement in server.

It is also possible to use Unix sockets directly in /location, like in

    location /soc1/ {
        proxy_pass http://unix:/home/ubuntu/soc1;
    }

however it has an unwanted limitation that you can not add trailing / to proxy_pass. And this means that URL will be passed as is, e.g. soc1 will receive /soc1/foo instead of /foo.

To avoid such limitation we can use named upstream and add trailing / to proxy_pass.

    location /soc1/ {
        proxy_pass http://soc1/; # Mind trailing "/".
    }