Network routing interaction on MacOS

# January 2, 2023

There are a series of resolution layers governing DNS, IP, and port routing on OSX. Here are some of the different interfaces to manipulate how you route traffic to the internet or to localhost.

/etc/hosts

The hosts file forms a direct association between domain and IP address. It is effectively used as a higher priority routing record to a record in a DNS lookup table. Note that this file does not support port routing. Commands will be routed 1:1 from synthetic domain name to IP.

Given the entry:

192.168.10.1 dev

Accessing the domain in curl or a browser will route accordingly:

curl dev -> 192.168.10.1:80
curl dev:1000 -> 192.168.10.1:1000

ifconfig

Allows you to analyze and manipulate the different networking interfaces on your computer. To view all of the available interfaces:

$ ifconfig -a

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
    options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    nd6 options=201<PERFORMNUD,DAD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
anpi1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
...

It also allows the creation of new synthetic IP values, depending on the support of the networking interface. So instead of having localhost be 127.0.0.1 you can also create 127.0.0.2 within the lo0 loopback interface.

sudo ifconfig lo0 alias 127.0.0.2 up

pfctl

Mac replacement to ipfw with a similar command structure. This utility focuses on filtering out packets from the packet filter but can also do much more to manipulate packets. The critical section of the man page:

The packet filter can also replace addresses and ports of packets. Replacing source addresses and ports of outgoing packets is called NAT (Network Address Translation) and is used to connect an internal network (usually reserved address space) to an external one (the Internet) by making all connections to external hosts appear to come from the gateway.

This allows you to route packets across IP/port combinations. Let's say you want to route from the new 127.0.0.2:80 synthetic IP to port 3000 mounted on standard localhost.

echo -n "rdr pass on lo0 inet proto tcp from any to 127.0.0.2 port 80 -> 127.0.0.1 port 3000n" > route_configuration.conf

sudo pfctl -e -f route_configuration.conf

The recommended approach is to supplement the default file that can be found in /etc/pf.conf to maintain Apple's built in routing filters.

/etc/resolver

Just for nameserver definition lookup. Add as independent files that have the domain as the filename:

$ cat /etc/resolver/dev.local

domain dev.local
nameserver 192.168.10.1

DNS servers are expected to respond on 80 and this list is no exception; custom ports are not allowed here.

Related tags:
#programming
The curious case of LM repetition
I was doing some OSS benchmarking over the weekend and was running into an odd issue. Some families of models would respond with near-gibberish, even with straightforward prompt inputs. This is a debugging session for LLM repetition.
Webcrawling tradeoffs
A couple of years ago I built our internal crawling platform at Globality, which needed to be capable of scaling to billions of pages each crawl. The two main types of crawlers that are deployed in the wild are typically raw or headless. We ended up implementing a hybrid architecture. Hybrid crawling can make use of the strengths of both while trying to minimize their weaknesses.
Headfull browsers beat headless
Twenty years ago a simple curl would open up the world. HTML markup was largely hand designed so id and name attributes were easily interpretable and parsable. Now most sites render dynamic content or use template defined class tags to define the styling of the page. Building a headfull browser container to more easily deploy and debug Chromium in a remote cluster.

Hi, I'm Pierce

I write mostly about engineering, machine learning, and company building. If you want to get updated about longer essays, subscribe here.

I hate spam so I keep these infrequent - once or twice a month, maximum.