Make tcp worker multithreaded, too

This shamelessly steals the model established in the UDP workers, i.e.
if the kernel is able to bind multiple listening sockets to the same
endpoint, we will share them across threads. If it can't, all threads
will accept on the same socket.

The fast path handles most of the connections, only the rare case of
no data yet on the socket is offloaded to the slow path (which used to
be the mainloop) for completion. This scales nicely, as the higher the
load is, the higher chances are that an accept happens only after the
network roundtrip to the peer completes, leading to more connections
that already have data waiting for us.

We also consolidate the workstruct init and deinit code, preparation
for later unifying all thread state structs for all protocols (mca,
udc and tcp).

The config file gains a per tcp connection worker count and some docs
explaining what this knob buys and costs, nudging the user to either
keep it 0 or dial it up to at least 2.

1 files changed, 16 insertions, 0 deletions

diff --git a/man4/opentracker.conf.4 b/man4/opentracker.conf.4
index aa82e1d..1bd1c7a 100644
--- a/man4/opentracker.conf.4
+++ b/man4/opentracker.conf.4
@@ -42,6 +42,14 @@ in the thread that also serves TCP. See
 .Sx TUNING
 before raising it.
 .Pp
+.It listen.tcp.workers Ar threads
+Specifies how many threads accept and answer HTTP connections, for every
+listen statement that does not carry a
+.Cm workers
+option of its own. Defaults to 0, which serves them from the event loop. See
+.Sx TUNING
+before raising it.
+.Pp
 .It listen.shared Ar boolean
 Permits another process to bind the same address, for every listen statement
 that does not carry a
@@ -101,6 +109,14 @@ group only where the answer is yes.
 in the source distribution sweeps the setting against a running tracker.
 Measure on the machine that will carry the traffic.
 .Pp
+A TCP worker accepts a connection and answers it on its own stack, without
+the event loop seeing it at all; only a connection it cannot finish is handed
+over. One worker serves less than none, because accepting leaves the event
+loop's thread without anything gained in return, so the setting is worth
+either 0 or at least 2.
+.Pa tests/http_load.c
+in the source distribution drives the tracker over HTTP to measure it.
+.Pp
 On FreeBSD the
 .Xr accf_data 9
 filter has the kernel hold a new connection until data arrives on it, so the