Make tcp worker multithreaded, too

This shamelessly steals the model established in the UDP workers, i.e.
if the kernel is able to bind multiple listening sockets to the same
endpoint, we will share them across threads. If it can't, all threads
will accept on the same socket.

The fast path handles most of the connections, only the rare case of
no data yet on the socket is offloaded to the slow path (which used to
be the mainloop) for completion. This scales nicely, as the higher the
load is, the higher chances are that an accept happens only after the
network roundtrip to the peer completes, leading to more connections
that already have data waiting for us.

We also consolidate the workstruct init and deinit code, preparation
for later unifying all thread state structs for all protocols (mca,
udc and tcp).

The config file gains a per tcp connection worker count and some docs
explaining what this knob buys and costs, nudging the user to either
keep it 0 or dial it up to at least 2.

1 files changed, 4 insertions, 3 deletions

diff --git a/ot_http.h b/ot_http.h
index 82d645b..9cacd21 100644
--- a/ot_http.h
+++ b/ot_http.h
@@ -40,9 +40,10 @@ void           http_interest_set(const int64_t s, struct http_data *cookie, int
 /* Close a connection and release everything that belongs to it */
 void           http_close(const int64_t s);
 
-/* Promote a stack cookie to a heap one for persistence and handing it over
-   to the event loop. */
-int            http_adopt(const int64_t s, struct http_data *cookie);
+/* Promote a stack cookie to a heap one, for persistence and for handing it
+   over to the event loop or to another thread. Does nothing to one that is
+   already on the heap. Returns the connection's state, or NULL. */
+struct http_data *http_adopt(const int64_t s);
 
 ssize_t        http_handle_request(const int64_t s, struct ot_workstruct *ws);
 ssize_t        http_sendiovecdata(const int64_t s, struct ot_workstruct *ws, size_t iovec_entries, struct iovec *iovector, int is_partial);