[Bro-Dev] libcurl and libev integration

[lauri]

Hi,

This is sort of a long story so grab a cup of coffee :)

I've been using ActiveHTTP::Request to issue HTTP requests from Bro
scripts so far, but as it's using shell invocation of curl it is a)
slow due to execution of external process b) unable to make use of
HTTP keep-alive and c) hence unable to multiplex requests to same
server. This causes whole lot of overhead with current
ActiveHTTP::Request making it unusable for various interesting
usecases, eg. querying data from ElasticSearch ;)

I eventually started tweaking the code to incorporate libcurl
functionality into Bro runtime and the blocking version came along
just fine. Now as I am trying to make the libcurl function calls
asynchronous, it's becoming really complex. There is
curl_multi_fdset(multi_handle, &fdread, &fdwrite, &fdexcep, &maxfd);
[1]  which can be used to fetch socket descriptors associated with
libcurl's connections. It is designed to populate existing fd_set
struct with proper file handles to be spoon-fed to select(). Note that
fd_set struct internal structure is not well defined (different on
Windows and POSIX); macros FD_SET, FD_CLR, FD_ISSET and FD_ZERO are
designed for fd_set manipulation and there is no macro to extract
socket descriptors associated with particular fd_set.

Trying to work with and around Bro's FD_Set and IOSource classes makes
it nearly impossible to integrate libcurl as it is without a lot of
dirty code. I also noticed that you've implemented asynchronous DNS
client on top of your event loop handler mechanism. I assume this was
built much earlier than any event handling library was conceived so
it's sort of a legacy. Now looking how stuff is done nowadays you see
there are a lot of event loop libraries out there which make more
efficient use of kernel such as epoll() and kqueue in contrast to
select() and already have built-in methods for asynchronous file
input/output. Note that libev [3] also has built-in async DNS client.

My conclusion at this very early stage is that it would make sense to
substitute Bro's event loop and DNS client with libev. This should
make it significantly easier to integrate with other libraries such as
libcurl, take a look at libuv example of libcurl [4]. I am not sure
how this would affect Bro runtime logic. Comments, questions and
feedback on the ideas presented above are very much welcome :)

1. http://curl.haxx.se/libcurl/c/curl_multi_fdset.html
2. http://www.mkssoftware.com/docs/man3/select.3.asp
3. https://github.com/libuv/libuv
4. http://curl.haxx.se/libcurl/c/multi-uv.html

-- 
Lauri Võsandi
tel: +372 53329412
e-mail: lauri.vosandi at gmail.com
blog: http://lauri.vosandi.com/