[Zeek] monitoring proxied web traffic
Konrad Weglowski
konrad.weglowski at gmail.com
Tue Nov 12 07:54:24 PST 2019
Thanks again Michał for the breakdown. I have the full picture now :)
Take care
Konrad
On Fri, Nov 8, 2019 at 3:21 AM Michał Purzyński <michalpurzynski1 at gmail.com>
wrote:
> You're welcome!!
>
> If the traffic from the client to the proxy is not encrypted and the proxy
> is not "transparent" (meaning you configured it on the client) then you
> have the same case as I do here.
>
> So there's this host
> source IP 10.48.75.81
>
> trying to talk to clr.telemetry.intel.com
>
> via our proxy at 10.48.122.10 / 10.48.74.16 / 10.48.74.17
>
> *(for anyone working at a place where disclosing an internal IP or an FQDN
> equal incident - you know a lot of great places are hiring, right?)*
>
> I used the "uid" to correlate this connection across three types of logs -
> conn.log, http.log and ssl.log. You're gonna get them due to the way
> proxies operate.
>
> Please ignore tons of metadata from our SIEM (MozDef = custom Python +
> pure ES), I'm going to C&P three logs as they are without any redactions.
> This post is HUGE!!
>
> *conn.log* - host -> proxy
>
> {
> "_index": "events-20191108",
> "_type": "_doc",
> "_id": "JmLRSW4BiWsbe3JQtC8S",
> "_version": 1,
> "_score": null,
> "_source": {
> "type": "nsm",
> "details": {
> "ts": 1573196540.178897,
> * "uid": "C9PFt711Odfwpbr0Ti",*
> * "proto": "tcp",*
> * "service": "http,ssl",*
> "duration": 0.183212,
> "orig_bytes": 1604,
> "resp_bytes": 7180,
> "conn_state": "SF",
> "local_orig": true,
> "local_resp": true,
> "missed_bytes": 0,
> * "history": "ShADadFf",*
> * "orig_pkts": 14,*
> * "resp_pkts": 13,*
> "peer": "nsm2-af_packet-enp18s0f0-8",
> * "orig_l2_addr": "b4:0c:25:e0:40:10",*
> * "resp_l2_addr": "00:50:56:a1:48:64",*
> * "sourceipaddress": "10.48.75.81",*
> * "sourceport": 55330,*
> * "destinationipaddress": "10.48.122.10",*
> * "destinationport": 3128,*
> * "originipbytes": 2340,*
> * "responseipbytes": 7864,*
> "sourceipv4address": "10.48.75.81",
> "destinationipv4address": "10.48.122.10"
> },
> "customendpoint": "bro",
> "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
> "tags": "bro",
> "category": "bro",
> * "source": "conn",*
> * "utctimestamp": "2019-11-08T07:02:20.178897+00:00",*
> "timestamp": "2019-11-08T07:02:20.178897+00:00",
> "receivedtimestamp": "2019-11-08T07:02:27.686061+00:00",
> "eventsource": "nsm",
> "severity": "INFO",
> "mozdefhostname": "mozdef5.private.mdc1.xxxxxxxxxx.com",
> "summary": "10.48.75.81:55330 -> 10.48.122.10:3128 ShADadFf 2340
> bytes / 7864 bytes",
> "plugins": [
> "customDocType",
> "broFixup",
> "ipFixup",
> "geoip"
> ],
> "processid": "UNKNOWN",
> "processname": "UNKNOWN"
> },
> "fields": {
> "receivedtimestamp": [
> "2019-11-08T07:02:27.686Z"
> ],
> "timestamp": [
> "2019-11-08T07:02:20.178Z"
> ],
> "utctimestamp": [
> "2019-11-08T07:02:20.178Z"
> ]
> },
> "highlight": {
> "summary": [
> "@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@:55330
> -> 10.48.122.10:3128 ShADadFf 2340 bytes / 7864 bytes"
> ],
> "details.sourceipv4address": [
> "@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@"
> ],
> "details.uid": [
> "@kibana-highlighted-field at C9PFt711Odfwpbr0Ti
> @/kibana-highlighted-field@"
> ],
> "category": [
> "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
> ]
> },
> "sort": [
> 1573196540178
> ]
> }
>
> *http.log* (because the connection from the client to the proxy was
> technically a http connection)
>
> {
> "_index": "events-20191108",
> "_type": "_doc",
> "_id": "jPDRSW4BC2j90LU9UdgL",
> "_version": 1,
> "_score": null,
> "_source": {
> "type": "nsm",
> "details": {
> "ts": 1573196540.202888,
> * "uid": "C9PFt711Odfwpbr0Ti",*
> "trans_depth": 1,
> "method": "CONNECT",
>
> *"host": "clr.telemetry.intel.com <http://clr.telemetry.intel.com>",
> "uri": "clr.telemetry.intel.com:443 <http://clr.telemetry.intel.com:443>",*
> "version": "1.1",
> "request_body_len": 0,
> "response_body_len": 0,
> * "status_code": 200,*
> * "status_msg": "Connection established",*
> "tags": [],
> "proxied": [
> "PROXY-CONNECTION -> Keep-Alive"
> ],
>
>
>
> *"sourceipaddress": "10.48.75.81", "sourceport": 55330,
> "destinationipaddress": "10.48.122.10", "destinationport": 3128,*
> "sourceipv4address": "10.48.75.81",
> "destinationipv4address": "10.48.122.10"
> },
> "customendpoint": "bro",
> "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
> "tags": "bro",
> "category": "bro",
> * "source": "http",*
> * "utctimestamp": "2019-11-08T07:02:20.202888+00:00",*
> "timestamp": "2019-11-08T07:02:20.202888+00:00",
> "receivedtimestamp": "2019-11-08T07:02:21.427537+00:00",
> "eventsource": "nsm",
> "severity": "INFO",
> "mozdefhostname": "mozdef7.private.mdc1.xxxxxxxxxx.com",
> "summary": "HTTP CONNECT 10.48.75.81 -> 10.48.122.10:3128",
> "plugins": [
> "customDocType",
> "broFixup",
> "ipFixup",
> "geoip"
> ],
> "processid": "UNKNOWN",
> "processname": "UNKNOWN"
> },
> "fields": {
> "receivedtimestamp": [
> "2019-11-08T07:02:21.427Z"
> ],
> "timestamp": [
> "2019-11-08T07:02:20.202Z"
> ],
> "utctimestamp": [
> "2019-11-08T07:02:20.202Z"
> ]
> },
> "highlight": {
> "summary": [
> "HTTP CONNECT @kibana-highlighted-field at 10.48.75.81
> @/kibana-highlighted-field@ -> 10.48.122.10:3128"
> ],
> "details.sourceipv4address": [
> "@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@"
> ],
> "details.uid": [
> "@kibana-highlighted-field at C9PFt711Odfwpbr0Ti
> @/kibana-highlighted-field@"
> ],
> "category": [
> "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
> ]
> },
> "sort": [
> 1573196540202
> ]
> }
>
> And the *ssl.log* - over the previously established http connection the
> client told proxy to establish a tunnel to the destination host - and Zeek
> parsed that as TLS (find me another IDS/NSM that can do that)
>
> {
> "_index": "events-20191108",
> "_type": "_doc",
> "_id": "sfDRSW4BC2j90LU9S8_Q",
> "_version": 1,
> "_score": null,
> "_source": {
> "type": "nsm",
> "details": {
> "ts": 1573196540.253057,
>
>
>
>
>
>
>
>
>
>
>
>
>
> * "uid": "C9PFt711Odfwpbr0Ti", "version": "TLSv12", "cipher":
> "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "curve": "secp256r1",
> "server_name": "clr.telemetry.intel.com <http://clr.telemetry.intel.com>",
> "resumed": false, "established": true, "cert_chain_fuids": [
> "FknnGH2L2UIKGERW3b", "FGtbhK1HoQMWdgyKW",
> "F8JbGM30JvqFVi41A6", "Fsc2fRRTyTDQrT1Gc" ],
> "client_cert_chain_fuids": [],*
>
>
>
>
>
>
>
>
>
>
>
>
> *"subject": "CN=c-all.telemetry.intel.com
> <http://c-all.telemetry.intel.com>,OU=Multi-Domain SSL,OU=Hosted by Intel
> Corporation,OU=SSG,O=Intel Corporation,street=2200 Mission College
> Blvd,street=FM1-110,L=Santa
> Clara,ST=California,postalCode=95054-1537,C=US", "issuer": "CN=COMODO
> RSA Organization Validation Secure Server CA,O=COMODO CA
> Limited,L=Salford,ST=Greater Manchester,C=GB", "validation_status":
> "ok", "server_cert_md5": "27ba7ede8ceab7c896ee4b6bcb509f63",
> "server_cert_sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21", "pfs":
> true, "ja3": "f436b9416f37d134cadd04886327d3e8", "ja3s":
> "303951d4c50efb2e991652225a6f02b1", "sourceipaddress": "10.48.75.81",
> "sourceport": 55330, "destinationipaddress": "10.48.122.10",
> "destinationport": 3128,* "sourceipv4address": "10.48.75.81",
> "destinationipv4address": "10.48.122.10"
> },
> "customendpoint": "bro",
> "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
> "tags": "bro",
> "category": "bro",
> "source": "ssl",
> "utctimestamp": "2019-11-08T07:02:20.253057+00:00",
> "timestamp": "2019-11-08T07:02:20.253057+00:00",
> "receivedtimestamp": "2019-11-08T07:02:21.464921+00:00",
> "eventsource": "nsm",
> "severity": "INFO",
> "mozdefhostname": "mozdef6.private.mdc1.xxxxxxxxxx.com",
> "summary": "SSL: 10.48.75.81 -> 10.48.122.10:3128",
> "plugins": [
> "customDocType",
> "broFixup",
> "ipFixup",
> "geoip"
> ],
> "processid": "UNKNOWN",
> "processname": "UNKNOWN"
> },
> "fields": {
> "receivedtimestamp": [
> "2019-11-08T07:02:21.464Z"
> ],
> "timestamp": [
> "2019-11-08T07:02:20.253Z"
> ],
> "utctimestamp": [
> "2019-11-08T07:02:20.253Z"
> ]
> },
> "highlight": {
> "summary": [
> "SSL: @kibana-highlighted-field at 10.48.75.81
> @/kibana-highlighted-field@ -> 10.48.122.10:3128"
> ],
> "details.sourceipv4address": [
> "@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@"
> ],
> "details.uid": [
> "@kibana-highlighted-field at C9PFt711Odfwpbr0Ti
> @/kibana-highlighted-field@"
> ],
> "category": [
> "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
> ]
> },
> "sort": [
> 1573196540253
> ]
> }
>
>
> Not enough? Let's look for the certificate's MD5
>
> {
> "_index": "events-20191108",
> "_type": "_doc",
> "_id": "_CwBSm4BC2j90LU9vfNt",
> "_version": 1,
> "_score": null,
> "_source": {
> "type": "nsm",
> "details": {
> "ts": 1573199701.992707,
> "id": "FdE5qt1b93z9QOJnz1",
> "san.dns": [
> "c-all.telemetry.intel.com",
> "c.telemetry.intel.com",
> "c1.telemetry.intel.com",
> "c2.telemetry.intel.com",
> "c3.telemetry.intel.com",
> "c4.telemetry.intel.com",
> "c5.telemetry.intel.com",
> "c6.telemetry.intel.com",
> "c7.telemetry.intel.com",
> "c8.telemetry.intel.com",
> "c9.telemetry.intel.com",
> "clr.telemetry.intel.com",
> "sur.telemetry.intel.com"
> ],
> "md5": "27ba7ede8ceab7c896ee4b6bcb509f63",
> "sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21",
> "certificate": {
> "basic_constraints_ca": false,
> "exponent": "65537",
> "issuer": "CN=COMODO RSA Organization Validation Secure Server
> CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB",
> "key_alg": "rsaEncryption",
> "key_length": 2048,
> "key_type": "rsa",
> "not_valid_after": "2020-08-02T23:59:59+00:00",
> "not_valid_before": "2018-08-03T00:00:00+00:00",
> "sig_alg": "sha256WithRSAEncryption",
> "subject": "CN=c-all.telemetry.intel.com,OU=Multi-Domain
> SSL,OU=Hosted by Intel Corporation,OU=SSG,O=Intel Corporation,street=2200
> Mission College Blvd,street=FM1-110,L=Santa
> Clara,ST=California,postalCode=95054-1537,C=US",
> "version": 3,
> "serial": "1406C0A906762390B7252C9598F2DB3C"
> }
> },
> "customendpoint": "bro",
> "hostname": "nsm1-ber3.private.ber3.xxxxxxxxxx.com",
> "tags": "bro",
> "category": "bro",
> "source": "x509",
> "utctimestamp": "2019-11-08T07:55:01.992707+00:00",
> "timestamp": "2019-11-08T07:55:01.992707+00:00",
> "receivedtimestamp": "2019-11-08T07:55:17.040196+00:00",
> "eventsource": "nsm",
> "severity": "INFO",
> "mozdefhostname": "mozdef1.private.mdc1.xxxxxxxxxx.com",
> "summary": "X509 certificate seen",
> "plugins": [
> "customDocType",
> "broFixup"
> ],
> "processid": "UNKNOWN",
> "processname": "UNKNOWN"
> },
> "fields": {
> "details.certificate.not_valid_after": [
> "2020-08-02T23:59:59.000Z"
> ],
> "receivedtimestamp": [
> "2019-11-08T07:55:17.040Z"
> ],
> "details.certificate.not_valid_before": [
> "2018-08-03T00:00:00.000Z"
> ],
> "timestamp": [
> "2019-11-08T07:55:01.992Z"
> ],
> "utctimestamp": [
> "2019-11-08T07:55:01.992Z"
> ]
> },
> "highlight": {
> "details.md5": [
> "@kibana-highlighted-field at 27ba7ede8ceab7c896ee4b6bcb509f63
> @/kibana-highlighted-field@"
> ],
> "source": [
> "@kibana-highlighted-field at x509@/kibana-highlighted-field@"
> ],
> "category": [
> "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
> ]
> },
> "sort": [
> 1573199701992
> ]
> }
>
>
> Let's try to splice the first and the second half of the connection
>
> category:bro AND source:dns AND details.query:clr.telemetry.intel.com
>
> clr.ats.infra-host.com, 52.37.21.110, 34.208.200.27
>
> Back to looking at conn.log with destination IP like above and source =
> proxies (from there you can again do the crazy dance with ssl.log but
> there's no need to)
>
> {
> "_index": "events-20191108",
> "_type": "_doc",
> "_id": "jvDRSW4BC2j90LU9UdgL",
> "_version": 1,
> "_score": null,
> "_source": {
> "type": "nsm",
> "details": {
> "ts": 1573196540.253057,
> "uid": "CrcIhj4lIsSLpPwPe5",
> "version": "TLSv12",
> "cipher": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
> "curve": "secp256r1",
> "server_name": "clr.telemetry.intel.com",
> "resumed": false,
> "established": true,
> "cert_chain_fuids": [
> "FpfU1u4yMxOBNKQJk2",
> "FYiimM3T6A5AKcTeze",
> "FfAQxKi177b5fl95a",
> "FWHqmB2emMx082lxO1"
> ],
> "client_cert_chain_fuids": [],
> "subject": "CN=c-all.telemetry.intel.com,OU=Multi-Domain
> SSL,OU=Hosted by Intel Corporation,OU=SSG,O=Intel Corporation,street=2200
> Mission College Blvd,street=FM1-110,L=Santa
> Clara,ST=California,postalCode=95054-1537,C=US",
> "issuer": "CN=COMODO RSA Organization Validation Secure Server
> CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB",
> "validation_status": "ok",
> "server_cert_md5": "27ba7ede8ceab7c896ee4b6bcb509f63",
> "server_cert_sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21",
> "pfs": true,
> "ja3": "f436b9416f37d134cadd04886327d3e8",
> "ja3s": "303951d4c50efb2e991652225a6f02b1",
> "sourceipaddress": "10.48.74.16",
> "sourceport": 58860,
> "destinationipaddress": "52.37.21.110",
> "destinationport": 443,
> "sourceipv4address": "10.48.74.16",
> "destinationipv4address": "52.37.21.110",
> "destinationipgeolocation": {
> "city": "Boardman",
> "continent": "NA",
> "country_code": "US",
> "country_name": "United States",
> "dma_code": 810,
> "latitude": 45.8491,
> "longitude": -119.7143,
> "metro_code": "Boardman, OR",
> "postal_code": "97818",
> "region_code": "OR",
> "time_zone": "America/Los_Angeles"
> },
> "destinationipgeopoint": "45.8491,-119.7143"
> },
> "customendpoint": "bro",
> "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
> "tags": "bro",
> "category": "bro",
> "source": "ssl",
> "utctimestamp": "2019-11-08T07:02:20.253057+00:00",
> "timestamp": "2019-11-08T07:02:20.253057+00:00",
> "receivedtimestamp": "2019-11-08T07:02:21.465325+00:00",
> "eventsource": "nsm",
> "severity": "INFO",
> "mozdefhostname": "mozdef7.private.mdc1.xxxxxxxxxxxxxxxxxxxx.com",
> "summary": "SSL: 10.48.74.16 -> 52.37.21.110:443",
> "plugins": [
> "customDocType",
> "broFixup",
> "ipFixup",
> "geoip"
> ],
> "processid": "UNKNOWN",
> "processname": "UNKNOWN"
> },
> "fields": {
> "receivedtimestamp": [
> "2019-11-08T07:02:21.465Z"
> ],
> "timestamp": [
> "2019-11-08T07:02:20.253Z"
> ],
> "utctimestamp": [
> "2019-11-08T07:02:20.253Z"
> ]
> },
> "highlight": {
> "category": [
> "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
> ]
> },
> "sort": [
> 1573196540253
> ]
> }
>
> I could now paste the third (?!) leg of this connection - after SNAT, but
> you get the idea.
>
> Unfortunately, this sort of visibility partially goes away soon, due to
> some pseudo-privacy mechanisms AKA a classic shoot in the foot, while
> trying to kill a mosquito. From a shotgun. <- my private opinion BTW. I'm
> well
> after hours ;)
>
> On Thu, Nov 7, 2019 at 10:00 AM Konrad Weglowski <
> konrad.weglowski at gmail.com> wrote:
>
>> Thank You Michał
>>
>> Client to Proxy would not be encrypted. So based on that I will have all
>> the logs, however my destination IP for proxied connections would be proxy
>> IP (that was my understanding). I guess ingesting proxy logs would then
>> provide full visibility as you mentioned. I am thinking maybe if for some
>> reason I cannot get the proxy logs ingested and also tap after the proxy, I
>> could possibly correlate both ends of the connections based on URI/SNI/etc
>> and same timestamps in my analytics platform. Anyways I will give that a
>> try as well. Thanks for your help again.
>>
>> Konrad
>>
>>
>>
>> On Wed, Nov 6, 2019 at 6:59 PM Michał Purzyński <
>> michalpurzynski1 at gmail.com> wrote:
>>
>>> There is no way for Zeek to really correlate if the proxy does
>>> everything at the application level, like Squid. The proxy
>>> terminates the connection and then it's free to start the second leg,
>>> to the destination host, as it wishes to.
>>>
>>> Is your traffic from clients to proxy encrypted as well?
>>>
>>> If the traffic between clients are the proxy is not encrypted (as it's
>>> usually the case) then even if the client will establish a TLS session
>>> _through_ the proxy you will see a destination with the "service" field
>>> http,ssl and all the conn + ssl + http logs.
>>>
>>> We have that + proxy logs and that's pretty much a full visibility.
>>>
>>>
>>>
>>> On Wed, Nov 6, 2019 at 1:11 PM Konrad Weglowski <
>>> konrad.weglowski at gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I need to monitor web traffic with Zeek that is going through an
>>>> implicit web proxy. I would like to be able to see real client IPs in which
>>>> case tap would be placed before the proxy, however I will not be able to
>>>> see the real destination IP as it will always be proxy IP. I can also tap
>>>> on the other side of the proxy where source will always be proxy IP at the
>>>> same time.
>>>>
>>>> Is there a way with Zeek to correlate the two sessions (before and
>>>> after proxy) somehow?
>>>>
>>>> I realize that UID or Community ID are probably not going do it since
>>>> different source/destinations pairs for each for each session, but maybe
>>>> based on the timestamp/URI/SNI/byte or packet counts/etc?
>>>>
>>>> I was also thinking "X-Forwarded-For" header but I guess that would
>>>> only work for HTTP and not HTTPS connections.
>>>>
>>>> Thank You,
>>>>
>>>> Konrad
>>>> _______________________________________________
>>>> Zeek mailing list
>>>> zeek at zeek.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191112/63377ea2/attachment-0001.html
More information about the Zeek
mailing list