[Zeek] monitoring proxied web traffic
Michał Purzyński
michalpurzynski1 at gmail.com
Fri Nov 8 00:21:05 PST 2019
You're welcome!!
If the traffic from the client to the proxy is not encrypted and the proxy
is not "transparent" (meaning you configured it on the client) then you
have the same case as I do here.
So there's this host
source IP 10.48.75.81
trying to talk to clr.telemetry.intel.com
via our proxy at 10.48.122.10 / 10.48.74.16 / 10.48.74.17
*(for anyone working at a place where disclosing an internal IP or an FQDN
equal incident - you know a lot of great places are hiring, right?)*
I used the "uid" to correlate this connection across three types of logs -
conn.log, http.log and ssl.log. You're gonna get them due to the way
proxies operate.
Please ignore tons of metadata from our SIEM (MozDef = custom Python + pure
ES), I'm going to C&P three logs as they are without any redactions. This
post is HUGE!!
*conn.log* - host -> proxy
{
"_index": "events-20191108",
"_type": "_doc",
"_id": "JmLRSW4BiWsbe3JQtC8S",
"_version": 1,
"_score": null,
"_source": {
"type": "nsm",
"details": {
"ts": 1573196540.178897,
* "uid": "C9PFt711Odfwpbr0Ti",*
* "proto": "tcp",*
* "service": "http,ssl",*
"duration": 0.183212,
"orig_bytes": 1604,
"resp_bytes": 7180,
"conn_state": "SF",
"local_orig": true,
"local_resp": true,
"missed_bytes": 0,
* "history": "ShADadFf",*
* "orig_pkts": 14,*
* "resp_pkts": 13,*
"peer": "nsm2-af_packet-enp18s0f0-8",
* "orig_l2_addr": "b4:0c:25:e0:40:10",*
* "resp_l2_addr": "00:50:56:a1:48:64",*
* "sourceipaddress": "10.48.75.81",*
* "sourceport": 55330,*
* "destinationipaddress": "10.48.122.10",*
* "destinationport": 3128,*
* "originipbytes": 2340,*
* "responseipbytes": 7864,*
"sourceipv4address": "10.48.75.81",
"destinationipv4address": "10.48.122.10"
},
"customendpoint": "bro",
"hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
"tags": "bro",
"category": "bro",
* "source": "conn",*
* "utctimestamp": "2019-11-08T07:02:20.178897+00:00",*
"timestamp": "2019-11-08T07:02:20.178897+00:00",
"receivedtimestamp": "2019-11-08T07:02:27.686061+00:00",
"eventsource": "nsm",
"severity": "INFO",
"mozdefhostname": "mozdef5.private.mdc1.xxxxxxxxxx.com",
"summary": "10.48.75.81:55330 -> 10.48.122.10:3128 ShADadFf 2340 bytes
/ 7864 bytes",
"plugins": [
"customDocType",
"broFixup",
"ipFixup",
"geoip"
],
"processid": "UNKNOWN",
"processname": "UNKNOWN"
},
"fields": {
"receivedtimestamp": [
"2019-11-08T07:02:27.686Z"
],
"timestamp": [
"2019-11-08T07:02:20.178Z"
],
"utctimestamp": [
"2019-11-08T07:02:20.178Z"
]
},
"highlight": {
"summary": [
"@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@:55330
-> 10.48.122.10:3128 ShADadFf 2340 bytes / 7864 bytes"
],
"details.sourceipv4address": [
"@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@"
],
"details.uid": [
"@kibana-highlighted-field at C9PFt711Odfwpbr0Ti
@/kibana-highlighted-field@"
],
"category": [
"@kibana-highlighted-field at bro@/kibana-highlighted-field@"
]
},
"sort": [
1573196540178
]
}
*http.log* (because the connection from the client to the proxy was
technically a http connection)
{
"_index": "events-20191108",
"_type": "_doc",
"_id": "jPDRSW4BC2j90LU9UdgL",
"_version": 1,
"_score": null,
"_source": {
"type": "nsm",
"details": {
"ts": 1573196540.202888,
* "uid": "C9PFt711Odfwpbr0Ti",*
"trans_depth": 1,
"method": "CONNECT",
*"host": "clr.telemetry.intel.com <http://clr.telemetry.intel.com>",
"uri": "clr.telemetry.intel.com:443 <http://clr.telemetry.intel.com:443>",*
"version": "1.1",
"request_body_len": 0,
"response_body_len": 0,
* "status_code": 200,*
* "status_msg": "Connection established",*
"tags": [],
"proxied": [
"PROXY-CONNECTION -> Keep-Alive"
],
*"sourceipaddress": "10.48.75.81", "sourceport": 55330,
"destinationipaddress": "10.48.122.10", "destinationport": 3128,*
"sourceipv4address": "10.48.75.81",
"destinationipv4address": "10.48.122.10"
},
"customendpoint": "bro",
"hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
"tags": "bro",
"category": "bro",
* "source": "http",*
* "utctimestamp": "2019-11-08T07:02:20.202888+00:00",*
"timestamp": "2019-11-08T07:02:20.202888+00:00",
"receivedtimestamp": "2019-11-08T07:02:21.427537+00:00",
"eventsource": "nsm",
"severity": "INFO",
"mozdefhostname": "mozdef7.private.mdc1.xxxxxxxxxx.com",
"summary": "HTTP CONNECT 10.48.75.81 -> 10.48.122.10:3128",
"plugins": [
"customDocType",
"broFixup",
"ipFixup",
"geoip"
],
"processid": "UNKNOWN",
"processname": "UNKNOWN"
},
"fields": {
"receivedtimestamp": [
"2019-11-08T07:02:21.427Z"
],
"timestamp": [
"2019-11-08T07:02:20.202Z"
],
"utctimestamp": [
"2019-11-08T07:02:20.202Z"
]
},
"highlight": {
"summary": [
"HTTP CONNECT @kibana-highlighted-field at 10.48.75.81
@/kibana-highlighted-field@ -> 10.48.122.10:3128"
],
"details.sourceipv4address": [
"@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@"
],
"details.uid": [
"@kibana-highlighted-field at C9PFt711Odfwpbr0Ti
@/kibana-highlighted-field@"
],
"category": [
"@kibana-highlighted-field at bro@/kibana-highlighted-field@"
]
},
"sort": [
1573196540202
]
}
And the *ssl.log* - over the previously established http connection the
client told proxy to establish a tunnel to the destination host - and Zeek
parsed that as TLS (find me another IDS/NSM that can do that)
{
"_index": "events-20191108",
"_type": "_doc",
"_id": "sfDRSW4BC2j90LU9S8_Q",
"_version": 1,
"_score": null,
"_source": {
"type": "nsm",
"details": {
"ts": 1573196540.253057,
* "uid": "C9PFt711Odfwpbr0Ti", "version": "TLSv12", "cipher":
"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "curve": "secp256r1",
"server_name": "clr.telemetry.intel.com <http://clr.telemetry.intel.com>",
"resumed": false, "established": true, "cert_chain_fuids": [
"FknnGH2L2UIKGERW3b", "FGtbhK1HoQMWdgyKW",
"F8JbGM30JvqFVi41A6", "Fsc2fRRTyTDQrT1Gc" ],
"client_cert_chain_fuids": [],*
*"subject": "CN=c-all.telemetry.intel.com
<http://c-all.telemetry.intel.com>,OU=Multi-Domain SSL,OU=Hosted by Intel
Corporation,OU=SSG,O=Intel Corporation,street=2200 Mission College
Blvd,street=FM1-110,L=Santa
Clara,ST=California,postalCode=95054-1537,C=US", "issuer": "CN=COMODO
RSA Organization Validation Secure Server CA,O=COMODO CA
Limited,L=Salford,ST=Greater Manchester,C=GB", "validation_status":
"ok", "server_cert_md5": "27ba7ede8ceab7c896ee4b6bcb509f63",
"server_cert_sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21", "pfs":
true, "ja3": "f436b9416f37d134cadd04886327d3e8", "ja3s":
"303951d4c50efb2e991652225a6f02b1", "sourceipaddress": "10.48.75.81",
"sourceport": 55330, "destinationipaddress": "10.48.122.10",
"destinationport": 3128,* "sourceipv4address": "10.48.75.81",
"destinationipv4address": "10.48.122.10"
},
"customendpoint": "bro",
"hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
"tags": "bro",
"category": "bro",
"source": "ssl",
"utctimestamp": "2019-11-08T07:02:20.253057+00:00",
"timestamp": "2019-11-08T07:02:20.253057+00:00",
"receivedtimestamp": "2019-11-08T07:02:21.464921+00:00",
"eventsource": "nsm",
"severity": "INFO",
"mozdefhostname": "mozdef6.private.mdc1.xxxxxxxxxx.com",
"summary": "SSL: 10.48.75.81 -> 10.48.122.10:3128",
"plugins": [
"customDocType",
"broFixup",
"ipFixup",
"geoip"
],
"processid": "UNKNOWN",
"processname": "UNKNOWN"
},
"fields": {
"receivedtimestamp": [
"2019-11-08T07:02:21.464Z"
],
"timestamp": [
"2019-11-08T07:02:20.253Z"
],
"utctimestamp": [
"2019-11-08T07:02:20.253Z"
]
},
"highlight": {
"summary": [
"SSL: @kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@
-> 10.48.122.10:3128"
],
"details.sourceipv4address": [
"@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@"
],
"details.uid": [
"@kibana-highlighted-field at C9PFt711Odfwpbr0Ti
@/kibana-highlighted-field@"
],
"category": [
"@kibana-highlighted-field at bro@/kibana-highlighted-field@"
]
},
"sort": [
1573196540253
]
}
Not enough? Let's look for the certificate's MD5
{
"_index": "events-20191108",
"_type": "_doc",
"_id": "_CwBSm4BC2j90LU9vfNt",
"_version": 1,
"_score": null,
"_source": {
"type": "nsm",
"details": {
"ts": 1573199701.992707,
"id": "FdE5qt1b93z9QOJnz1",
"san.dns": [
"c-all.telemetry.intel.com",
"c.telemetry.intel.com",
"c1.telemetry.intel.com",
"c2.telemetry.intel.com",
"c3.telemetry.intel.com",
"c4.telemetry.intel.com",
"c5.telemetry.intel.com",
"c6.telemetry.intel.com",
"c7.telemetry.intel.com",
"c8.telemetry.intel.com",
"c9.telemetry.intel.com",
"clr.telemetry.intel.com",
"sur.telemetry.intel.com"
],
"md5": "27ba7ede8ceab7c896ee4b6bcb509f63",
"sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21",
"certificate": {
"basic_constraints_ca": false,
"exponent": "65537",
"issuer": "CN=COMODO RSA Organization Validation Secure Server
CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB",
"key_alg": "rsaEncryption",
"key_length": 2048,
"key_type": "rsa",
"not_valid_after": "2020-08-02T23:59:59+00:00",
"not_valid_before": "2018-08-03T00:00:00+00:00",
"sig_alg": "sha256WithRSAEncryption",
"subject": "CN=c-all.telemetry.intel.com,OU=Multi-Domain
SSL,OU=Hosted by Intel Corporation,OU=SSG,O=Intel Corporation,street=2200
Mission College Blvd,street=FM1-110,L=Santa
Clara,ST=California,postalCode=95054-1537,C=US",
"version": 3,
"serial": "1406C0A906762390B7252C9598F2DB3C"
}
},
"customendpoint": "bro",
"hostname": "nsm1-ber3.private.ber3.xxxxxxxxxx.com",
"tags": "bro",
"category": "bro",
"source": "x509",
"utctimestamp": "2019-11-08T07:55:01.992707+00:00",
"timestamp": "2019-11-08T07:55:01.992707+00:00",
"receivedtimestamp": "2019-11-08T07:55:17.040196+00:00",
"eventsource": "nsm",
"severity": "INFO",
"mozdefhostname": "mozdef1.private.mdc1.xxxxxxxxxx.com",
"summary": "X509 certificate seen",
"plugins": [
"customDocType",
"broFixup"
],
"processid": "UNKNOWN",
"processname": "UNKNOWN"
},
"fields": {
"details.certificate.not_valid_after": [
"2020-08-02T23:59:59.000Z"
],
"receivedtimestamp": [
"2019-11-08T07:55:17.040Z"
],
"details.certificate.not_valid_before": [
"2018-08-03T00:00:00.000Z"
],
"timestamp": [
"2019-11-08T07:55:01.992Z"
],
"utctimestamp": [
"2019-11-08T07:55:01.992Z"
]
},
"highlight": {
"details.md5": [
"@kibana-highlighted-field at 27ba7ede8ceab7c896ee4b6bcb509f63
@/kibana-highlighted-field@"
],
"source": [
"@kibana-highlighted-field at x509@/kibana-highlighted-field@"
],
"category": [
"@kibana-highlighted-field at bro@/kibana-highlighted-field@"
]
},
"sort": [
1573199701992
]
}
Let's try to splice the first and the second half of the connection
category:bro AND source:dns AND details.query:clr.telemetry.intel.com
clr.ats.infra-host.com, 52.37.21.110, 34.208.200.27
Back to looking at conn.log with destination IP like above and source =
proxies (from there you can again do the crazy dance with ssl.log but
there's no need to)
{
"_index": "events-20191108",
"_type": "_doc",
"_id": "jvDRSW4BC2j90LU9UdgL",
"_version": 1,
"_score": null,
"_source": {
"type": "nsm",
"details": {
"ts": 1573196540.253057,
"uid": "CrcIhj4lIsSLpPwPe5",
"version": "TLSv12",
"cipher": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"curve": "secp256r1",
"server_name": "clr.telemetry.intel.com",
"resumed": false,
"established": true,
"cert_chain_fuids": [
"FpfU1u4yMxOBNKQJk2",
"FYiimM3T6A5AKcTeze",
"FfAQxKi177b5fl95a",
"FWHqmB2emMx082lxO1"
],
"client_cert_chain_fuids": [],
"subject": "CN=c-all.telemetry.intel.com,OU=Multi-Domain
SSL,OU=Hosted by Intel Corporation,OU=SSG,O=Intel Corporation,street=2200
Mission College Blvd,street=FM1-110,L=Santa
Clara,ST=California,postalCode=95054-1537,C=US",
"issuer": "CN=COMODO RSA Organization Validation Secure Server
CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB",
"validation_status": "ok",
"server_cert_md5": "27ba7ede8ceab7c896ee4b6bcb509f63",
"server_cert_sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21",
"pfs": true,
"ja3": "f436b9416f37d134cadd04886327d3e8",
"ja3s": "303951d4c50efb2e991652225a6f02b1",
"sourceipaddress": "10.48.74.16",
"sourceport": 58860,
"destinationipaddress": "52.37.21.110",
"destinationport": 443,
"sourceipv4address": "10.48.74.16",
"destinationipv4address": "52.37.21.110",
"destinationipgeolocation": {
"city": "Boardman",
"continent": "NA",
"country_code": "US",
"country_name": "United States",
"dma_code": 810,
"latitude": 45.8491,
"longitude": -119.7143,
"metro_code": "Boardman, OR",
"postal_code": "97818",
"region_code": "OR",
"time_zone": "America/Los_Angeles"
},
"destinationipgeopoint": "45.8491,-119.7143"
},
"customendpoint": "bro",
"hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
"tags": "bro",
"category": "bro",
"source": "ssl",
"utctimestamp": "2019-11-08T07:02:20.253057+00:00",
"timestamp": "2019-11-08T07:02:20.253057+00:00",
"receivedtimestamp": "2019-11-08T07:02:21.465325+00:00",
"eventsource": "nsm",
"severity": "INFO",
"mozdefhostname": "mozdef7.private.mdc1.xxxxxxxxxxxxxxxxxxxx.com",
"summary": "SSL: 10.48.74.16 -> 52.37.21.110:443",
"plugins": [
"customDocType",
"broFixup",
"ipFixup",
"geoip"
],
"processid": "UNKNOWN",
"processname": "UNKNOWN"
},
"fields": {
"receivedtimestamp": [
"2019-11-08T07:02:21.465Z"
],
"timestamp": [
"2019-11-08T07:02:20.253Z"
],
"utctimestamp": [
"2019-11-08T07:02:20.253Z"
]
},
"highlight": {
"category": [
"@kibana-highlighted-field at bro@/kibana-highlighted-field@"
]
},
"sort": [
1573196540253
]
}
I could now paste the third (?!) leg of this connection - after SNAT, but
you get the idea.
Unfortunately, this sort of visibility partially goes away soon, due to
some pseudo-privacy mechanisms AKA a classic shoot in the foot, while
trying to kill a mosquito. From a shotgun. <- my private opinion BTW. I'm
well
after hours ;)
On Thu, Nov 7, 2019 at 10:00 AM Konrad Weglowski <konrad.weglowski at gmail.com>
wrote:
> Thank You Michał
>
> Client to Proxy would not be encrypted. So based on that I will have all
> the logs, however my destination IP for proxied connections would be proxy
> IP (that was my understanding). I guess ingesting proxy logs would then
> provide full visibility as you mentioned. I am thinking maybe if for some
> reason I cannot get the proxy logs ingested and also tap after the proxy, I
> could possibly correlate both ends of the connections based on URI/SNI/etc
> and same timestamps in my analytics platform. Anyways I will give that a
> try as well. Thanks for your help again.
>
> Konrad
>
>
>
> On Wed, Nov 6, 2019 at 6:59 PM Michał Purzyński <
> michalpurzynski1 at gmail.com> wrote:
>
>> There is no way for Zeek to really correlate if the proxy does everything
>> at the application level, like Squid. The proxy
>> terminates the connection and then it's free to start the second leg, to
>> the destination host, as it wishes to.
>>
>> Is your traffic from clients to proxy encrypted as well?
>>
>> If the traffic between clients are the proxy is not encrypted (as it's
>> usually the case) then even if the client will establish a TLS session
>> _through_ the proxy you will see a destination with the "service" field
>> http,ssl and all the conn + ssl + http logs.
>>
>> We have that + proxy logs and that's pretty much a full visibility.
>>
>>
>>
>> On Wed, Nov 6, 2019 at 1:11 PM Konrad Weglowski <
>> konrad.weglowski at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I need to monitor web traffic with Zeek that is going through an
>>> implicit web proxy. I would like to be able to see real client IPs in which
>>> case tap would be placed before the proxy, however I will not be able to
>>> see the real destination IP as it will always be proxy IP. I can also tap
>>> on the other side of the proxy where source will always be proxy IP at the
>>> same time.
>>>
>>> Is there a way with Zeek to correlate the two sessions (before and after
>>> proxy) somehow?
>>>
>>> I realize that UID or Community ID are probably not going do it since
>>> different source/destinations pairs for each for each session, but maybe
>>> based on the timestamp/URI/SNI/byte or packet counts/etc?
>>>
>>> I was also thinking "X-Forwarded-For" header but I guess that would only
>>> work for HTTP and not HTTPS connections.
>>>
>>> Thank You,
>>>
>>> Konrad
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191108/b3c98ee7/attachment-0001.html
More information about the Zeek
mailing list