Differences

This shows you the differences between two versions of the page.

--- app:auto_fingerprint [2017/11/29 17:00] – created vivek
+++ app:auto_fingerprint [2017/11/29 22:58] (current) – [Web Server Access Log] veera
@@ Line 2: / Line 2: @@
-TLS Fingerprinting is still in its early days therefore the coverage of known prints is not too deep.  The Trisul [[app:tlsfingerprint|TLS Fingerprint App]] ships with a known fingerprint database of about 500 entries but you will likely find anywhere from 50-70% unknown fingerprints. We need a way to resolve prints to client ID.
+TLS Fingerprinting is still in its early days therefore the coverage of known prints is not too deep.  The Trisul [[app:tlsfingerprint|TLS Fingerprint App]] ships with a known fingerprint database of about 500 entries ([[https://github.com/trisulnsm/trisul-scripts/blob/master/lua/frontend_scripts/reassembly/ja3/prints/ja3fingerprint.json|ja3prints.json]]) but you might find anywhere from 50-70% unknown fingerprints in your network. We need a way to resolve prints to client ID and build up the mappings.
-Internally we use a Ruby TRP script that can automate this process. The script is available at [[https://github.com/trisulnsm/trisul-scripts/tree/master/lua/frontend_scripts/reassembly/ja3/prints|mk_ja3print.rb]]
-===== How this works =====
+===== Resolving TLS Fingerprints =====
+Some of the techniques of resolving unknown fingerprints
+  - If you have access to a Web Server log  - Look at the ''User-Agent'' field
+  - If not, see if you can find a ''User-Agent'' vertex nearby, for example if the client followed a HTTP 302 Redirect to a https site.
+  - Look at Hosts using the fingerprint, see if you can detect a pattern or an application. Many web scanners, bots, and applications like Git, Dropbox can be detected this way.
+===== Web Server Access Log =====
+Internally we use a Ruby TRP script that can automate this process if given access to web server logs. The script is available on Github at [[https://github.com/trisulnsm/trisul-scripts/tree/master/lua/frontend_scripts/reassembly/ja3/prints|mk_ja3print.rb]]
+The script isnt too complicated. It works in the following way.
+  - Connects and gets list of unresolved JA3 TLS Prints in a 24 hour period.
+  - For Each unresolved print
+    - Use Graph Analytics to get list of Hosts using it
+    - Look for the Host in the Web Server Access log and pick out the first User-Agent
+    - Look for the print in the TLS Print App log - this contains the print and the Print String
+    - Print it out in JSON format
+Running the script.
+//Usage : mk_ja3fingerprint.rb  TRP-Server-Endpoint  Webserver-IP  Webserver-Access-Logs  Trisul-TLSPrint-Log-Pattern//
+A sample run of the script is shown below
+<code>
+$ ruby mk_ja3fingerprint.rb  tcp://74.207.234.90:12006  138.68.45.27    'trisul_access.log*'  'lua.stdout.jahash.lua.11*'
+"Found 29 Unresolved JA3 TLS Prints"
+"Sending EdgeGraph request vertex key=35c0a31c481927f022a3b530255ac080"
+"35c0a31c481927f022a3b530255ac080 resolved to  Mozilla/5.0 (compatible; RSiteAuditor)  771,49192-159-158-157-156-49195-49187-49191-49172-49171-61-60-53-47-49196-49188-49162-49161-106-64-56-50-10-19-5-4,65281-0-10-11-13-35,23-24,0"
+"Sending EdgeGraph request vertex key=37f691b063c10372135db21579643bf1"
+"37f691b063c10372135db21579643bf1 resolved to  urlgrabber/3.10 yum/3.4.3  771,49196-49162-49195-52393-49161-49200-49172-49199-52392-49171-159-57-56-107-158-52394-51-50-103-22-19-157-53-61-156-47-60-10-5-4,0-65281-10-11-13,29-23-24-25,0"
+"Sending EdgeGraph request vertex key=c2769dbd398f0b72e409887ceb9eb8ad"
+"Sending EdgeGraph request vertex key=05af1f5ca1b87cc9cc9b25185115607d"
+"05af1f5ca1b87cc9cc9b25185115607d resolved to  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1  769,49172-49171-57-51-53-47-49162-49161-56-50-10-19-5-4,0-10-11-23-65281,23-24,0"
+"Sending EdgeGraph request vertex key=1885aa9927f99ed538ed895d9335995c"
+"1885aa9927f99ed538ed895d9335995c resolved to  Mozilla/55 (Windows NT 10.0; WOW64; rv:55.0) Gecko/20100101 Firefox/55  771,49195-49199-158-49162-49161-49171-49172-49159-49169-51-50-57-156-47-53-10-5-4-255,0-11-10-35-13-15,14-13-25-11-12-24-9-10-22-23-8-6-7-20-21-4-5-18-19-1-2-3-15-16-17,0-1-2"
+"Sending EdgeGraph request vertex key=05e15a226e00230c416a8cdefeb483c7"
+"Sending EdgeGraph request vertex key=1543a7c46633acf71e8401baccbd0568"
+"Sending EdgeGraph request vertex key=f22bdd57e3a52de86cda40da2d84e83b"
+..
+"Output written to file /tmp/fingerprint.json"
+</code>
+Once the script is finished, the JSON output is written to ''/tmp/prints.json'' this can be easily appended to the TLS Prints database.
+<code json>
+{"desc":"RSiteAuditor","ja3_hash":"35c0a31c481927f022a3b530255ac080","ja3_str":"771,49192-159-158-157-156-49195-49187-49191-49172-49171-61-60-53-47-49196-49188-49162-49161-106-64-56-50-10-19-5-4,65281-0-10-11-13-35,23-24,0"}
+{"desc":"urlgrabber/3.10 yum/3.4.3","ja3_hash":"37f691b063c10372135db21579643bf1","ja3_str":"771,49196-49162-49195-52393-49161-49200-49172-49199-52392-49171-159-57-56-107-158-52394-51-50-103-22-19-157-53-61-156-47-60-10-5-4,0-65281-10-11-13,29-23-24-25,0"}
+{"desc":"Feedly/1.0","ja3_hash":"f22bdd57e3a52de86cda40da2d84e83b","ja3_str":"771,49188-49192-61-49190-49194-107-106-49162-49172-53-49157-49167-57-56-49187-49191-60-49189-49193-103-64-49161-49171-47-49156-49166-51-50-49196-49195-49200-157-49198-49202-159-163-49199-156-49197-49201-158-162-49160-49170-10-49155-49165-22-19-255,10-11-13-0,23-24-25-9-10-11-12-13-14-22,0"}
+</code>
+Iteratively running this script for a few days can resolve most of the unknown prints. That makes outlier detection much easier.
+===== Other methods to resolve =====
+Once you get the unknown prints down to 10-20% you can use Trisul's excellent Graph Analytics manually to explore and nail down each print.  We will see that in another article.