Differences

This shows you the differences between two versions of the page.

--- admin:debuggingcrash [2018/10/03 17:18] – [Where to look] veera
+++ admin:debuggingcrash [2018/10/03 18:50] (current) – [Debugging with a binary with symbols] veera
@@ Line 1: / Line 1: @@
 ====== Debugging crashes and other problems on the Trisul Probe nodes ======
-Trisul probes are running on hundreds of different networks and are generally very stable.  We also put them though fuzz tests and mangled packets to test the stability. However, it is possible that sometimes the Probes stop. This document describes how you can debug such events.
+Despite our best efforts with testing , sometimes the Trisul-Probes can stop for a number of reasons. This document describes how you can debug such events.
 ===== Where to look =====
-If you find Trisul Probe has crashed, you can collect the following information before restarting and see if you find some clues.
+If you find Trisul Probe has crashed, you can collect the following information before restarting.  If nothing jumps out , you can contact [[https://www.trisul.org/support/|Trisul Support]] and send us the logs.
-  - Trisul probe log files
-  - Version number of Trisul Probe
-  - Platform (Ubuntu/CentOS version)
-  - dmesg output
-  - Some information about the frequency of crashes
 ==== Log files ====
-The log files on the trisul probe are stored under ''/usr/local/var/log/trisul-probe/domain0/probe0/context0''  The latest ''ns-xxx.log'' contains the most recent logs before the crash.
+The log files on the trisul probe are stored under ''/usr/local/var/log/trisul-probe/domain0/probe0/context0''  The latest ''ns-xxx.log'' contains the most recent logs before the crash. You can also use the [[https://www.trisul.org/docs/ref/trisbashrc.html|trisbashrc bash aliases]] to navigate the directories as shown below:
-Another option is to use our helper aliases
 <code bash>
+# add the helper aliases
 source /usr/local/share/trisul-probe/trisbashrc
-# to change to log dir
+# to change to probe log dir
 cd.l
-# to tailf log file
+# to tailf probe log file
 tailf.ns
+# the latest log file
+cd.l
+ls -lrt ns*
 </code>
+When contacting Trisul Support send us the latest log file ns-xxx.log
+==== dmesg : view messages from the kernel ====
+Sometimes useful clues can be found in the kernel message log.  For example, when trisul is stopped by an OOM - Out of Memory killer.  To view dmesg output
+<code bash>
+# to dump the kernel message log
+dmesg -T
+# to view trisul messages
+dmesg -T | grep trisul
+</code>
+==== Software versions ====
+To get probe version  ''trisul --version''
+To get platform version ''uname -a''
+To get installed packages on probe
+<code bash>
+# on ubuntu
+dpkg -l | grep trisul
+# on centos
+rpm -qa | grep trisul
+</code>
+===== Debugging with a binary with symbols =====
+If we find nothing in the Trisul and Kernel log files, we (Trisul Support) will send you a special binary image with symbols. You can then run it under ''gdb'' to help pinpoint the location of the crash. This could be a bug
+<note tip>We recommend you run this under the 'screen' utility, so you can detach the terminal and log off.
+</note>
+<code>
+# install gdb
+apt install gdb
+# run the special trisul binary we send you like this
+gdb --args /home/mike3/trisul -nodemon \
+    /usr/local/etc/trisul-probe/domain0/probe0/context0/trisulProbeConfig.xml \
+       -mode online_rxring
+(...wait...)
+# after it crashes - type bt on the gdb prompt
+gdb:  bt
+..output here.. send us this output
+</code>