CSAM Month of False Positives: Appropriately Weighting False and True Positives, (Fri, Oct 31st)

This is a guest diary submitted by Chris Sanders. We will gladly forward any responses or please use our comment/forum section to comment publicly.”>”>If you work with any type of IDS, IPS, or other”>detection technology then you have to deal with false positives. One”>common”>mistake I see people make when managing their indicators and rules is”>relying”>solely on the rate of false positives that are observed. While false”>positive”>rate is an important data point, it doesnt encompass everything you”>should”>consider when evaluating the effectiveness of a rule or indicator. For”>instance, consider a scenario where you have a rule that looks for a”>specific”>”>alert tcp $HOME_NET any – $EXTERNAL_NET any”>(msg:Random Malware content:|AB BF 09″>B7|”>”>You can see that this rule isnt incredibly”>specific as it examines all TCP traffic for four specific outbound bytes.”>As a”>result, there might be potential for false positives here. In this case, I”>ran”>this rule on a large network over the course of a month, and it generated”>58″>false positive alerts. Using that data point alone, it sounds like this”>rule”>might not be too effective. As a matter of fact, I had a few people who”>asked”>me if I could disable the rule. However, I didnt because I also”>considered the”>number of true positive alerts generated from this rule. Over the same”>period of time this rule generated 112 true positive alerts. This means”>that the rule was effective at catching what it was looking for, but it”>still”>wasn”>”>I mention the word precise, because the false”>positive”>and true positive data points can be combined to form a precision”>statistic”>using the formula P = TP + (TP + FP). This value, expressed as a”>percentage,”>can be used to describe exactly how precise a rule is, with higher values”>being”>more desirable. In the case of our example rule, the rule has 65.9%”>precision,”>meaning that it successfully detected what it was looking for 65.9% of the”>time. That doesnt sound like a rule that should be disabled to me.”>Instead, I”>was able to conduct more research and further tune the rule by looking for”>the”>”>When examining rules and indicators for their effectiveness, be sure”>to consider both true and false positives. You might miss out on favorable”>detection if you don”>”>Blogs:”>”>http://www.chrissanders.org

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

Reposted from SANS. View original.