In order to better detect compromised and malicious users the market for log management and security analytics is shifting away from vertically integrated SIEM vendors to a split architecture consisting of log repository vendors and security analytics vendors. The success of log repository vendors enables a new breed of start-ups who can focus on security analytics to improve threat detection without having to deal with log management.

For the last ten years organizations have been collecting and analyzing the logs of security controls, systems, and applications in order to (1) meet compliance requirements, (2) perform forensics, and (3) detect compromised devices and malicious users.

Vertically integrated Security Information and Event Management (SIEM) solutions have dominated the market. They perform best at providing compliance reports and are fair at forensics. Regarding threat detection, they have performed poorly due mainly to their primary form of analysis – rules. We understand that some vendors have added basic statistical techniques like mean/standard deviation. However, the threat detection problem we are trying to solve is not Gaussian.

For evidence, just review the Mandiant M-Trends and Verizon’s DBIR reports. The 2013 Mandiant M-Trends report says that “63% of victims were notified by an external entity,” and the median time from earliest evidence of compromise to discovery of compromise (dwell time) was 243 days!!

Here is what has to be a stunning chart from page 54 of the 2013 Verizon DBIR report below. For large organizations, Log Review as the discovery method accounted for only 4%.  I’ll be generous and add the 4% found by NIDS, since they can bubble up into the log system. That means 92% of breaches were detected by means other than log analysis!!

Verizon 2013 DBIR Figure 44 - Breach Discovery Methods

More shocking is the dwell time issue. Here is the Verizon DBIR chart from page 55. Of all of the breaches analyzed, only four were discovered in minutes. The mode was months!!

Verizon 2013 DBIR Figure 45 - Breach count by discovery method and time to discovery

In true Innovator’s Dilemma fashion, we are seeing independent security analytics software companies leveraging the modular architecture offered by companies like Splunk. (While Splunk does offer a security analytics application , it’s priced separately. Where SIEM solutions are vertically integrated, we are seeing a split architecture gaining popularity. One group of vendors provides log repository management (log collection, parsing and enrichment , storage, and query functionality) with not only APIs but also language SDKs, that enable another group of vendors to focus on analytics and visualization. And we surely are seeing new companies forming who are focused on improving the detection of compromised and malicious users through the use of advanced analytics without having to deal with log management issues.

These next-generation analytic solutions are more adept at detecting compromised and malicious users because their algorithms are non-deterministic. Rule-based systems are deterministic in the sense they will only alert on predefined situations. In other words, if you don’t know the question to ask, you will not get an answer. Unsupervised machine learning algorithms do not need learning periods or thresholds. They can find anomalous behavior that you have not modeled.

In conclusion, enterprises will be able to select a log management solution without locking themselves into a single vendor’s analytics capabilities.  And enterprises will be able to avail themselves of new and innovative analytics solutions much more quickly and at lower cost since they will not have to change their log management infrastructure.

If you have a question or a comment, or would like more information, please let us know by completing the Contact Us box on the right side of this page.