In security we have preventative and detective controls, and you're encouraged to use both. The problem with the detective controls is that nobody reads the logs let alone try to interprete the results. I've been involved in building NOC and SOC environments in the last 10 years and the recurring error is what are we going to log? Everything. If you have the money, resources and computer power like amazon or google you can. If you don't, try to filter that what is interesting. Sounds easy but it is not. So people buy expensive correlation engines, SIEM solutions or other tools alike. Which is good, but you need intelligence and intelligence does not comes out of green, orange or even a black box. It comes from people, experience, bright minds, situations, reports or even statistics.
Pattern detection can be automated to a certain extend but needs to be interpreted by human minds and the hard thing is that if humans look over endless long log files you become not only numb but you do not see the patterns anymore. If you look at a stream of black balls for minutes and we throw in a coloured ball in every now and then you will not detect it. Same problem with our pattern detection.
If we narrow down the log amounts to something useful and search for a pattern for a very specific service, attack vector, action, software or error we increase the success rate. Sounds simple no? My question, why don't we ?
Because for whatever reason we're spending our budgets on fancy security devices that generate the logs but not on those that can handle the logs.
I'll be back for more cynical views on security!