Searching your logfiles and your knowledge management sources

A friend of mine pointed me to Splunk for log file analysis, thanks for that :-)

I havnt had a chance to install and try Splunk, but looking around, Splunk could be the util to combine knowledge management searches with real time event searches from servers. A single point of entry for searching is crucial, but not easy to up and running in the day to day use.

To benefit from a search engine, that engine should be able to reach all the different places that people put knowledge. And it must be able to crawl all file formats, eg. Open office, MS office, excel, pdf etc. We can get the file indexing working from all kinda places, but the hurdle seems to be indexing mailboxes! The example being a public mailbox archive of all the support answers to customers, with many years of useful knowledge! Indexing mailboxes, eg. Lotus Notes, should be possible with enterprise search engines like Google and Yahoo Omnifind.

For logfile analysis, i usually stick with simple tools ala fetchlog, our own grep scripts on centralized syslog servers, and some OSSEC. Other utils I have played with for correlating of information is prelude.

Perhaps Splunk can combine the above (search engine and logfile analysis) into one application?

Splunk provides a free edition, so I will keep it around, in case I get a chance to try it :-) It sure seems worth a try for an enterprise! Of course, being an open source and community fan, I am more biased toward an open source alternative for Splunk? Prelude and OSSEC are both open source free software.

While looking around I stumbled upon an interesting open source site, Softpanorama.org:

Mission and Vision Statement This is a self-education oriented site (see
about for more info) that contains resources for the independent study in
computer science and programming. The latter is the area were open source really
shines: the academic value of open source software (OSS) cannot be
overestimated.

Softpanorama.org has some Splunk entries in their Log Analyzers News:

[Apr. 17, 2006] Splunk Welcome
Splunk is search software that
imitates Google search engine functionality on logs. Can be considered as
the first specialized log search engine. It can correlate some
alerts:
Splunk Splunk User's Guide
Splunk Administrator's Guide

[Feb 16, 2006] Splunk, Nagios partner on open-source systems-monitoring tools
Log file search and indexing software vendor Splunk Inc. announced Tuesday that it will soon add systems
management host, network and service monitoring capabilities to its software
through a partnership with the Nagios open-source project. ...