To prepare pancakes (my favorite dish) you can find a lot of recipes on the internet. Some are more complicated to prepare other more easy, some tasty others not. My favor pancake recipe is really simple to prepare, it's composed of only few basic ingredients and it tastes all. When introducing a central log management infrastructure to a productive system it's also a question of ingredients driven by requirements you have to meet for your environment, business etc.
Alike for pancakes I like simple approaches to start small and grow as you go. Thus the underlying recipe for a central log management setup is quite pragmatic and easy to introduce. It consists of the following elements:
Capturing logs on each component in the infrastructure and forwarding them in near real-time to a central log destination server using syslog as the most popular, simple and very fast log transport protocol.
Storing received logs as plain files on the central log destination server and retaining them for some weeks.
Enabling access to centrally stored log files using logsniffer. It provides an end to end web interface for parsing, viewing, tailing and searching log data.
Monitoring logs for events of special interest with optional alerting to channels like mail, http etc.
Centralizing / forwarding logs
Your infrastructure will be usually composed of several components like application servers, web servers, databases etc. deployed in the cloud or on-premise on multiple nodes. Each generates logs which have to be centralized and forwarded to the dedicated central log destination server for further processing. The most popular log transport protocol is syslog. It's the standard logging solution on Unix-like systems. Support for Windows platforms is also available. The modern syslog implementations like Rsyslog or syslog-ng are usually installed by default in common distributions of Linux. These support collecting logs from any source and delivering them to a wide variety of destinations ...Read More