Log Management Tools (Graylog)
As you probably know by now keeping track of logs and alerts is a crucial part of security. Therefore, having tools that allows administrators to create an efficient way to manage their logs. The tool we are going to discuss in this post is Graylog, but first let’s look at Log Management Tools in general.
Log management (LM) is the action of dealing with large volumes of computer-generated log messages. These include messages such as; audit records, audit trails, event-logs, etc. and these all are then stored in a centralized location. This allows for ease of use for the administrator by enabling their ability to conduct; log collection, centralized log aggregation, long-term storage and retention, log rotation, log analysis, and log search/reporting. (https://en.wikipedia.org/wiki/Log_management)
For those of you who don’t know, log rotation is very similar to have if you save two documents with the same name it will append the name with a “(1)”. Well log rotation does a similar action by basically labeling each log with a number and then the administrator can set a max for how many of that type of logs should be saved. When it hits the max, it will begin to remove the oldest version. Administrators may also use log rotation to email administrators logs after the rotation has completed.
Now the following is from 2014 so it is certainly outdated but just seeing the four main components and the diagram you can kind of see how this tool works.
Graylog consists of four main components:
- Graylog Server Nodes: The server nodes receive and processes messages while communicating with all other non-server components. The performance of these nodes is dependent on the CPU.
- Elasticsearch Nodes: These nodes store all the logs and messages. The performance of these nodes is based off RAM and disk I/O (input/output or write/read)
- MongoDB: Graylog utilizes MongoDB to store metadata.
metadata – a set of data that describes and gives information about other data
- Web Interface: This interface provides easy access for the administrator to configure/access the Graylog tool.
Now, we will discuss the current Graylog and an overview of its futures. On the GrayLog website you can see the overview which I will be discussing below, in more detail, and dumbed down when needed. https://www.graylog.org/overview
Collect & Process – Collects logs, wire data, and event data from any data source while improving clarity as well as analyzing segments of the collected data. While this is a major part of the Graylog the other benefit is that Graylog provides a centralized configuration management for 3rd party collectors. Graylog utilizes a processing pipeline which allows for greater flexibility in routing, blacklisting, modifying, and enriching messages in real-time as they are transmitted to Graylog.
Pipeline – set of data processing elements connected in series. In the series the output of one element is the input of the next one. The elements of the pipeline are often executed in parallel or in time-sliced fashion. (https://en.wikipedia.org/wiki/Pipeline_(computing))
Analyze & Research – Can search through terabytes of log data to discover and analyze information. Graylog has a powerful search syntax to find exactly what you are looking for. The ability to save search queries to share results. With the search syntax an administrator can find application errors across all servers with a single query if configured accordingly. The administrator could also investigate the activity of a suspicious user ID within a timeframe. Discover a misconfigured firewall on your network. These are just some simple examples of what Graylog can complete in the Analyze & Research department but there is truly so much more.
Drill Down & Visualize – Create dashboards to allow for visualization of metrics and observe trends in one central location. Can also use field statistics, quick values, and charts from the search results page to achieve a deeper analysis of your data. The simple user interface enables other users to easily access the information and add new charts. An example of utilizing this tool could be finding all IP addresses that were blocked by a specific firewall. Another would be getting the average response time of your application components. Lastly, discover users with the most failed logins with the last 24 hours. As you can see the visualization and deeper analysis allows administrators to find crucial information regarding the administrator’s environment. Graphs and other visualization tools create a much easier environment for analyzing data.
Alert & Trigger – Can trigger actions or get notified when something needs attention. An example of this would be failed login attempts, exceptions, or performance degradation. An administrator can than configure their Graylog environment to send an email or slack message to his team. May also do actions such as spawn a new machine to balance the processing load or block IP ranges in your firewalls automatically when an attack is detected.
Enterprise Ready – Graylog has a huge amount of potential in the enterprise field. Luckily, Graylog takes this into consideration and ensures that the Graylog platform is enterprise ready. Below you can see the Compliance Ready details which show two key features that Graylog contains that allows it to be used safely in an enterprise environment.
Compliance Ready – (Features for corporation compliance)
- User Audit Log – Capabilities to audit log records and store actions taken by a user or administrator that makes changes to the Graylog system.
- Offline Log Archival – Ability to store everything older than 30days on slow storage and only re-import it into Graylog when you need it. This is a major feature for efficiency and storage.
- LDAP – Graylog can be integrated with existing LDAP (lightweight directory access protocol) user directories.
- REST API – Configuration settings as well as log data are available through the Graylog REST API (Representational State Transfer Application Programming Interface). This allows the ability to integrate Graylog seamlessly into evolving architecture as well as build your own reports and analysis. The REST API is the only dependency of the Graylog web interface which means guaranteed high quality and completeness.
- Outputs – Forward data to specialized systems or anything that could utilize a real-time stream of data. The example they use is forwarding metrics to a time series database such as an SIEM (Security Information and Event Management) or APM (Application performance management).