Log management refers to managing the log events via log management tools. It involves log collection, log parsing, searching, and log archiving, amongst others. While there are many log management tools available to choose from, users should pick a log management tool based on factors such as capabilities of the tool, features, pricing, organization size, use cases, and legal or regulatory obligations. Log management tools also play a quintessential role in intrusion detection. Companies can leverage the log data and address hostile activities beforehand. Companies with hybrid infrastructure and multiple applications also need to ensure outages are resolved quickly and there’s almost 0% downtime. Log files and log management tools are also critical to addressing incidents.
In this blog, we’ve outlined a detailed comparison between a few standard features of Papertrail™, Datadog, and Loggly®, but first, let’s understand the basics of these tools.
1. Papertrail
SolarWinds® Papertrail consolidates logs at a central location, making it easier for users to diagnose and resolve issues quickly. Users can back up and archive log data along with controlling global retention policies through its intuitive UI. Papertrail provides support for various log formats and parses logs automatically while helping eliminate the requirement of grep or AWK to view, search, or tail events in real time from a single UI. A few of its featured languages include IIS Log Viewer, NGINX Logging, Windows Syslog Server, Linux Log Management, and JavaScript Logging.
2. Loggly
Loggly is a cloud-based service designed to automatically structure and summarize the log data across the stack. It lets users send logs and metrics from multiple sources and is capable of integrating with the software development life cycle. Users can also create a Jira ticket based on a specific log data within Loggly. Its unique dynamic field explorer capability provides a structured summary of the parsed logs, allowing users to answer key questions, spot trends, and track SLA compliance. It allows users to create charts in the form of percentiles, averages, and more through structured, semi-structured, and unstructured log data. Loggly also allows users to create specific parsing rules.
3. Datadog
Datadog lets users see across their apps, systems, and services with turnkey aggregations. It offers more than four hundred built-in integrations. It allows users to track performance with auto-generated service views and a comfortable navigation facility across logs, metrics, and request traces.
Now let’s take a deep dive into some standard features these tools offer.
Log Parsing
Datadog automatically parses JSON formatted logs. For other formats, it lets users augment the records via the Grok Parser, helping them extract attributes from semi-structured text messages. Papertrail supports JSON, MySQL, Windows Event Log, Common Log Format, and W3C Extended Log Format, amongst others. Loggly, on the other hand, is continually working to integrate more log types into the ecosystem and recognizes Apache, JAVA, JSON, Nginx, NodeJS, PHP, Rails, and Python, amongst others. Loggly automatically parses many kinds of data, which allows users to perform advanced functions like statistical analysis on advanced values and faceted search. In case automated parsing isn’t available for a log type, Loggly lets users log and perform a full-text search, and is thus a better choice than the other two tools.
IIS Log Management
IIS Log File format is a fixed text-based format in the form of ASCII text-based format wherein users cannot customize the file format. It logs more data than other log formats and contains information like IP and username, service status, request date, and time. Getting a centralized view of IIS log management is a difficult task. Papertrail provides a single view of all IIS log files, which helps discover data patterns and trends. They’re searchable by time, date, IP address, and more in real time and eliminate the need to sift through individual log files spread across the environment. Users can forward logs to Papertrail by NXlogs, or event log-to-Syslog. Datadog lets users analyze IIS log data, graph it, and aggregate it by any facet. Users can also monitor IIS metrics and logs in a single place and can configure Datadog to get alerts when IIS encounters performance issues. Loggly offers advanced IIS logging capabilities including indexing, parsing, organizing, and visualizing the IIS logs to solve problems faster. Users can send IIS logs to Loggly via NXlog, which converts them to JSON format after extracting each field in the IIS logs. Loggly hence provides great value from IIS logs as it helps solve problems and proactively prevents them. It also has advanced IIS logging options and can be considered as a better choice for users.
Live Tail Feature
The Datadog live tail displays the number of events received per second and a sampling rate with high throughput log streams. These log streams can be filtered, paused, or played. It also offers the option of customizing the log tail views. The Papertrail live tail capability gives users a real-time view of events. These logs can be paused and resumed. The time seek feature is also available for users. The live tail version is available in Loggly on the command line version and browser version and supports abilities like real-time updates, pattern-based filtering, and color-coding. Users can also group similar events and identify the patterns quickly. Users can also arrange the live tails as per theme or grouping settings where theme settings imply dark and light mode, and grouping settings indicate exact match and string match. Live tail in Loggly is also available as a command-line interface (CLI) program for Mac OS X, Linux, and other Unix-type and Windows operating systems. Users can also put the Loggly live tail into an interactive mode to display matching patterns, and send live tail output to third-party messaging and collaboration tools like HipChat and Slack.
Conclusion
Even though both Papertrail and Datadog offer extensive log management features, the Loggly features can help organizations achieve much more through fast and powerful searching over massive volumes of log data. It needs no proprietary agents, and dozens of log sources are visible at a single location. Users can also analyze the log data through charts, filters, and range searches, and monitor the logs by setting up alerts and dashboards. We suggest users check out the free versions of these tools and their pricing plans before making a choice.
*As of June 2020