MongoDB Logs: Basics, What to Monitor, & Tools to Analyze

MongoDB is a popular NoSQL database that is document-oriented and cross-platform. Unlike relational databases like MySQL, MongoDB stores data in JSON-like documents with optional schemas, allowing data storage and usage without the need for relationships or tables.

It is quicker than SQL databases, simple to scale, and written in C++. Its open-source nature and superior performance make it a top choice for big businesses. MongoDB is free to use and eliminates the need to manage relationships and scalability.

MongoDB logs are valuable for diagnosing issues, monitoring deployments, and fine-tuning performance. Tools like mtools provide free analysis of MongoDB log messages, aiding in effective database management.

What is MongoDB, & How Does It Store Data?

Technically speaking, MongoDB is a document-oriented NoSQL database, meaning that documents are used in a schema to describe models. MongoDB stores documents (objects) in a format called BSON.

In addition, its primary selling factors include simple indexing, replication, ad-hoc queries, schema lessness, cross-platform support, and superior performance.

Consider a document an object whose keys and values fall under the same taxonomy. Here is an example of a JSON-encoded document model for a typical user and its corresponding BSON:

Here is a JSON document example.

{
"hello” : "world"
}

MongoDB is innovative by incorporating this structure into a new format known as BSON, a binary representation of JSON data optimizing the speed, schema flexibility, and governance, resulting in enhanced scalability. 

When storing the JSON document, it will be converted to the following.

\x16\x00\x00\x00             // total document size
\x02                         // 0x02 = type String
hello\x00                    // field name
\x06\x00\x00\x00world\x00    // field value (size of value, value, null terminator         
\x00                         // 0x00 = type EOO ('end of object')

MongoDB partitions data by breaking up more enormous shards into smaller pieces which can be performed automatically or manually, depending on certain conditions.

What Are MongoDB Logs?

• When you use MongoDB, it records some messages in its logs, just like any other database, including MySQL. 
• It is helpful to be aware of the issue before it arises and causes any negative impact. You can fix the problem in advance if you are aware of it.
• You can find information in the logs that, when used properly, can save time and effort in case of issues. 

What Types of Event Messages Should You Monitor in MongoDB Logs?

• MongoDB encodes numerous log messages. You will not use every log message to address your application’s issue. You must know what information from MongoDB logs is crucial. 
• You’ll primarily search for log messages like fatal, error, warning, and debug.
• According to the official MongoDB documentation, the log messages have various levels ranging from fatal to debug, where debug is the lowest level. Beginning with MongoDB 4.4, all log messages are output by mongod/mongos instances in structured JSON format. 
• The log entry format is a set of key-value pairs, where each key denotes a field type of the log message, like “severity,” and each corresponding value contains the logging data related to that field type, like “informational.”

What are MongoDB Logs Message?

• You can find MongoDB logs in the MongoDB log files at /var/log/mongodb/mongodb.log or else; check the mongodb.conf.- a configuration file specifying the location of logs.
• The log file structure is the first thing you’ll need to understand. The log file is a JSON structure, as described earlier. Each key in a key-value combination that makes up a log entry identifies a different sort of log message field, such as “severity.” 
• For each appropriate value, such as “informational,” the accompanying logging data gets recorded. Log entries appear as plain text, which isn’t always readable.

What are mtools?

You can use Unix commands as a MongoDB log analyzer to do a fair analysis. However, several tools make log analysis more accessible. mtools is a fantastic open-source utility.

Suppose that slow queries executing against MongoDB are hurting the performance of the database; you can determine where MongoDB is slowing down by using mtools.

mtools is a collection of Python utility scripts to analyze and filter MongoDB log files, display information from log files, and rapidly build up sophisticated local MongoDB test setups. Look at the “queries” section of mloginfo first.
Use command – mloginfo mongod.log –queries

Each line shows the namespace, the query pattern, and statistics of a particular namespace/pattern combination. The rows get sorted by the “sum” column, descending. Sorting by sum is an excellent way to see where the database spent most of its time.

Further mlogfilter assists in refining log file searches. The script allows you to filter log messages based on their namespace (database and collection names), operation (queries, inserts, updates, commands, etc.), and individual connection.

Lastly, mplotqueries takes a log file (mlogfiltered or not) and visualizes the data in various ways. Several graph kinds are available, including scatter plots (showing all operations over time versus their length), histograms, event and range plots, and other specific graphs such as connection churn and replica set changes.

Visualizing MongoDB Logs File with mplotqueries

mtools can provide visualization of query performance and other operations graphically. mplotqueries’ scatter plot shows the duration of any operation (y-axis) over time (x-axis), thus making it easy to spot long-running operations. The following plot gets generated with

mplotqueries mongod.log

and then press L for the “logarithmic” y-axis view:

The blue dots stand out, showing hundreds and thousands of seconds, unlike others which are sub-seconds—clicking on any of the blue dots prints out the relevant log line to stdout for analysis and action.

Conclusion

• Similar to other log messages, MongoDB logs are pretty helpful for resolving problems. You may use the terminal to analyze the log file by displaying its contents.
• Even though this is functional, it is difficult to view or search data through the terminal. It becomes considerably worse when the log file contains important data.
• The commands included in mtools MongoDB log analyzers, such as mloginfo, mlogfilter, mplotqueries, mlogvis, and mlaunch, may help you do more with a tool like mtools.
• This program has several limitations, including the inability to show data on the terminal, despite its vast command set. You may also explore commercial mongodb log analyzer tools for MongoDB log analysis if required.