MongoDB Filtering: A Comprehensive Guide 101

With the exponentially growing semi-structured and unstructured data on the web, it becomes essential for businesses to look out for a robust database management system. MongoDB has become the popular choice as a NoSQL & document-oriented database for handling this type of data. As you try to query, aggregate & analyze data from MongoDB, it becomes critical to optimize your queries to get faster results with minimum computational overhead.

One solution is to perform MongoDB Filtering which returns only the documents that match your specified condition. You can also execute MongoDB Filtering on MongoDB Compass Filter to leverage the powerful GUI tool. Other MongoDB products such as MongoDB Atlas App Services also allow you to carry out MongoDB filtering and optimize your query performance.  

In this article, you will learn about MongoDB Filtering and how you can apply filters to your queries.

Table of Contents

• What is MongoDB?
  • Key Features of MongoDB
• Why do you need MongoDB Filtering?
• MongoDB Filtering: 3 critical aspects
  • MongoDB Filtering: MongoDB Filter Aggregation Operator
  • MongoDB Filtering: MongoDB Compass
  • MongoDB Filtering: MongoDB Realm
• Conclusion

MongoDB is a popular Free and Open-Source Cross-Platform Document Oriented Database built for efficiently storing and processing massive volumes of data. Unlike traditional Relational Databases, MongoDB is classified as a NoSQL Database Management System that uses Collections and JSON-like Documents instead of tables consisting of rows and columns. Each collection consists of multiple documents that contain the basic units of data in terms of key and value pairs. 

Officially introduced as an Open-source development model in 2009, the MongoDB database is designed, maintained, and managed by MongoDB.Inc under a combination of the Server Side Public License and the Apache License. MongoDB is widely used by organizations such as MetLife, Barclays, Viacom,  New York Times, Facebook, Nokia, eBay, Adobe, Google, etc to efficiently meet their exponentially growing data processing and storage requirements. MongoDB is highly flexible as it supports several programming languages such as C, C++, C#, Go, Java, Node.js, Perl, PHP, Python, Motor, Ruby, Scala, Swift, and Mongoid.

Key Features of MongoDB

MongoDB offers a wide range of eye-catching features:

• High Data Availability: MongoDB’s Replication feature provides multiple servers for disaster recovery and backup. Since several servers store the same data or shards of data, MongoDB provides greater Data Availability & Stability. You don’t need to worry about cases of server crashes, service interruptions, or even good old hardware failure as Real-time Replication of data ensures uninterrupted data access and security. 
• Supercharged Analytics: You may need to consider thousands to millions of variables while running Ad-hoc queries. MongoDB indexes BSON documents and utilizes the MongoDB Query Language (MQL) that allows you to update Ad-hoc queries in real-time. MongoDB offers full support for field queries, range queries, and regular expression searches along with user-defined functions.
• Indexing: With Multiple Indices and Language-specific Sort Order Features to support complex data set access patterns, MongoDB provides optimal query performance. For the real-time ever-evolving query patterns and application requirements, MongoDB also provides On-Demand Indices Creation.
• Horizontal Scalability: Using Sharding, MongoDB provides horizontal scalability by using shard keys to distribute data across multiple server combinations. Each shard in every MongoDB Cluster stores portions of the data, thereby acting as a separate database. This collection of comprehensive databases allows efficient handling of growing volumes of data with zero downtime. The complete Sharding Ecosystem is monitored and managed by Mongos that direct queries to the correct shard based on the Shard Key.
• Load Balancing: Real-time Replication and Sharding greatly influences large-scale Load Balancing. Ensuring top-notch Concurrency Controls and Locking Protocols, MongoDB can effectively handle several concurrent read and write requests for the same data.  
• Aggregation: Similar to the SQL Group By clause, MongoDB can easily batch process data and present a single result even after executing several other operations on the group data. MongoDB’s Aggregation framework consists of 3 types of aggregations i.e. Aggregation Pipeline, Map-Reduce Function, and Single-Purpose Aggregation methods.

Why do you need MongoDB Filtering?

By applying a filter to your query, you will be essentially modifying the MongoDB query to return only a subset of the results matched by the query. These filters include additional query parameters that allow you to omit fields from your query. You can apply filters to:

• Optimize queries & minimize compute overhead.
• Secure sensitive data by omitting the critical or unused fields.
• Restrict queries to a subset of all documents.

MongoDB Filtering: 3 Critical Aspects

To effectively understand MongoDB Filtering, you can go through the following aspects:

• MongoDB Filtering: MongoDB Filter Aggregation Operator
• MongoDB Filtering: MongoDB Compass
• MongoDB Filtering: MongoDB Realm

MongoDB Filtering: MongoDB Filter Aggregation Operator

With version 3.2, you can start MongoDB Filtering using the new $filter aggregation operator.  This operator selects a subset of an array based on the condition you specify and will return it. It is to be noted that this array of elements is in the original order. You can use the following syntax for the $filter operator:

{ $filter: { input: <array>, as: <string>, cond: <expression> } }

The above-mentioned syntax consists of the following parameters:

• input: This points to the array from which you want to filter out the data.
• as: This is an optional parameter that represents the variable name of an element in your input array. If you don’t specify a name for it, “this” is assigned as the default name.
• cond: You can use this parameter to define whether an element of an array should be included in the query result or not. The filter operator individually references each element of your input array with the variable name you have provided in the As parameter. 

You can use the following conditions in the cond parameter for MongoDB Filtering: 

• Greater than ((>) or $gt)
• Less than ((<) or $lt)
• Greater than equal to ((>=) or $gte)
• Less than equal to ((<=) or $lte)

To understand MongoDB Filtering, you can consider the sample collection inventory that has the following documents:

{
   _id: 101,
   products: [
     { prod_id: 56, quantity: 10, cost: 200 },
     { prod_id: 3, quantity: 5, cost: 340 }
   ]
}
{
   _id: 102,
   products: [
     { prod_id: 40, quantity: 4, cost: 120 },
     { prod_id: 20, quantity: 5, cost: 3 },
     { prod_id: 30, quantity: 2, cost: 200 }
   ]
}
{
    _id: 103,
    products: [
       { prod_id: 4, quantity: 3, cost: 25 }
    ]
}

Using the $filter operator for MongoDB filtering, you can retrieve the products whose cost is greater than 100. 

Query: 

db.inventory.aggregate([
   {
      $project: {
         products: {
            $filter: {
               input: "$products",
               as: "prod",
               cond: { $gte: [ "$$prod.cost", 100 ] }
            }
         }
      }
   }
])

Output: 

{
   "_id" : 0,
   "products" : [
     { prod_id: 56, quantity: 10, cost: 200 },
     { prod_id: 3, quantity: 5, cost: 340}
   ]
}
{
   "_id" : 1,
   "products" : [
      {  prod_id: 40, quantity: 4, cost: 120 },
      {prod_id: 30, quantity: 2, cost: 200 }
   ]
}
{ "_id" : 2, "products" : [ ] }

MongoDB Filtering: MongoDB Compass

An effective method to query, aggregate, and analyze your MongoDB data is by using a powerful GUI like MongoDB Compass. It is completely free to use and is compatible with macOS, Windows, and Linux. You can simply specify the filter document in the Filter field of the Query bar and click Run to execute the query. For instance, you can consider an example query for MongoDB filtering that returns documents that have a Country value of Brazil:

It is to be noted that the Compass Filter used for MongoDB Filtering supports the mongo shell mode representation of the MongoDB Extended JSON BSON data types. For instance, you can consider the following filter returns documents where start_date is greater than the BSON Date 2020-07-04:

{ "start_date": {$gt: new Date('2020-07-04')} }

By specifying the Date type in both start_date and the $gt comparison operator, Compass performs the greater than comparison chronologically, returning documents with start_date later than 2020-07-04.

However, when you try to execute the query without the Date type, MongoDB Compass compares the start_dates as strings lexicographically, instead of comparing the values chronologically.

MongoDB Filtering: MongoDB Atlas App Services

MongoDB Atlas App Services allows you to easily write and host an application in a completely managed cloud ecosystem. This cloud environment includes Atlas Device Sync, serverless cloud functions, declarative data access rules, and more. A filter in the App services consists of the following components:

• Apply When: You can use this expression to check if the filter applies to an incoming request. You can use variables such as %%user and %%request, though you cannot use MongoDB variables like %%root as the App Services applies your filters before it reads any data.
• Optional Query Document: You can provide the standard MongoDB query syntax for this query. This Query document merges with the existing query of any request the filter applies to.
• Optional Projection Document: This document utilizes the standard MongoDB projection syntax and merges with the existing projection of any request the filter applies to.

To understand the MongoDB Filtering for the App Services, consider the following sample collection:

{ "_id": ObjectId(...), "name": "suraj", age: 34, "vote": "no", "shareVoteAnonymous": true }
{ "_id": ObjectId(...), "name": "anand", age: 56, "vote": "yes", "shareVoteAnonymous": true }
{ "_id": ObjectId(...), "name": "jenny", age: 44, "vote": "no", "shareVoteAnonymous": false }
{ "_id": ObjectId(...), "name": "rob", age: 52, "vote": "no", "shareVoteAnonymous": true }
{ "_id": ObjectId(...), "name": "tim", age: 28, "vote": "no", "shareVoteAnonymous": false }
{ "_id": ObjectId(...), "name": "bain", age: 54, "vote": "yes", "shareVoteAnonymous": true }

Filter: 

{
  "name": "AnonymizeVotes",
  "apply_when": true,
  "query": {
    "shareVoteAnonymous": true
  },
  "project": {
    "_id": 0,
    "age": 1,
    "vote": 1
  }
}

Output:

{ age: 34, "vote": "no" }
{ age: 56, "vote": "yes" }
{ age: 44, "vote": "no" }
{ age: 52, "vote": "no" }
{ age: 28, "vote": "no" }
{ age: 54, "vote": "yes" }

Conclusion

In this article, you have learned about MongoDB Filtering and how you can effectively filter out documents in MongoDB. MongoDB provides the $filter aggregation operator that allows you to filter out your documents and extract the fields you want based on the condition you have specified. You can also MongoDB Compose to easily query your MongoDB data in a visual environment by simply providing the filter in the Query bar. MongoDB Atlas App Services also allows you to filter the incoming MongoDB query. The filter can add extra query parameters and omit fields from query results before App Services runs the query. 

As you collect and manage your data filter in MongoDB and across several applications and databases in your business, it is important to consolidate it for complete performance analysis of your business. However, it is a time-consuming and resource-intensive task to monitor the Data Connectors continuously. To achieve this efficiently, you need to assign a portion of your engineering bandwidth to Integrate data from all sources, Clean & Transform it, and finally, Load it to a Cloud Data Warehouse, BI Tool, or a destination of your choice for further Business Analytics. All of these challenges can be comfortably solved by a Cloud-based ETL tool such as Hevo Data.   

Visit our Website to Explore Hevo

Hevo Data, a No-code Data Pipeline can replicate data in Real-Time from a vast sea of 100+ sources like MongoDB & MongoDB Atlas to a Data Warehouse, BI Tool, or a Destination of your choice. It is a reliable, completely automated, and secure service that doesn’t require you to write any code!  

If you are using NoSQL Database Management Systems like MongoDB & MongoDB Atlas and searching for a no-fuss alternative to Manual Data Integration, then Hevo can effortlessly automate this for you. Hevo, with its strong integration with 100+ sources and BI tools(Including 40+ Free Sources), allows you to not only export & load data but also transform & enrich your data & make it analysis-ready in a jiffy.

Want to take Hevo for a ride? Sign Up for a 14-day free trial and simplify your Data Integration process. Do check out the pricing details to understand which plan fulfills all your business needs.

Share your experience of executing MongoDB Filtering! Let us know in the comments section below!