Upload Files in MongoDB with Mongoose GridFS & NodeJS: 3 Easy Steps

If you want to store big data files into a database without worrying about conforming to structures and schemas, MongoDB is an excellent option for you. Thanks to features like GridFS, even extremely large data files have a place on MongoDB. This tutorial will show you how to upload files easily into MongoDB with a NodeJS application that uses a Mongoose GridFS Prometheus.

Overview of MongoDB, Mongoose GridFS & NodeJS

MongoDB is a non-relational database manager used to store big data files. Unlike SQL databases which sort data into rows and columns, MongoDB stores data in the form of documents and collections.

Although MongoDB supports various data types, the most suitable data form for the database manager is JSON (Javascript Object Notation). In MongoDB, JSON files are stored in binary formats called Binary JSON or BSON. Each BSON document can only occupy about 16MB worth of memory.

For one reason or another, you may want to store a big data file that is larger than 16MB in MongoDB. This is where GridFS comes in. Grid FS is a feature in MongoDB that breaks data files more than 16MB into smaller parts. This lets you store your data file as multiple documents in MongoDB.

Node.JS is a runtime environment that lets developers run Javascript code outside a web browser. With Node.JS, programmers can write commands for their websites and create unique responses for user requests on their web pages. So, different users may see various web pages when they log on to your website. 

Mongoose is an Object Model Library for MongoDB. This means that it allows developers to attribute schema and structures to data in MongoDB documents. You can also use Mongoose to convert code in Node.JS to data representations in MongoDB.

Read along to understand the steps to load data into MongoDB using Mongoose GridFS and NodeJS.

Steps to Upload Files into MongoDB Using Mongoose GridFS Prometheus

Here are the steps to take when uploading a file into MongoDB using Mongoose GridFS Prometheus:

• Step 1: Build a Node.JS Application
• Step 2: Upload Your Files into FileU 
• Step 3: Storing Your Files in MongoDB through Grid FS

Step 1: Build a Node.JS Application

We need a Node.JS application to run the code for uploading our files into MongoDB. You can create a Node.JS application with your npm package. 

The packages you need to run the Node.JS application are third-party modules like mongoose. That said, you have to install these packages into npm to use them. Before you start installing these packages, you need to create a node project, and configure it to npm. Follow these steps to build your node project:

• Create a folder on your computer and navigate to it. You can quickly do this by running a command line on your Command Prompt tool. Let’s assume that the title of the folder is ‘building npm project’. Type the following in Command Prompt:

mkdir building-npm-project (to create the folder)
cd building-npm-project (to navigate to the folder)

• Enter ‘npminit’ command in Command Prompt to configure the project folder to npm.
• After creating the node project, you need to install the packages you need for building your node.js application. These packages are:
  • Mongoose (This package will translate the node.JS code to MongoDB)
  • Express (You’ll need this package for any HTTP requests you want to run)
  • BodyParser (This package lets you receive content from HTML forms)
  • Multer (This package enables easy file upload into MongoDB
  • Multer-gridfs-storage (You need this package to implement the MongoDB gridfs feature with multer).

• Apart from the packages above, you also need to install nodemon as a dependency for your node.js application. 
• Enter the following command line to install any of the packages into npm:

 C : NodeProject> npm install  <package name>

For instance, you would write this code to install mongoose into npm:

C: building-npm-project> npm install <mongoose>

• You should also use this command line to install express and multer packages. The command line for multer-gridfs-storage is different from the other packages in this section. To install multer-gridfs-storage, you need to enter the code below:

npm   i  -S  express mongoose dotenv multer multer-gridfs-storage

• Finally, install nodemon as a dependency for your Node.JS application:

npm  i   --save-dev  nodemon

• Once you have installed the essential packages and dependencies for your Node.JS application, you can start building the app. Enter the command line below to create the Node.JS application:

const express = require("express");
const app = express();
const mongoose = require("mongoose");
const multer = require("multer");
const {
  GridFsStorage
} = require("multer-gridfs-storage");
 
require("dotenv")
  .config();
 
const mongouri = 'mongodb+srv://User1:' + process.env.MONGO_PASS + '@cluster0.wakey.mongodb.net/myFirstDatabase?retryWrites=true&w=majority';
try {
  mongoose.connect(mongouri, {
    useUnifiedTopology: true,
    useNewUrlParser: true
  });
} catch (error) {
  handleError(error);
}
process.on('unhandledRejection', error => {
  console.log('unhandledRejection', error.message);
});
 
//creating bucket
let bucket;
mongoose.connection.on("connected", () => {
  var db = mongoose.connections[0].db;
  bucket = new mongoose.mongo.GridFSBucket(db, {
    bucketName: "newBucket"
  });
  console.log(bucket);
});
 
//to parse json content
app.use(express.json());
//to parse body from url
app.use(express.urlencoded({
  extended: false
}));
 
app.listen(process.env.PORT, function () {
  console.log(`Application live on localhost:{process.env.PORT}`);
});

This should set up the Node.JS application. After you finish building the application, the structure should contain the following: :

• File_U
• Node_Modules
• Views
• Index.js
• Package.json

Where, 

• File_U is the place where you upload your files into the node.js app;
• Node_modules contain the code that will communicate with MongoDB;
• Views contain your template files;
• index.js hold the HTML code for your command lines
• package.json contains all the packages used, such as multer and mongoose;
• app.js covers basic functionalities like routing. 

Step 2: Upload Your Files into FileU

After building your Node.JS tab, you need to upload your files into FileU. From FileU, your files will be submitted into MongoDB. 

Write the HTML code in the Index.js page to upload your files into FileU. If you are wondering what the “enctype = “multipart/form-data” means, you should know that the command line lets you send your files through a post into fileU. 

Now that you have uploaded your files into FileU, the next step is to store them in MongoDB through Mongoose GridFS.

Step 3: Storing Your Files in MongoDB through GridFS

By default, MongoDB only allows you to upload files below 16MB. But with GridFS, you can divide your file into different parts and upload each part as a separate file. GridFS divides large files into chunks of 255KB, with the final chunk only carrying the remnant data size.

GridFS stores your files in 2 collections: one collection holds the file chunk, while the other contains the metadata. This helps GridFS assemble all your file chunks into a single file when you query the system for a file. 

Remember all the packages you downloaded into npm? These packages will come into play when you are storing your files in MongoDB. View the code below to understand how the rest of the uploading process into MongoDB works:

const express = require('express');
const bodyParser = require('body-parser');
const path = require('path');
const mongoose = require('mongoose');
const multer = require('multer');
const GridFsStorage = require('multer-gridfs-storage');
const Grid = require('gridfs-stream');
const methodOverride = require('method-override');

const app = express();  

// Middleware
app.use(bodyParser.json());
app.use(methodOverride('_method'));
app.set('view engine', 'ejs');

// Mongo URI
const mongoURI = 'mongodb://localhost:27017/fileU';

// Create mongo connection
const conn = mongoose.createConnection(mongoURI);

// Init gfs
let gfs;

conn.once('open', () => {
  // Init stream
  gfs = Grid(conn.db, mongoose.mongo);  
  gfs.collection('uploads');
});

// Create storage engine
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
        const filename = file.originalname;
        const fileInfo = {
          filename: filename,
          bucketName: 'uploads'
        };
        resolve(fileInfo);
    });
  }
});

const upload = multer({ storage})

Express is the framework for uploading the files into MongoDB, bodyparser retrieves essential content from HTML forms; multer handles the file upload, and multer-gridfs-storage integrates GridFS with multer to let users store large files in MongoDB.

Now you have all the commands necessary to upload your files into MongoDB. But how do the files move from your Node.JS application into MongoDB? 

You need a command line for Mongoose to build a link between Node. JS and MongoDB. As seen in the code above, you have to declare a mongoURI to connect your node.JS app to MongoDB. The mongoURI is the location where your files will be stored in MongoDB. Your mongoURI is unique to your database. The mongoURI in the code above is merely a sample. 

How do you know what your Mongo URI is? Take the following steps to discover your MongoURI:

• Log on to the MongoDB website and sign in.
• Next, click on the Control Panel and find the application where you want to transfer the files.
• Select Database & Users to find the database that will hold the files. If you don’t have any databases in your target app, create a new database. You should also associate some users with this database.
• Open the database and select Overview in the side Menu bar.
• Look for your MongoURI under Connector Information.
• Once you know what your MongoURI is, you can input it into your code. 
• After entering the mongoURI, you will assign some variable gfs to the MongoDB Mongoose GridFS connection.
• Next, you have to build the storage engine. The components of the storage engine are the mongoURI and the file you want to upload. This file is returned through a Promise.
• Now, enter the filename and declare the upload function through multer, just like in the backend code above. 
• The final step is to create an upload route for your file. You can do this by using the following code:

app.post('/upload', upload.single('file'), (req, res) => {
  res.redirect('/');
});

The command, “upload,single( )” will break your files into chunks and store them in your MongoDB database.

Hurray! You have now uploaded your file to MongoDB using the Mongoose GridFS and NodeJS integration.

Conclusion

This post provided an overview of MongoDB, NodeJS, and Mongoose GridFS. You understood the steps to upload files into MongoDb using Mongoose GridFS and NodeJS. Now that you understand how MongoDB works, you can start uploading files on it without any hassle.

However, as a Developer, extracting complex data from a diverse set of data sources to your MongoDB Database can seem to be quite challenging. This is where a simpler alternative like Hevo can save your day! Hevo Data is a No-Code Data Pipeline that offers a faster way to move data from 150+ Data Sources such as MongoDB and other 60+ Free Sources, into your Data Warehouse to be visualized in a BI tool. Hevo is fully automated and hence does not require you to code.

Frequently Asked Questions