When writing files to disk in node you have to be careful. Many times file writes require insertion of large chunks of data over a (relatively) long period of time. The key thing to avoid is opening the file multiple times which is resource intensive.
It is far better to open the file once and hold that file handle in memory, ready to be used the next time. For that we use the createWriteStream async method.
createWriteStream async in Node fs
This method gives you ultimate control over file handling, which means more responsibility. Therefore you have to handle file opens, closes, errors and everything else that may go wrong.
Let’s first look at the single line of code needed to create a createWriteStream:
//cteate your sreite stream with optional optionsfs.createWriteStream (myFilePath,myOptions)------------------------------------------myOptionscanbeanyofthe following:options<Object>encoding<string> Default:'utf8'//file encodingautoClose<boolean> Default:true//autoclose after write? false = close it yourselfemitClose<boolean> Default:true//emit event to notify file closedstart<integer>//where should you insert the content?highWaterMark<number> Default:16384//how much data should this method gather before it's flushed to the fileflush<boolean>// File descriptor flushed prior to closing. Default = false.
Most of these will not be specified by you, just leave them as defaults. However, you may want to play with highWaterMark depending on your use case. That sets the buffer size and should be matched to the size / frequency of data streaming you have.
NOTE: Always use a resource monitor such as atop or htop to know if you have issues!
createWriteStream async Full Example
Before we start, note that to use createWriteStream in its async promise based version (instead of callback versions) you use the fs module and not the fs.promises module! I know, I know it’s counterintuitive!
The code below opens a write stream, writes something to a file on 2 separate calls then closes the file.
constfs=require('fs');//the main fs libtry{//always wrap in try catch as file writes can be error prone, especially on linux hosts...//create the writer object we can reuseletwriter=awaitfs.createWriteStream('log-stream.txt',{encoding:'utf8',highWaterMark:16384,flags:'a',//open and append (don't overwrite file)});//use the writer objectawaitwriter.write("Log 10: what happened in last 10s");awaitwriter.write('\r');//create a new line//sometime laterawaitwriter.write("Log 20: what happened in last 10s");//finally close the file to avoid memeroy leaks and file access issues!awaitwriter.close()}catch (err) {console.error('Error occurred while writing file:',err);}
Important notes: The above code shows a simplified step by step process to make it clear for you. To implement this in a real app, you would never write it this way, instead you’d create separate functions to handle file open, write and close – then call each function as you need them in your code.
createWriteStream Events and Listeners
FS gives us a lot of control over the file handling process, which means we can manually do everything ourselves. One of those options is choosing whether fs should close the file automatically or if we should listen to events and do it ourselves.
If your file write is long in duration, over multiple chunks, then you can tell FS to leave the file open, saving the overhead of opening it over and over.
This is perfect for server logs, for example. To do this we will listen to events called by createWriteStream. Here’s an example (code below):
Open the file writer with createWriteStream
Tell createWriteStream that we want to handle file closing ourselves (autoClose flag)
Register event listeners for open, ready and close
Write some text to a file
Use the listeners and callbacks to decide when to perform next actions
Close the writer once a write has been completed
Remove all writer listeners to avoid memory leaks!
Next, I have written the code in a step by step manner which is not realistic. You would separate this out in production to be called by the main program, as needed:
Example createWriteStream async Code (Events)
constfs=require('fs');//the main fs libtry{letwriter=awaitfs.createWriteStream("log-stream.txt",{encoding:"utf8",highWaterMark:16384,autoClose:false,emitClose:true,flags:"a"})//add a listener with options such as open, close, readyawaitwriter.addListener("open",async()=>{console.log("Log file OPEN!")})awaitwriter.addListener("ready",async()=>{console.log("Log file READY!")})awaitwriter.addListener("close",(fd)=>{console.log("Log file CLOSED")})writer.write("hello world \r\n",async()=>{awaitcloseWriter(writer)})}catch (err) {console.error('Error occurred while writing file:',err)}asyncfunctioncloseWriter(writer){awaitwriter.close(async()=>{awaitwriter.removeAllListeners() // important to avoid memory leaks!})}
Worth mentioning again: DO NOT forget to unsubscribe to your writer listeners on writer close or you’ll get memory leaks!
Summary
createWriteStream gives you power over all aspects of file writes but with great power comes great responsibility!
As long as you handle events appropriately and monitor resource usage then you should be able to avoid pitfalls of this method and retain complete control over your file writes.
The FIRST thing you do with any app is to create authentication for it, after all you will have users and they will need accounts!
In this article I will show you the basic idea with a standalone one file express server. We will use several popular off the shelf libraries such as express and express-session to hold the users authentication.
As this is a demo it’s not suitable for production so please do not use in a production app! See the last paragraph for things you’d have to do to make it production ready!
Setup (if already done then skip to ‘Start’ below)
Setting up a Node.js environment on Windows is a straightforward process. Here are the steps to get you started:
Download the LTS (Long-Term Support) version and follow the installation wizard.
2. Verify Installation:
Open a command prompt or PowerShell window.
Check that Node.js and npm (Node Package Manager) are installed by running the following commands: bash node -v npm -v
These commands should print the installed Node.js and npm versions, confirming a successful installation.
3. Install a Code Editor (Optional):
You can use any code editor or Integrated Development Environment (IDE) of your choice. Popular options include Visual Studio Code, Atom, or Sublime Text.
4. Create a Simple Node.js Application:
Create a new file for your Node.js project named index.js and open it in your code editor.
Start – Express Server
Define our Dependencies
The first step is to define the libraries we need. Put this into the top of your index.js file. The comments give you a basic understanding of what each one does.
constexpress=require('express') //popular server frameworkconstsession=require('express-session') //to store the session details on the clients browser (that get sent with every request)constbodyParser=require('body-parser') //to 'decode' the contents of the requestconstbcrypt=require('bcrypt') //enables hashed password storage and comparison. NEVER store plain text passwords!constapp=express() //creates an express server objectconstport=3000;//defines the port our app will use (localhost:3000 in this case)
Now open a terminal in VS Code (or whatever you use), navigate to your project folder and install the libraries by running:
npmi
Reading the Request Body – Middleware
When registration or login requests come in their data payloads will be basic strings. Use this line to intercept them and change to objects that our app can use much more easily.
app.use(bodyParser.urlencoded({extended:true}));
Once this step is defined then you’ll be able to get the request data using “req.body” from anywhere within your app.
Setting a Session Object & Mock Database
Next, we tell our app to use a session object which defines the secret key used to encrypt user data. This is found in the “your-secret-key” in the below code. Feel free to change the key to your liking.
Note: You should never, ever store this key in your code (although I have, for simple demo purposes). In production you would probably store this key in environment variables!
Following the session setting we will store our registered user in a simple array called “users”. Again, this is just for dev / demo purposes. In production this data would be stored in a database.
//set teh session keysapp.use(session({secret:'your-secret-key',resave:true,saveUninitialized:true}));// Sample in-memory database for storing usersconstusers= [];
Your server should start and you get a message in the console telling you which port the server is listening on.
From now on you’ll need to restart your server after each code change. Do this with Control + C to stop the current server and then rerunning node index.js.
Alternatively, you can download a library that watches your files for changes and auto reloads for you (such as pm2 or nodemon). I use pm2 for my production servers but both libraries work well.
Setup an HTML Template for Registration and Login
For now our server doesn’t actually serve anything useful. Let’s set up 2 routes that return registration and login forms:
The /register endpoint is a “get” route that returns an HTML template for the user to register.
The /login endpoint is also a “get” route for that new user to login.
The following code is VERY basic HTML and consists of a simple form with email and password inputs. In production you’d never have something this basic – you’d at least want to validate the email address and force a minimum password length / complexity.
// app.get defines the 'GET' request - which is what your browser sends when you type an address into the bar. The '/register' part tells this block to deliver the HTML if the user goes to 'yoursite.com/register'.app.get('/register',(req,res)=>{// res.send responds to the request with the HTML form contentres.send(` <form method="POST" action="/register"> <label for="username">Email:</label> <input type="text" id="email" name="email"><br> <label for="password">Password:</label> <input type="password" id="password" name="password"><br> <input type="submit" value="Register"> </form>`);});app.get('/login',(req,res)=>{res.send(` <form method="POST" action="/login"> <label for="username">Email:</label> <input type="text" id="email" name="email"><br> <label for="password">Password:</label> <input type="password" id="password" name="password"><br> <input type="submit" value="Login"> </form>`);});
Registering the User and Storing Serverside
The HTML endpoints we’ve defined do nothing right now, as we need to setup code to receive and process the data.
Remember the req.body mentioned earlier? That’s what carries the users email and password when they submit the registration form.
Our server will check if that user is already registered and if not then we proceed to create a new user.
IMPORTANTNOTE: We never store passwords in plain text on a server – no matter how secure you think it is! We always ‘hash’ them (put them through a one way algorithm, using bcrypt in this case) and then store the resulting hash.
Finally we store the user in our in-memory database. In production you would use a database such as mysql, Postgres or mongo. However, databases are beyond the scope of this article.
app.post('/register',async(req,res)=>{// get submitted form data from request bodyconst{email,password}=req.body;// Check if the email is already registeredif (users.find(user=>user.email===email)) {returnres.status(400).send('Email already registered');}// Hash the password before storing itconsthashedPassword=awaitbcrypt.hash(password,10);// Store the user in the in-memory database (you might want to use a database in a real-world application)users.push({email,password:hashedPassword});console.log(users)res.send('Registration successful');});
You’re all set to navigate to /register. Go ahead and register an email and password and you should see the users array printed out to your console, complete with hashed password! For example:
Note: The password hash is ultra secure and cannot be decrypted within a billion years with all the computing power in the world. That’s the power of maths and encryption 😉
On to the next step!
Checking Login Details
Now, if you navigate to the ‘/login’ page you are presented with a form to login – which does nothing! Let’s create the backend code in the /login endpoint that allows this to actually do something.
We will use the bcrypt compare function to see if the user password matches our database hash.
app.post('/login',async(req,res)=>{const{email,password}=req.body;// Find the user by emailconstuser=users.find(user=>user.email===email);// Check if the user existsif (!user) {console.log("User doesn't exist")returnres.status(401).send('Invalid email or password');}// Compare the provided password with the hashed password in the databaseconstpasswordMatch=awaitbcrypt.compare(password,user.password);if (passwordMatch) {// Set a session variable to indicate that the user is authenticatedreq.session.authenticated=true;res.send('Login successful');}else{res.status(401).send('Invalid email or password');}});
Restart the server, register a user and navigate to /login. You should be able to enter your details and receive back a message about having had a successful login.
Middleware to Protect User Area
Now that we have login credentials we need somewhere to send the user so they can see all their private stuff. We will create a /dashboard route that will only allow access to the user if logged in.
Now, seeing as we have to check the users credentials for each protected route we should create what’s called a middleware. This is an express.js feature that executes a function between when the route is called and when it returns some data.
Middlewares are extremely useful and you will use them everywhere in express JS! Let’s create a ‘requireAuth’ middleware that will perform authentication:
// Middleware to protect routes that require authenticationconstrequireAuth=(req,res,next)=>{if (req.session.authenticated) {next();// tells express to perform the next function, which is calling the dashboard in this case}else{res.status(401).send('Unauthorized');}};
By itself this middleware does nothing and needs to be called somewhere. Create a /dashboard endpoint, noting that the second parameter is the middleware we just created.
// Example of a protected routeapp.get('/dashboard',requireAuth,(req,res)=>{res.send('Welcome to the dashboard!');});
Save your file and restart your server.
Navigate to /dashboard and you’ll see that you are unauthorised! Go through the registration -> login -> dashboard loop to see that it all works now!
Optional – Niceties
It’s a bit annoying to manually navigate to pages so let’s make a simple link page at the root of our site:
Restart server and now when you navigate to localhost:3000 you will get a page with those links.
In my code linked below I’ve also changed the registration ‘success’ message to instead tell the browser to redirect to /login. I’ve also made a redirect for the login success case which takes the user to their dashboard.
Full Code – Express JS Session Server
constexpress=require('express');constsession=require('express-session');constbodyParser=require('body-parser');constbcrypt=require('bcrypt');constapp=express();constport=3000;app.use(bodyParser.urlencoded({extended:true}));app.use(session({secret:'87fir87r8f',resave:true,saveUninitialized:true}));app.get('',(req,res)=>{res.send(` <a href="/register">Register</a><br> <a href="/login">Log In</a><br> <a href="/dashboard">Dashboard</a>`)})app.get('/register',(req,res)=>{res.send(` <form method="POST" action="/register"> <label for="username">Email:</label> <input type="text" id="email" name="email"><br> <label for="password">Password:</label> <input type="password" id="password" name="password"><br> <input type="submit" value="Register"> </form>`);});app.get('/login',(req,res)=>{res.send(` <form method="POST" action="/login"> <label for="username">Email:</label> <input type="text" id="email" name="email"><br> <label for="password">Password:</label> <input type="password" id="password" name="password"><br> <input type="submit" value="Login"> </form>`);});// Sample in-memory database for storing usersconstusers= [];app.post('/register',async(req,res)=>{const{email,password}=req.body;// Check if the email is already registeredif (users.find(user=>user.email===email)) {returnres.status(400).send('Email already registered');}// Hash the password before storing itconsthashedPassword=awaitbcrypt.hash(password,10);// Store the user in the in-memory database (you might want to use a database in a real-world application)users.push({email,password:hashedPassword});console.log(users)res.redirect('/login');});app.post('/login',async(req,res)=>{const{email,password}=req.body;// Find the user by emailconstuser=users.find(user=>user.email===email);// Check if the user existsif (!user) {console.log("User doesn't exist")returnres.status(401).send('Invalid email or password');}// Compare the provided password with the hashed password in the databaseconstpasswordMatch=awaitbcrypt.compare(password,user.password);if (passwordMatch) {// Set a session variable to indicate that the user is authenticatedreq.session.authenticated=true;res.redirect('Dashboard');}else{console.log("User password doesn't match")res.status(401).send('Invalid email or password');}});// Middleware to protect routes that require authenticationconstrequireAuth=(req,res,next)=>{console.log(req.session)if (req.session.authenticated) {next();}else{res.status(401).send('Unauthorized');}};// Example of a protected routeapp.get('/dashboard',requireAuth,(req,res)=>{res.send('Welcome to the dashboard!');});app.listen(port,()=>{console.log(`Server listening at http://localhost:${port}`);});
Finally, if you have any questions or issues then please say so in the comments!
If you use replit then you can also access this server below. Note that replit has some ‘issues’ around node js so you may see some weird redirect loop behaviour.
Creating a basic server in Express.js is very straightforward. Express.js is a minimal and flexible Node.js web application framework that provides a robust set of features to develop web and mobile applications. To create a basic server, follow these steps:
Install Node.js and npm: Ensure that you have Node.js and npm (Node Package Manager) installed on your machine. You can download them from the official website: Node.js.
Initialize Your Project: Open your terminal and create a new directory for your project. Navigate into the project directory and run the following command to initialize a new Node.js project and create a package.json file:
npminit-y
Install Express: Install Express.js as a dependency for your project. In your terminal, run:
npminstallexpress
Create Your Server Script: Create a file (e.g., app.js or server.js) where you’ll write your server code.
Write the Express Server Code: Open your server script file and write the following basic Express server code:
constexpress=require('express');constapp=express();constport=3000;// Choose a port number (e.g., 3000)// Define a routeapp.get('/',(req,res)=>{res.send('Hello, Express!');});// Start the serverapp.listen(port,()=>{console.log(`Server is running at http://localhost:${port}`);});
Run Your Server: Save your file and run your server using the following command in the terminal:
nodeapp.js
Your server will start, and you should see the message “Server is running at http://localhost:3000” (or the port you chose).
Test Your Server: Open a web browser and navigate to http://localhost:3000 (or the port you chose). You should see the message “Hello, Express!”
Congratulations! You’ve created a basic Express.js server.
This is a simple example, and Express.js provides many features for building more complex web applications, such as routing, middleware, and template engines.
As your project grows, you may want to explore these features to enhance your application.
About Express JS
Express.js is a minimal and flexible Node.js web application framework that provides a robust set of features for building web and mobile applications. It is designed to make it easier to develop server-side applications and APIs (Application Programming Interfaces) with Node.js. Here are some key aspects of Express.js:
Web Application Framework:
Express.js is a web application framework for Node.js. It simplifies the process of creating web servers and handling HTTP requests and responses.
Middleware:
Middleware functions in Express are functions that have access to the request and response objects. They can modify these objects, end the request-response cycle, or call the next middleware function in the stack. Middleware allows you to add functionality to your application’s request-response cycle.
Routing:
Express provides a simple and intuitive way to define routes for handling different HTTP methods and URIs. Routes are used to map URL patterns to functions that handle requests.
Template Engines:
While Express itself does not mandate a specific template engine, it supports integration with various template engines like EJS, Handlebars, Pug, etc. Template engines allow you to dynamically generate HTML on the server.
Static File Serving:
Express makes it easy to serve static files (like HTML, CSS, and images) using the express.static middleware.
Middleware Ecosystem:
There is a rich ecosystem of middleware available for Express.js. These middleware modules can be easily integrated into an Express application to add additional functionality, such as authentication, logging, compression, etc.
HTTP Methods:
Express supports standard HTTP methods like GET, POST, PUT, DELETE, etc. This makes it suitable for building RESTful APIs.
Community and Documentation:
Express.js has a large and active community. It is well-documented with a wealth of tutorials, guides, and examples available online.
Simplicity and Flexibility:
Express.js is designed to be minimal and unopinionated, providing developers with the flexibility to structure their applications as they see fit. It allows developers to use other libraries and tools alongside Express to meet specific project requirements.
This server will print out a “hello world” to a local webpage on your PC. The steps are simple:
Step 1: Install Node JS
Go here https://nodejs.org/en/download/ and download the installer for your system (Max / Windows / Linux). Once installed verify by opening a cmd or terminal and typing
“node -v && npm -v”
This will verify that node and node package manager (npm) are installed. Any errors should be searched for on Stack Overflow.
Step 2: Setup the project
Many tutorials miss out this step. If you’re new here then what you need to know is:
Node / React / Angular / many other projects start from a single file called package.json
package.json is simply a configuration file that tells your system
Some details about the project, such as name
What packages or “dependencies” your project needs, generally these come from NPM
Many other possibilities (but that’s not needed right now)
Step 3: Create a Basic Node Project
Create a folder on your computer and name it (for example) basic-express-app.
Open that folder in Visual Studio Code, or whatever code editor you’re using.
Step 4: Use NPM to Start the Project
Open a terminal or command line. If using VS Code you can do this is the “terminal” menu.
Type “npm init” and accept the basic answers to the questions.
Npm will create a basic package.json file and our project is ready to install
In the terminal type “npm i” which will install the dependencies (there are none right now but the system will tell you that everything is up to date anyway)
Step 5: Create an index.js file
You have a package.json and in that file it should have a “main” entry. This tells the app where to start when it is run. It should say index.js right now.
Create the “index.js” file in the root of your folder, leave it empty for now.
Step 6: Install Express
Express is a framework that sits on top of Node JS and makes creating a server MUCH easier! There are alternative framworks such as Hapi JS etc but Express is very popular and much loved.
(Developer lesson: stick with much loved projects because they’re most likely to still be around in a decade, when you come back to your ancient code!)
Grant @ iamdev
In the terminal type: “npm i express” and hit enter
This will fetch Express from npm and install it in “node_modules” folder
Congratulations, all we have to do now is write our first server code!
Step 7: Paste into index.js
Take the following code and paste it into index.js:
Here we tell our app to pull in the Express dependency (1st line) and then initialise the express server (2nd line). Finally we tell the app which port to use out of the thousands available on our PC. This is important as we may have multiple servers on one system (although I wouldn’t advise it) so each one must have a unique port it can use.
Behind the scenes express JS is doing a lot of heavy lifting and abstracting away all the painful tasks!
app.listen(port, () => { console.log(`App listening at http://localhost:${port}`); });
The last 3 lines in this section tell the express app to listen to the port we specified. Whenever a packet reaches that port the app captures it and then later on will route it on to its destination.
The first 3 lines define a single destination for the app. The destination is ‘/’ in this case, which means the root destination. For example: facebook.com is also a root destination, but facebook.com/messenger is not.
Within the block we see req, res which are objects that hold information about the request and the response. Request is what the person using your site sent (eg, address, header, body, IP etc). Response contains data and methods we can use to return an answer to the person who sent the request.
Finally you can see that the “res.send” line returns a message to the user. In this case it’s a simple string with a message. Most of the time you’ll have more structure than a string – we usually send data formatted in JSON (javascript object notation) – but that’s for another day.
Step 8: Run the Server
Go to your terminal and type “node index.js”
Your app will now run, and stay running. You should get a message like: App listening at http://localhost:3001
Congratulations you have built a server! To see it running:
Open a browser and go to localhost:3001
You should see a message welcoming you to the dark world of back end server creation 🙂
Note that this will only work on your current PC. It doesn’t work on the web (yet).