createWriteStream async – Guide

by

When writing files to disk in node you have to be careful. Many times file writes require insertion of large chunks of data over a (relatively) long period of time. The key thing to avoid is opening the file multiple times which is resource intensive.

It is far better to open the file once and hold that file handle in memory, ready to be used the next time. For that we use the createWriteStream async method.

createWriteStream async in Node fs

This method gives you ultimate control over file handling, which means more responsibility. Therefore you have to handle file opens, closes, errors and everything else that may go wrong.

Let’s first look at the single line of code needed to create a createWriteStream:


//cteate your sreite stream with optional options
fs.createWriteStream (myFilePath, myOptions)

------------------------------------------
myOptions can be any of the following:
  options <Object>
    encoding <string> Default: 'utf8' //file encoding
    
    autoClose <boolean> Default: true //autoclose after write? false = close it yourself
    
    emitClose <boolean> Default: true //emit event to notify file closed

    start <integer> //where should you insert the content?
    
    highWaterMark <number> Default: 16384 //how much data should this method gather before it's flushed to the file
    
    flush <boolean> // File descriptor flushed prior to closing. Default = false.

Most of these will not be specified by you, just leave them as defaults. However, you may want to play with highWaterMark depending on your use case. That sets the buffer size and should be matched to the size / frequency of data streaming you have.

NOTE: Always use a resource monitor such as atop or htop to know if you have issues!

createWriteStream async Full Example

Before we start, note that to use createWriteStream in its async promise based version (instead of callback versions) you use the fs module and not the fs.promises module! I know, I know it’s counterintuitive!

The code below opens a write stream, writes something to a file on 2 separate calls then closes the file.

const fs = require('fs'); //the main fs lib

try { //always wrap in try catch as file writes can be error prone, especially on linux hosts...

  //create the writer object we can reuse
  let writer = await fs.createWriteStream('log-stream.txt', {
    encoding: 'utf8',
    highWaterMark: 16384,
    flags: 'a', //open and append (don't overwrite file)
  });
  
  //use the writer object
  await writer.write("Log 10: what happened in last 10s");
  await writer.write('\r'); //create a new line
  
  //sometime later
  await writer.write("Log 20: what happened in last 10s");
  
  //finally close the file to avoid memeroy leaks and file access issues!
  await writer.close()
  
} catch (err) {
  console.error('Error occurred while writing file:', err);
}

Important notes: The above code shows a simplified step by step process to make it clear for you. To implement this in a real app, you would never write it this way, instead you’d create separate functions to handle file open, write and close – then call each function as you need them in your code.

createWriteStream Events and Listeners

FS gives us a lot of control over the file handling process, which means we can manually do everything ourselves. One of those options is choosing whether fs should close the file automatically or if we should listen to events and do it ourselves.

If your file write is long in duration, over multiple chunks, then you can tell FS to leave the file open, saving the overhead of opening it over and over.

This is perfect for server logs, for example. To do this we will listen to events called by createWriteStream. Here’s an example (code below):

  1. Open the file writer with createWriteStream
  2. Tell createWriteStream that we want to handle file closing ourselves (autoClose flag)
  3. Register event listeners for open, ready and close
  4. Write some text to a file
  5. Use the listeners and callbacks to decide when to perform next actions
  6. Close the writer once a write has been completed
  7. Remove all writer listeners to avoid memory leaks!

Next, I have written the code in a step by step manner which is not realistic. You would separate this out in production to be called by the main program, as needed:

Example createWriteStream async Code (Events)

const fs = require('fs'); //the main fs lib

try {
  let writer = await fs.createWriteStream("log-stream.txt", {
    encoding: "utf8",
    highWaterMark: 16384,
    autoClose: false,
    emitClose: true,
    flags: "a"
  })

  //add a listener with options such as open, close, ready
  await writer.addListener("open", async () => {
    console.log("Log file OPEN!")
  })

  await writer.addListener("ready", async () => {
    console.log("Log file READY!")
  })

  await writer.addListener("close", (fd) => {
    console.log("Log file CLOSED")
  })

  writer.write("hello world \r\n", async () => {
    await closeWriter(writer)
  })
} catch (err) {
  console.error('Error occurred while writing file:', err)
}

async function closeWriter(writer) {
  await writer.close(async () => {
    await writer.removeAllListeners() // important to avoid memory leaks!
  })
}

Worth mentioning again: DO NOT forget to unsubscribe to your writer listeners on writer close or you’ll get memory leaks!

Summary

createWriteStream gives you power over all aspects of file writes but with great power comes great responsibility!

As long as you handle events appropriately and monitor resource usage then you should be able to avoid pitfalls of this method and retain complete control over your file writes.

If you want a deep dive into all aspects of Node FS then I have the Ultimate FS Node Guide here.