Category: node js server

fs (file system) module is a much misunderstood part of node JS, especially the asynchronous versus the synchronous methods. This mega post will cover everything you need to know about fs and more! I include lot’s of architecture guides and potential gotchas (many of which have gotten me in the past).

fs stands for file system, so (almost) anything you can do to a file using your terminal or mouse can also be done with fs.

This article will start with basic file writes and move on to complex writes, file reads and file management. Please do read all the text between code examples as there are many many places you can be tripped up when using fs.

What can fs do?

Because fs is part of node JS on the server it has elevated privileges that allow it to access the file system. That said, if you find that certain file operations fail then please check the linux privileges of your node JS process / server app folder.

The fs module can do many things including:

• Open files
• Read files
• Create and write to files
• Update files
• Delete files
• Rename files

fs Callbacks, Async and General Notes

In this mammoth article we will cover the basic methods needed to read, write and manipulate files. Bear in mind that there are 2 variations of most file manipulation commands in fs, synchronous and asynchronous. Or async and sync for shorthand.

Important Developer PSA: You should always aim to use the asynchronous file manipulation methods. This is because node JS runs on a single thread, is non blocking and event based. Ie, node expects your code to not block the main thread unnecessarily, which is very important for the responsiveness of your server.

You will very rarely, if ever, need to use synchronous methods in fs so this guide focuses exclusively on the async methods.

Asynchronous methods are further sub divided into 3 versions – callback, promise or async based. These are all essentially the same thing, just different ways of writing code. If you’re not yet familiar with synchronous vs asynchronous programming then please click this to see my most excellent guide on it.

Personally, I always default to async await code. It’s easier to read, flows better and helps you avoid nasty problems like race conditions or cross scope variable assignment, bugs from which are very hard to diagnose.

In the first section (writing basic files) I will also show you the callback version of the sample code but future sections will only use the promise / await syntax. Just know that you can always revert to the callback versions by implementing the relevant structure (but why would you??!?!)

Basic File Writes with fs

Writing Basic Files – writeFile Function

In order to write a basic file in fs we use the writeFile method, which takes 4 parameters:

writeFile (filePath, text to write, encoding (optional, defaults to utf8), return function)

NOTE: This callback version is for reference only, I strongly advise you to use the promises / await styles in the next section!

The callback below is implemented by passing in a function as the last argument of the writeFile method. That function is executed when the tasks completes:

const fs = require("fs");

let name = "Grant iAmDev";

//option 1: without specifying the file encoding
fs.writeFile("person.txt", name, (err) => {
  if (err) {
    console.log(err);
  }
  console.log("File saved");
});

//option 2: explicitly specify some options such as file encoding
fs.writeFile("enc-person.txt", name, { encoding: "utf8" }, (err) => {
  if (err) {
    console.log(err);
  }
  console.log("File saved");
});

Gotcha 1: writeFile will replace the file if it already exists! There will be no warnings and no thrown errors to let you know!

Gotcha 2: Do not run this method on a file whilst it is being written to by another writeFile call. You should have a flag to lock the file, preventing simultaneous writes!

Developer note: If you’re storing high frequency incoming data, or large data chunks then you should not use this method. Instead opt to use the createWriteStream method described later on.

writeFile – Promise Await Version (Avoid Callbacks)

If you prefer to avoid callback hell you can instead use the promises module included in fs. Note that the import statement changes to “fs/promises”. Quite often I’ve been caught out when visual studio auto imports just “fs” and I’m left scratching my head as to why the await statement doesn’t work!

Obviously, the promise version takes one less parameter as you don’t need to specify a callback function. Also note that we always enclose the writeFile call in a try-catch statement. This is necessary as access to a file is never guaranteed so could be a significant source of app crashes. If you have server logs ensure that a crash like this always gets recorded in there.

import fs from 'fs/promises'

async function writeData(name) {
  try {
    return await fs.writeFile("person.txt", name, "utf8") //options can use the shorthand version here, just a string automatically assigns file encoding
  } catch (err) {
    console.error('Error occurred while writing file:', err)
  }
}

In order to call the above “writeData” function you would use the following line:

await writeData("Grant @ iAmDev")

This call will run asynchronously, without tying up your main thread, but will pause execution of your code until a result is received. This function call will not block code in other functions from running, and keeps your app / server responsive.

If there is no error thrown then you can assume that the operation was successful. HOWEVER, I find this code a bit problematic as I usually need to know explicitly if an operation failed or succeeded. The simplest version of this is to return a boolean indicating success or failure. This is the most basic version of this code and there are other things I may do, but those are beyond scope of this article:

async function writeBasicFilePromise() { //always return a boolean so you know if something failed or not
  try {
    await fsp.writeFile("person-promise.txt", textToWrite, "utf8")
    return true
  } catch (err) {
    console.error('Error occurred while writing file:', err)
    return false
  }
}

//call above function and look at returned "success" to determine code flow
const success = await writeBasicFilePromise()
console.log("File write successful? : " + success)

If you’re super fussy you may want to also assert that the file now exists and its content matches what you expect but I hardly ever find this to be necessary.

writeFile in fs – Complete List of Flags & Details

The options object passed in to writeFile allows you to specify filesystem flags that change the default behaviour when opening a file. Here’s a simple example:

await fsp.writeFile("person-promise.txt", textToWrite, 
  {
    flag: "ax" //causes a thrown error if the file already exists!
  }
)

The following flags are available for you to use – be careful to choose the right one!

• 'a': Open file for writing (adding to the end of existing content) + the file is created if it does not exist (default)
• 'ax': Like 'a' but fails if the file path already exists
• 'a+': Open file for reading and writing. The file is created if it does not exist.
• 'ax+': Like 'a+' but fails if the file path exists.
• 'as': Open file for writing in synchronous mode (be careful not to tie up your main thread). File is created if it does not exist.
• 'as+': Open file for reading and writing in synchronous mode. The file is created if it does not exist.
• 'r': Open file for reading. An exception is thrown if the file does not exist so you need to catch that exception.
• 'rs': Open file for reading in synchronous mode. An error is thrown if the file does not exist.
• 'r+': Open file for reading and writing. Exception is thrown if the file does not exist.
• 'rs+': Open file for reading and writing in synchronous mode. Instructs the operating system to bypass the local file system cache. This is useful for opening files on NFS mounts as it allows skipping the potentially stale local cache. It has a big impact on I/O performance so using this flag is not recommended unless it is really, really needed.
• 'w': Open file for writing. The file is created (if it does not exist) or truncated (if it exists).
• 'wx': Like 'w' but fails if the path exists.
• 'w+': Open file for reading and writing. The file is created (if it does not exist) or truncated (if it exists).
• 'wx+': Like 'w+' but fails if the path exists.

Add Content to File with fs (Append)

Now that we’ve covered basic file writes let’s look at how we add content to the end of a file. This is only really convenient if you want to add to the end of a file. Adding to the middle of a file is notoriously difficult and something you’d need to manage yourself.

Side note: If you ever find you need to write a line in the middle of a file then you should really revise your architecture / storage choices. Usually a structured database, of any kind, would be a much better choice.

We can add content to a file with the appendFile method:

const fs = require('node:fs/promises');

async function appendTextToFile() {
  try {
    const content = 'Extra content at end of file';
    await fs.appendFile('/Users/iamdev/data.txt', content);
  } catch (err) {
    console.log(err);
  }
}

appendTextToFile();

Note: You can also append to a file using the writeFile method and including the flag ‘a’.

Add Content to Start of a File with node fs (Prepend)

Unfortunately there is no simple way to add content to the start of a file using the standard fs module. However there are libraries that will do this for you such as prepend-file.

Another alternative is to read the content from your existing file into memory, insert your new data and write all that back to the same file. Watch out though, a large file will occupy a lot of memory and if you don’t garbage collect properly then you’ll have mega memory leaks!

Continuous File Writes in node fs

Many times your file writes will require insertion of multiple data blocks over a longer period of time. For this operation you will want to open up a stream that allows addition of data into a file over time, whilst avoiding the overhead of re-opening the same file.

This is achieved in fs with the createWriteStream asynchronous method.

createWriteStream async for Writing Large / Continuous Files in fs

This function is fairly complicated as you are moving into a level below high level writeFile. Therefore you have more control, but also more responsibility in terms of handling file closes, errors etc.

First let’s cover the basic createWriteStream method and all the arguments it can take (with explanations):

fs.createWriteStream (filePath, OPTIONS)

OPTIONS can include any of the following:
  options <Object>
    //Specify file encoding
    encoding <string> Default: 'utf8'
    
    //Should the file be autoclosed after write? If you set
    // to false then you must close the file yourself!
    autoClose <boolean> Default: true
    
    //An event telling our code if the file has been closed
    emitClose <boolean> Default: true
    
    //What location should this content be inserted to?
    // Takes an integer from 0-MAX_ALLOWED_INT
    start <integer>
    
    //highWaterMark is another way of saying buffer size before
    // actually writing to file
    highWaterMark <number> Default: 16384
    
    // File descriptor is flushed prior to closing. Defaults to false.
    flush <boolean>

Of particular note here is the highWaterMark option, aka: buffer size. This sets the amount of data to hold in memory until flushed to file. Note that you can get amazing memory leaks in the order of gigabytes if incorrectly specified for your use case. Whenever your server is running always use something like atop or htop to monitor resource usage!

Now let’s move onto a more substantial example. In the following we are trying to architect createWriteStream in an asynchronous manner. The code below shows you the logical steps you need to do this.

const fs = require('fs'); //the main fs lib

try { //always wrap in try catch as file writes can be error prone, especially on linux hosts...

  //create the writer object we can reuse
  let writer = await fs.createWriteStream('log-stream.txt', {
    encoding: 'utf8',
    highWaterMark: 16384,
    flags: 'a', //open and append (don't overwrite file)
  });
  
  //use the writer object
  await writer.write("Log 1: what happened in last 10s");
  await writer.write('\r'); //create a new line
  
  //sometime later
  await writer.write("Log 2: what happened in last 10s");
  
  //finally close the file to avoid memeroy leaks and file access issues!
  await writer.close()
  
} catch (err) {
  console.error('Error occurred while writing file:', err);
}

Important notes: The above function combines the creation of the write stream (createWriteStream), the write processes and the file close, to make it clear for you. In production you would put the file open, write and close in separate functions.

Then you can call each function as required by your application.

createWriteStream Events, Manual Close etc

As we’re operating at a lower level we can specify if we want file closes to be done automatically or if we should handle them. To do this we must handle these tasks to have a complete file writing system that we control all aspects of:

• Open the writer with createWriteStream
• Tell that writer we wish to handle file close ourselves, specifying the autoClose flag as “false”
• Register our own event listeners for open, ready and close
• Write content via that file writer (normally through a separately called function)
• Use listeners and callbacks to monitor state of our writes and file handling
• Close the writer on completion of a subset of writes / whenever is right for your application
• IMPORTANT: call removeAllListeners on completion to avoid memory leaks!

const fs = require('fs'); //the main fs lib

try {
  let writer = await fs.createWriteStream("log-stream.txt", {
    encoding: "utf8",
    highWaterMark: 16384,
    autoClose: false,
    emitClose: true,
    flags: "a"
  })

  //add a listener with options such as open, close, ready
  await writer.addListener("open", async () => {
    console.log("Log file OPEN!")
  })

  await writer.addListener("ready", async () => {
    console.log("Log file READY!")
  })

  await writer.addListener("close", (fd) => {
    console.log("Log file CLOSED")
  })

  writer.write("hello world \r\n", async () => {
    await closeWriter(writer)
  })
} catch (err) {
  console.error('Error occurred while writing file:', err)
}

async function closeWriter(writer) {
  await writer.close(async () => {
    await writer.removeAllListeners() // important to avoid memory leaks!
  })
}

The example code above does things in a stepwise, logical sequence to make it clear. Production code would never do it like this, instead separating it all into functions to be called when required by your app.

Reading files with fs

How to Use readFile in node fs

As with the writeFile function we looked at earlier the readFile function is much the same. Here’s some basic async await code:

const fs = require('node:fs/promises');


async function readAndPrintFile() {
  try {
    //print out the index.js file that contains your server code!
    const data = await fs.readFile('index.js', { encoding: 'utf8' });
    console.log(data);
  } catch (err) {
    console.log(err);
  }
}

await readAndPrintFile();

The above code will print out your index.js file, which should be your server entry point. CHange to any file in your directory as needed.

Note: This will read the whole file into memory. If you don’t have enough memory then your app will slow down at best, crash at worst! Obviously only use this for small files!

A better alternative is to use a stream to read files that are larger.

How to Use createReadStream Async to Read a File

A stream in Node creates a pipeline between content and your application. This avoids having to read a whole file into memory and you can instead listen out for chunks coming from your file:

async function streamTextFromFile() {
  //NOTE: This function combines createReadStream AND reads the file.
  // in reality you should separate these 2, only creating the stream once!
  try {
    let reader = await fs.createReadStream("index.js", 'utf8')
    reader.on('error', function (error) {
      console.log(`error: ${error.message}`);
    })

    reader.on('data', (chunk) => {
      //NOTE: this outputs RAW hex data if you DON'T set the encoding above!
      console.log(chunk);
    })

  } catch (err) {
    console.error('Error occurred while stream reading file:', err)
    return false
  }
}

await streamTextFromFile()

If you don’t specify file encoding then you will receive RAW file output (in hex).

Finally

If you enjoyed this but still need help with your application or server then feel free to contact me or find me working as a software developer in Gibraltar.