Node JS Fabric JS Memory Leaks - Prevent ENOMEM

I’ve been working on tracking down a memory leak in my Fabric JS app running on Node JS hosted on a Rackspace cloud server.

Contents

1 The Problem
2 Previous Attempts To Fix This Issue
3 Memory Leak Debugging Process
4 Other Useful Information
- 4.1 NodeJS Streams
- 4.2 Garbage Collection

The Problem

This server would never free up it’s used memory. I watch the memory climb and climb on the server monitor until it hits the dreaded drop off point. The server runs out of memory, it throws an ENOMEM error, and forever restarts the server.

Previous Attempts To Fix This Issue

I’ve looked into this issue before and discussed it in a post about garbage collection in NodeJS, however that sadly didn’t solve all of the issues.

The other solution I’ve had in place for some time is utilizing a CRON job that will restart the server once a day in the off hours. This will reset the memory at that time. This hack works, but as usage has increased enough that it’s crashing before we ever get to that point.

Memory Leak Debugging Process

Strip Out Code

Today I essentially stripped out code line by line to try an figure out where there could be an issue. In your code, take out function after function, line after line until you start to notice some differences in your memory monitor.

This is a manual process. Take out some code, restart the server to get back to a baseline, run your process, and compare what the final memory usage is to what it was from your previous test. You’ll want to grab a paper and pencil to write some numbers down. If you think you’ve found something that made a change, undo the code change, retest, and prove that what you think you saw did happen.

Memory Monitoring

I’m utilizing htop to monitor the servers memory usage on a dev server. (I changed my setup settings from the defaults so if the images below look different than you see that is why.)

Today’s Culprit – FabricJS canvas.clear()

I found something that made a difference! Sadly I think there still are some lingering issues out there, but this is a sizable enough change that I’m happy to have found a big issue here.

It appears the canvas object isn’t being cleared properly. I had been clearing the canvas manually via Javascript, but apparently that either isn’t the correct method or sufficient enough. What I was doing is trying to set the canvas to null and then forcing the garbage collector to run.

// Create canvas
canvas = fabric.createCanvasForNode( width, height );

// Process the canvas with some code here

// Cleanup the canvas
canvas = null;
global.gc();

// Create canvas

canvas = fabric.createCanvasForNode( width, height );

// Process the canvas with some code here

// Cleanup the canvas

canvas = null;

global.gc();

What I found today though in the FabricJS Github Issue #1997 is that there is a FabricJS method that will better cleanup these memory issues.

canvas.clear();
canvas.dispose();

1 2	canvas.clear(); canvas.dispose();

According to the source code, canvas.dispose() calls canvas.clear() and thus technically you would only need canvas.dispose();

For me, I found that canvas.dispose(); didn’t exist. I am running an older version so it must have been added somewhere along the way.

So for me, the solution I found is:

// Create canvas
canvas = fabric.createCanvasForNode( width, height );

// Process the canvas with some code here

// Cleanup the canvas
canvas.clear();
canvas.interactive && canvas.removeListeners();

// Create canvas

canvas = fabric.createCanvasForNode( width, height );

// Process the canvas with some code here

// Cleanup the canvas

canvas.clear();

canvas.interactive && canvas.removeListeners();

Memory Usage Results

Baseline memory usage — This is the memory usage just after a server reboot.

Memory usage with FabricJS memory leak — After running the server process, you can see there is a memory leak. The server memory goes up and stays up. As pointed out above, this is due to not clearing the canvas created by FabricJS properly.

Memory usage after clearing FabricJS canvas — I restarted the server to get back to the baseline usage above, and then ran the process – this time using canvas.clear(); to clear the FabricJS canvas from the NodeJS server memory. As you can see there is a significant drop in the retained memory. As you can also see it didn’t go back to what the baseline was and thus there still is another leak to find. But this was a big step in the right direction!

Other Useful Information

NodeJS Streams

I didn’t find this to be the golden ticket in my debugging, but I did see plenty of discussion by other developers that using streams in NodeJS to be a better practice in regards to memory usage as opposed to reading/writing directly from the disk. I’m not going to try and act like an expert on this topic however – there are already great resources out there including:

Garbage Collection

I’ve written garbage collection in NodeJS in the past, but wanted to add a few points to the topic here.

Garbage Collection can be a server intensive process. Improvements have been made in the backend code that have improved this, but it’s still not recommended to run this more than necessary. Many developers say you should never manually run it as the V8 engine that NodeJS runs on is good enough. I’m still not sure what my verdict on the topic is however.
Garbage collection is a very complex topic. I’m not saying I fully understand it. But this article by Jay Conrod about V8 Garbage Collection is a start for helping to understand how it operates.