Node.js as helper for developers – theory and practice

Date: 15.05.2020

gridscale node.js

In our tutorial Node.js for beginners we showed you what Node.js is, how to install and use it. In this article we will show you how to use Node.js in practice – specifically as a helper in (web) development.

What do I need Node.js for?

The everyday life of a web developer has changed in the last years: If a few years ago you developed a few loosely connected – at best dynamic – websites, which you either modified “live” on the server or transferred to the production server via FTP after a short local test, the requirements have grown in the meantime.

Today a website is often part of a whole application on which several developers are working. The application should be versioned, bugfixes and new features should be integrated into the running application as fast as possible without destroying it or even causing a longer downtime. “Continuous Integration” (CI) and “Continuous Delivery” (CD) are the keywords here.

As much as this practice increases the quality of the product, it also leaves the developers with additional processes. The danger that the developers bypass one or other process for lack of time or convenience is high. Tools that make the life of a developer worth living again can help here.

And why Node.js?

The advantage of Node.js is that most developers – not only in the web area – have already written something with Javascript. The basics of the language are well known, the learning curve for Javascript is not very high, and reference books on the Internet are a dime a dozen.

Node.js now combines this simplicity with a power close to the system. Suddenly I can use Javascript not only to build popups and animations for the browser, but also to access the file system, read system properties, launch applications and execute shell commands!

So Node.js empowers every programmer to build arbitrary tools for himself or his team without much effort. These tools can then be executed by all colleagues on their computers without any effort, almost regardless of the operating system they use. These scripts can also be executed on servers or CI runners, e.g. at Github or Gitlab. In combination with Docker Containers, the tools become conveniently shippable applications – but this leads too far at this point.

What can Node.js do?

Scripts written with Node.js can initially access all functions known from the current JavaScript standard (ECMAScript). Thus, extensive functions are already available, e.g. for displaying and manipulating date and time, for mathematical tasks or for string manipulation.

Node.js extends these possibilities by a lot of additional functions, which are divided into different packages. For example, there is the package “fs” for manipulating the file system, or the package “child_process” for managing child processes. So these modules are the bridge between JavaScript and the lower system levels.

To use a functionality from one of the packages, you first have to import the package (require() function). You can find out which packages are available in the documentation of your Node.js version, for example here: https://nodejs.org/dist/latest-v12.x/docs/api/

THIRD PARTY PACKAGES

Besides the packages already available in Node.js there are a lot of other packages from other developers. With NPM (=Node Package Manager) Node.js has its own package manager including a public package database at https://npmjs.com. The offer of free and paid packages is huge. Packages can be found by searching on npmjs.com.

They are then installed with the npm command line tool, which is directly installed with the installation of Node.js. On the one hand there is the possibility to install packages globally. This is useful for packages that are to be executed directly on the command line for all projects.

On the other hand you can install the packages project-related and enter the package with the desired version as “requirement” in a file named package.json. This file is then checked into the version control system with the project and allows other developers to install all necessary dependencies for your application by simply calling npm i.

Install package globally (-g flag):

npm i -g sat

Install the package on a project basis and save it in package.json (-save flag)

npm i -save @gridscale/api

Some packages are only needed for development, e.g. for testing your application. You can install them with the –save-dev flag instead of the -save flag. These packages are not installed with an “npm i -prod“.

EXAMPLES FROM THE PRACTICE

Example 1: Deployment

Let’s say you have an application that needs to go through some build processes for production operation.

Start your script by first defining a few configurations and including all required packages:

 
// global settings
const DIR_DIST = __dirname + '/dist/';
const DIR_JS = __dirname + '/js/';
const FILE_MAIN_SCSS = __dirname + '/css/main.scss';
 
// import 1st party requirements (are included in Node.js)
const childProcess = require('child_process'); // required to spawn additonal processes / execute shell commands
const events = require('events'); // required for event handling
const fs = require('fs'); // required for file system operations
const os = require('os'); // required to gather information from operating system (we use it to get the username)
const readline = require("readline").createInterface({
   input: process.stdin,
   output: process.stdout
}); // required (and set up) to gather some input from the user
 
// import 3rd party requirements (have to be installed via `npm i --save-dev browserify sass`)
const browserify = require('browserify'); // this compiles our javascript files, including all dependencies
const sass = require('sass'); // this compiles our CSS files
  


Since it is suitable for such a script, we choose a sequential programming style. The script is therefore processed from top to bottom. However, JavaScript is actually an asynchronous programming language – i.e. operations that cannot be performed immediately, such as file operations, do not normally block the execution of the script, but callbacks after completion, or work with so-called promises. Thus, no sequential flow would be possible, because the program would simply continue to run, although the compilation of the JavaScript is still running in the background.

Node.js therefore often offers a synchronous variant of the individual methods for its own packages – in addition to an “fs.writeFile()”, which works with callbacks, there is also an “fs.writeFileSync()”, which stops the execution of the program until the operation is completed and returns the result as a return value. In case of an error an exception is thrown and – if not handled separately – the program is aborted. These …Sync() variants exist for most Node.js methods.

For the methods, which do not offer these synchronous variants, we use the await syntax. For this we encapsulate the following code in an anonymous, asynchronous function:

 
(async function () {
    // ensure that dist directory exists and is empty
   childProcess.execSync('mkdir -p ' + DIR_DIST); // we use shell command here, because this silently quits if directory exists
   childProcess.execSync('rm -rf ' + DIR_DIST + '*'); // we use shell command here, because recursive removal is experimental in `fs`
 
   // - the following code is inserted here -
})(); 


But now back to our script. The application needs some third party libraries, whose versions are exactly defined in the package.json file you provide in your repository. You have to make sure that every developer builds the application with the correct libraries in the correct versions. To do this we simply run the node.js package manager `npm` from our script. This will install or update all required packages:

 
// ensure all packages of the main app are up-to-date before building
   childProcess.execSync('npm i --prod', { cwd: __dirname });


Before the application is built, the developer should indicate whether it is a patch, minor or major release and which features or bugfixes are included in this release:

 
   var releaseType;
   var releaseDescription;
   readline.question("What kind of release are you deploying (P)atch, (M)inor, M(a)jor?\n", _releaseType => {
       if (!_releaseType.match(/^[PMA]$/i)) throw new Error('Invalid answer!');
       releaseType = _releaseType;
 
       readline.question("Short description of whats included in the release?\n", _releaseDescription => {
           if (!_releaseDescription.match(/^(\w+\s?){10,}/)) throw new Error('Please enter a short description, minimum 10 characters!');
           releaseDescription = _releaseDescription;
 
           readline.close(); // resolves the promise we are waiting for further execution
       });
   });
   await events.once(readline, 'close');
 
switch (releaseType.toLowerCase()) {
       case 'p':
           childProcess.execSync('npm version patch', { cwd: __dirname });
           break;
       case 'm':
           childProcess.execSync('npm version minor', { cwd: __dirname });
           break;
       case 'a':
           childProcess.execSync('npm version major', { cwd: __dirname });
           break;
   }
 


Then the application is built. Here we build a fictitious application by simply compiling with Javascript Browserify (a tip by the way, if you want to use NPM packages in the Javascript of the web application), and SCSS with Sass:

 

   // collect all the javascript files we have and add to browserify
   var b = browserify();
   fs.readdirSync(DIR_JS)
       .forEach(file => b.add(DIR_JS + file));
   // compile javascript!
   var distJS = fs.createWriteStream(DIR_DIST + 'main.js');
   b.bundle().pipe(distJS);
   await events.once(distJS, 'close'); // wait with further execution until stream is closed
 
 
   // compile sass to CSS, using the `sass` package
   var sassResult = sass.renderSync({file: FILE_MAIN_SCSS});
   fs.writeFileSync(DIR_DIST + 'style.css', sassResult.css);
 
   // create a ZIP file from our dist dir
   childProcess.execSync('zip ' + __dirname + '/dist.zip ' + DIR_DIST + '*');
 


After the application is built, some basic integration tests should be performed and only if successful, a deployment to the production server should be initiated. In our example application, the deployment works by sending a POST HTTP request to a fictitious deployment service.

We send the service not only the application as a ZIP file, but also the result of the integration test, the name of the developer who initiated the deployment, and the description of the release entered by the developer:

 
// run integration tests
   childProcess.execSync('myTestSuite --resultfile=' + __dirname + '/tmp/integrationtest.json'); // "myTestSuite" is a placeholder here - testing is out of scope for this article
   var testResults = JSON.parse(fs.readFileSync(__dirname + '/tmp/integrationtest.json', 'utf8'));
 
   if (testResults.tests === testResults.passed) {
       // all tests passed, trigger the deployment
 
       // initialize POST request to our deploy service
       const req = https.request('https://myUrl.com/myDeployService', {
           method: 'POST',
           headers: {
               'Content-Type': 'application/json'
           }
       }, (result) => {
           console.log('Deploy Service result: ' + result.statusCode);
       });
 
       req.on('error', (e) => {
           console.error(`problem with deploy request: ${e.message}`);
       });
 
 
       // set request body
       req.write(JSON.stringify({
           deployer: os.userInfo().username,
           testResults: testResults,
           releaseDescription: releaseDescription,
           applicationZip: fs.readFileSync(__dirname + '/dist.zip')
       }));
      
       // executes the request
       req.end();
   }


Now our Deploy-Script is ready. Without script the deployment of our application would be a complex and error-prone process for every developer. You can find the complete example here: https://github.com/gridscale/tutorials/tree/master/nodejs-as-developer-tool/Sample1-Deploy

Note: Our script uses some shell commands which may have different names on Windows systems, e.g. `mkdir`. For Windows systems you might have to adapt the script. The script also uses some placeholder commands and URLs, and is therefore not immediately executable.

Example 2: Switch customer environment

Another fast script is only used to quickly test the application for different customers during development. Imagine, you provide the same application to all customers, but with an adapted layout in company colors and some special settings.

This script even works without any 3rd party packages:

 
const DIR_APP = __dirname + '/app/';
const DIR_CUSTOMER = __dirname + '/customers/';
const DIR_DIST = __dirname + '/dist/';
 
const childProcess = require('child_process');
const fs = require('fs');
const http = require('http');
const sass = require('sass');
 
 
// gather customer key from argument
var customer = process.argv[2] || undefined; // `process` is a global available variable
 
if (customer !== undefined && !fs.existsSync(DIR_CUSTOMER + '/' + customer)) {
   throw new Error('Customer ' + customer + ' does not exist');
}
 
// create temp dir and ensure it's empty
childProcess.execSync('mkdir -p ' + DIR_DIST);
childProcess.execSync('rm -rf ' + DIR_DIST + '*');
 
// copy app files to temp dir
childProcess.execSync('cp ' + DIR_APP + 'index.html ' + DIR_DIST);
childProcess.execSync('cp -r ' + DIR_APP + '*.scss ' + DIR_DIST);
childProcess.execSync('cp -r ' + DIR_APP + '*.js ' + DIR_DIST);
 
if (customer !== undefined) {
   childProcess.execSync('cp ' + DIR_CUSTOMER + '/' + customer + '/_theme.scss ' + DIR_DIST);
}
 
// compile sass
var sassResult = sass.renderSync({ file: DIR_DIST + 'style.scss' });
fs.writeFileSync(DIR_DIST + 'style.css', sassResult.css);
 
// merge settings json
var baseJSON = JSON.parse(fs.readFileSync(DIR_APP + 'settings.json', 'utf8'));
var customerJSON = {};
if (customer !== undefined) {
   customerJSON = JSON.parse(fs.readFileSync(DIR_CUSTOMER + '/' + customer + '/settings.json', 'utf8'));
}
 
var mergedJSON = Object.assign(baseJSON, customerJSON);
fs.writeFileSync(DIR_DIST + 'settings.json', JSON.stringify(mergedJSON));
 
// start simple webserver
http.createServer(function (req, res) {
   var file = DIR_DIST.replace(/\/$/, '') + req.url || '/index.html';
   fs.readFile(file, function (err, data) {
       if (err) {
           console.error(file, ' not found');
           res.writeHead(404);
           res.end(JSON.stringify(err));
           return;
       }
       console.info('Serving ', file)
       res.writeHead(200);
       res.end(data);
   });
}).listen(8080);
 
console.info('Open http://localhost:8080 to see the result');


You can find the complete, executable example here: https://github.com/gridscale/tutorials/tree/master/nodejs-as-developer-tool/Sample2-CustomerEnvironment

Task automation – simple and portable

If you are basically familiar with JavaScript and are looking for a simple and portable way to automate or at least simplify certain recurring tasks, you should consider if Node.js can be a solution.

If you don’t want to script everything yourself, there are several tools, so called “Task runner” which are based on Node.js. Examples are here grunt or gulp. Here there is already a certain predefined script structure and many common tasks have already been cast in addons for these tools.

Happy scripting!


Jan Stuhlmann

Jan Stuhlmann | Frontend, UX & GUI
Jan created websites and applications while still at school. In 2007 he finally turned his hobby into a profession. At gridscale he mainly takes care of the frontend of the different panels, but he also likes to help out in the backend and the area of the website.

Back to overview