Authors: Shmulik Biton, Ron Sevet

It’s not very often in the career of a programmer that one has the privilege to deploy a brand new application to production. We were lucky to be in that position. This was even more special since this was a completely new technology stack that we were introducing here at Tipalti. Our entire server-side stack is based on .NET and the various Microsoft products that go along with it. We have an extensive experience with .NET’s best practices, tools, how to run our applications and how to secure them.

This wasn’t the case here. Our new application is written in Node.js and we wanted to run it on Linux for better performance. This meant we had to find how to apply the practices and capabilities from our .NET stack on the Linux stack.

We had two major tasks, the first was to find a capable solution for running our node application. We searched around and found PM2. PM2 is a very comprehensive process manager with many useful features. It’s popular, well maintained and looked very promising.

Securing Configurations

After setting up PM2, we needed a way to secure our sensitive configuration keys like DB credentials and alike. In .NET, you can simply use the Windows Protected Configuration API for a seamless experience. Your web.config is encrypted using the user’s credentials and .NET handles it without too much fuss. You don’t need to store any encryption keys next to the config so it’s a pretty good solution.

In Linux, there isn’t something similar, and after some research, we found a best practice that seemed reasonable to us: storing sensitive configurations as environment variables. The reasoning here is that accessing this file requires either root access or the ability to run code on the machine, which in that point there is not much you can do to keep those variables a secret.

At first, all went well. PM2 performed as expected, and using the environment variables was an easy solution.

The issues started after we had duplicated our server and had to change some of those environment variables. We started getting weird errors that we did not see before. We found out that we were still using the old configurations. Using the `pm2 reload` command did not have any effect. Nor the `restart` command or even restarting the server itself. This was very baffling. It turns out that PM2 was the culprit here. One of PM2’s features forces you to explicitly reload environment variables in order to get the updated values. The flag to use is `–update-env`. This seemed to fix the issues we thought the problems are behind us.

We were wrong. After some time has passed another config change was required and this time the flag did not work. Nothing seemed to work. Rebooting the server did not have any effect either. We figured there must be some sort of a cache PM2 was using to store the application state. This was indeed the case.

We used the command `pm2 save` which creates a dump file on ~./pm2/. When the app started/reloaded/restarted it used that dump file to continue where it left off. This behavior was not desired for two reasons: First, it stopped us from updating our configuration. Secondly, it stored sensitive configurations in the home directory of the user running our app. This can potentially expose us to file traversal attack vectors.

Solving the Issue

We solved it by deleting the dump file and stopped using the `pm2 save` command. We also changed the default startup script PM2 creates when you call `pm2 startup`.

This is our final startup script:

[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
Type=forking
User=
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
EnvironmentFile=/your/env/file
Environment=PATH=/usr/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/home/integration/.pm2
PIDFile=/home/integration/.pm2/pm2.pid

ExecStart=/usr/lib/node_modules/pm2/bin/pm2 start /home/integration/.pm2/ecosystem.config.js
ExecStartPost=/usr/lib/node_modules/pm2/bin/pm2 reload /home/integration/.pm2/ecosystem.config.js
ExecReload=/usr/lib/node_modules/pm2/bin/pm2 reload /home/integration/.pm2/ecosystem.config.js --u
ExecStop=/usr/lib/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

Note that you need to specify in the startup script the environment file explicitly, otherwise, it won’t load it, and since we are already specifying the file path, we decided to use a dedicated file with root permissions instead of /etc/environment. This would prevent a general process to have access to our configuration.

A quick note on key store solutions: We thought it would add another layer of complexity without adding much security. If someone got root access to the machine they could get the configuration no matter what we do. This is why we decided not to go that route.

In summary

If you want to run a Node.js application under Linux using PM2, these are the steps that we found worked best:

1. Put sensitive configuration variables in a root accessible file
2. Do not use the `pm2 save` command
3. Change the startup script as shown above
4. Reload your application after changing the configuration using
   `pm2 reload <your config file name> –update-env`