Open Source Productivity Automation with Joplin + Node-RED
In this article I’ll show you how to use free open source free and open source software to build productivity automation workflows to make Evernote and IFTTT jealous whilst keeping your data safely on your own machine(s).
Introduction
Many people are familiar with apps like Evernote and more recently Notion that allow you to keep notes on pretty much everything — almost like a second “digital” brain. There are also plenty of mature tools like IFTTT and Zapier that allow you to chain together the many digital resources in your life. E.g. send emails containing a particular keyword to your evernote account or add items to your todo list via SMS.
However, relying on these third party tools can sometimes have its downsides. When you use tools like Evernote and Notion all of your notes reside on their servers and that means that you have to trust that they’re not doing anything nefarious with your data and that they will look after it properly — even in the event of a cyberattack. Also in most cases you will have to pay for these services one way or another.
Thankfully, in the year 2020 there are many other options provided as free and open source software that allow you to keep your data safely on your own machine(s) and are completely free (except for the cost of your computer/phone) to use. In this article I’m going to show you how to use Joplin, a free open-source Evernote/OneNote/Notion alternative and Node-RED, a very flexible free, open-source task automation platform and our IFTTT/Zapier alternative in this context, to build very flexible productivity automation workflows.
Install and configure Joplin
First you’ll want to install and configure Joplin. You can download the latest version of the app from the website for Windows, Mac and Linux so just pick whichever one you need for your personal setup and follow the installation instructions.
In order to use Joplin for task automation we need to enable the WebClipper API. If you’re not already familiar with it, the Joplin WebClipper is a very powerful browser plugin for Firefox and Chrome that allows you to take local copies of entire websites and store them inside Joplin for future reference (perfect for offline browsing or in case the original gets taken down). I’m a heavy user of this plugin and I highly recommend it.
The web clipper uses a REST API to communicate between the browser and the Joplin app and this API is powerful enough that it can be used for all sorts of other use cases (as you’ll see below).
Once you fire up Joplin you’ll want to open the Options menu (CTRL + , or CMD + , on Mac) and bring up the Web Clipper tab. You need to click “Enable Web Clipper Service” and should see the text update and say “Started on port 41184” — the port number may be different and that’s ok — whatever the number is make a note of it for later.
Under the “Advanced options” tab there is an Authorization token which looks like a long random string of letters and numbers, we’re going to need this later too. The easiest thing for now is to keep this pane open and minimized and come back for the token later.
Install and Configure Node-Red
Next we’re going to install Node-Red our IFTTT alternative. Node-RED installation can be a little bit fiddly if you’ve never worked with node.js before but it has really good documentation and getting started guides here
If you’re running on Windows you might want to try out the windows installation guide here and there’s a really good unofficial guide here.
Once you have installed Node-Red on your operating system of choice you can run it by opening up a terminal window (or on windows the command line interface) and typing node-red. Now if you go to a browser and type:
http://localhost:1880 you will be able to see the node-red user, it will look a little like this:
The idea of node red is that you use the nodes in the palette to the left to build out workflows on the canvas (white squared area).
Node-red comes with a basic set of nodes installed but we will add more so that we can interact with Joplin and our email provider. You’ll want to go to the hamburger menu in the top right and click “manage palette”.
This will bring up a new dialog showing a list of nodes that you can install. There’s a huge catalogue including nodes for sending and receiving emails, nodes for interacting with IoT devices like smart fridges and nodes for using AI services like IBM Watson or Google Cloud Vision.
Go to the “Install” tab and type in “joplin” — we’re going to install node-red-contrib-joplin by clicking the “install” button in the bottom right.
Following the same process we’re also going to install node-red-node-email which will allow us to send and receive emails via node-red.
Once you’ve added the packages, scroll down in the pallete and you should see new nodes for both Joplin and email have appeared.
Now we’re ready to build our first flow.
Receiving emails in Node-RED
It can sometimes be very useful to archive emails in your note-taking app. Evernote has long supported this by assigning you a pseudo-random email address to forward your messages to. Here we will configure a node-red flow that imitates this behaviour with your existing email provider.
We will use GMail as our email provider (the irony of using GMail in an article about hosting-your-own FOSS services has not escaped me) but you can set this flow up using any IMAP or even POP email provider and there are plenty of articles out there about how to set yourself up an email server if this is something that interests you.
Firstly we may need to turn on “Less secure app” access for your google mail account via the control panel here — the “less secure” bit may sound scary but basically it’s anything that communicates with Gmail using a username/password rather than a randomly generated security token that can be revoked remotely. There’s a pretty good article here about what it is and best practices for using Less Secure apps.
Next we’re going to drag an email node onto the canvas- the one with the envelope to the left and the square “port” to the right. Double click on the node to configure it. I’ll explain a bit more about the significant of these “ports” below.
The server should be imap.gmail.com and port 993 . Your userid and password are your gmail email address and password.
By default the node will import everything that is unread in the INBOX and mark it as read. We can change this so that it looks at messages we put in a specific folder. In this case I am going to create a new folder in my inbox by opening gmail’s labels settings page and clicking “Create new label”
Once I’m happy with my label I am going to enter the same thing into the “Folder” field in the node-red configuration prompt. We’ll leave Disposition as “Mark Red” and Criteria as “Unseen” for now but you can experiment with these later if you want to.
Now we’re going to drag a debug node (green with a “port” on the left) from the palette onto the canvas to the right of the email node and click and drag from the port on the email node to the port on the debug node to connect them together with a wire.
Double click on the debug node and change the output from msg.payload to “complete msg object” and hit Done to confirm this change.
Next I’m going to move an email from my inbox into the new folder and mark it as unread to test our flow.
Finally we’re going to click “Deploy” in the top right and we should see a success message pop up like the one below (if you see the prompt about unused configuration nodes, ignore it for now — Successfully deployed is the important bit).
Now if you bring up the debug tab with the little bug icon on it in the side bar (you may have to click View -> Show sidebar in the hamburger menu if you don’t see the sidebar) you should see the output from your test message
If you managed to stick with me and got this far then congratulations, that was the most fiddly bit. Next we’re going to connect up Joplin but before that I’m going to do a quick Node-RED primer to try to help you make sense of what just happened.
Before we continue: Node-RED in 30 seconds
If you’re familiar with node-red you can skip this section and go to “Saving emails to Joplin” below.
The ideabehind node-red is that message objects (codified as msg ) are created by input nodes (like the email node we just created) and these messages are passed between nodes that are wired together. We can manipulate different properties of the message at will using the nodes in the palette and power users can write their own javascript functions to manipulate the messages.
Different types of nodes handle messages in different ways. An input node has a port on the right only indicating that it produces messages. An output node has a port on the left only to indicate that it sends the message off somewhere (e.g. to Joplin or an IoT device or to twitter). Function nodes have ports on both left and right to indicate that they can receive and send messages (i.e. a transformation is happening inside).
In the basic flow above our email node listens for emails and when it gets one it creates a message containing data about that email. It passes that message down the wire to our debug node which outputs it to the sidebar. If we want to we can manipulate that email before we output it to debug.
Saving emails to Joplin
Different nodes expect different msg properties to be present on input and produce different properties on output. Our email node puts the email subject in a property called msg.topic and the actual body of the email in msg.payload.
We will use a Joplin create node to create a note in our notebook. Joplin create is expecting the title of the new note we want to create to be present in msg.payload.title and the body of the new note to be present in msg.payload.body
We can use a change node to move different properties of messages around and rename them. Drag a change node onto the canvas between the email; and msg nodes (if you intersect the wire it will automatically hook itself up between the nodes so that the email flows into the change node and the output from the change node flows to the debug node). Double click the change node and we can configure it to make our email node “speak Joplin”.
With a bit of juggling we’re able to set the fields up in the correct way. You can test it by marking your message as unread in gmail and clicking Deploy. the new debug output should have a payload object with a body containing the text
Now we can configure our joplin node to take the email and save it to a notebook. Drag a “joplin-create” node from the palette onto the canvas and position it above the debug node. Draw a wire from the change node to the joplin node.
Double click on the joplin node to configure it. First we need to set up our server connection. The pencil icon will bring up the server config which will ask you for your hostname which is localhost and port number which is either 41184 or whatever you wrote down earlier.
You’ll also need to copy the access token from your Joplin settings page into the API Key box here. Once you’re done click “Add”.
Now here’s another cool feature of Node-RED. Most of the nodes in the palette let you set options in the editor but also allow you to pass them in programatically. Here we can set the title and body of the note to be created but if we pass these values in via msg.payload.title and msg.payload.body respectively we don’t need to set them here.
The other thing we can do is set the parent notebook that we want to import the new note into (e.g. we could have a Recipes folder in our email inbox and a corresponding Recipes folder in Joplin). If the Parent Notebook tree doesn’t show up, try setting the server, clicking “Done” and then re-opening the config for the joplin node.
We’re now finally ready to test our end-to-end flow. Mark your email unread and hit deploy, give it a few seconds and BAM! You should see the email appear in the debug output again but also…
Hooray! If you got this far congratulations, you now have a working node-red flow that imports emails into Joplin. Hopefully you now also have an idea of what Node-RED is and the power that it offers. There are so many possibilities and cool automation flows that you can build and with the power of third party nodes and flows you’ll be able to do stuff that IFTTT users can only dream of.
Closing Thoughts on Joplin + Node-RED
Using DIY solutions like Node-RED and Joplin allows you to retain power over your data. Once that email arrives on your machine (irony of using Gmail again duly noted) it does not leave your machine to be bounced around between third parties.
The main caveats of self-hosted solutions are data retention and backups (e.g. Evernote is all in the cloud so if you lose your laptop you can easily get it back). However, Joplin has a mature sync protocol that you can use with providers like Nextcloud and Dropbox and end-to-end encryption so that if you do end up backing up to a third party you can be ~relatively~ comfortable that they can’t access your inner-most secrets.
The other thing to beware of is that Node-RED running on your local machine will only be operational when your computer is switched on. There’s a great guide for making Node-RED start automatically when your computer starts up or you could also try running Joplin and Node-RED on a server or Raspberry Pi and having your desktop Sync with them.
Thanks for reading, I hope you enjoy exploring Joplin and Node-RED — feel free to leave tips, recipes and ideas in the comments below.
Tune in next time when I’ll be showing you how to to use the Joplin-search Node-RED node with Telegram to build a chatbot that can access your personal knowledge store.
