Creating a web app takes lots of moving parts, and the way that these moving parts hang together is often hard to picture. Talking about different frameworks and stacks and the hot new javascript library – if you’re not a web developer who diligently reads their tech blogs every morning, it is easy to get lost.

To this end, I wanted to do a quick tour of the technology stack we used to create, to give you an idea of exactly how a data-driven app like it functions. The focus of the architecture underlying our Perception platform is repetition, and the ability for all the components to support this repetition will become clear.

Factory and shopfront

At the highest level, Perception is split into two parts – the server and the host. The server is where the application components are built, and the host is simply the location where they are stored, ready for consumption from the outside world. A good metaphor here is the idea of the server being the factory where the goods are made, and the host being the shopfront where they are sold. Word of warning – I’m going to flog this factory metaphor to death.

A well organised warehouse

Databases give you a structured way to ask for data, so you can be sure you are getting the right thing in the same way every time you ask. It lets you store and access all your data consistently, over and over again. Compare a well organised warehouse to a hole in the ground where you throw stuff to keep it safe. PostgreSQL is a powerful leading open source database that has been around since the 1990s. Its advantages and disadvantages are hard to quantify in many respects, but an easy one is comparing the licensing cost. Common proprietary databases are hideously expensive to license (an enterprise licence for an Oracle database can set you back $40,000 plus, yikes), and PostgreSQL gives you all the functionality you need for relatively simple data storage (relatively simple, here, means in the realm of megabytes or even gigabytes of data rather terabytes). As an atlas, we also needed a way to store geographic information (i.e. complex polygon geometries) in this database. The beauty of using an Open Source database is that the community has already provided us with this functionality, in the form of the PostGIS plugin for PostgreSQL.

Machinery driving the factory

The server environment is then your toolbox and workflow – it’s the system that holds together the different functional bits. It’s the machinery that drives the factory. In Perception, the server environment is used to extract data from the database, build the web application from its source code to be ready for use in a browser, and then deploys the built application. NodeJS is a widely used server environment that is created from V8, the very javascript engine that Google uses to build its Chrome web browser, so it leverages the power of Googles’ computing engine into a system that can be freely used on your own servers. Scripts to do the processing are then created to operate within the environment. One of the other upshots of NodeJS is that these scripts are written in Node’s own flavour of Javascript, so web developers feel right at home coding in NodeJS.

The logistics team

The final component in the server environment is Devops. Devops is the way in which you guarantee that your build process is repeatable and triggerable. In our factory metaphor, it’s probably best comparable to the team that handles logistics. Devops is used to set up what processes need to be done, when, and gives you an interface to manage the jobs – it bridges the gap between “development” (i.e. coding and app creation) and “operations” (i.e. the people actually using the app). The advantage of putting Devops in place is that you can rapidly, efficiently, and consistently re-release your app, with new data or new features. For example, when I have a new dataset to add to Perception, I work with the data locally until it looks the way that I want, then I upload it to the database. Once it is in the database, all I need to do is say “Hey, Devops. There’s some new data.” and in 15 minutes, without having to twiddle so much as one knob, there is a new version of Perception on the website ready for its adoring public. We use Jenkins here at SV, which is fantastically solid Devops software styled after a fictional butler who organises all your jobs for you (which I suppose makes me the bumbling aristocrat that he’s constantly picking up after)

Using Jenkins, we can define jobs, set job triggers, and then have them cascade into each other, so that you can set off a whole chain of deployment events just by sending an update of your changes. For example, we have a build-data job, which runs a NodeJS script to extract data from the database. After this job is finished, the build-app job automatically starts, and creates a functional version of the web application. Once this is done, the whole package is arranged and shipped to the host server, where it is ready for consumption. The whole update process is automated and easily repeatable.

The Shopfront

So that’s how things work behind the scenes (at the factory), let’s talk about the application itself, A.K.A. “the shopfront” that your shoppers visit to check out your products. How’s your customer service? How are your product displays? Is it laid out in a sensible way? Can people find what they want? The driving principle in the design of the Perception app is, once again, repetition. For this, we employed the AngularJS javascript framework. AngularJS is a library used to create data-driven web apps. It puts repetition at the forefront of its functionality, allowing developers to define and use data structures throughout the application, rather than needing to recreate them every time. It means you don’t need to arrange every display in your shop – you make one display and then all your other products slot right into it. You can template up a page of your website in a few lines of code, throw some data at it, and it can output a page with thousands of data items on it. What this means is that the work put into designing the application is done a total of one time, but it can produce as many pages of content as you need. It’s a perfect framework for Perception, because it is designed to be reproducible scalable and, importantly, reproducible.

For mapping in the browser, Perception makes use of the popular Leaflet mapping library. Leaflet plugs into AngularJS, converting the standalone mapping package into a data-driven machine. Likewise, the charts you see in Perception are produced with a third-party library plugged into AngularJS. D3 is a visualisation library created by the famous Mike Bostock (well, famous in certain circles). Used to create SVG charts, it comes pre-canned with a huge variety of graphs that use the data that is already driving your AngularJS app.

And finally, the whole app needs to look nice, doesn’t it? That’s where Bootstrap comes in. One of the greatest challenges for a web developer for what felt like the longest time was making sure that your website looked good on all devices. Often (and you’ll still see this occasionally) you would need to produce two completely different versions of your website – one for desktop and one for mobile. Good riddance to that. Nowadays, consistently styled layouts and responsive sizing can all be catered for from a single composition, thanks to libraries like Bootstrap. Bootstrap is the system of measurements and standardised elements that you use for arranging your shop so that it suits anybody walking into it (although the metaphor does break down slightly here since nobody designs a shop to shrink and grow depending who the customer is…)

And that is the tech stack for Perception!