Andrew Nisbet - Web Developer

Choosing a Suburb by Bike Commute Time

Mon, 03 Oct 2016 00:00:00 -0700

I recently moved to a new city, and I'm looking to live close enough to work that I can bike. Although I can eyeball on a map the straight-line commute distance for a potential house, often the actual time you'd spend biking it is quite different. Dedicated bike paths are super fast to ride, bypassing stop signs and traffic lights. While airports and highways can result in time-consuming detours.

So I put together this map overlaying contours of equal commute time onto the streets and suburbs around Mountain View.

━ 15 minutes travel time

━ 30

━ 45

There's a few neat things that stand out:

I can live further away if I go south or northwest. This is probably because of bike-friendly routes: Steven's Creek bike path heads directly south from work, and there's a couple of bike routes from Mountain View to Palo Alto with barriers to cars.
It takes 15 minutes to bike to downtown Mountain View, even though it's less than 2km in a straight line. Within that distance there are two major highways as well as train tracks for both Caltrain and VTA, all with limited crossings for cyclists.

The code for this project is on GitHub. My approach was to build a grid of points on the roads around work, compute the bike time for each one, and take contours of the resulting data.

Map Data

I needed two different geographical data sources:

Directions data, for querying the real travel time between two points.
Map data, for finding valid road points to query.

Google has an API for this sort of thing, but the number of requests is limited and you don't have any control over how the travel times are calculated.

Instead I used Project OSRM, an open source routing project that uses OpenStreetMap data to give travel directions. I pulled all my road bike journeys from Strava and used the average speed for the bike travel times. I also assigned a large penalty for intersections, as I've found that many traffic lights around here don't recognise bikes and have long cycle times.

Building a Grid

I used a square grid of uniformly spaced latitude and longitude values. Because of the Earth's not-quite-spherical shape, a uniform grid results in two errors (which unhelpfully work in the same direction so don't cancel out):

Longitude values are closer together at higher latitudes.
The Earth's radius gets smaller at high latitudes, making longitude values even closer together.

Fortunately, the difference isn't much for this project, it works out to about 0.5% over a 40km grid.

A bigger problem is that when you pick a point randomly on a map, there's a good chance it won't be somewhere you can bike to. Silicon Valley is full of stuff like buildings, airports, and ponds. So I fed the grid into OSRM and replaced each point with the nearest valid biking location. This mostly worked, though I had to manually tidy up lots of maintenance tracks around the shoreline that are incorrectly tagged as public access. My next project is to figure out how to contribute to OpenStreetMap.

Shifting my grid like this loses the uniformity. Large forbidden areas like the NASA airbase result in heaps of points getting redistributed just outside the boundaries with much higher density than the rest of the grid. This isn't really an issue, but mostly to keep the plots clean I pruned the resulting grid by dropping any points that were too close to another point:

for point in points
    for neighbour in points
        if euclidian_distance(point, neighbour) < threshold
            remove neighbour from points

Here's the development of the grid in an area with some forest and a lake. There's a hikers-only trail in the bottom-left that OSRM correctly avoids snapping to, with the rest of the roads and tracks evenly covered by points.

Uniform grid.

Snapped points.

After pruning.

Travel Times

I fed the evenly-spaced grid into OSRM to find the travel times:

● Up to 5 minutes travel time

● 10

● 15

● 20

● 25

● 30

● 35

● 40

● 45

● 50

Contouring

The resulting contours are overly detailed, and have weird varying scale thanks to the non-square grid and areas without points. I tidied up the lines by linearly interpolating back to a square grid, and applying a Gaussian blur to the travel times.

Interpolated contours.

After smoothing.

Stray Observations

OpenStreetMap is a fantastic project. I can't think of any open dataset that comes close in terms of size or quality.
OSRM is pretty neat too. It calculated 1000s of directions per second. Over HTTP, in a Docker container, in a VirtualBox machine, on an old laptop. It would be interesting to do a comparison with the Google Directions API to see how different the travel times are.

Modelling Animal Home Ranges

Sat, 10 Sep 2016 00:00:00 -0700

The goal of this project was to develop a mathematical model for animal behaviour that would result in realistic and stable home ranges. In this writeup I’ll explain my approach and the different models I tried.

The code for the project is in GitHub.

Approach

The home range of a herd of animals is simply the area where the animals regularly spend their time. Home ranges cover hunting grounds, nesting sites and breeding sites. This information is useful for researchers in understanding the group movement of animals.

My aim was to develop a model of individual animal movement that would result in a home range emerging.

A random walk model was used for its proven accuracy in simulating animal movement. The direction of the movement would be biased be the environment and presence of other animals.

Defining a Home Range

Before home range models can be evaluated, a definition for a home range is needed.

Data on real animals is typically collected in the form of distinct location coordinates at given times. The models developed for this project produce the same type of data: animal locations in time and space. In defining a home range the time data was discarded, leaving a set of coordinates.

Two main methods exist for relating this spatial data to a well-defined home range shape: minimum convex polygon fitting, and kernel density estimation.

Minimum Convex Polygon

The simplest method is to construct the smallest possible convex polygon (shown in black) around the locations visited by the animal (shown in blue):

This is a well-defined method that doesn’t require subjectively choosing parameters. However, because the home range includes all locations, it tends to be larger than necessary. This effect is particularly noticeable if the animal ever wanders (perhaps before forming a stable home range):

Wandering can be excluded by ignoring initial or uncommon points, but this introduces additional complexity and parameters, defeating the advantages of the method. Because this simulation project looks at animals converging to a stable home range, minimum convex polygon is not suitable as it cannot handle the initial travelling to a home range.

Kernel Density Estimation

It seems reasonable that an animal may occasionally move outside of its home range. If this is the case, a better approach to home range definition would be to consider where an animal spends most of its time, rather than all of it. Alternatively, this can be interpreted as how likely it is to find an animal at a given position.

This concept leads to a more robust method of defining home ranges: Kernel Density Estimation (KDE). Given a set of points, KDE produces a probability density function for the entire domain. Sampling the density function results in a set of points with similar distribution to those fed into the estimation.

Here’s what a KDE looks like on the same wandering walk that the convex hull failed to represent:

where darker red regions more likely to contain the animal, and therefore more likely to be in the home range. Choosing a threshold probability value gives a single line:

which appears to be a more accurate representation of a home range.

Random Walk Models

Each location in the grid can be assigned a quality q(x,y). This value represents the desirability of a certain position as perceived by the animal, and may incorporate factors such as food availability, predator territory, and climate. These qualities are combined into an environment matrix Q, with Q_xy = q(x,y).

The environment quality is the only parameter for the walk, and it acts to bias the movement direction. At each timestep the animal moves to a neighbouring grid cell with a probability proportional to the neighbouring cells’ respective quality.

So now the model of animal movement becomes a method for defining the quality of the environment.

Environment Quality

2D environments were created by taking random noise:

and applying Gaussian blur:

to make the variation more realistic. In this example, dark areas have high quality and attract the animals - which might represent an area with plentiful food or good shelter. Animals are unlikely to move towards the lighter white areas, which might represent a predator’s territory or dangerous terrain.

Let’s release a herd of 10 animals into the centre one of these environments and see what happens:

They travel together to a local maximum, with their travel history shown again in blue. The model is a form of gradient descent, where the environment is the objective function and the random walk adds stochasticity to the optimisation.

However, this nice convergence is not always seen. Multiple animal convergence depends on the steepness, size, and uniqueness of the local maximum. Here’s different random environment, with the final positions of the animals shown in red:

Because of the two local maxima found on either side of the starting point, the herd splits and animals end up in different areas with a fractured home range. This seems like an unlikely result; it’s difficult to imagine a small group of tightly packed herding animals splitting in opposite directions.

It seems that simply heading towards desirable areas isn’t enough for home ranges to emerge.

Olfactory Model

Some form of animal interaction is needed, which is often realised in nature by leaving some sort of scent. This scent diffuses and decays over time, which can be considered as a dynamic environment quality that changes each iteration.

I assumed the animals produced scent at their current position at a constant rate. The scent was then spread according to the 2D diffusion equation

which is total overkill for this project, but I wanted to get some use out of my fluid dynamics course.

The resulting model causes the animals to remain close to each other because they are attracted to the scent of the other animals. They tend to follow each other because of the scent of their tracks. And because the scent is slowly diffusing they remain largely in the same place.

Here’s how the scent attraction quality looks like at a particular point in time:

Combined model

The next logical step is to combine the environment quality model with scent attraction. This is done by combining the two qualities:

Q = (Q_environment)^m₁ +(Q_scent)^m₂

with a weighting m for each one. The result looks good:

This is the same environment that the animals failed on with the basic environment model. And now they remain together in a stable home range. Success!

Stray Observations:

This is essentially a 2D optimisation problem, which pops up a lot in mathematics. It’s cool to see that animals have to deal with this stuff too.

A Break From Personal Projects

Fri, 09 Sep 2016 00:00:00 -0700

One of the first personal projects I worked on (bookfetch) was a massively beneficial for me. What I was trying to do was so far beyond my skill level that I learned an enormous amount about programming and web development. I used these skills and experiences to land my first few jobs in web development with no formal background in software.

Since then I’ve constantly had some other personal project on the go, mostly in search of a passive income. I’ve spent hundreds of hours on these projects, but I have little to show for that time. The projects I’ve started haven’t been challenging enough for me to learn from, but were too ambitious to easily succeed.

Going forward, I’d like to spend time working on small blog-post-sized exercises. It’s much easier to find something this size that I find interesting and that I can learn from, while avoiding spending time managing servers and writing boilerplate.

This is a collection of all those sunsetted projects.

Snip

I often bookmark sites with good design or UX, and I wanted to be able to organise that visually without dealing with dead links. There would be an extension to take screenshots of websites, and then a Dribbble-like website to manage them.

Sussed

A helpdesk/CMS for freelancers. Design liberally inspired by Stripe and Invision.

Spindel

A company like ScrapingHub offering custom datasets. I still have the scrapers running, and would like to explore some of the datasets when I have time.

Stockyard

Free stock photo aggregator.

Portfolio

So many portfolio redesigns, mostly borrowed from others.

Passing Multichoice Exams Without Studying

Sat, 05 Mar 2016 00:00:00 -0800

One of my psychology lecturers required our class to buy the textbook Writing for Psychology (O’Shea et al., 2006). He referred to it as “The Bible”. The book talked about things like how to structure a written argument and format a research proposal, and it was the key to passing undergraduate psychology assignments.

The otherwise serious textbook devoted half a page to multiple-choice exam technique, with advice such as “pick the longest answer” and “if all else fails, choose C”. Like all good humour, it was hard to tell if the authors were joking or not. And the information was not referenced, even though the book has an entire chapter on how to reference information. But it got me wondering if there really was any pattern to multichoice tests that could be exploited to give the test taker an edge.

I decided to test my idea by throwing a couple of machine learning techniques at some sample questions.

Best case scenario would be to find a model that was both simple enough to use in real life (“always choose the longest answer”), and interpretable enough to reveal something about the psychological biases of a test writer (“lies come to us more easily that the truth”). But I’d settle for p < 0.05.

The code and a presentation of the results are on GitHub.

Finding a Dataset

Rather than overfitting a model to a particular style of exam (“half the answers to Prof. Smith’s behavioural economics tests are C”), I wanted my results to be applicable to multiple choice tests in general. So I was looking for a dataset of multichoice questions and answers that was typical of a wide range of tests.

I settled on the Australian Science Olymiads, which I remember taking at school. Here’s the sort of question they ask:

The organisation publishes PDFs of past exams and solutions on their website. When I downloaded the data in 2014, there were 400 questions and 1800 answers, spread over 19 exams. The exams cover three subjects: biology, chemistry, and physics. This ought to be broad enough to reduce picking up patterns in a single subject.

I was also hoping to improve individual representativeness by ensuring the exams were written by different people. To begin with, it seems unlikely that a single person would write the exams for three diverse subjects over size years. But we can go one step further by looking at the author field of the PDF documents: the 19 papers show 9 unique authors, with several papers listing no author at all. Now that’s no guarantee of distinct authors, I could be detecting the person who compiled each document, or set up the computer. But short of full on writing style analysis (which apparently is called stylometry), I’m pretty confident that the results won’t pick up on the biases and preferences of a single person.

Data Wrangling

In an extreme display of naivety and laziness, I decided to parse the question and answer data out of the PDFs programmatically, rather than doing it by hand. What should have been a couple of hours of data entry ended up taking me several days of programming, and I still ended up with a fairly large error rate.

First I tried parsing the PDF binary, but was bitten by the “presentation over semantics” philosophy of PDFs. The biggest problem was getting the answer labels to correspond with the answer text. The extracted text would look something like this:

a.
Provision of shade for their root systems.
b.
c.
Elimination of excess water that is entering via the roots.
To allow for leaf damage by insects.
Acquisition of as much light as possible for photosynthesis.
d.

which is parsable, but there were about five different variations of this in each document requiring special parsing logic. All to render an identically-formatted result. So I supplemented the parsing method with optical character recognition, taking whichever result didn’t throw an error and favouring OCR to break conflicting results.

Finally, there were lots of questions that were too abstract to be parsed by any method:

In total, I managed to parse 80% of the questions into a structured digital format. About 10% were impossible to parse, and the remaining 10% is where my program failed.

Feature extraction

For each answer I extracted 32 features from both the answer text and the question text, such as

Number of words
Average word length
Inverse question logic: the question is worded something like: Which of the following statements is INCORRECT?
Value rank: for numerical answers, the size of the number relative to the other answers

I avoided adding metadata about the exams, like the subject or the author. Missing values were replaced with mean values from the other questions. As labels I had whether each answer was correct or incorrect, from the provided answer keys.

Modelling

A regression analysis showed no significant correlation for any one parameter, nor for all the parameters as a whole. This likely rules out the existence of a simple rule-of-thumb model.

So next I tried a neural network with a single hidden layer of 16 nodes. I suspected there would be some nonlinear interactions between the features I selected - in particular a feature of the question interacting with a feature from the answer. Neural networks can automatically tune for higher order features, however using a neural network practically rules out a psychologically interpretable result.

Results

If the labels were chosen randomly, you’d expect to score 21%. The neural network had a test score of 26% averaged over ten runs. The model is 5 percentage-points better than random choice. So if you had to guess 20 questions in an exam, you would get one more correct with the neural network model that randomly guessing.

That’s a lot of effort for a small increase. As far as time allocation goes, it’s probably not an improvement on studying. And also not an improvement on actual cheating. However the results seem significant, despite the small effect size, suggesting that there are exploitable biases in multichoice exams.

Hindsight

I did this project a few years ago, and have learnt a lot about this sort of thing since then. Here’s what I’d do differently today:

Manually define nonlinear features. The nonlinearity was my motivation for using a neural network, but there were only a few interactions that I had in mind. I might have been better off just assigning the ones I could think of, which would allow for use of simple interpretable methods like regression.
Use a NLP approach. Rather than looking for relationships between features and labels, another approach I’d like to try is treating the data as two different corpuses (correct and incorrect answers) and trying to classify unseen answers into one or the other. You’d miss out on some of the more complicated features like numerical value and anything to do with the question, but it might capture writing style better than I could design into a feature.

Static Site Hosting Speed Test

Fri, 05 Feb 2016 00:00:00 -0800

I’m running this blog as a static site, which offers awesome performance and security for very little effort. Static sites also give you the option of hosting on a static host, which tend to be faster, cheaper, and easier to manage than traditional servers. The question is which host to use?

I compared the performance of various static hosting offerings from Google, Amazon, GitHub, and CloudFlare - some of the biggest players in the hosting business.

The analysis is saved in this IPython Notebook.

The Hosts

Here are the services I tested:

Amazon S3

S3 is an online file service from Amazon, and it’s super cheap to host a static site.
Amazon Cloudfront

CloudFront is a content delivery network (CDN) that sits in front of other Amazon offerings (for this test I used S3). CDNs your files on servers in multiple locations which should mean better speeds and uptime for users. CloudFront is slightly more pricey than S3, but Amazon will give you free SNI SSL for your custom domain which is pretty neat.
Google AppEngine

Google AppEngine is a PaaS offering, but it’s really easy to set up for static hosting. You get a ton of customisation options, and it’s easy to make parts of your site dynamic later if you need. It comes with a free quota which will cover a significant amount of static traffic.
Google Cloud Storage

GCS: basically S3 by Google. Similar features and pricing
CloudFlare

CloudFlare is a CDN that will cache requests going to another service. The performance of the host behind CloudFlare shouldn’t matter - for this test I chose AppEngine and Google Cloud Storage because they were the easiest to set up. CloudFlare comes with a functional free plan including SSL, though you’ll need a paid plan for any serious customisation.
GitHub Pages

GitHub’s Pages serves a git repository on your domain. It’s free, and sites are hosted via Fastly’s CDN. However, Pages has few options for customisation: you can’t set headers, choose cache times, or use SSL. And although using git for deployment seems like a good idea, trying to do something like keeping draft posts private will have you wishing for separation between version control and deployment.

Many of these providers are available in multiple regions, so I spun up services in the US and Europe where possible. This gave a total of 10 hosts, each on a separate subdomain. To each one I uploaded an draft version of my homepage: a 3KB HTML file.

Method

I expected that the three CDN-backed options (CloudFront, CloudFlare, and GitHub Pages) would outperform the rest. To test this, I set up a script to request my homepage from one of the 10 hosts every 10 seconds. I recorded how long it took to download the page, as well as the response headers and any errors.

The script ran on servers located in West US, East US, and Europe. The idea was to roughly represent the distribution of “Western” internet users, so that values averaged over all the regions would be a first-order approximation of real usage.

I rented 3 VPSs each from Digital Ocean and Linode to lessen the effect of one host being particularly close to a requesting server’s datacentre. After running the script for a day to warm up servers and fill caches, I measured 14 days of data.

Results

In total, almost 600,000 requests were made. This works out at 4000 requests per day for each host, which seems like reasonable traffic for a personal blog.

To evaluate the hosts, I’ll compare error rates, average performance, and worst-case performance.

Errors

First, let’s look at when the server responded with a code other than 200 OK (in all plots, lower is better):

Aside from GitHub Pages, all hosts perform really well. The 3 errors experienced by and AppEngine EU and S3 US still correspond to an uptime of 99.995%, and these errors are distributed in time so a browser refresh would fix any issues.

GitHub pages had a massive number of 404 errors, all within a 10 minute period:

All requests from all servers failed during this time, so no users would have been able to access the site. To GitHub’s credit, the issue was reported on their status page. However, they seem to have a similar issue every month.

Winners: AppEngine US, CloudFlare, GCS.

Average Speed

Time to Last Byte (TTLB) is the time taken to fetch all the data of a webpage, so is a pretty good proxy for when a user will see a completely downloaded site. (I also recorded Time to First Byte, but they weren’t significantly different for such a small payload.)

The requests are made from well-connected datacentre rather than something more realistic like a mobile network. These timing values represent a lower bound for real life performance, though the ordering of the results should remain the same even under high latencies.

Here’s the TTLB for each host:

It looks like S3’s Europe region is really struggling! It’s possible that this is made worse by the majority of the requests coming all the way from the US, so here’s the TTLB from each host to just the European requesting servers:

Even for requests within Europe, the US regions for S3 and AppEngine perform better than the EU regions, and there’s little difference for CGS. Perhaps these EU regions were created for data privacy reasons, though the AppEngine setup page hints at performance:

I’ll exclude S3 EU for the rest of speed analysis, as it’s scaling the plots too much. Let’s try the TTLB again:

That’s better. Now we can see that CloudFlare and GitHub Pages are both coming close to the all-important 100ms threshold for human perception of instantaneous change, with little variation shown by the quantiles. Amazon’s offerings (S3 US and CloudFront) lag behind, along with AppEngine’s EU region.

Winners: CloudFlare, Pages.

Edge Case Responses

Average times are a useful metric for overall performance, but the extremes are important too. A high 90th percentile response time may not affect average TTLB, but could cause that 10% of your userbase to get frustrated and leave.

Here’s the 90th percentile TTLB for the hosts: 10% of users wait longer than this to make a request. And with a typical website requiring 10s of requests for scripts and other files to render a complete page, a large delay in any one of these critical assets will delay the entire page.

I feel like we’ve seen this plot already. GitHub pages performs extremely well, with Amazon and AppEngine Europe brining up the rear. It’s unusual to see such a large difference between the two CloudFlare hosts, perhaps this indicates a high level of variability in the data.

We can look at the 99th percentile too: The numbers are bigger, though these events occur much more rarely. The CloudFlares are together again. Actually I’m surprised by how well all the hosts perform, 99% of requests being completed under a second.

CloudFront has risen to the highest response times, as it seems to refresh its cache fairly frequently: 2% of request miss the cache (one every 20 minutes), compared to 0.11% for CloudFlare and 0.3% for GitHub Pages. A cache refresh incurs the regular latency of CloudFront plus a round trip to the S3 backend.

Finally, the proportion of requests that take longer than an instantaneous-feeling 100ms:

Our leaders return slow-feeling responses in only a quarter of requests. Behold the power of static sites.

Winners: Cloudflare, GitHub Pages, GCS.

Results

CloudFlare is the clear winner of this test, even on the free plan. Considering the effect size of the results, you probably also wouldn’t notice any meaningful difference with Google Cloud Storage or AppEngine’s US region.

GitHub pages was just as fast as CloudFlare but seems to have downtime issues. I also feel the lack of configuration would be a problem in a lot of use cases.

S3 is slow, but has a lot else going for it as the backbone of Amazon’s AWS infrastructure. I was surprised how poorly CloudFront performs though, given how frequently I see it deployed.

So now this blog is hosted with CloudFlare, backed by Google AppEngine for ease of deployment.

Initial Commit

Thu, 04 Feb 2016 00:00:00 -0800

We are fortunate in the web development industry that the quality of our work can be so easily assessed. Instead of relying on references and qualifications, it’s easy to share source code and deployed web work.

What’s missing is the human component: the reasoning and explanation behind what we do. I think this is important for demonstrating competence, and a blog seems an excellent place to share that information.

But mostly this blog is an excuse to experiment with data, writing, and maths. Both for professional development and because I genuinely enjoy these things. For some reason, writing about something you’ve done seems to make the whole activity justifiable amid other deadlines and commitments.

Not sure why that is. Maybe I’ll write a blog post about it.

Andrew Nisbet - Web Developer

Choosing a Suburb by Bike Commute Time

Map Data

Building a Grid

Travel Times

Contouring

Stray Observations

Modelling Animal Home Ranges

Approach

Defining a Home Range

Minimum Convex Polygon

Kernel Density Estimation

Random Walk Models

Environment Quality

Olfactory Model

Combined model

Stray Observations:

A Break From Personal Projects

Passing Multichoice Exams Without Studying

Finding a Dataset

Data Wrangling

Feature extraction

Modelling

Results

Hindsight

Static Site Hosting Speed Test

The Hosts

Amazon S3

Amazon Cloudfront

Google AppEngine

Google Cloud Storage

CloudFlare

GitHub Pages

Method

Results

Errors

Average Speed

Edge Case Responses

Results

Initial Commit