When the startup shut down there were still dozens of netbooks out there in the wild collecting data on the residential houses fitted with our adaptive heating control algorithms, hopelessly attempting to connect to our VPN server that didn’t exist anymore in order to upload all that data to our now-defunct database. That’s a lot of data, sitting and growing on a lot of internet-connected devices.

Some of us came together and figured it could be possible to resume collecting that data, and showcase the benefits of having our system installed on your house. The first problem was, how do we connect to these netbooks? And at near-zero cost?

Warning: hacks ahead.

We figured that step one would be to establish a reverse SSH tunnel to each of these netbooks. A reverse SSH tunnel is set up when an otherwise-inaccessible device (in our case, the netbooks) connects to a publicly available SSH server, opens a port on the server, and forwards (“tunnels”) all incoming connections to that port back to the device. This is the best solution to connect to a device that’s not exposed to the public internet short of setting up a proper VPN solution.

To set up a reverse SSH tunnel you first need a publicly available machine running an SSH server and that will accept reverse tunnels. The good news is that you can all have one by signing up to Amazon Web Services (AWS) and going to the Elastic Cloud 2 (EC2) service:

Next you want to launch an instance:

You really want the smallest, freeest possible machine here that runs Linux:

Make sure you have generated a key pair for this instance (and that you have saved the private key!) and that the machine accepts SSH from anywhere:

But when you set up an SSH tunnel you will also need to make sure the EC2 instance accepts SSH traffic on the ports that will be opened by the tunnel. These are up to you; I have created two tunnels, one on port 7030 and one on 7040, so navigate to the settings for the security group of your instance and make sure the instance will accept TCP traffic to these ports:

That’s all on the server side. On the netbook side you need to do three things: 1) get the private key, 2) change the file permissions on the key, 3) establish the tunnel.

Getting the private key to the netbook is entirely up to you. What I did, and which is absolutely not recommended, was to place the private key neurobat.pem on the same web server hosting this blog. Then I was able to get the key with

wget --no-check-certificate davidlindelof.com/<path-to-key>

(Notice the --no-check-certificate argument. Those netbooks are hopelessly out of date and won’t accept HTTPS certificates anymore.)

Next you need to set the right permissions on the key, or SSH will not accept them:

chmod 400 <path-to-key>

And finally you can set up the tunnel, say on port 7000:

ssh -i <path-to-key> -fN -R :7000:localhost:22 ec2-user@<ec2-ip-address>

If all went well you’ll now be able to ssh into the remote device by sshing to your EC2 instance on port 7000:

ssh <username-on-device>@<ec2-ip-address> -p 7000

As an extra precaution you might also want to look into using the autossh program, which can detect connection drops and attempt to reconnect.

Clunky? Sure. Hacky? You bet. Brittle? Oh my god. But it did the job and I can now work on doing things the “right” way, i.e. setting up a proper VPN solution, probably based on OpenVPN or something.

It’s Monday afternoon. It’s a holiday but I have a couple of things to catch up from last week that I didn’t finish. The rest of the family is either on holiday camp or taking a nap in the bedroom. I’m working from home. But the home is anything but silent.

I can hear the girls’ muffled chatting, from the sound of it they’re making up some story with their dolls. The village church bell just tolled a single note for the quarter past the hour. My phone’s notification just dinged, and in a rare moment of self discipline I don’t pick it up. Some birds are chirping outside. The convection oven in the kitchen has had a malfunction in years and emits a beep every 10 seconds that I have learned to ignore. Occasionally a plane comes in overhead to land on Geneva’s airport; there’s only one landing strip and depending on the direction of the wind, planes come in from the direction of our village. And on top of it all I hear some kind of background whine that’s very soft–I usually don’t notice it but it’s definitely there and I don’t know if it comes from outside of me or from inside my head.

That’s a lot of noise. It’s also the best possible working conditions I’ve ever experienced. Today I’ve chosen to deliberately notice all these sounds and now I cannot unhear them.

Then there’s the visual distractions. I’ve been working for the past three years from a corner in the living room, the rest of which fills my field of view, as well as parts of the kitchen.

These working conditions sound bad but they can be fixed. I usually set a screen between me and the rest of the living room, and almost always do my deep focus work wearing noise-canceling over-the-ear headphones, playing focus-friendly music. My family knows that when daddy wears the headphones, he is not to be disturbed unless there’s blood or fire. It mostly works.

Like many others, I used to work in an open-space office. Noise-wise and visual distraction-wise, open-space offices are possibly better than working from home. On more than one occasion, visitors from abroad have been impressed by the museum-grade silence filling a Swiss open-space office. But open-space offices offer a richer set of options for not concentrating on your deep work. Entire days can go by, being interrupted by colleagues, taking a walk to the cafeterias, listening in on neighboring conversations, attending more meetings than you should because you fear you’ll miss out. And the siren song of office perks, of course.

The choice is between perfect quiet filled with distractions, or constant information-free background sounds that you can learn to ignore with monk-like focus. I’ve tried it all and I know what works for me. Do you?

In my current team we have decided to split up the work in a number of workstreams, which are in effect subteams responsible for different aspects of the product. One workstream might be responsible for product instrumentation, another for improving the recommendation algorithms, another responsible for the application’s look and feel. Each workstream has its own backlog and its own set of quarterly commitments, which map nicely to quarterly OKRs.

Workstreams aren’t necessarily disjoint: the same person might contribute to more than one work stream. Indeed for specialists (UX researchers, UI specialists, data science), that is almost the norm. As an aspiring data scientist myself, I contribute to several workstreams; I may entirely own a key result assigned to a workstream, or provide input (e.g. statistical advice, experiment sizing, etc) to another.

We don’t do daily standups, not even among the software engineers. Instead we meet twice weekly for 30 minutes and review the current plans, update the board, and make sure no one is blocked.

We’ve adopted this process early this year. The response from the team has been generally positive. Compared to a more traditional front-end vs back-end division of labour, the team has cited the following benefits:

• tighter team cohesion
• better understanding of what the others are working on
• more productive team meetings
• greater sense of accomplishments

The main drawback with this system affects those of us in a more specialized role, such as UI, UX, or Data Science, who contribute to more than one workstream. We find ourselves compelled to attend the semi-weekly meetings of all the workstreams we are involved with, and never know which ones we can safely skip. On top of this I also have a weekly Data Science sync with the product manager.

At a recent retrospective we have agreed to mitigate these issues by the following:

• notes should be taken at all meetings, and the note-taker should remember to tag any team member who might be absent but who might need to know something important;
• we will shorten the sync meetings to 15 minutes, and defragment them so that two workstreams could have their syncs done in the same half-hour (and sometimes the same room).

I can’t say that this is the final perfect solution to embed a data scientist in a product team but at least we have an adaptive process in place: a system to regularly iterate on our processes and give the team permission to adapt their working agreements.

Are you a specialist embedded in a product team mostly made up of software engineers? How do you interact with the rest of the team? I’d love to hear your story in the comments below.

A while back I had the pleasure to address a team of user experience researchers at YouTube, and I got asked for a few resources that could help someone pretty good at science, math, and programming who wanted to get into data science. Here’s the list I gave. These have worked for me in the past, with the caveat that I’m very partial towards books.

Absolute must-reads

Both are freely available, outstanding books that cover a LOT of ground. The former uses R and goes somewhat deeper in theory, while the latter uses Python and is perhaps more practical, covering iPython, Numpy, and the scikit-learn ecosystem.

Great too

One of the clearest expositions of fundamental statistical concepts I’ve read. It’s also well written and avoids dry, lifeless prose; the author does a great job at discussing the pros and cons of each technique, and frequently gives templates on how to present the results. One of the most memorable passages was his/her (read the text to understand…) rant against the use of p-values AFTER looking at the data. Free book.

Hadley Wickam’s companion book to the tidyverse. Essential reading if you’re into R and use the tidyverse. More oriented towards data manipulation and programming than actual statistical modeling. Free book.

For the brave

The “grown-up” version of ISLR (mentioned above). Covers a lot of theoretical ground, including a great discussion of the variance-bias tradeoff so beloved of interviewers. That book taught me to stop blindly normalizing covariates before running clustering algorithms.

Harrell is to statistics what Wickham is to data manipulation: the opinionated author of some amazing R packages that do a better job than the ones provided in base R. It’s a very dry text though, and probably better read in conjunction with some explanatory blog posts. Furthermore, it can be difficult to find resources online because these packages are not as widely adopted as the tidyverse.

Summer reading

Joel Grus is amazing. In this book he shows how to code (and test!) many constructs used in Data Science, culminating with a pseudo-relational database.

Oh you think you know statistics?

I’m including these two books because I think reading them will make you a better statistician. The former is a short but mind-blowing read that will make you rethink every analysis you’ve ever done. The latter is the must-read text if you’re going to do any kind of causal inference.

Non-book resources

Machine Learning

Deep Learning

AI nanodegree

These are some online courses I’ve taken and which I can wholeheartedly recommend, especially the first one which covers pretty much most concepts used in DS / ML. The Deep Learning specialization is more oriented towards neural networks, while Udacity’s AI nanodegree has probably nothing to do with DS but is a great intro to topics like building game-playing AI or path-finding algorithms.

Am I missing something? Feel free to add your own recommendations in the comments below.

Dear future self,

I’ve just lost (again) about half an hour of my life trying to find a vaguely remembered formula that generalizes the law of total probability to the case of conditional probabilities. Here it is. You’re welcome.

The law of total probability says that if you can decompose the set of possible events into disjoint subsets (say $B$ and $\overline{B}$), then (with obvious generalization to more than two subsets):

$$\Pr(A) = \Pr(A \mid B) \Pr(B) + \Pr(A \mid \overline{B}) \Pr(\overline{B})$$

But what if you’re dealing with $\Pr(A \mid C)$ instead of just $\Pr(A)$? What’s the formula for the law of total probability in that case? What you’re searching for can be found by googling for “total law probability conditional”:

$$\Pr(A \mid C) = \Pr(A \mid B, C) \Pr(B \mid C) + \Pr(A \mid \overline{B}, C) \Pr(\overline{B} \mid C) $$

There’s a great derivation here: https://math.stackexchange.com/questions/2377816/applying-law-of-total-probability-to-conditional-probability.

I’ve been collecting all XKCD comics related to Data Science and/or Statistics. Here they are, but if you think I’m missing any please let me know in the comments. Use at will in your data visualizations but remember to attribute. Sorted in reverse chronological order.

Cross-validation—the act of keeping a subset of data to measure the performance of a model trained on the rest of the data—never sounded right to me.

It just doesn’t feel optimal to retain an arbitrary fraction of the data when you train your model. Oh and then you’re also supposed to keep another fraction for validating the model. So one set for training, one set for testing (to find the best model structure), and one set for validating the model, i.e. measuring its performance. That’s throwing away quite a lot of data that could be used for training.

That’s why I was excited to learn that bootstrapping provides an alternative. Bootstrapping is an elegant way to maximize the use of the available data, typically when you want to estimate confidence intervals or any other statistic.

In “Applied Predictive Modelling“, the authors discuss resampling techniques, which include bootstrapping and cross-validation (p. 72). The authors explain that bootstrap validation consists in building N models with bootstrapped data and estimating their performance on the out-of-bag samples, i.e. the samples not used in building the model.

I think that may be an error. I don’t have Efron’s seminal book on the bootstrap anymore but I’m pretty sure the accuracy was evaluated against the entire data set, not just the out-of-bag samples.

In “Regression Modelling Strategies“, Frank Harrell describes model validation with the bootstrap thus (emphasis mine):

With the “simple bootstrap” [178, p. 247], one repeatedly fits the model in a bootstrap sample and evaluates the performance of the model on the original sample. The estimate of the likely performance of the final model on future data is estimated by the average of all of the indexes computed on the original sample.

Frank Harrell, Regression Modelling Strategies

Ask any programmer what their favourite programming book is, and their answer will be one of the usual suspects: Code Complete, The Pragmatic Programmer, or Design Patterns. And rightly so; these are outstanding and highly-regarded works that belong to every programmer’s bookshelf. (If you’re just starting out building up your bookshelf, Jeff Atwood has some great recommendations).

But once you get past the “essential” books you’ll find that there are many incredibly good programming books out there that people don’t talk much about, but which were essential in taking me to the next levels in my professional growth.

Here’s a partial list of such books; I’m sure there are many others, feel free to mention them in the comments.

Growing Object-Oriented Software, Guided by Tests

Here’s a fun weekend project: scrape the real estate classifieds of the website of your choice, and do some analytics on the data. I did just that last weekend, using the Scrapy Python library for web scraping, which I then let loose on one of the major real estate classifieds website in Switzerland (can’t tell you which one—not sure they would love me for it).

After about 10 minutes I had the data for 12’124 apartments or houses for sale across Switzerland, with room count, area, price, city, and canton.

I’ve imported the data in R, and log-transformed the room count, area, and price because of extreme skewness. Here’s the resulting scatterplot matrix, obtained with ggpairs():

There’s a number of interesting features, even from this raw, unclean dataset:

• there are about twice as many apartments for sale than houses
• the room count comes in discrete values in steps of 0.5 (half rooms are frequently used for “smaller” rooms such as a small kitchen, a small hallway, etc)
• the room count is highly correlated with area, as expected
• the price is more correlated with the area than with the room count
• there are several extreme outliers:
  • a property with 290 rooms (was a typo; the owner meant an area of 290 m²)
  • some properties with abnormally low area (one of them was a house with a listed room count of 1 and area of 1 m²—obviously didn’t bother to enter correct data)
  • and more interesting, several properties with abnormally low prices; the lowest-priced item is a 3.5-room, 80 m² apartment in Fribourg priced at CHF 99.-.

Before we go any further, we’ll obviously have to clean up these faulty data points. There doesn’t seem to be many of them so I’ll do that manually, and write a follow-up post if I find anything interesting.

This piece is a writing assignment for the Learning How To Learn online class, in which we are asked to reflect on a recent learning challenge.

Shorthand—the ability to write at possibly over 200 words per minute—is a dying skill. The ubiquitous use of computers and laptops for taking notes and meeting minutes has turned shorthand into a curiosity, a skill reserved for a dying generation or some die-hard hobbyists. Which is a shame—there’s a kind of elegance and beauty to some of the shorthand systems out there, and who wouldn’t want to be able to write and read scripts like this:

Shorthand belongs to a family of skills that were considered essential perhaps 50 years ago, but have been made all but obsolete by technology, such as:

• Using a slide ruler
• Note taking
• Touch typing
• Handwriting
• Mnemotechnics
• Shorthand

Yet I claim that many of these, if not most, should still be taught in our primary schools; in this piece I reflect on my experience in learning the Gregg Shorthand system.

As far as knowledge work goes, I’ve had a rather typical education: Master in Physics, PhD in Physics, self-taught in Computer Programming, Statistics, and Data Science. I’ve always taken my professional development very seriously and have almost always got some MOOC going on.

Being something of a compulsive note taker, I became interested in the various shorthand systems in 2005. I researched the different systems, and concluded that the Gregg system would be ideal for me, striking a good balance between ease of learning vs writing speed. So I began to learn the system, relying at first on the vast collection of free resources available online.

But in the last 14 years or so, my enthusiasm for learning shorthand has ebbed and flowed. My commitment to learning went through spikes and valleys. I never lost interest, but other interests would inevitably take priority. With hindsight, I believe the three largest mental hurdles were the following:

• No incentive: I never entertained any illusion of gaining something tangible from learning shorthand, so my only motivation was my own curiosity.
• Lack of resources: in spite of the website mentioned above, I feel that there aren’t that many resources out there for learning shorthand. I couldn’t find any reading material written in shorthand, for example. Nor could I find any online class.
• Lack of priority: just as with anything else, the first excuse for dropping out will be the lack of time. But that’s seldom the root cause. More likely, I would frequently let other things take priority over the regular practice time needed for learning a new skill such as shorthand.

So what to do? How to get good at shorthand, when the only tangible benefit, to be honest, is the satisfaction of having learned something cool? Here is what seems to be working for me:

• The book: the free resources available online are absolutely incredible, but they’re, well, free. When I download a free book I’m not vested in it; there’s no sunk cost, so no compulsion to make something good come out of my “loss”. Not so with a physical book. I bought The GREGG Shorthand Manual Simplified, so that I would feel bad whenever I saw the book on my desk gathering dust.
• The community: sharing a ridiculous obsession with others is always more fun than being alone. I discovered a Reddit group dedicated to shorthand in general, and I joined it. Being part of such a community was a great boost to my motivation, and provided me with a place to ask questions about difficult reading exercises.
• Self-testing: the book mentioned above features many reading exercises, but doesn’t give the answers. It made it difficult for me to assess whether I was making progress. Instead, I discovered that AnkiApp, one of many flashcards apps out there, would let me download and install a deck of flashcards for practising shorthand reading.
  But what about the book’s reading exercises? How could I make sure I understood them correctly without bothering the Reddit community? I discovered a website where you can enter text in English, and it would be rendered into Gregg shorthand (to this day, I have been unable to locate a tool that would read Gregg and turn it into English). I now had all the necessary means to test myself.

Practising Gregg shorthand has now been part of my daily routine for the past couple of months; I can read the Lord’s Prayer above, albeit slowly. I am still far from being able to take meeting notes in shorthand, but I’m confident I will be able to do so in a few months.