What I learned from 2 years of 'data sciencing'

Last week was my last one at uSwitch.com. I first became aware of 'data scientist' as a valid job title when I read it on my own offer letter, two years ago.

sad day today as @uSwitchEng had to say goodbye to the amazing @Quantisan. you will be missed and best of luck in the next great adventure!

— Tim Goodwin (@timrgoodwin), December 18, 2013

I spent most of those two years cleaning and shaping data. Some takeaway lessons, then, while the experience is still fresh.

It's not about the science but the data.

In my first year our team of two delivered a handful of data projects. We built a dashboard showing lifetime values for all of our millions of customers. We demonstrated a 6 percent revenue gain with a product showcase sorting algorithm modelled on the multi-armed bandit problem. We simulated offline advertising impacts to online sales for optimising marketing spend, saving £20,000 a month.

None of them gained traction within the company. All of them were abandoned. I still don't have a clean answer for why.

Much of the effort on those projects went somewhere else entirely. We needed to capture events across applications on different technology stacks, associate individual events to unique customers, and be able to process all that data in an ad hoc manner. Over the course of that year we built and evolved a distributed data architecture and scalable data workflow based on open source tools and publications from companies like Google, LinkedIn and Twitter. I scratched enough of my own itch along the way to end up as one of the maintainers of Cascalog.

Congratulations to @Quantisan for becoming a Cascalog committer today

— Nathan Marz (@nathanmarz), March 14, 2013

On a rather "by the way" note, we structured lifetime views of customer data from the disparate signals across company verticals. Word had been going around the company that we had this new business intelligence tool, and more and more people asked us to help them answer questions with data on their side of the business. It satisfied a widespread need to understand customer behaviours. Little did I know that we'd be cleaning and shaping data for most of my second year. Our commercial team has since released an external data product off the back of it that I can't say much about, but which might bring in sizable benefits for the company soon.

It is glamorous to talk about the latest and greatest machine learning or data visualisation. In practice, I was just cleaning and shaping data. Enabling more people to make use of deep and structured data was the part that delivered value to the company.

Figuring out the right problems to solve is not easy.

So why didn't we do the customer behaviour work in month one? I don't think it would have worked. Nobody knows a data team exists yet, so nobody comes asking.

The fact that the data science role is so vague doesn't help. The marketing crew think we are mining for customer insights. Developers think we're toying with Riemann, Storm, or something bleeding edge. Product managers think we are plotting graphs. Each of those first-year projects was an answer to one of those guesses. They failed as products. My guess is that they worked as advertising (i.e. people can't ask for something they can't picture) and that they built the plumbing besides. The customer data existed because the abandoned projects had needed it first.

That's a flattering story, and I might be fooling myself.

Seeing that this is data science, why not dive right into the data like they would often say in hackathons? I made the mistake in the first few weeks of my data science career of hacking away at the data and then trying to persuade people to make use of the result ... somehow.

Some interesting graphs came about. But as Marc often likes to ask, "so what?" Unless someone or something can act on the data, results can only satisfy intellectual curiosity. A business can't survive by funding people to carry out academic studies forever.

Nowadays, we talk to different stakeholders to try and dig as deep as possible into their needs before writing any code for a new project. This is me handwaving. Frankly, I'm still learning my way and rely a lot on luck through trial and error in discovering the right problems to solve.

It is a humbling experience.

Working with Paul Ingles (fun fact: I'm one of 3 Pauls amongst like 30 devs) and Abigail Lebrecht has been frustratingly awesome. Paul is opinionated about doing things as simply as possible. On more occasions than I can remember, we implemented our own little Clojure libraries because the open source ones available were "trying to do too much". Abigail was adamant about getting the data and analyses right. "What do you mean this data is only 99 percent correct?" Working day in and day out with Paul and Abigail showed me that I still had much to learn in efficient problem-solving, and taught me to question all hidden assumptions.

I had worked in a multidisciplinary team before, as a biomedical engineer. But on my haptic-robotic therapy project, "multidisciplinary" meant a small group of professionals coming together with each person doing different tasks to get the thing to work. I never even considered going into a workshop to build my robot or providing clinical therapy for the stroke patients. Data science was the opposite. One week I might be pair programming with Paul and fighting to keep him away from my keyboard, another week integrating Riemann to monitor our data architecture. Other days I was debating with Abigail on the data mining side – usually because she had found flaws in the materialised tables that I produced from Cascalog, and then I would have to come up with a better estimation model for the missing data.

So do you want to be a data scientist?

This is it for me formally as a data scientist. I am moving back across the Atlantic to the States to co-found a new venture and continue my journey to make information accessible.

If you're starting a role that nobody at your company can define, spend the first year in other teams' huddles. That's where I found the work worth doing. Someone half-describes a problem, and between us we work out that the data we'd already built for something else could answer it. Nobody was ever going to schedule that conversation.

If cleaning vast amounts of data, being clueless as to what to do, and debating with brilliant colleagues all add up to a challenge that you want to take on, then yes, I'd recommend it.