Simple, easy, quick: using Go along with Clojure

We had questions about how Go compares with Clojure at the Boston Clojure Meetup earlier this week. My default answer to any this-vs-that technology debate is -- it depends. But it just so happens that our backend system at Glassy Media consists of an even split of services written between Go and Clojure. However, this isn't due to any technical reasoning. It’s just that using the two languages together feels right for our particular needs.

Rich Hickey made clear the distinction between simple and easy in his famous talk, Simple Made Easy. Clojure makes simple easy. But sometimes I don't want simple and easy. I want quick and easy. That is exactly where Go comes in to complement Clojure.

Simple, Easy, Quick

How Clojure made simple easy is a path well-beaten. The rest of this post is about why I think Go is quick and easy.

Our path to Go

Once upon a time (i.e. a few months ago) our backend cogs and gears consisted of a bunch of command line programs that we would run on our laptop every now and then to churn our data. We then distributed those tools to our non-developer helpers to serve our clients. Those command line tools started out in Python or Node.js. But distributing, maintaining, and updating those programs to other people's laptops eventually became a real pain. So one weekend I rewrote one of our command line programs in Go. On the Monday, I shared a link to a single executable file on Dropbox and people were able to run it directly from there. No more asking people to go on command line and typing pip install or npm install. That was how we started using Go at Glassy Media.

I had no prior experience with Go before that. The fact that I was able to learn the language and then produce something useful over a weekend was not to my credit but lies in the smart tooling, the clear documentation, and the practicality of the language. These features make Go easy to ramp up.

Language Practicality

Take the for loop as an example, you can iterate over a map like so. (code sample taken from Effective Go)

for key, value := range oldMap {
    newMap[key] = value

The same range can be used to iterate over an array:

sum := 0
for _, value := range array {
    sum += value

No need for additional syntax to do the same thing. And there are no while or do loops either. for, alone does all of that, as shown:

// complete `for` struct
for init; condition; post { }

// only check condition like a `while`
for condition { }

// or, nothing like a `do`
for { }

The Go core syntax is consistently minimal like this.

Clear Documentation

From The Go Programming Language,

 The Go project takes documentation seriously. Documentation is a huge part
 of making software accessible and maintainable. Of course it must be
 well-written and accurate, but it also must be easy to write and to
 maintain. Ideally, it should be coupled to the code itself so the
 documentation evolves along with the code. The easier it is for
 programmers to produce good documentation, the better for everyone.

Go's source code documentation has simple convention and automatic publishing. Take our trivial go-arrays library for manipulating arrays. Here is one of the functions and its docstring:

// Contains check if 's' is in 'coll' string array
func Contains(s string, coll []string) bool {

What's amazing is that a corresponding project documentation page is automatically scraped from our source code and hosted on with no extra work needed on our part. So as soon as you push a project onto Github, you can request your project documentation on website. Here is a screenshot of one: for go-arrays

Because software documentation is baked into the process, I find most of the Go libraries are less frustrating to pick up.

Smart Tooling

I don't know why GoFmt isn't more common in programming. GoFmt is a tool that automatically formats Go source code. It is like jshint or pyflakes but goes a step further and actually fixes your code styling for you too. No more gg=G in Vim.

Consistent source code styling makes life so much easier when digging into other people's projects.

Overall, I enjoy the workflow developing in Go. There is no interactive REPL. What you get is an almost instant go test and go run execution time. After doing everything in Vim, switching between my editor and the command line isn't as annoying as I would have expected.

One reason for this relative ease is that the workflow is different when developing in Go. GoFmt is integrated in Vim (and other typical editors). A second reason is that Go is statically typed, by the time when your code passes GoFmt without error, most of the time it would work as expected. So it's not very often that I need to resort to doing go test to check my code.

Go for web services

Having had Go in production for only a few months, our experience with it is limited. Still, what stands out most with Go is using it to build web services. Take our SMTP callback verification service, for example. The HTTP server code for it is under 50 lines using go-json-rest and a couple core packages. Not only that, Go has mock HTTP server testing functionality built right in.

I am aware that this is subjective. But having used a handful of languages and frameworks to build RESTful web services, Go is by far the easiest for getting a server up and running.

Complementing Clojure with Go

The one time I had to choose between Go vs. Clojure was when the Boston Go Meetup happened on the same night as the Boston Clojure Meetup.

Otherwise, there is no reason why Go and Clojure can't work alongside each other as part of a toolset suite.

Everyone has their tools of choice. Your taste and circumstance are probably different from mine. What matters is that the tools in your toolbox fit the problems that you face. Go and Clojure each answer different needs for us.

Posted 12 May 2014 in computing.

More problem solving, less solution glorifying

I have been growing fond of this trendy Go programming language lately. We use it at Glassy Media to build services over HTTP and command line. So when we needed a Amazon Mechanical Turk interface I thought why not use Go for that too? I found an existing library in Go for interacting with Amazon Web Services. But its Mechanical Turk component is incomplete. So I dove into the source code and started patching to make it work.

A weekend of coding later, I still haven't finished. Then I realised what I was doing. I was building a cool solution but not actually solving the problem at hand. This might have been beneficial for a capable company to help contribute back to the open source community. But not where I'm at -- a new startup that's stretching thin on resources. We simply couldn't afford to lose focus at this phase. So I switched over to good ole Python. The boto Python interface to AWS worked just fine. The result was that there were no "I did this in Go" blog post to write about but our problem was solved and we moved on to the next task.

This sense of building cool solutions just for the sake of it seem common in tech. Just last week at a Go meetup, someone spoke about Using Reflections in Go. Even Rob Pike, the man himself, wrote that reflection "should be used with care and avoided unless strictly necessary". Somehow that was taken as a challenge.

challenge accepted

I am guilty of glorifying new technologies and putting down on tried and true solutions too. On more occasions than I dare to admit at data conferences, I participated in bashing at SQL databases with the attendees. I can't speak for others, but for me it was due to ignorance. I used SQL a few times before but was put off by the data warehousing aspect of it. Having gotten more familiar with MySQL in particular over the past couple years, SQL can be simple and beautiful too. I have actually come to prefer it in some cases. Use the right tool for the right task, right?

An example of focusing on solving the problem. Here is one of the distilled tech specs we have for a component.

  1. A webpage for users to view a dataset of their private contacts
  2. Sorting capabilities on some of the data fields
  3. Expected number of contacts per user is several hundreds
  4. Limit to no more than 10 - 20 users at beta launch

Most of the efforts went into distilling our problem to make it as easy as possible. Like letting the user edit the data? Nope. UX design? Not until we figure out what data to surface. Because the problem specs are concise, solving this is almost trivial. Our current technical stack to solve this is embarassingly simplistic:

  • HTML/CSS/Javascript for front end hosted on Heroku,
  • Go for REST-based backend on Heroku; and
  • Google Spreadsheet as database.

Nothing fancy. Yes, we didn't even bother with spinning up a database.

People build more complicated things in a hackday than this. That's because the real problem we have is that we think this is needed with no real proof yet. Perhaps all they need is just having CSV files emailed to them? We tried that too. We want to get this out to our customers as soon as possible for them to use and give us feedback. What is the task that this component is doing for our customers in the scope of our product experience? Abstract problems like these are not trivial and validating them is why we are building things in a startup. Not the other way around.

As I was reminiscing earlier this year, effective problem solving is more about defining a clear problem than coming up with a smart solution. "A problem well stated is a problem half-solved." In a pursuit of the latest and greatest, I sometimes find myself swung too far in the pendulum towards glorifying the solution rather than appreciating the effectiveness of clarifying the problem. Sometimes the best solution is to realise not to do something.

At this early stage of our startup, the biggest problem should clearly be shipping and learning. Being a Clojure enthusiast myself, we've only used Clojure for 2 of the dozen or so of projects shipped. Most of the rest are a mix of Python, Go, Javascript, Bash scripts, or even Google Doc macros. However, don't mistaken getting things done quickly with cutting corners or committing spaghetti code. We are adament about keeping our software architecture as simple and decoupled as possible. If that's done right, the individual implementation don't matter because we can always rewrite any of them without impacting the system. That Mechanical Turk project? You can bet that we'll come back to refactor it later if it proves useful. Maybe we'll pick up on goamz again. But at least it's shipped now. I am focusing on the problem and not let building a startup become a technical challenge to satisfy my vanity.

Posted 16 March 2014 in startup.