Eating your own dog food

Early feedback is important. The earlier in the life cycle of development feedback comes in, the faster you can iterate, figure out what is working and what is not working, improve, and iterate again. You should release early and release often.

Releasing early and often usually aims at release cycles of something like 2 weeks. Depending on your kind of system, this can be shorter, but especially for native apps, much shorter release cycles aren’t really feasible. An even quicker way to get feedback is to give your software into the hands of your own colleagues and selected testers – constantly. Within your own organization, nobody prevents you from releasing continously, as often as multiple times in a day, without the overhead of an official release. You can then take the feedback of your own peers to iterate even faster. In modern tech slang this has become known as “eating your own dog food”.

Here at Klamr we try to get ongoing development into the hands of all our colleagues as fast as possible. The key to do this is Continuous Integration, that’s where everything ties in. Here’s how we do it:

  1. Jenkins: We’re using Jenkins as our continuous integration server and use it to automate most of our tasks. For each project there is a Jenkins job that pulls the latest code regularly, builds it, tests it, and then distributes it. Jenkins is amazingly easy to set up and configure, yet incredibly flexible and powerful. Ever since we started using it, it has grown with us into dozens of very different jobs for pretty much every project we’re working on.
  2. GIT branching strategy: while working on new features we need to decide when exactly changes should be made available internally. The general requirements are never to break builds altogether and not to break core functionality. We don’t pull every single change that is made anywhere in the project. We hook our Jenkins job into our GIT branching strategy to give the responsibility to decide which change is ready to our engineers. They have control over it by pulling changes into certain branches when they are ready.
  3. Schedule your distribution: depending on the project we either distribute immediately on every new change, or nightly. This is configured in Jenkins. My personal rule of thumb is: the more transparent new versions are for your (internal) users, the quicker and easier distributions/deployments are, and the less frequent commits to your distribution branch are, the better is to distribute changes immediately. When starting a new project, I generally start with this. Once problems appear that can be solved by slowing down, go to nightly distributions. Everything running server-side like a web app, for example, is completely transparent for users (just as they are in your production environment), new versions aren’t disrupting anybody. That’s a good candiate for very frequent distributions. An iOS application, on the other hand, needs to installed manually, hence pushing out 20 new versions every day tends to be disrupting for everybody. The last thing we want to do is make our co-workers feel disrupted and annoyed, that just leads to less and worse feedback.
  4. Distribute: the actual deliveries are all automated, but differ quite a bit depending on the type of software. Some examples of what we do:
    • Backend application: this get built and deployed to internal servers. This is the most complex deployment process we’ve got, especially things like database migrations don’t make it exactly trivial.
    • Web application: our klamr.to web application is deployed on every new change to an internal, protected web server. It is then connected to our live database, so everybody in the company can use this web application instead of our live production web application. Changes on here have sometimes only been finished for minutes until they get available.
    • Android: our Android app is distributed in two ways: new APK files are sent out directly via email (Android makes installing new APK’s directly from email attachments so much easier than iOS) and via the service Appaloosa Store. The latter has some nice advantages like providing a custom store app and push notifications for new versions.
    • iOS: our iOS app is distributed via Testflight. There’s a few catches for iOS, for example that you need to build on a machine running Mac OS. That’s why we have a separate Jenkins instance only for building the iOS app. Most other Jenkins jobs are running on one Linux-based instance hosted on Amazon EC2. Also, devices must be explicitly registered in your ad-hoc provisioning and Apple restricts the number of internal devices to 100. No rocket science once it’s all set up, but a few extra hoops to jump through.
  5. Real data: It’s important to allow internal users to use these early builds against their real Production data. Our web application, for example, runs on an internal URL, but is configured against our Production servers and database. This allows us to test drive new features early on with our real accounts. This leads to much better feedback than asking people to test features on isolated servers with fake data and helped a lot with internal acceptance.
  6. Automate: the key to all of this is automation. If it’s not automated, regular distribution either doesn’t happen, or it wastes valuable engineering time. And as mentioned already above, this all ties into continuous integration. Much of the process and infrastructure described above should be in place anyways to continuously build and test your software in an automated way.
  7. Release notes: for us it proved incredibly helpful to automate release notes for each internal distribution. Remember that one of the main reasons to do all this in the first place is to get early feedback. Without release notes, it’s not possible for anybody to know what has changed and to know which part of your apps to pay attention to. We’re not doing this in all places but if we do it, we’re using GIT commit comments. They aren’t suitable for end users, but they are more than good enough for internal users.
  8. Respect: although these builds are only internal, we highly respect them. This means we never try to break them (see above), we try to make using and updating them as easy as possible for our co-workers, and our engineers are quickly reacting to any kind of feedback that comes in.

Regular internal distribution helps us to keep the feedback cycle as short as possible, sometimes even down to minutes. Automation of all the tasks involved helps us to keep moving fast, even as the number of systems and their complexity grows. I would highly recommend trying to automate as much as possible right from the start.

Are you eating your own dog food? What is your experience with this? Are you using different techniques and tools? Leave a comment, I’m very interesting to hear what you’re doing.

A/B Testing on iOS and Android (and others)

A/B testing (aka. split testing) has been around on the Web for a long time. The reason to do it is simple: much of how you design your user interface and your user experience is based on assumptions and no matter how good you think your assumptions are, you don’t know what works better in reality. And let’s be honest, most assumptions aren’t thought through carefully in the first place. After all, nothing can tell you better than real data from real users.

You’re already measuring what users are doing? That’s not quite enough yet. The reason why it’s “A/B testing” and not just “A testing plus analytics” is simple: numbers alone don’t always say much (e.g. vanity metrics). Comparing 2 numbers on the other hand is easy and reliable and even if neither of the 2 is really good, at least you know which one is the lesser of 2 evil and the one you should continue with (read: continue with and iterate again, fast).

One presentation that caught my interest early on was Marissa Mayer’s keynote at Google I/O in 2008. She gives insights on how far Google actually goes to fine-tune and optimize their user interfaces using A/B testing. That way, Google found proof for details that would usually be decided upon by a design or UX specialist, even down to the “right” whitespace between different parts of the page on google.com:

(…) you can test interfaces and be able to tell in a mathematical way which one works better for your users.

Fast forward to today. A/B testing is old news for websites, but still only just beginning for native mobile apps. It’s easy to get started on reasons, like:

  • Native apps get bundled as binaries, no code is running on your own servers where you can easily control them, implementing different behaviors of the same feature is arguably more difficult.
  • Release cycles are different. New versions still get “shipped” (just like software in the old days) and you don’t to send your users through this every other day or so.
  • And even if you would want to, on iOS it’s Apple who prevents you from doing so with a 1 to 2 weeks approval time.
  • It’s just relatively new and there isn’t much written, there aren’t many tools and frameworks and there isn’t much of a best-practice and universally established mindset existant around it.
  • And more…

The bright side: it’s changing now.

Tools and frameworks are appearing and making this whole story a whole lot easier. Some of the ones that look great are clutch.io, arise.io or Pathmapp. And even Amazon joins with their own A/B testing service.

I’m excited to give these tools a shot and see what they can do. We’ve recently been accepted into Pathmapp’s beta program and I’m thrilled to see how good it works. I’m going to follow up with experiences in later posts. In the meantime I’d be more than happy to hear your experiences and recommendations.

3 steps I should have taken to get started with Web and App analytics

In the last weeks I wrapped my head around client-side analytics here at Klamr. Looking back, probably the biggest Gotcha! was the realization that ignoring all the cool out-of-the-box stats and colorful graphs would have been better. It’s all to easy to get paralyzed by the sheer amount of information each tool provides you. The answer you’re  looking for isn’t between visits and pageviews and referrers and user flows and mobile data and custom segments, so don’t spend your time looking for it there (just yet). Eric Ries calls them “Vanity Metrics” and tells you why they are dangerous. I should have listened earlier.

It actually starts with making sure you know what your app is supposed to do and what user actions are valuable for you. That’s all that really matters in the beginning. Whether you’ve got 50 or 5000 sessions and whether your median session length is 34 or 58 seconds, that doesn’t tell you much yet. So, better start with what matters and ask yourself: what are the few important actions I want my users to do with my app?

Hence, if I would start over again, I would do the following:

  1. Make a list with the 3, 4, 5 most valuable user actions in your app, on a very high level. For a shopping app, a completed purchase should obviously be on this list. For a photo sharing app, a shared photo should be on. A blogging platform would include creating a new blog and publishing a post. Note that this list should contain the successful  final action at the end of activities, not the actual activity leading up to it. Analytics tools often call them “events”. A purchase, for example, usually consists of a relatively long flow, but for now only the successful completion matters.
  2. Determine where and how these events are happening in your app, i.e. the lines of code that actually perform them. Find out whether you wanna instrument your client application (web or native) or whether you better do it on your server backend. Instrumenting your client often tends to be easier tool-wise, but sometimes it’s just better done on the server. But keep in mind that you try to find out how to improve your client. Getting the number of purchases, for example, isn’t the actual goal here. What you actually want to do in later iterations is find out where you can improve user flows, UI and UX to optimize. That’s why server-side instrumentation doesn’t work out well later.
  3. Instrument your system with the tool of your choice, e.g. murm.io, Google Analytics or Flurry. Then release it and wait for data. Instead of tons of numbers you should now see your few major numbers showing up in the tool.

See what this gives you and then iterate. You can add more (refined) events, categorize them like a Pirate, add funnels and goals, and go deeper. And in one of your next iterations you can look at what you can do to improve.

The Best Damn Ship In The Navy?

Recently I’m trying to soak up lots of material about Leadership. After stumbling upon It’s Your Ship: Management Techniques from the Best Damn Ship in the Navyagain and again and now have it gotten recommended by a colleague as well, I finally decided to read it. I actually picked up the Audible audio book since there wasn’t a Kindle version available and I needed it immediately and take it on a trip.

Bad news first:

  • The US Navy as a context for a book using real-world examples seems refreshing at first, but it’s getting more and more difficult to digest. At least for me as Non-American, turned out to be a big turn-off eventually.
  • The author keeps emphasizing his own work and contribution and achievements as opposed to talking about him and his team. Given that leadership actually means working with your team, this is very surprising and ended up being another annoyance for me.

On the upside, the book contains a lot of good examples and useful advice to become a better leader. It’s not exactly innovative and doesn’t bring up points nobody else did before, but it’s a good and easy to read (or: to listen) summary of much an aspiring leader can start doing immediately. AFAICT at least, I’m aspiring myself.

“If all you give are orders, then all you will get are order takers.”. Agreed.

A New Attempt

It’s been 492 days ago since I published my last blog post. Recently I’ve started to become motivated again to do more blogging. I moved away from hosting the blog myself at hendrikbeck.com and I’ve cleaned up and renewed everything a little bit.

One more change: I always wanted to keep my dedicated to articles related to my work and use other resources (like hendrikbeck.tumblr.com) for personal posts. In now I think that’s just BS, given that I don’t blog every day. So, I’m just gonna combine it somehow and hope that I keep up with it this time and don’t stop again after some months. Not sure how much faith I’ve got in myself… ;)

BarcampSaigon 2010

Yesterday I spent (half) a day at BarcampSaigon 2010 on the campus of RMIT in Ho Chi Minh City. I have never been to a Barcamp before and had high hopes for this event. In 2010 I had many opportunities to dive into tech communities in Saigon through similar events and I realized over and over again how much passion and determination there is around. Vietnam not only produces lots of tech talent, it also provides a fruitful ground for entrepreneurial minds. Barcamp seemed to be the ideal platform to meet, to share, to learn, and to boost the momentum that is already there.

I was more than impressed to see my expectations exceeded. Around 480 participants brought a lot of passion and participation, excellent speakers and presentations, and a noteworthy focus on entrepreneurship and startups. I’m happy to have participated, I’m sad that it’s over, and I’m looking forward to BarcampSaigon 2011.

The closing session at around 4PM

Check out more pictures of BarcampSaigon 2010 here on Facebook.

Testability is a design goal

Testing is important, let’s for a moment agree on that. Untested code might more likely break, now or at some point in the future. If I say “might”, then Murpy immediately corrects me and tells me it “will”. Definitely. Even if we manage to write perfect code right now, it’s still nice to have regression tests and be much more confident and agile for future changes and refactorings. Also, tests are always some kind of a proof that we tried our best to produce a good product, for us, for the team, for our boss, and, wait for it, for our customers. After all, if code breaks, every argument is obsolete, because no matter what, code that breaks is always wrong.

Besides my all-time favorite that there is no time to test, I also hear the following again and again: “We can’t really test that, it’s just not, um, testable!” Be it the Java application that doesn’t allow dependent services to be replaced by mock implementations. Be it that we’re hard-coding configurations in a way that we can’t switch to test configurations. Or be it that our web application can’t be easily tested because we don’t have ID’s in our HTML elements. Yes, that’s all legitimate reasons why code isn’t easily testable. But is it also legitimate not to test because of those reasons?

In short: It’s not.

If code isn’t (easily) testable, we have a bunch of options: we don’t write tests and ignore what we said earlier in this post. We test it in another context, maybe on an integration test level (for example). We test it manually, which is often an option, but then we’re missing out on all the advantages regarding repeatability, usage as regression tests, and continuous integration. Or, and that’s the point of this post, we’re changing our code to make it testable. Not always, but often that means introducing something that doesn’t purely serve functionality but is additionally introduced in order to make it testable. We might end up with 5 lines of code or even with an indirection more that we wouldn’t need otherwise. At least we can test it now. Or, even better, we think about testability from the very beginning (oh, and: test first, anyone?) and aligning design and implementation accordingly. Very quickly we will recognize untestable code, untestable patterns, and untestable designs at first sight and we’ll make slight but important changes. With best regards to our test coverage.

Don’t hesitate to make changes to your code to increase testability. After all, testability is also a design goal.

EDITED: looking back the few months since I’ve written this post I made more experiences manifesting testability and a very important yet mostly underestimated goal in software development. Today I stumbled upon a presentation by Michael Feathers of Object Mentor talking about the Deep Synergy between Testability and Good Design. I realize how much more further he goes with his thoughts, and how much more experience has to speak out of. 100% agreed, Michael, nice presentation!

A brand-new Groovy / Grails user group Vietnam

In previous posts I’ve already been trying to explore software development communities in Vietnam. Since I am working with Grails in one of my current projects (very enjoyable!) I thought I’d start an experiment and found the “Groovy/Grails Usergroup Vietnam“.

After letting it sit for a couple of days I almost forgot about it until someone actually found the site, became a member and started a discussion! Admittedly, the discussion wasn’t about Groovy and Grails but rather why it’s so freakin empty on this page but still, it’s a member and a discussion.

So, Java people out there, let’s just continue with the experiment, invite more people and see where it goes. The actual community might really be very very small in Vietnam but maybe even more of a reason to get together here.

You’re not sure what Grails actually is? In very short I would describe it as “Ruby on Rails for Java” (is that ok to call it that?) and as a very entertaining and light-weight way of developing web applications based on Java. For more please check out the grails.org or the Wikipedia article or Google.

If you’re interested now, please join here: http://groups.google.com/group/groovy-grails-vietnam. And if you think the idea’s even more stupid than offering shoeshine for flip-flops, then join the comments below.

My 5 reasons not to use short variable names anymore

Every other day I stumble upon lines like that reviewing code:

sysMgrH = vFgrMan

or

parse1(l, e, arr);
parse2(l2, e2, arr);

Please don’t do that anymore! Choose long and clear variable names, nothing else! Why?

  1. It’s actually allowed to have variable names longer than 1, even longer than 10 characters, if you need that.
  2. Your job is not about saving a couple of keyboard hits or 30 seconds of time, it’s about producing something of high quality. And something maintainable. And sustainable.
  3. Actually writing out the code is only 1% of your time compared to the rest, like e.g. thinking about what you should write. You can as well spend a few seconds more on the writing part.
  4. If you worry about having to write the same 12-characters variable name over and over again: make friends with Ctrl+Space.
  5. But the actual reason, my personal #1 reason, is the following: you don’t write that piece of code for yourself and for right now. It will be existing for the next 5 or 10 years and if you’re lucky dozens of developers will see it. And some of them have to understand it. In 2 months, in a year, in 5 years. But for them it’s not a matter of 30 seconds (that’s the time you saved 5 years ago, remember?), for them it can easily become a matter of hours. Keep in mind that on average more than 50% of software development costs are spent after the initial version is finished.

So here is my appeal: be nice and write your code as if you’d write a novel. Choose nice words, choose long words, and format everything nicely. Think of your fellow developer in the future who will thank you for that.

    Also, be aware of The world’s two worst variable names by Andy Lester.

    Coffee shops and other tools for small software startup teams

    In all my years back home I haven’t really noticed something that is very apparent in Vietnam: people working in coffee shops. Sure, you see students in cafe’s, people with laptops or people reading documents but here it’s a very common thing to do actual work in a coffee shop. No upfront investement in real estate, no contracts and it scales well whenever you need more space, or less, or just want to go somewhere else. Sounds a little bit like cloud computing.

    While a decent office has some advantages on its own, coffee shops are definitely a way to go for small startup-like teams, at least here in Vietnam. In case you’re developing software you’re probably going to need a little more than just a chair, a table, a laptop and a Ca Phe Sua Da. If you want to keep your flexibility and make sure, your whole team has access to all they need where ever they are, it’s often best to chose hosted solutions. Let’s have a look at what you might need:

    Version control repository plus bug tracking

    unfuddle.com: the free plan comes with SVN/GIT repositories as well as tools for milestone planning, bug tracking and a calendar. One project and two developers are included, that’s often enough to set up a project and get going. For 9 USD / month 10 developers can work on a project and for 24$ a month it comes with SSL (if that’s necessary). It’s rather slow here in Vietnam but I never had any problems connecting to my repository. Your source code is definitely something that needs to be on the web, it just gives you so much freedom to work outside and collaborate with your colleagues who just happen to prefer a different coffee shop.

    Some other repository providers worth checking out: BeanStalk, Assembla, or ProjectLocker.

    Some bug trackers worth checking out: well, you tell me, I’ve got no experience with a) free b) web-based c) hosted bug trackers somehow. Or maybe I never produced bugs and that’s why.

    Requirements management

    pivotaltracker.com: requirements tool for agile projects, especially those closely following Scrum. Although everything could as well be packed into an Excel sheet, Pivotal Tracker comes with some nice features around iterations and velocity tracking that make it worth checking it out. It’s free to use and it there’s even an Android app so that you can access your user stories and product backlog wherever you are. Limited use cases for that one, I know, but I just loved to be able to add product ideas directly to my icebox whenever I happen to have those ideas.

    Some others worth checking out: a Google Docs spreadsheet. It might sound like a joke, but hey, you’re just starting up, it takes 5 minutes to create the right document and then you get going and stop thinking about the right tool for the job, because you have it already.

    Continuous integration

    mikeci.com: An optional yet strongly recommended part in between developing something and running it in production is integration testing, or even better: continuous integration. There aren’t many providers out there who offer that as a service. One I’ve recently joined is MikeCI.com. If you use Maven, Ant or Rake (this one still in beta) and have it hosted in a SVN, GIT or a Mercurial repository, then setting up CI is as easy as pie. It also supports different polling cyles and email notification to make sure you don’t miss a failed build. MikeCI is not free but starts with a reasonable 10 USD / month. Two more plans with 49 and 99 USD / months make sure it scales with you. It’s very nice to have this service provided, although the power and extendability of Hudson make it a though choice whether you really outsource continuous integration. You might just end up doing CI on your laptop instead.

    Some others worth checking out: as mentioned, do it on your laptop. Install a Tomcat, drop in Hudson, configure your project, activate polling to your repository and then let it run in the background. You will fall in love with it very soon.

    Web hosting

    Hosting. A lot is changing right for hosting of web applications, all towards something cloud-based. But hosting is still something worth to take the time and look into it. It might often not be the cheapest to go for a cloud provider and prices are probably something more important in early stages of startup projects. Or you choose to decide that your own time is more important than a few extra dollars a month and decide to go for a cloud provider to save time for administration, setup and operation of own servers. I am a big fan of CloudFoundry because it reduces all that to an absolute minimum. Java applications only here, but there might be others out there.

    Google App Engine: Google’s platform-as-a-service offering comes with free setup and basically won’t cost any money until you’re really starting to get traffic to your website. This might sound tempting but App Engine’s absolute strengths are elsewhere, in creating web applications that will scale perfectly inside Google’s cloud infrastructure. But this requires some serious changes in the way we learned to develop web applications over the last years, so you should think about it twice. Especially if experience with app engine is limited and time-to-market is crucial, it’s probably a bad idea to go for it now.

    Some others worth checking out: Amazon EC2 (plus 3rd-party tools on top, like CloudFoundry), all hosting provider with VPS offerings (no cloud, but often more power for a lot less money), vmforce.com (a promising choice for Java applications, coming in fall 2010)

    Good old office documents, spreadsheets and presentations

    Google Docs: Documents, spreadsheets and presentations are always needed, no matter what you’re about to create in particular. While MS Office is still the office suite of choice in business, Google Docs is perfect for distributed and collaborating teams. Shared document folders, concurrent editing of documents and nice little features like in-document chatting make it the perfect choice for your team. Another plus: Google is always fast, here in Vietnam and most likely everywhere else.

    Alternatives: if you’re not into concurrent editing of documents and you don’t mind still sending documents around via email instead of just links to a common workspace, then you can as well stick to MS Office or Open Office.

    Backups

    dropbox.com: dropbox.com is popular, it’s easy to use and the usual freemium model offers a 2 GB plan for free. Millions of users somehow tell me they are reliable enough to use them (they use Amazon S3 as storage), only connection speed was rather slow from Vietnam.

    While remote backups (both words are important) are something crucial, just take a look at what we talked about so far and think about how important it actually really is: your source code is by now in a safe remote repository, your documents are on Google, your requirements in Pivotal Tracker (or also on Google), what is really left on your laptop to back up and protect? Not much, isn’t it.

    Communication and collaboration

    Google Wave: Everyone knows Google Wave as one of Google Labs’ products that produced a lot of waves in the beginning and quickly ended up being forgotten. So, why the hell does it appear here? It’s a shame that it doesn’t work in practice as it has some nice approaches that might have just needed some fine-tuning here and there to make it really usable.

    Another reason why it appears here is because I want you to complain and tell me what you’re using to collaborate. Of course there are the usual suspects that might not be worth mentioning anymore, such as Yahoo, Skype, emails and, wait for it, meeting in person (like for example working in the same coffee shop).

    I’d be interested to hear who else is working in coffee shops and how your infrastructure looks like.