Analytics

The Web / Mobile Feedback Loop

Backlogs for Web and mobile products don’t exclusively contain new features. One eye should always be on what has been done and how that is working out. A proper feedback loop gives valuable input that helps to determine what should be done next.

On one hand, of course, there’s high level goals and vision that define new features and the larger chunks of upcoming work (which just reminds me of this great article about how Spotify has done Prioritization in their early days). But then there’s more. For example. there’s bugs, there’s A/B testing results, there’s the Google Analytics account that somebody should actually have a look at, and there’s more. Most people know most of these, but mostly, they aren’t managed really well all together. So I thought a good start would be to all those sources of input on the feedback loop that may (or may not) affect our priorities:

  1. The product vision (this is what your management and product managers want to do, the longer term goals, this isn’t actually on the feedback loop, I just wanted to have it on the list)
  2. Business figures (e.g. your sales numbers; I dare say this input is usually indistinguishable from #1 (because it comes from the same people?), but I’d argue that it’s “feedback”, unlike #1)
  3. Analytics (the likes of Google Analytics)
  4. Feedback that is built into your product (without being explicit feedback, it’s basically extracted from normal usage of the app)
  5. A/B Testing (e.g. Optimizely or a variety of other ways to do them)
  6. Explicit customer feedback (lots of sources here incl. all the feedback your customer support and sales teams gather, but there’s also tools you can use that allow your customers to give feedback online, e.g. murm.io (for specific feedback on your existing features) or tools a la Uservoice and ZenDesk)
  7. Crash reporting tools (Crashlytics, Crittercism, …)
  8. Dogfooding (your own company using your product, often this is a much smaller feedback loop since it allows you to get feedback on unfinished work that wasn’t even released yet)
  9. External ratings (e.g. what your users say about your app on Google Play and iTunes)
  10. Customer opinions out on the web (blogs, social media, very similar to the point before but wide-spread on the Internet)
  11. Beta testers and special user groups (there’s a bunch of tools that help you, e.g. Testflight)

This was just a first shot and I’m merely thinking out loud.

It’d also be interesting to see how all of these can be managed more effectively than having different people “keep an eye on it” or having 13 different tools at our disposal to log in and check regularly. I’d greatly welcome less overhead to collect them, a better way to manage and follow up and make them a part of the development process, and create a lot more transparency for teams and stakeholders around them.

I’d be interested to hear what others think or whether there’s anything missing on the list above.

3 steps I should have taken to get started with Web and App analytics

In the last weeks I wrapped my head around client-side analytics here at Klamr. Looking back, probably the biggest Gotcha! was the realization that ignoring all the cool out-of-the-box stats and colorful graphs would have been better. It’s all to easy to get paralyzed by the sheer amount of information each tool provides you. The answer you’re  looking for isn’t between visits and pageviews and referrers and user flows and mobile data and custom segments, so don’t spend your time looking for it there (just yet). Eric Ries calls them “Vanity Metrics” and tells you why they are dangerous. I should have listened earlier.

It actually starts with making sure you know what your app is supposed to do and what user actions are valuable for you. That’s all that really matters in the beginning. Whether you’ve got 50 or 5000 sessions and whether your median session length is 34 or 58 seconds, that doesn’t tell you much yet. So, better start with what matters and ask yourself: what are the few important actions I want my users to do with my app?

Hence, if I would start over again, I would do the following:

  1. Make a list with the 3, 4, 5 most valuable user actions in your app, on a very high level. For a shopping app, a completed purchase should obviously be on this list. For a photo sharing app, a shared photo should be on. A blogging platform would include creating a new blog and publishing a post. Note that this list should contain the successful  final action at the end of activities, not the actual activity leading up to it. Analytics tools often call them “events”. A purchase, for example, usually consists of a relatively long flow, but for now only the successful completion matters.
  2. Determine where and how these events are happening in your app, i.e. the lines of code that actually perform them. Find out whether you wanna instrument your client application (web or native) or whether you better do it on your server backend. Instrumenting your client often tends to be easier tool-wise, but sometimes it’s just better done on the server. But keep in mind that you try to find out how to improve your client. Getting the number of purchases, for example, isn’t the actual goal here. What you actually want to do in later iterations is find out where you can improve user flows, UI and UX to optimize. That’s why server-side instrumentation doesn’t work out well later.
  3. Instrument your system with the tool of your choice, e.g. murm.io, Google Analytics or Flurry. Then release it and wait for data. Instead of tons of numbers you should now see your few major numbers showing up in the tool.

See what this gives you and then iterate. You can add more (refined) events, categorize them like a Pirate, add funnels and goals, and go deeper. And in one of your next iterations you can look at what you can do to improve.