thanks for your reply and your questions. I ended up only getting hands-on experience in Optimizely.

The me, they definitely seemed good to A/B test variations on your UI. The typical example use cases like “Does a red or a green button work better” are definitely doable. Often though we ran into situations where things we’re a bit too complex that they could easily be done with Optimizely. Especially if you wanna test slightly different backend behavior, it gets a lot more difficult all of a sudden.

Also, I’ve been trying to use it in early stage startup scenarios. But there, you often deal with small numbers of users, so focusing too much on statistical hints at what works better doesn’t get you very far. In my opinion at least, I know Lean Startup et. al. motivate you to A/B test early on. But not sure if something like Optimizely is the right tool for that.

Lastly, one more comment about Mobile Apps vs. Web: since you can’t easily roll-out changes there, it seemed so logical to me to integrate A/B testing, basically to reduce your average cycle time and learn faster. But as well, didn’t end up using it long and intensive enough to really have a sound opinion.

I’d be thrilled to hear your opinion and learn about your experiences with it.

Cheers and hope to hear from you again. And good luck with your project!
Hendrik

Im making changes and improving an existing e-commerce app and I’m trying to figure out the best way to do the AB testing on the UI changes and stumbled across your blog, seems like you’ve been playing around with different AB testing platforms, was just wondering if you could share some insights that you’ve gained so far? does it appear to have any clear winner ? there seems to be a couple of different companies are working on it at the moment

On a side note, how’s life in Vietnam been treating you so far ;), I’m from Vietnam too, not living there atm but spent the first 20 years of my early life there

Joey

Happy new year & Happy A/B testing
The Arise.IO team