Archive for 2012

mobile: a few comments on testing, emulators & tools

Recently a colleague within the testing community emailed me three questions re: mobile testing. After answering the email, I realized that our exchange of questions and answers might be helpful to share – so following is the Q &A from our exchange (minus personal identification).

Hello -

I’ve shared my thoughts on each of your questions. I have to say – this is all based on my experiences and not necessarily reflective of the broader and global mobile market. Testing like so much else about mobile, will continue to change significantly in the next 1 and 2 years time.

1. What is mix of simulator based testing vs actual device based testing ? What factors decide/influence this mix?

In general, emulators have not been used with clients I have worked with. Occasionally a developer will use an emulator during unit testing but otherwise, emulators do not seem to be used or trusted.

The factor used seems to be a general distrust that an emulator will be accurate and this is based on opinions vs experiences encountered. It is also well known that through Device Anywhere and Perfecto Mobile, you can get your hands on nearly any device and so why tolerate an emulator?

Unreliable to depend upon but a typical experience I’m seeing is that within most teams, there is a natural mix of devices and people generally use their own devices at least part of the time. Bigger companies have funding and I know of a couple of companies that are building out mobile test labs – accumulating what they can.

There are a few “quick and free tools” on the market to execute mobile readiness tests. See the W3C, http://validator.w3.org/mobile/ and Gomez, http://www.gomez.com/mobile-readiness-test/ and dotMobil, http://mtld.mobi/emulator.php See a comment on the next question for my “reaction” to these tools.

2. What are the class of defects or problems simulator based testing cannot discover?

I cannot fairly comment on this as I have more experience with devices than emulators.

I have seen issues reported from the quick tools that I mentioned above but found in most cases, the issues detected were either obvious or more obscure GUI-issues that we (the team) were not likely to address – the net result, not so helpful.

3. With so many tools claiming to do automation for mobile apps – how does one pick? In what ways automating (driving GUI from a tool) mobile app is diff from other software apps/web apps?

HP seems to be pushing their QTP app to handle mobile automation. I do not know anyone using it. Perhaps it works? Also a shocking but true fact – at this point in time, which I do expect will change – I do not know of any company that is successfully using automation in mobile. I know a couple of companies trying different tools, wanting to find a tool but at this time – and again, I think this will change before 1 or 2 years time – most companies are trying to figure out what their app and/or mobile site is going to do, what they need to test, what devices they need to cover, how to gather decent statistics and then will move onto mobile automation. The old expression – you have to walk before you can run seems to be the situation.

And although you didn’t directly ask me, I know that you are keen to continue to learn, as am I – so what do I read and where do I continue to learn?

I learn by other people sharing with me (why I feel it is important to get back to you and share as well.) I look to my colleagues Julian Harty (UK) and Jonathan Kohl (CA) and share with them as all of our schedules permit. Google their names, track down their work and articles.

I learn by experience and am currently (which has been true for the past couple of years) most eager for any mobile work I can get my hands on. When I do not have client work that gets me that experience, I continue with experimenting with a variety of apps and my own devices.

I follow Twitter closely because mobile is so current nearly anything in print is already dated. (You might look to see who/companies I follow as much of my Twitter following is mobile-centric.) That said, I think I own at least the “core” books on mobile offered from Amazon and O’Reilly as of today. I scan every testing newsletter for articles and attend webinars as feasible due to my sometimes crazy schedule. In this last 24 hours I devoured one book from Amazon in e-format only as the book is only days old, titled: Mobile Analytics by Jesus Mena. Straightforward title and rock-solid information.

Like most things I suppose, we find what we are truly interested in – no matter the day of week or time of day. At this time, mobile is a top interest of mine.

Now on an unexpected final note, given that I have taken the time to reply to you – I realize that your questions are not so very different than questions other people have asked me – I have decided to take my replies and post them into my blog. I won’t identify your name as it would be inappropriate of me to do that – but the replies I have shared with you are not different than I would share in my blog – and perhaps this exchange will help another tester? And so the circle of sharing continues.

Happy Testing~

Share
Posted in mobile | No Comments »

Experience Report: Sony S tablet versus the iPad

I expected to love the Sony S tablet; I had eagerly waited for the tablet arrival but wow, it didn’t turn out to be a good experience at all.

I’m detailing my personal account of a Sony S tablet versus an iPad in hopes it will help someone else who might be on the edge of trying to decide what tablet to go with.

I was curious how the unboxing experience would be because even the initial opening of a device seems to set the tone. When I opened the device and powered up, the opening screen of the Sony S tablet was a system crash message. A restart and the device turned on fine; I completed setup with no further issue but geez that first bad experience left me with doubts.

Like any new device – beyond the setup, I seem to have the same reaction – ok, cool now what? It half makes me smile because I think the issue is data – devices on their own don’t do much – instead it is the applications combined with either our own data or the data we look up that makes a device so interesting. I have no doubt that the commercial market and users whose occupations are outside of the tech arena don’t see it that way – but think about even the iPad ads – it’s the photos, the videos, the books, the contacts, and other bits of data that make a device interesting. It is not the raw merits of the technology as much as the sweet combination of technology and data that make technology compelling.

Ok back to the Sony S tablet experience – the screen was dark, I adjusted the device to be as bright as I could but it just didn’t matter. The Sony S tablet is dark. Also in almost all cases – from using an app to watching video – the entire viewing screen of the device is not put to use – which adds to a disappointing visual experience. Each time I used the tablet, I had this inclination (if not need) to put my reading glasses on because I just couldn’t interact with the device readily and easily visually. I was surprised, I mean this is Sony – shouldn’t the visual experience be great?

After mucking around with the device for two days and having about the same reactions of “not being wowed” I decided I needed to do something. I needed to take my Sony S tablet and go to the Apple store and do my own side by side comparison, app by app if it took that – to compare the Sony tablet versus the iPad – which of course is not just a device comparison but a fundamental OS comparison. It is after all, almost a religious difference to be on a Droid (Google) vs. iPad (Apple) device. It might seem more logical that I would go to Best Buy or somewhere else and compare the Sony S tablet to say the Samsung or Motorola tablets but it wasn’t the comparison I wanted. And as much as I love my Droid phone, I was willing to give up a Droid tablet and move to Apple if I was happier with the device.
Meanwhile I have several issues with Apple’s interfaces and there are some aspects of working with either an iPod or an iMac that I find flawed and not as intuitive as Apple touts. But I do own and use a mix of PC and Mac, Droid and Apple (even before this tablet experiment) – so I’m religiously a bit undefined. In fact, I believe in building and maintaining some knowledge in both technology camps because I test in the mobile space and to completely disavow either part of the market would render me too much of a zealot which I think is inappropriate in my role as an independent consultant and as a software tester in general.

I was a bit concerned how I might look in the Apple store smuggling in an Android tablet but figured I wasn’t stealing anything and besides if a company is so confident of their own product, wouldn’t they welcome the challenge?

Somehow I found a corner of the Apple store and was able to step through what I wanted to in my own side-by-side comparison. Since it was Christmas Eve and the store was busy with last minute shoppers, not too many people cared what I was doing which was perfect for me. In fact the best holiday gift I gave myself this season was the time to run my own comparison tests – this was important to me. If you haven’t noticed by now, I like my gear.

Here goes, first stop: You Tube. The experience of comparing the iPad to the Sony S was almost a closed case by comparing video display and handling on You Tube alone, the iPad has it hands down. The Sony S has a yellow hue to all video, graphics and display – it’s not just the overall darkness but an additional coloring that renders sepia-like-experience. I found myself wanting true coloring. I queued up several videos in a side by side checking visuals as well as sound.

Second stop. Photos, same disappointing graphical experience. What’s with the yellow hue Sony?

At this point, I knew everything else would pale. I figured the data – mail, contacts and calendar would be easy with the Sony S and it was – as long as you have a Google account and use each of those (email, contacts and calendar) then loading up data on the Sony S was easy. But that easy data loading is not compelling enough because Apple handles that too – although Apple’s limited handling of Google calendar is annoying.

Next: device handling and covers. This seems like a minor point but I know that a large amount of ongoing ownership time gets spent in pulling out a device, turning it on/off and that the device cover or protection can influence and impact the overall device experience. So much so I don’t know why most smartphones don’t have a more rugged shell to begin with so we can stop buying cheap plastic cases for devices worth so much more.

After having one phone case that left me disliking a phone, I know the wrong device accessory/case can influence how I interact with the device itself. The Sony S tablet case is one of the worst cases I have experienced. There’s a large Velcro strip loud enough to hear across a room – that’s a deal breaker itself because at this point there are no other cases designed for that particular tablet and the Sony case is awful. One of the most important aspects of owning a tablet is to be able to access and use it quickly – so if it’s a hassle to pull out the tablet and embarrassingly loud – then the case is a detriment to the device.

Meanwhile in addition to being loud, the case itself is not snug enough to protect the device. And finally, the Sony case does nothing to help if the device is left on, the case will store the device and happily drain the battery if you forget to shut the device off before stowing it away. The iPad’s smart cover is cool, works well and it is that simple.

At this point, an Apple staff person came over to help me. Her eyes widened when she realized what I was doing – comparing devices right in the store. Then she asked if she could join me, seems she’d never held a Sony S tablet and she was curious too. My daughter arrived in the store at the same time and I asked both of them to help me with the comparison. It was an interesting hour and at the end, I bought an iPad2. Wow, I’m surprised, still surprised because my Droid phone has been great.

The last step was to remove all my data from the Sony S and return the Sony tablet. I was able to uninstall apps and handle some cleanup easily but not my Gmail. Looking up what other people have written I discovered a horrible “feature” of Gmail – and that is there is no way to sign out or remove your account from a tablet. Search for what other users have discovered and you’ll probably find what I did – people furious because they can’t share their Droid tablet with anyone else because there is no way to switch what Gmail account is logged in. And there is no way to remove email, contacts or calendar data without a factory reset. Not cool, not acceptable.

I have an iPod Touch and iMac so the Apple is familiar but I have been “adjusting” to having an iPad, and have to say, I’m becoming pretty attached to my iPad and have to admit, it is simply amazing.

Share
Posted in mobile, Uncategorized | No Comments »

Switch to our mobile site