Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I developed one application for iPhone and now I want it on App Store. So many my iOS geek friends told me to test it on actual device i.e. on iPhone.

So I wonder that why it is necessary to to test my iPhone app on actual iPhone device though they (Apple) have given "simulator" which is near about same as my device?

share|improve this question
3  
The problem is with "near about same as my device". Near the same isn't good enough. The small difference can affect the great things of your application. Not only do you have to actually test it on hardware, but you have to think of testing on different versions of the hardware and software (iOS version). –  Coral Doe Aug 17 '12 at 12:20
add comment

11 Answers

up vote 50 down vote accepted

You need to test application on real device to at least to look how it behaves with:

  • Real device hardware
  • Real internet connection (including the use of a cell network vs WiFi)
  • Your fingers instead of mouse
  • Performance with other apps running in the background
  • The limitations of the iPhone, like cpu, disk capacity and memory (A Simulator is not an Emulator).
  • Real context: is it easy to use your app on the train, or while walking down the street? How about in bright sunlight or in the rain?

iOS developers, please continue this list.

share|improve this answer
    
that's nice.. :). The very simple thing I must think about... :) Thnaks –  NSS Aug 17 '12 at 10:30
2  
Actually he needs to test it with all devices it's intended for (iphone3/3g/4/4s/5) as well as all ios versions 3/4/5/6, or he must explicitly exclude that device/version. –  ott-- Nov 3 '12 at 18:03
    
Thanks for the simulator not emulator link.. I never knew there was a difference between both. –  Krishnabhadra Feb 20 '13 at 10:02
    
There are open device testing labs springing up, read e.g. mobile.smashingmagazine.com/2012/09/24/…. Maybe there's one in your neighbourhood. Google "mobile testing lab" "open device lab" –  Jan Doggen Feb 20 '13 at 14:26
add comment

The iPhone Simulator implements some APIs that the iPhone itself does not (The main one that comes to mind being the DOM XML API, where the iPhone only supports SAX to my knowledge, this might have changed though now.)

It will also let you 'feel' the app, are your buttons the right size? Do the right buttons fall under one's thumb? Is the iPhone up to running the app? The simulator is not an emulator and works by just adding Cocoa Touch to your desktop Mac for that app. You have to simulate out of memory warnings and the like.

share|improve this answer
add comment

One thing you'll never know when testing with emulator is how it really feels for a user holding a real device in their hand, sliding fingers on its screen. As a result, user actions that seemed smooth when simulating with touchpad on your laptop, may turn out quite cumbersome for a real device usage. To make sure that your application is OK, test it with real device.

Another thing worth testing with real device is battery consumption. That one is really safer to just test with real device than rely on how well simulator developers managed to reproduce it in their tool.

There may be other things that just aren't close enough at simulator. Audio volume and balance for example - the way it sounds on your laptop, may differ from how it will be on a real phone. Vibration is another example that is barely possible to get right with simulator. The way how gyro sensors work at the real phone. GPS / location related stuff. Etc etc etc...


Simulator vs real device testing is quite an important issue. In one of my past projects, an important part of commercial success was a careful balancing between these kinds of testing, which essentially boils down to permanently re-asking questions like why on device?

Real work starts when one asks why, picking the reasons to choose between on-device and simulator testing in particular cases and situations.

Ignoring device testing exposes your product to a (pretty high) risk of breaking right in the hands of end user, completely destroying all the efforts you put into development. But thing is, simulator testing is much MUCH cheaper and much easier for automation. If one blindly sticks to on-device-only testing, their releases might become substantially later and more expensive than those of competitors.

share|improve this answer
3  
+1 for emphasising the user experience part. It is a very important part when thinking about sharing an application in a app store. –  Coral Doe Aug 17 '12 at 12:22
1  
You had a great comment on the whiteboard about the much greater expense of testing a device vs. a simulator. I think that would be great to include in your answer. +1 even without it. –  psr Nov 2 '12 at 16:51
    
@psr glad that you liked it - updated an answer as you suggested –  gnat Nov 2 '12 at 17:14
add comment

When we implement things for iOS (or android, or windows phone), we develop not for the desktop but for the device. For some calculation/resource-heavy applications, this may mean to normal behavior on the simulator BUT issues on the actual device.

So situations like this can be encountered in later stages if we do not test on device from the beginning:-

  • Memory warnings/crashes
  • Single-digit OpenGL frame rates
share|improve this answer
add comment

practical reason:

1) You do not have "send mail" functioning.

2) You cannot put device upside-down..

and of course already said reason:

3) low bandwidth

4) very small computational power compared to simulator

5) Open GL calls are implemented a bit different in simulator

6) disk space / RAM..

share|improve this answer
add comment

There are some features like PushNotification, using the camera, etc. which we can only test on a device; these are features that can't be tested on a simulator.

And testing your app on an actual device before submission to the App Store reduces the chances of app rejection; sometimes an app works fine in the simulator but it crashes on an actual device, which is the main reason for app rejections.

share|improve this answer
add comment

There can be real performance differences between the actual device and the emulator. We found that only testing with the emulator resulted in a very slow app in many cases, which we didn't expect.

share|improve this answer
3  
Ironically, one of my apps has better performance on the actual device than on the emulator on my laptop. –  Michael Itzoe Nov 2 '12 at 15:43
1  
@nathan : I think we use SIMULATOR rather than EMULATOR! Android sdk have emulator while iOS sdk provides simulator.. and there is difference in between emulator and simulator. isnt? so I read ur ans by replacing word emulator with simulator.. n tnx 2 ans :) –  NSS Nov 5 '12 at 14:19
add comment

From experience, and a take off from the best voted answer:

  • Your fingers instead of the mouse was the largest difference when we developed Decimation X2 for the Windows Phone 7. It was coded on an emulator, as we did not have a WP7 and it was before the WP7 was released. We could have potentially received a free WP7 before release, in case that last sentence made no sense to you, as we were invited by Microsoft to have a launch title on their phone. It turned out that, what we wanted the user to do with their fingers was very hard on the real phone, but easy with a mouse. And it had to do with screen-edge cases. Unfortunately our game required your fingers to be at the screen-edge all the time, which some phones made hard to do due to their sunken screen as well as thick cases. We actually put out a patch to fix this. This means all our first-time users were shown a bad, and potentially unusable, version of our game. :(

  • Different hardware speed was the next largest difference. We had to literally guess by the inaccurate method of taking our Xbox 360 version of the game and downgrading it accordingly to half the frame-rate (60fps to 30fps) and one-third the GHz (3.0GHz to 1.0GHz), and we guessed wrong. The processors were different, of course, and we knew this. Without the hardware, we were left with lame guesswork. It was not our choice as we had no WP7, but we learned the lesson I'm sharing with you now. On some phones, during the most intensive parts of the game, it dropped frames. :( Nobody seemed to care as they assumed the slowdown was appropriate for such intense parts. So it was no big deal, but the point is: Had this been a big deal, the "app" would have been broken from our lame guesswork.

Test on real hardware. And when you code for various phone hardware, test on the lower end ones if performance is an issue.

share|improve this answer
add comment

Because you are not going to have a lot of users walking around with a simulator in their pocket.

EDIT: Whenever you're testing your app on a simulator (or emulator), you're using a fake device that cannot be, by definition, a 100% accurate representation of the real thing. An emulator might be more accurate than a simulator, but there will still be differences. The only 100% accurate emulator is the device itself.

Designing, testing, optimizing code on a simulator results in an application that is finely tweaked to optimally work on a.. simulator. Your users will not have simulators, though; you're targeting the wrong device. A very similar one; but not the very same device that your users will use.

This can lead to several kind of issues. Severe issues like bugs, crashes are certainly your top priority. But there are others; such as ergonomics. Try to hold the Simulator in your hand. Just try. UI elements are rendered on a different screen, with possibly different color rendering, and certainly different dimensions (a problem exacerbated by the beautiful retina displays, not entirely solved by also using a retina macbook, for instance). Are those exquisite shades of gray equally distinguishable on a device in the sun?

Subtle differences in speed, and different sensor emulation (or lack thereof), can sometimes dramatically alter the experience.

If your application relies on internet connectivity, there is no way for you to switch between LTE, 3G, EDGE or GPRS, to test different scenarios, or even test different carriers.

Are you going to support jailbroken devices? Perhaps you aren't, but if you are, you're probably willing to test your app with one. Or, if you aren't, are you confident that you're detecting a jailbroken environment?

Is the iPad game you're developing on the simulator equally usable when a user is holding its weight and using her fingers to play? Are unintended multiple touches able to break your app, something that you weren't able to anticipate in the safe single-touch (or symmetric double-touch) simulator environment?

Would you be comfortable boarding a plane commanded by a pilot who has never actually left the ground?

The bottom line is: before shipping, please do use the same device that your users are going to use. None of them is going to use a Simulator.

share|improve this answer
4  
Simple and valid sentiment but detail beyond a single sentence makes answer's more valuable to the community at large. –  Jimmy Hoffa Nov 3 '12 at 1:26
add comment

While it's been mentioned that hardware's performance is generally worse, it must be noted that this is not the case with OpenGL ES. The simulator implements it in software so it is not uncommon, to notice huge performance increase while running on the device itself.

Moreover, there are some minor differences between the software and hardware implementations of Open GL ES, e.g. shader precision hints might have different outputs.

share|improve this answer
add comment

The user experience varies from one device to another due to different OS and

hardware specifications. Hence, it is necessary to test an iPhone app on the actual

devices – mobile devices running various versions of iOS in the marketplace.

Although a simulator is useful in identifying any issues faced by the end-user,

testing the app on the original device will help in identifying and addressing major

concerns of the user.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.