As soon as you release the application, it might be reverse engineered. This means that there is nothing you can do to be 100% protected if the same application (same binaries, same settings) is distributed to all of your users.
If you can customize the application for every user, then you have a chance to maybe not forbid some other application to use your API, but at least limit this application by number of requests it can do to the API.
Imagine the following schema:
- Client connects and sends its unique identifier (one identifier by user).
- Server replies by sending a challenge encrypted with a public key. This public key is associated with the unique identifier previously sent.
- Client solves the challenge by decrypting the data using a private key and sends the decrypted secret in plain back to the server.
- Server verifies that the submitted secret corresponds to the one originally generated.
The developer who will hack your application and successfully obtain the private key would be able to use your API from his own application, but he will be identifier as himself for your server.
If the same user can do 10 000 requests to your API per day and in average, an active user makes 2 000 requests per day, it means that this developer would be able to use his application himself and maybe give it to his friends, but he wouldn't be able to, say, sell it to thousands of people, just because it will work only for a few minutes in the morning.
While this helps, it's not 100% proof neither. What if the hacker finds a way to extract the private key from your app when his own app is installed on the device?
Side note which doesn't answer your question, but still might be useful: don't think about an API as a tool for your primary product (mobile application). Think about it as a first class product itself, a product which may be paid. The same model is used for years by Amazon and Google, it starts to be actively used by Microsoft with Azure, etc.
As soon as you consider the API not as a secondary tool reduced to slavery for your shiny new mobile apps, but the actual product, at the same level as any application the user actually sees, you start thinking less about how to protect the API against the usage by other apps, and more about the monetization of the API itself. Such API can be used by your apps which are its customers, or any other apps, freely developed by anyone. This has several benefits:
Making an API so that it would be used only by your applications is difficult and expensive. This time and money can be used for something more useful.
Opening your API to the public can have a great benefit for both you and the world. Imagine you're a great architect and a great developer, so you've created an astonishingly great API, but your visual designer skills suck and you really don't understand anything about interaction design, etc. If you hide your API, the only thing people will know is that you've created a mobile application which is unusable and ugly. If your API is public, other developers will be attracted by its quality and write great applications for it, bringing you lots of money.
You never imagine how other people may use your APIs. This is what happened with Kinect. Originally, Microsoft created Kinect for games. When Microsoft opened the API to the public, they never imagined it would be used a few years later by scientific applications, health sector, etc. It's similar for web APIs: more developers are using it, more widespread the ideas would be.