SDK testing

[andrew]

Hi All
I am in a but of a quandary and my head is spinning with mixed messages. I have been looking at testing a mobile SDK which requires 2 way interaction. So it is not really useful to go the route of utest which looks a good way to test mobile apps.
At the moment in it is an android solution so have looked at what my options are with the likes of deviceanywhere, openstf, and Amazon device labs.
I have not really found much honest feedback on these sort of labs so asking the question to wider testing community whats the experience of these labs? I dont want to be in the position of having to buy lots of devices but at the same time I am interested to see how other people are doing testing with mobiles, especially with SDKs where I dont have a proper app to test.
Thanks
Andrew

[Deepak]

Andrew, you question is not so clear to me… but i kind of understand

you want to test SDK and perhaps want to verify the results, you could do this under some development IDE in source code format ( link below like i said i don’t know whole of your question), or you have write a wrapper app (phonegapp, etc) which is good enough to test the logic/paths/functionality of your SDK.
The tools you mentioned in question are: Black box testing tools on REAL WORLD devices and work only at app level.
Finally for your two way connection you could use IDE, Wrapper app approach with the output in Json format.

http://developer.android.com/sdk/index.html
http://phonegap.com/

[Péter]

Hi Andrew!

Just like Deepak said, it’s hard to figure out what is the actual problem.
What do you mean by no proper app to test? Are they just libraries that you would like to test?
What kind of tests would you like to execute? What would you like to achieve with these tests?

If you could answer these questions and give more insights of the software under test, that would allow us to give you proper feedback and tips.

All the best!
– Péter

[andrew]

Thanks for the feedback, it is hard to explain so sorry for the confusion.
The SDK has a dummy app of 3 buttons i can test Voice, Video and text. From the SDK point of view it requires 2 way interaction, which means i need to have a person at the other end using the PC software to interact with SDK and verify Voice, and video is feeding back.

Having a device in my hand gives me that feedback, where I can hear and see the output. However some of the Device labs do not provide that level of feedback. The idea of our SDK is customers can use it within their own app so hence why I do not have a proper app to test as ownus is on the customer. However I need to have confidence that the sdk will work as expected. So although I dont have an app as such I have a dummy one which has been designed to specifically call methods to ensure that method returns the response.

I have had Nativetap suggested as one, and I am also looking at phone gap too,
Thanks
Andrew

[Chandra]

Hi Andrew,

Below Android SDK testing frameworks can be considered for your testing

UIautomator
Espresso
Appium

Please give a try and let me know if you need any help in testing

Thanks,
Chandra

[Luana]

Hello Andrew, I think you are doing right. I mean, you need an app or some dummy (mock) to test all scenarios that your SDK can do..

I prefer APP + smartphone because there are many OS versions, and sometimes these OS are customized by manufacturer, and there is no patern to consume a DLL or SDK function.

In all case, you need to know if functional behavior is correct. Using only mock function (or dummy) you can’t garantee that it works good, to your clients.

Br,
Luana

[Jon]

Andrew, I am not sure I accept your first statement about not being able to use crowd source testing (you said Utest). I think it might be possible, though you would need to work with the crowd source provider so they know to provide teams of “2” testers at a time. I’ve not seen this done, but it might be worth talking with a vendor. It seems possible to me.

I would also think you would need to provide the testers tours, attacks, or scenarios and statements of what risk you are worried about. It seems to me from what you have provided you are worried about interactions of different variables (devices, vendors, OSs, networks, other apps, etc). This points toward the use of combinatorial testing. As Luana has pointed out you might be able to do some testing with moch-dummies to rule out some errors, but it is not clear to me from what you have provided what other real risks, info and bugs you are looking to find. A risk analysis might be needed. In any cases testing at different levels (e.g. developer, dog fooding, combinatorial-internal integration testing, test appliance networks, and crowd sourcing) should be considered.

[Author]

Posts