API Testing – Tools and Tips

[Kim]

Hi all,

I am interested if any of you test APIs on a regular basis and what tools do you find most useful?

I have recently used soapUI the free version and it was a little bit difficult to understand and get used to but is definitely a powerful tool.
We also trialed the paid for version ReadyAPI but this seemed overkill for our purposes.

Next up was Postman with the Jetpacks add on. This seemed to be a great compromise. We can full fill a lot of our requirements with this tool.

Jmeter seems to be a good too. Once you get it up and running it can do a lot and not just load test! (which was my misunderstanding).

What do you look for in an API test tool?

Also do you have any experience with API Gateways? Any tips and tricks on the subject?

Let’s share! 🙂

[Sammy Connelly]

Hi,

I’m an embedded tester on a dev team, we are moving towards microservices and continuous delivery. My team are creating contract driven tests at the moment. I think other people have been calling them producer and consumer tests. So a consumer of an API end point can say I need X and Y from a producer. This test will verify that the producer actually supplies those details under certain scenarios. If the consumer later on changes to need Z aswell either the team who created the producer will get alerted that a new contract is needed or the test will be able to confirm that the producer can already provide it. It seems to be a way for the developers to try and automate backwards compatibility testing.

We also tend to use postman for checking our API end points. We will also mock/stub other artefacts in a dev environment for testing with.

I hope that helps.

[Kim]

Awesome thank you!
Do you know if you are using an API gateway at all,mot manage the subscriptions to your API?
We are currently researching the subject and it is very interesting. I hadn’t even heard the term a week ago. 🙂

I like the idea of backwards compatibility testing and being able to automate it. I will bring that to our discussion tomorrow! Thank you.

[Ronan Healy]

@kim I was at a talk recently by Ole Lensmar of Smartbear titled “The Internet of Things – Challenges for Testers”. It was an interesting talk with API coming up a lot. You can find his Twitter here. Might be of use.

[Kim]

Thanks @Ronan !
I will add him and I think I had seen a review of the talk somewhere as well! Going to try and hunt it down!

[Victor]

I started using free SoapUI but it end up being a maintenance nightmare so I switch to a suite of functional tests in C# using an RetSharp as the http client. You can get more info here:

API Functional Testing – The Black-Box approach

[Jon]

So tools are important, and I don’t recommend any tool over another, except to say most tool have a context they work well in and contexts they do not work so well it. However, in using the API to do testing, I will say I view it as one “layer” that testing should be done at. You might have the layers of unit, integrated units, APIs, integrated elements (say hardware, software, 3rd party software, etc), the UI level (if that exists), and the system (where we might test many qualities both functional and non-functional). Give this the testing must have a strategy as part of the plan. Also when testing at the API the API must be defined. This does not necessarily mean in a formal document such as a IRS or ICD, but it might. The testing of the API must be positive (against the functionality and any document) and negative (what bad inputs can we provide that messes with the API. The early API testing, as the software and API unfold, should be more functional and the later become more negative.

Just some thoughts

[Padmaraj]

I am testing API and like to bring more light on it:

Test methods – GET, POST, PUT, DELETE, and HEAD. (options, trace, patch) & Response`s
Test strategy – Black box testing ( functional side ) Acceptance environment & Whiter box ( more at code levels ) Development environment
There are several API testing advantages – Test for Core Functionality, Time Effective, Language-Independent

I am using SOAP UI &Fiddler – Free tool for API testing.

😛

[Ben Klaasen]

I use curl wrapped in bash for situations where I don’t need to parse the server response (but just log or eyeball the responses) and the Python Requests library when I do. Thankfully I haven’t had to work with SOAP since 2010 – it’s been JSON all the way since then!

If you’re taking the time to learn SoapUI or JMeter, I’d highly recommend learning a general purpose scripting language like bash or Python. You can be surprisingly productive in Python in a very short time.

[Kim]

Thanks all for the varied suggestions.

I am actually learning a bit of python, groovy and javascript at the moment. Often the tools alone have been a bit limiting and in order to write effective tests, values may need to be created and passed along the tests.
That is what I use scripts for and also for test assertions.

We are coming to the point now where we have defined API testing as a layer of testing. I created a regression suite in postman recently and it has been great. I can check at certain points during the day that the core functionality has not been affected by any changes and carry on exploratory testing the GUI.

[Janez]

Try HttpMaster. Fine tool for API and REST testing.

[Kim]

Thanks Janez! Will also look at it! 🙂

[Bas]

I’ve been doing API testing for a while now, using various tools. I have quite a bit of experience using Parasoft SOAtest, which I like but might be a little too much for testing just a couple of APIs. There’s also SoapUI / ReadyAPI of course, which I have a bit of experience with.

And for when you’re building a test framework in Java and you want to perform some validations on REST APIs, I highly recommend using REST Assured.

Still looking for a Java library that helps me do similar things for SOAP-based APIs.

[Nicholas]

Ben Klaasen wrote:

I use curl wrapped in bash for situations where I don’t need to parse the server response (but just log or eyeball the responses) and the Python Requests library when I do. Thankfully I haven’t had to work with SOAP since 2010 – it’s been JSON all the way since then!

If you’re taking the time to learn SoapUI or JMeter, I’d highly recommend learning a general purpose scripting language like bash or Python. You can be surprisingly productive in Python in a very short time.

Hi Ben,

I’m curious if you could provide more detail on what you’re doing with the Python Requests Library as it may be what I need to do soon enough. I’m guessing just exercising the API with set requests and validating the response against an expected one or pieces of what would be expected?

[Nicholas]

Hey Kim you may find this material useful if you haven’t all ready come across it, I unfortunately missed the weekend testing session also 🙁

http://weekendtesting.com/archives/3898
http://blog.eviltester.com/2015/01/some-api-testing-basic-introductory.html

I’m doing testing of a simple API at the moment also. Using a manual subset to confirm things when it goes to new environments. Otherwise its largely taken care of by a suite of functional tests built in robot (ewww)

Not a big fan of robot yet as its just slowing me down but its been very good at picking up tweaks. I’ve been using this extension in Chrome mostly:

https://chrome.google.com/webstore/detail/dhc-resthttp-api-client/aejoelaoggembcahagimdiliamlcdmfm?hl=en

[Kim]

Thanks Nick!

I liked the evil tester post! I have no idea what robot is so need to have a look to find out! 😀

I just found a resource I found really useful!

It is more of a beginners guide to API testing written by SmartBear:
http://www.soapui.org/testing-dojo/welcome-to-the-dojo/overview.html

If anything it helped me to understand that API testing on a basic level is testing but with different tools! This may sound stupid but I was worried it is something special and scary but it doesn’t seem too bad. 🙂

[Martin]

Hi Kim,

Interesting thread! I think most points have been covered by now. Ultimately I think your choice of tool comes down to understanding your API testing requirements. How will your customers use your APIs? What technologies will be used to make the calls? Will you have a client sitting on top of the APIs or will your customers be using them to build their own clients?

Also what are your own skills and will you get assistance from developers? Or are development already covering their own set of API tests, perhaps at a lower level?

I’ve found SOAPUI perfectly adequate for SOAP testing. It is lightweight and fairly intuitive to use. You can automate pretty easily using properties and groovy script and I find it pretty easy to maintain.

I’d be interested to know about anyone’s experience of using Ready API, what extras does it give you?

Also very interested to learn more about Sammy’s experience of testing microservices? Sammy, do you have any good pointers to any good documentation on the subject or could you share more of your experiences?

Also Kim, what do you mean by the API gateway? Is this like a proxy which handles different technologies?

Regards,
Martin

[Alin]

Hi Kim,

Regarding API testing, I have always used SoapUI Pro Version. Its interface is not very complicated and it has a lot of features. I can handle it very fast when I have to assert specific information from a response by using the XPATH, XQuery match or HTTP valid status codes just by a right-mouse click..

Also the concept of working with the “Get data” option from a response and transfer it to properties or other calls is very useful I think it is very important for all testers to be able to generate and process specific data needed in the tests like names, emails, time/date. The groovy script is very good for that.

Basically, in API testing I think it is important to test everything that means SOAP and HTTP requests (POST, PUT, GET) , to have an easy access to WADL and WSDL and to be able to connect to a database for. handling any needed data. On all these topics I think SoapUI performs perfectly.
A big advantage of SoapUI is that the tester can run tests that call other tests and decide for each run which parameters should be used as input data or output data. This is a huge improvement for a tester that must test many features and cover many test cases. A useful option is for debugging: I can partially run a test that failed by using the option “Run from here”. So, I can run only the steps I want in order to see a specific result.

Regarding the maintainability, when requirements are changed the tests can be easily modified accordingly. If needed, you can also develop and run load tests using various thread numbers.

As drawbacks, I find the UI not very friendy. It is rather simple in terms of design. Also, the groovy script does not have too many options for advanced scripting like a proper IDE (e.g. eclipse). There is also a lack of customization regarding the test reporting because the options for generating test reports is quite limited. But with a bit of practice, a tester can get used with these limitations.

You can also have a look at HP UFT (Unified Functional Testing). It can do also API testing and it can be integrated with HP ALM (Application Lifecycle Management). I saw a short comparison between SoapUI and HP UFT. The SmartBear solution was definitely better but HP is developing further its solution for API testing.
Other tools I did not use too much. That’s why I will not go into details.

I hope you have found my answer useful.

Regards,
Alin

[James]

Hi @punkmik,

I saw this post and was interested to how you have got on 1 year later. What tool(s) did you settle on and what have been your experience.
Do you have any recommendation for someone starting to look at API Testing?

Thanks,
James

[sneha shinde]

A few pointers to API testing:

– Getting to know the functionality of a particular API program and defining the scope of the program in a clear manner.

– Make use of testing techniques such as boundary value analysis, equivalence classes and error guessing along with writing test cases for the API.

– Input parameters are required to be planned and defined accurately.

API testing tools: Runscope, CTESK, Curl, dotTEST.

API testing related test cases:

API related test cases are based on following factors-

Returns nothing – When the return value does not exist, the effect of API on the system needs to be verified.

Modifying particular resources– If an API call alters some resources, then it ought to be validated with access to respective resources.

Return value depends on input condition– It is simple to test, that’s because input can be defined and authentication of results is possible.

Updating the data structure– Data structure updation will affect the system and that requires to be authenticated.

Initiation of some kind of event/API/interrupt– In the event that the outcome of an API initiates some kind of interrupt or event, then such events and interrupt listeners need too be tracked.

[Anita]

There are a bunch of new API testing tools, like ReqBin – you can test an API by sending requests directly from your browser.

[Razvan]

Hello,

I’ve been testing Rest API endpoints for years now, with both manual and automation tools (based on java / javascript / php).

So far, in terms of manual testing, Postman goes without saying.

In terms of automation, I strongly recommend RestAssured (Java), or Karate Framework (it’s a combination of Java and JavaScript – only basic knowledge required). Both ones have huge communities and very nice documentation.
RestAssured handles only REST API, but Karate promises to handle both REST and SOAP as well.

Even though the trend tends to microservices and rest api endpoints, we might still face some ‘old’ approaches that use SOAP. For those, I’ve heard that the SoapUI tool does the trick.

I hope it helped 🙂 for any other info, please let me know

Cheers,
RV

[Author]

Posts