Test Management Tools and automated tests – friends or foes ?

[Andreas]

Hi,

I’m desperately looking for a test management tool that supports automated tests. As this sounds simple, I still fail to do so.

Let me explain my setup. We have a bunch of automated tests, which are run as part of our continuous integration pipeline. This is fine and works as wanted, but we miss the benefits of test management tools, such as

– mapping requirements with test cases

– getting test coverage for requirements

– linking test cases/test results with issues

– easy retesting of fixed reported issues

and so on and so on. So simply said, the good things about a test mangement tool.

So I went for the hunt to find one, but for me it seems that all out there support mostly manual testing and/or have only limited support of automated testing.

Duplication of test cases

We have now a bunch of test cases already defined in natural language (via http://gauge.org). I want to integrate those into the TMT and not define them again, to avoid having duplicated my test cases and maintain two (or more) copies of them.

Our test cases residing in git and I have not found a tool to import test cases from git.

Automation results

All TMTs offer a workflow to add results in an intuitive manner (for manual tests), but I have not found a tool that supports automation, like trigger tests from the tool and retrieve the results.

 

All that is quite frustrating and I cannot think that it is only me fighting this, is it ?

Does anyone has any experience with any tool for automated tests ?

Please be so kind and do not simply post links, but I would really appreciate, if you can drop some lines, why this tool helps and how this tool would help me in the problems described above.

Thanks for any input

 

[Roman]

I observed two attempts of linking automated tests to TMT.

First one: custom framework to Quality Center. It was awful. Everyone except the developers hated both framework and QC.

Second one: custom framework (different one) and TestRail. Linking framework to test runs in TestRail proven itself too cumbersome, slow and inconvenient, and was abandoned.

Solution(s):

– Test cases in TestRail can have AutomationID field. That allows to track failed tests and find in TestRail (or other TMT) what they are about.
– Test cases in TestRail can have RequirementID field. That’s how you may track requirements coverage.
– Issues in Jira can have links to TestRail in some field or comments.
– TestRail has CSV import functionality with customizable mapping. If you can somehow convert existing test cases to CSV, you can import them into TestRail.

But I can’t recommend (emphasis) to link automation to any TMT — I hadn’t see any big good from that so far.

[Andreas]

Thank you @rpwheeler for your answer.

I used TestRail in my previous company and actual had the AutomationId field for automated tests. It worked ok-ish, but I totally understand your defensivness.

As for your recommendation – do you then simply do not have any linking between test cases and requirements or test results and issues or is it then all custom made ?

[Roman]

We had links from requirements to TestRail cases and link  from TestRail back to the requirements page(s) (one can put some link to any TestRail test suite folder description). I think that was what you call custom-made linking. One could open requirements on one’s left screen and TestRail folder on one’s right screen and see what is covered. However in our current reality, where words “agile” and “fast-paced” became ubiquitous description of what is going on, it is not mandatory anymore unless our customers sign off requirements.”Working software over comprehensive documentation”, you know.

We create test runs for manual test results, but do not have to link automated checks results to TestRail. Tickets created may or may not have custom links to TestRail.

 

[Author]

Posts