Grouping automated tests, your experiences?

[Shane]

Complete beginner here in the world of testing pretty much. Basically I’m working with visual studio and selenium to automate a series of regression tests for a site. I’m wondering what peoples experiences are in terms of grouping tests, the two options I can see are to create my methods, place them in a test method and then test them in order this way, or to create an ordered test in visual studio and run them this way. I’m just curious to find out what opinions people have on either of these methods of grouping, and what experiences people have had with either of these methods of grouping, positive or negative.

[Krister]

Hi @edwrd01

It’s the unexpectedly failed tests and unexpected events during testing that complicates automation. Following test cases or test suites become blocked and script stops.
My experience is that test suites shall have an initiate procedure, taking you to a desired precondition, and an error recover procedure, taking you back to a known status, a point from you can run the next test suite.
I like to think of it as a Daisy flower. You make a tour like a leaf out from centre and then returns to a safe state before taking next tour.

Rgds
Krister

[Alin]

Hi Shane,

A good approach would be to have no dependency between tests. This means that each test will use its own data; in case of a failure there will be no other tests affected by this problem since there is no connection between tests. You can take as example a test that checks the order process in a shop: the test starts by creating a new user, makes and validates all necessarily steps for ordering and at the end it deletes all created data (databases, logs, virtual machines etc.). The next test that validates other functionality will create its own user and so on. Of course, you can choose to delete data only when all tests are run (not at each test).

Another important thing is to avoid having very long tests. If a test has many steps you should try to split it into shorter tests.

Generally, unexpectedly failed tests and events are a big challenge for automation, as Krister says. You will see situations when they are randomly failing and that’s why is good to avoid any dependency between them.

A negative side of this approach is the overhead that is added because it requires more time at each test to create/delete data. If there is a time constraint for running all regression tests, you can handle this issue by using more systems to run them in parallel. Thus, it will take shorter to run the complete set of regression tests.

Regards,
Alin

[Joerg]

Hi,

in the different projects where I have worked for, we found mostly an approach like this (shorten):

1. Tests without database/master data we called resource-tests because they check only if a control is visible, a dialog can be used etc.
2. Tests with master-data but no transaction data, if you need settings to get specific business cases running you need access to a well defined dataset which has to be defined
3. Tests with master- and transaction data (business process tests) – these are the<span style=”text-decoration:underline;”> most costly but best end-to-end test cases</span> and you need to control the growth of these test cases often because they need everything from the system- and database-layer and eating time. Split them also into daily executing and release-executing test cases (e.g. huge load tests do not have to be executed every day but within defined iterations)

Hope that is an approach you can start with. We have also defined in some projects more special test classes but the basics also were like the approach described above.

Yours,
Jogi

[Author]

Posts