Tools Support For Testing CTFL Tutorial

The tools are grouped by the testing activities or areas that are supported by a set of tools, for example, tools that support management activities, tools to support static testing, etc.

It is not required to have a one-to-one relationship between a type of tool described here and a tool offered by a commercial tool vendor or an open-source tool. Some tools perform a very specific and limited function (sometimes called a ‘point solution’), but many of the commercial tools provide support for many different functions.

For example a ‘test management‘ tool may provide support for managing testing (progress monitoring), configuration management of testware, incident management, and requirements management and traceability. Similarly, another tool may provide both coverage measurement and test design support.

There are few things that people are good at in comparison to the computers. For example, when you see your friend in an unexpected place, like a shopping mall, you can immediately recognize their face. This is because people are very good at pattern recognition, but it’s not easy to write software that can recognize a face.

On the other hand there are things that computers can do much better or more quickly than people can do.

For example, to add up 20 three-digit numbers quickly. This is not easy for most people to do, there are chances that you make some mistakes even if the numbers are written down.

A computer does this accurately and very quickly.

Consider another example, if people are asked to do exactly the same work over and over, it will be monotonous and tedious they soon get bored and then start making mistakes.

So, the idea is to use computers to do things that they are really good at. Tool support is very useful for repetitive tasks – the computer doesn’t get bored and will be able to exactly repeat what was done before and that too without any mistakes.

Since the tool will be fast, this can make those activities much more efficient and more reliable.

Probe effect

A tool that measures some aspect of software may have few unexpected side-effects also on that software. Like, for example, a tool that measures timings for performance testing needs to interact very closely with that software in order to measure it.

A performance tool will set a start time and a stop time for a given transaction in order to measure the response time. But by taking that measurement, that is storing the time at those two points, could actually make the whole transaction take slightly longer than it would do if the tool was not measuring the response time. Of course, the extra time is very small, but it is still there. This effect is called the ‘probe effect’.

Other example of the probe effect is when a debugging tool tries to find a particular defect. If the code is run with the debugger, then the bug disappears; it only re-appears when the debugger is turned off, hence making it very difficult to find. These are sometimes known as ‘Heizenbugs’ (after Heizenberg’s uncertainty principle).

Following are the classification of different types of test tools according to the test process activities. The ‘(D)’ written after the types of tool indicates that these tools are mostly used by the developers. The various types of test tools according to the test process activities are:

1.Tool support for management of testing and tests:

• Test management tools
• Requirements management tools
• Incident management tools
• Configuration management tools

2.Tool support for static testing:

• Review process support tools
• Static analysis tools (D)
• Modelling tools (D)

3.Tool support for test specification:

• Test design tools
• Test data preparation tools

4.Tool support for test execution and logging:

• Test execution tools
• Test harness/ Unit test framework tools (D)
• Test comparators
• Coverage measurement tools (D)
• Security tools

5.Tools support for performance and monitoring:

• Dynamic analysis tools (D)
• Performance testing, Load testing and stress testing tools
• Monitoring tools.

Software Testing Tools used by ISTQB certified Experts like Test Analysts and Technical Test Analysts

Many tools for the support of software development are available, and the selection is growing every day.

It is therefore extremely difficult to list down specific tools. Here we are discussing the classification of Nine different types of tools according to the primary users of the tools specifically ISTQB certified software testing experts like Test analysts and technical test analysts.

These nine types of tools being described are as under:

1. 1)Test design tools
2. 2) Test data generation tools
3. 3) Test oracles
4. 4) Simulation and emulation tools
5. 5) Test execution tools
6. 6) Keyword-driven automation tools
7. 7) Comparison tools
8. 8) Fault-seeding and fault-injection tools
9. 9) Web tools

1)Test Design Tools:

Test design tools support the creation of test specifications. They can analyze a specification of the product, often expressed as a model in a formal way, and generate high-level test cases and possibly test procedures

or scripts based on this analysis.

This type of software testing tool can, for example:

• Derive high-level test cases from formally specified requirements, often managed by the same tool
• Generate test cases based on the specification of a model, for example, UML or state machines
• Generate input for test cases based on input models, for example, input distribution specifications
• Derive high-level test cases from actual source code

The test design tools require very formally formatted basis documentation, and that can be regarded as both an advantage and a disadvantage.

2)Test Data Preparation Tools:

Test input data preparation tools support:

• Selection (e.g., from an existing database)
• Creation
• Generation
• Manipulation
• Editing

of test data for use in setting up preconditions for test procedures and individual test cases.

Some of these tools are data tool-dependent, while some of the sophisticated tools have ability to handle a wide range of the file & database formats.

Test data can be selected and extracted from live data and scrambled to hide person-sensitive information. This enables tests to be performed on real data, something that can be essential for systems in, for example, the public sector.

For example – A test data preparation tool is able to extract live data from the tax authoritys database according to specific selection criteria for test runs of the implementation of a new tax law. The criteria may be 100 families with one income and at least three children, 100 people over 80 years of age with an income over a certain amount, and the 40 people with the highest income in a specific city. The tool scrambles the information that can identify the people in the test data (e.g., Social Security number, before the data may be used).

3)Test Oracles:

A test oracle is a special concept in test automation; it is used to determine expected results from inputs. Some say that the best test oracle is the tester, studying the test basis documentation and deriving the expected result from this. This is, however, sometimes not possible for time and/or cost reasons. Automated test oracles are tools that can generate the expected result for specific input and hence facilitate the creation of test cases. Such “oracles” are hard to find. In principle they must do exactly the same as the object under testing and may therefore seem redundant.

One of the situations where an oracle can be found and can be very useful is when an old system is being replaced by a new one providing the same functionality. This is seen more and more often when old legacy systems are replaced with systems using new technology, for example, Web access. In such a case test input may be given to the old system and the result be regarded as the expected result for the new system.

Oracles can also be created in situations where nonfunctional requirements can be disregarded and a system simulating the functionality only can be developed at a much lower cost. This is especially the case where the real system has very strict performance requirements.

4)Simulation Tools and Emulation Tools:

Simulation tools are generally deployed for supporting the tests when the desired code or some other systems are not possible or not available or even harmful for use.

Test harnesses and drivers fall into this category of tools. They are used where components or other test objects cannot be executed directly. It can be for testing of a component in isolation, embedded software without a user interface, or execution of many unrelated automated test scripts.

Some testing tools on the market provide harness and driver facilities, especially component testing tools. Very often, however, these tools are homemade and tailored precisely to needs. The principle in stubs and drivers has been explained in the following figure.

 A special type of simulators is called emulators because they are used to mimic hardware to which the software under testing interfaces.

Simulation tools are almost always bespoke systems made for a specific assignment.

5)Test Execution Tools:

This type of software testing tool have many popular names like: test execution tools, or test running tools, or capture and replay tools, and is probably the most widespread category of testing tool.

The test execution tools are primarily used for automation of regression testing. They can execute test scripts much faster and more reliably than human beings, and they can therefore reduce test execution time when tests are repeated and/or allow more tests to be executed.

All the tools of this category work according to the same basic principles, namely:

Capture:

A recording of all the testers manual actions and the systems responses into a test script

Control points:

A number of checkpoints added to the script by the tester during the capture

Playback:

Automatic re-execution of the test script

Test execution tools exist for graphical user interface, GUI, and for character-based interfaces. For GUI applications the tools can simulate mouse movement and button clicks and can recognize GUI objects such as windows, fields, buttons, and other controls.

When a script has been captured once, it may be executed at any given time again. If the software under testing reacts differently from what was expected at the inserted checkpoints, the execution will report a failure. It is usually also possible to log information during execution.

Test scripts are captured in a specific scripting language. In newer versions of these tools it has become possible to get access to the scripts that have been captured. These are often in a C or Visual Basic like code, and this offers the possibility for editing the scripts, so that, for example, forgotten operations, further control points, or changed values, may be added.

Experience shows that if the scripts are written from scratch, rather than captured, and good and systematic development principles are used, the scripts will be more maintainable. More and more of these tools are therefore used as test execution tools of coded scripts, rather than capture/playback.

6)Keyword-Driven Automation Tools:

Keyword-driven test is a way to execute test scripts at a higher level of abstraction. The idea is similar to that of a service or subroutine in programming where the same code may be executed with different values.

Keywords are defined to represent a script, and a tool can then act as a link between the keywords and the tool executing the corresponding test script. Values may be assigned for parameters associated with the keywords.

The tools make it possible to use parameter-driven test scripts without having to change the scripts in the execution tool. Keywords are usually related to higher level functionality or business procedures. They may also reflect use cases.

The tools for keyword-driven software testing are also known as script wrappers, because they wrap the technical part of the test (the actual test scripts and the test execution tool) so that the testers only need to know about the high-level keywords.

Keywords may be held in spreadsheets or tables, and longer sequences executions of test scripts can be specified by sequences of keywords. Example of a test sequence defined by Keyword-driven software testing requires a good overview of the test assignment and a high level of abstraction as all parameterization does. This is demanding but can be rewarding for the test in the long run.

7)Comparison Tools:

Comparison tools are used to find differences between the expected and the actual results.

These tools range from very simple comparison facilities, like in Word, for example, to very advanced, dedicated tools. Test execution tools normally have some comparison facility included.

The tools may be able to compare, for example, values in files or on screens, bitmaps, and positions.

8)Fault-Seeding and Fault-Injection Tools:

These types of software testing tools are used to support the defect-based test technique fault-injection or fault seeding.

The tools create or inject faults or defects into the software component under testing. The tools can work either on the source code, changing the code in pre-specified ways, or on the compiled code, changing the structure of the code.

In both cases new versions of the component under test are created with the specified defects.

9)Web Tools:

These days testing never stops. With more and more Web-based products around, we need to constantly monitor that the products are doing well. Products being Web-based means that some issues are out of our hands (for example, hyperlinks and server and network availability). Hyperlink testing tools are used to check that no broken hyperlinks are present on a Web site.

These tools often have additional functionality such as HTML validation, spelling, and availability check. The facility is often built into other tools e.g., HTML development tools.

Monitoring tools are used for Web-based products, most typically e-commerce and e-business applications. The tools monitor the products availability to customers and the service level i.e. performance and resource usage. The tools will issue warnings if the monitoring shows that something is not as expected.

Many free Web tools are available for this type of tool. Many web tools are proprietary, and are quite sophisticated as well.

Test management tool :

Test management tools are used to store information on how testing is to be done, plan testing activities and report the status of quality assurance activities. The tools have different approaches to testing and thus have different sets of features. Generally they are used to maintain and plan manual testing, run or gather execution data from automated tests, manage multiple environments and to enter information about found defects. Test management tools offer the prospect of streamlining the testing process and allow quick access to data analysis, collaborative tools and easy communication across multiple project teams. Many test management tools incorporate requirements management capabilities to streamline test case design from the requirements. Tracking of defects and project tasks are done within one application to further simplify the testing.

Structuring the test process :[edit]

Test management tools give teams the ability to consolidate and structure the test process using one test management tool, instead of installing multiple applications that are designed to manage only one step of the process. Test management tools allow teams to manage test case environments, automated tests, defects and project tasks. Some applications include advanced dashboards and detailed tracking of key metrics, allowing for easy tracking of progress and bug management.

Implementation:[edit]

A test management tool that includes everything needed to manage the test process can save testers the problems of installing separate applications that are necessary for the testing process, which can also be time consuming. They can be implemented with minimal programming ability, allowing for easy installation and monitoring of the test process across multiple project groups. Once installed, teams have instant access to a user interface and can immediately start running and recording test cases. These types of applications are designed to simplify the test management process with high levels of automation and tracking built in, yet don’t require advanced programming skills or knowledge to implement. They are useful for teams who manage a variety of test cases and for larger teams who need an all-inclusive application for project management.

Using:[edit]

Once a project has kicked off, a test management tool tracks bug status, defects and projects tasks, and allows for collaboration across the team. When administering test cases, users can access a variety of dashboards to gain access to data instantly, making the test process efficient and accurate. The type of dashboard used is determined by the scope of the project and the information and data that needs to be extracted during the testing process. Data can be shared and accessed across multiple project teams, allowing for effective communication and collaboration throughout the testing process.

Testing Tools in the V-Model

As seen in the image below, requirement testing tools are used for requirement analysis, functional design, technical design, and coding activities.

These tools often help in capturing requirements using intuitive models. Once requirements are recorded, the tool can generate high-level architecture design and code. The inbuilt logic of the tool helps in identifying gaps in the requirements.

Test design and test data preparation tools can be used once requirements are finalized. These tools can generate automatic test cases from requirements and support automatic creation of test data.

Static analysis tools become useful during design and coding phases by identifying gaps in the code. Coverage measure tools and debugging tools can be used during component testing.

Test harness and drivers, and dynamic analysis tools are used during integration testing. Test run and comparison tools are used at all levels of testing to compare results. Performance measurement tools are used in system and acceptance testing to measure the system performance.

Tools Support for Management of Testing

Tools that support Test Management activities are:

• Test Management Tools
• Requirement Management Tools
• Configuration Management Tools
• Incident Management Tools

Test Management Tools

Features provided by test management tools include managing tests, scheduling, logging test results, traceability support, interface with test execution, incident management, requirement management, configuration management, status report generation, and version control.

Examples include HP Quality Centre and IBM Quality Manager.

Requirement Management Tools

Features provided by requirements management tool include storing requirement statements, generating unique identifiers, checking the consistency of requirements, prioritizing requirements for testing purposes, and managing traceability through levels of requirements.

Examples include Analyst Pro, CaliberRM, and WinA&D.

Configuration Management Tools

Features of configuration management tools include storing information about versions and builds of the software and testware, traceability between software and testware, release management, baselining, and access control.

Examples include Visual Source Safe and Clear Case.

Incident Management Tools

Incident management tool is also known as a defect-tracking, a defect-management, a bug-tracking, or a bug-management tool. Incident management tools make it much easier to keep track of the incidents over time.  

Characteristics of incident management tools include incident and logging support, change request, providing incident or anomaly report, and other incident management reports.

Examples include Star Team and Bugzilla.

Introducing the Tool–Example

ABC Corp invested $1 million towards purchasing a new tool for Test Management Process. After this substantial investment, it did not want to waste time before the tool was released to all teams within the organization.

The teams were also excited about learning something new and hence quickly accepted the implementation.

However, as the teams started to use the tool, each team realized that the tool did not match the project process. Hence, they requested for tool workflow customizations.

With multiple requests coming in from teams, the tool support team was unable to define a consistent workflow.

Over a period of time, the main workflow was not followed by any team, and each team had its own exception flow being followed. The maintenance of the tool became difficult and eventually, the tool had to be removed from the organization.

Summary

Here is a quick recap of what we have learned this lesson:

• The quality and effectiveness of testing increase with an increase in the degree of independence.
• The factors to be considered while building a test plan are test policy of the organization, the scope of testing, and objectives of testing.
• Test progress monitoring is a test management task that periodically monitors the status of a test project.
• Configuration Management is a disciplined approach to the management of software and the associated design, development, testing, operations, and maintenance of testing.
• Risks related to the management and control of a project that impacts project’s capability to deliver its objectives is known as project risks.
• Incident management is the process to ensure that incidents are tracked through all the steps of the incident lifecycle.