Test Case Reduction Techniques

Technote (troubleshooting)


Problem(Abstract)

A test case consisting of a small program written in C/C++/Fortran can often be used to demonstrate a compiler defect. Generating a small, concise test case is very important to the fast turnaround time for its fix. This document will take you through some of the more common reduction techniques to help you reduce your test case.

Resolving the problem

The most effective way to demonstrate a compiler defect to IBM is through a small, self-contained program or a test case. An ideal test case contains only the code necessary to reproduce the error. It typically has fewer than 100 lines of code in a single file without any dependencies on third-party code. This small test case helps IBM developers to quickly determine the cause of failure and thereby provide a faster fix or explanation to the customer.

Most users, depending on their experience level and code familiarity, will be able to make educated guesstimates on which sections of the code to omit without damaging the integrity of the problem itself. However, there are certain program characteristics you may focus on to make this reduction process a faster and easier one. You may find most of these techniques can be used interchangeably between C/C++ and Fortran test cases.

Common Test Case Reduction Techniques

1. Remove Compiler Options

Start by removing as many options as possible. More often than not, you should have no options or have found a bug in the use of an option. Best options to remove first are the optimization related options (that is, -O, -O4, etc.). Unless the problem involves incorrect code generation at high optimization levels, in which case the optimization options will be required to reproduce the problem. Next options you should be looking to eliminate are the preprocessing related options (for example, -D, -U). Modify the source so that these options will not be needed. This step should be re-visited as more source is removed from the test case to have the fewest options possible.

This step is relatively easy and will often allow you to gain valuable information about the problem. If the problem is option related, this step will allow the IBM developers to quickly track down the defected area.

2. Remove #include Files

Once the test case is reduced down to one or a few files, eliminate the headers. A one line source file that includes a single Standard Template Library header file actually represents anywhere from 2000 to 8000 lines of code for the developer to work through.

Removing header files may be more tricky with a large include hierarchy. Taking away header files may just make the problem disappear. You have two methods you can apply at this point: 1) Trace through the hierarchy and decide which files can be eliminated while still keeping the problem; or 2) Produce a preprocessed file ( <filename>.i ) by using the -P option. The latter may be easier for a large include hierarchy. The -P option will pull in all the source code in the header files and combine them into one preprocessed file (.i). The reduction process can be then continued with this file. This method also makes backup easier as you will only have to backup one file instead of all the header files. It is a good idea to save your work every few steps as you may hit a dead-end and have to reverse previous steps.

3. Remove Source Code

By this point, your test case should contain a single file with no includes. Each tested removal, in your judgement, should have a good chance of reproducing the original compiler defect. Attempting a reduction that introduces a syntax error will almost guarantee a failure to reproduce the original defect. An excellent knowledge of the programming language will be very useful here.

With C/C++ problems, you may find inserting #if 0 ... #endif tags useful for testing the removal of large blocks of code. The block commenting (/* ... */) can also be used but you may run into problems when comments are still in the source. Start removal attempts at the bottom of the file. It is more natural for people to code in order and we should use this characteristic to our advantage. With ordered code, declarations at the end of the file cannot have any references to them further down, so they will be easier to remove. If the reduced file successfully reproduces the original symptom, delete the commented out code and repeat the process. Remember to always save your work after every few steps.

If you find a piece of code that is critical to reproducing the problem, try simplifying it before going back to remove other blocks of code from above it. The reason for this is that the fewer references the piece of critical code makes to the code above, the easier it will be to remove that code. For example, if removing this template class makes the problem go away:

template <class T, class U=, int i = A::i> class C:public D<U,T> {

void f(int i, float ff, B<T> bb);

... // many more member functions

... // many data members

};

try simplifying the template class C by using all of the items in the list below. If removing one of these seemingly harmless C++ structures makes the problem disappear, you have found potentially very useful information.

Remove extra code in the following order. This order will get rid of most code as quickly as possible and helps focus in on the code related to the problem.

Tips in reducing a C/C++ test case:

  1. Function bodies. If necessary, keep the function declaration until all the calls to the function are removed. The only function body that is needed is the one that produces the problem. All other functions should be reduced as much as possible.
  2. Unused user defined data types (enums, typedefs, classes, etc.). These should be removed continually. You may find it helpful to turn typedefs into the underlying base type. Only keep typedefs that improve the readability of the test case, which helps in the reduction process.
  3. Class access control. Turn classes into structs and take out all the public/protected/private labels. Friend declarations can also be removed here.
  4. Base classes. Try to reduce the size of the class hierarchy. Start with the most derived class, which should be the primary class you are working with anyway, and remove as many base classes as you can.
  5. Unused struct members. Remove all the unused data members and function members, unless the layout needs to be kept to identify a runtime problem. If the problem is related to virtual functions, you may need to keep one virtual function to get the vft.
  6. Templates. Convert the remaining templates into structs. By this time you should only have one or two small template classes that are only instantiated for a single type. If the problem is template-related, reduce the number of template parameters. If not, convert the templates to structs.
  7. Default initializers. Remove these from all function and template parameters. Change call and instantiation areas so that they are supplied explicitly.


Tips in reducing a Fortran test case:
  1. If module variables are used, remove the declaration from the module and declare the variables in the scoping unit that the variables are used.
  2. If derived types are used, remove the irrelevant components. If there is only one component left, try to replace the derived type by an object that has the intrinsic type of the component.
  3. If module procedures are used, move the procedure definition out of the module and declare it as an external procedure.
  4. If external/internal procedures are referenced, inline the procedures as much as possible.
  5. If statement functions are used, replace the references by the corresponding expressions.
  6. If comment form directives are used, remove the directives provided that the original symptom is reproducible.

4. Simplify Identifiers

Identifiers should be approximately scaled with project size. In a large project, descriptive identifiers are needed to distinguish between similar but different things. However, these long identifiers are less meaningful when the test case is reduced down to a small size. Rename the identifiers.

Symptom Specific Test Case Reduction Techniques

Test case reduction can also be categorized and handled according to their symptoms. The three main categories of defect symptoms are:

1) Invalid message

2) Internal compiler error (ICE) or infinite loop generation

3) Bad code generation (program execution fails) or unexpected runtime output

Invalid Message

This type of problems are usually the easiest. The compiler tells you which line(s) in the source file that is causing the problem. Use step 3 (above) to remove the irrelevant code.

Internal Compiler Error (ICE) or Infinite Loop Generation

An ICE occurs when the compiler hits an unexpected problem and crashes. The error message generated for this type of problem is not useful in identifying the problem at all. Since you probably have no idea where in the source is causing the failure, you will have to use the trial-and-error method. Eliminate code (using the techniques mentioned earlier) until you have found the problem area. And you should be able to remove the irrelevant code at a much faster rate after this.

Bad Code Generation (program execution fails) or Unexpected Runtime Output

By the time you have noticed this problem, you should have already identified the object file(s) that is generating incorrect code. Apply the removal techniques to reduce the rest of the program. Then find some way to produce the correct code. This may be done by using compilers at different maintenance level (PTF level) or compiling the code under different optimization levels (for example, use -O2 instead of -O4). This information is potentially useful to the IBM developers.

For problems with runtime output, it is often a good idea to start reducing from the main of the program. After reducing the size of the main, eliminating the rest of the project should be much easier.

Cross Reference information
Segment Product Component Platform Version Edition
Software Development XL Fortran Documentation AIX 6.1
Software Development XL C/C++ Documentation AIX, Linux, Mac OS X All
Software Development XL C Enterprise Edition for AIX Not Applicable AIX, Linux, Mac OS X All

Rate this page:

(0 users)Average rating

Add comments

Document information


More support for:

VisualAge C++
Documentation

Software version:

All versions

Operating system(s):

AIX, Linux Red Hat - i/p Series, Linux SUSE - i/p Series

Software edition:

All

Reference #:

1084174

Modified date:

2010-10-27

Translate my page

Machine Translation

Content navigation