Unidade de Testes

As of November 2007 we require all new features going into master to be accompanied with a unit test. Initially we have limited this requirement to qgis_core, and we will extend this requirement to other parts of the code base once people are familiar with the procedures for unit testing explained in the sections that follow.

Infraestrutura de testes do QGIS - um resumo

Unit testing is carried out using a combination of QTestLib (the Qt testing library) and CTest (a framework for compiling and running tests as part of the CMake build process). Lets take an overview of the process before I delve into the details:

  • There is some code you want to test, e.g. a class or function. Extreme programming advocates suggest that the code should not even be written yet when you start building your tests, and then as you implement your code you can immediately validate each new functional part you add with your test. In practive you will probably need to write tests for pre-existing code in QGIS since we are starting with a testing framework well after much application logic has already been implemented.
  • You create a unit test. This happens under <QGIS Source Dir>/tests/src/core in the case of the core lib. The test is basically a client that creates an instance of a class and calls some methods on that class. It will check the return from each method to make sure it matches the expected value. If any one of the calls fails, the unit will fail.
  • You include QtTestLib macros in your test class. This macro is processed by the Qt meta object compiler (moc) and expands your test class into a runnable application.
  • You add a section to the CMakeLists.txt in your tests directory that will build your test.
  • You ensure you have ENABLE_TESTING enabled in ccmake / cmakesetup. This will ensure your tests actually get compiled when you type make.
  • You optionally add test data to <QGIS Source Dir>/tests/testdata if your test is data driven (e.g. needs to load a shapefile). These test data should be as small as possible and wherever possible you should use the existing datasets already there. Your tests should never modify this data in situ, but rather may a temporary copy somewhere if needed.
  • You compile your sources and install. Do this using normal make && (sudo)  make install procedure.
  • You run your tests. This is normally done simply by doing make test after the make install step, though I will explain other aproaches that offer more fine grained control over running tests.

Right with that overview in mind, I will delve into a bit of detail. I’ve already done much of the configuration for you in CMake and other places in the source tree so all you need to do are the easy bits - writing unit tests!

Criação de unidade de teste

Creating a unit test is easy - typically you will do this by just creating a single .cpp file (not .h file is used) and implement all your test methods as public methods that return void. I’ll use a simple test class for QgsRasterLayer throughout the section that follows to illustrate. By convention we will name our test with the same name as the class they are testing but prefixed with “Test”. So our test implementation goes in a file called testqgsrasterlayer.cpp and the class itself will be TestQgsRasterLayer. First we add our standard copyright banner:

/***************************************************************************
 testqgsvectorfilewriter.cpp
 --------------------------------------
  Date : Friday, Jan 27, 2015
  Copyright: (C) 2015 by Tim Sutton
  Email: [email protected]
 ***************************************************************************
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 ***************************************************************************/

Next we use start our includes needed for the tests we plan to run. There is one special include all tests should have:

#include <QtTest/QtTest>

Beyond that you just continue implementing your class as per normal, pulling in whatever headers you may need:

//Qt includes...
#include <QObject>
#include <QString>
#include <QObject>
#include <QApplication>
#include <QFileInfo>
#include <QDir>

//qgis includes...
#include <qgsrasterlayer.h>
#include <qgsrasterbandstats.h>
#include <qgsapplication.h>

Since we are combining both class declaration and implementation in a single file the class declaration comes next. We start with our doxygen documentation. Every test case should be properly documented. We use the doxygen ingroup directive so that all the UnitTests appear as a module in the generated Doxygen documentation. After that comes a short description of the unit test and the class must inherit from QObject and include the Q_OBJECT macro.

/** \ingroup UnitTests
 * This is a unit test for the QgsRasterLayer class.
 */

class TestQgsRasterLayer: public QObject
{
    Q_OBJECT

All our test methods are implemented as private slots. The QtTest framework will sequentially call each private slot method in the test class. There are four “special” methods which if implemented will be called at the start of the unit test (initTestCase), at the end of the unit test (cleanupTestCase). Before each test method is called, the init() method will be called and after each test method is called the cleanup() method is called. These methods are handy in that they allow you to allocate and cleanup resources prior to running each test, and the test unit as a whole.

private slots:
  // will be called before the first testfunction is executed.
  void initTestCase();
  // will be called after the last testfunction was executed.
  void cleanupTestCase(){};
  // will be called before each testfunction is executed.
  void init(){};
  // will be called after every testfunction.
  void cleanup();

Then come your test methods, all of which should take no parameters and should return void. The methods will be called in order of declaration. I am implementing two methods here which illustrates two types of testing. In the first case I want to generally test the various parts of the class are working, I can use a functional testing approach. Once again, extreme programmers would advocate writing these tests before implementing the class. Then as you work your way through your class implementation you iteratively run your unit tests. More and more test functions should complete sucessfully as your class implementation work progresses, and when the whole unit test passes, your new class is done and is now complete with a repeatable way to validate it.

Typically your unit tests would only cover the public API of your class, and normally you do not need to write tests for accessors and mutators. If it should happen that an acccessor or mutator is not working as expected you would normally implement a regression test to check for this (see lower down).

//
// Functional Testing
//

/** Check if a raster is valid. */
void isValid();

// more functional tests here ...

Next we implement our regression tests. Regression tests should be implemented to replicate the conditions of a particular bug. For example I recently received a report by email that the cell count by rasters was off by 1, throwing off all the statistics for the raster bands. I opened a bug (ticket #832) and then created a regression test that replicated the bug using a small test dataset (a 10x10 raster). Then I ran the test and ran it, verifying that it did indeed fail (the cell count was 99 instead of 100). Then I went to fix the bug and reran the unit test and the regression test passed. I committed the regression test along with the bug fix. Now if anybody breakes this in the source code again in the future, we can immediatly identify that the code has regressed. Better yet before committing any changes in the future, running our tests will ensure our changes don’t have unexpected side effects - like breaking existing functionality.

There is one more benefit to regression tests - they can save you time. If you ever fixed a bug that involved making changes to the source, and then running the application and performing a series of convoluted steps to replicate the issue, it will be immediately apparent that simply implementing your regression test before fixing the bug will let you automate the testing for bug resolution in an efficient manner.

To implement your regression test, you should follow the naming convention of regression<TicketID> for your test functions. If no redmine ticket exists for the regression, you should create one first. Using this approach allows the person running a failed regression test easily go and find out more information.

//
// Regression Testing
//

/** This is our second test case...to check if a raster
 *  reports its dimensions properly. It is a regression test
 *  for ticket #832 which was fixed with change r7650.
 */
void regression832();

// more regression tests go here ...

Finally in our test class declaration you can declare privately any data members and helper methods your unit test may need. In our case I will declare a QgsRasterLayer * which can be used by any of our test methods. The raster layer will be created in the initTestCase() function which is run before any other tests, and then destroyed using cleanupTestCase() which is run after all tests. By declaring helper methods (which may be called by various test functions) privately, you can ensure that they wont be automatically run by the QTest executable that is created when we compile our test.

  private:
    // Here we have any data structures that may need to
    // be used in many test cases.
    QgsRasterLayer * mpLayer;
};

That ends our class declaration. The implementation is simply inlined in the same file lower down. First our init and cleanup functions:

void TestQgsRasterLayer::initTestCase()
{
  // init QGIS's paths - true means that all path will be inited from prefix
  QString qgisPath = QCoreApplication::applicationDirPath ();
  QgsApplication::setPrefixPath(qgisPath, TRUE);
#ifdef Q_OS_LINUX
  QgsApplication::setPkgDataPath(qgisPath + "/../share/qgis");
#endif
  //create some objects that will be used in all tests...

  std::cout << "PrefixPATH: " << QgsApplication::prefixPath().toLocal8Bit().data() << std::endl;
  std::cout << "PluginPATH: " << QgsApplication::pluginPath().toLocal8Bit().data() << std::endl;
  std::cout << "PkgData PATH: " << QgsApplication::pkgDataPath().toLocal8Bit().data() << std::endl;
  std::cout << "User DB PATH: " << QgsApplication::qgisUserDbFilePath().toLocal8Bit().data() << std::endl;

  //create a raster layer that will be used in all tests...
  QString myFileName (TEST_DATA_DIR); //defined in CmakeLists.txt
  myFileName = myFileName + QDir::separator() + "tenbytenraster.asc";
  QFileInfo myRasterFileInfo ( myFileName );
  mpLayer = new QgsRasterLayer ( myRasterFileInfo.filePath(),
  myRasterFileInfo.completeBaseName() );
}

void TestQgsRasterLayer::cleanupTestCase()
{
  delete mpLayer;
}

The above init function illustrates a couple of interesting things.

  1. I needed to manually set the QGIS application data path so that resources such as srs.db can be found properly.
  2. Secondly, this is a data driven test so we needed to provide a way to generically locate the tenbytenraster.asc file. This was achieved by using the compiler define TEST_DATA_PATH. The define is created in the CMakeLists.txt configuration file under <QGIS Source Root>/tests/CMakeLists.txt and is available to all QGIS unit tests. If you need test data for your test, commit it under <QGIS Source Root>/tests/testdata. You should only commit very small datasets here. If your test needs to modify the test data, it should make a copy of it first.

Qt also provides some other interesting mechanisms for data driven testing, so if you are interested to know more on the topic, consult the Qt documentation.

Next lets look at our functional test. The isValid() test simply checks the raster layer was correctly loaded in the initTestCase. QVERIFY is a Qt macro that you can use to evaluate a test condition. There are a few other use macros Qt provide for use in your tests including:

  • QCOMPARE ( atual, esperado )
  • QEXPECT_FAIL ( índex de dados, comentário, modo )
  • QFAIL ( mensagem )
  • QFETCH ( tipo, nome )
  • QSKIP ( descrição, modo )
  • QTEST ( actual, testElement )
  • QTEST_APPLESS_MAIN ( TestClass )
  • QTEST_MAIN ( TestClass )
  • QTEST_NOOP_MAIN ()
  • QVERIFY2 ( condition, message )
  • QVERIFY ( condition )
  • QWARN ( message )

Some of these macros are useful only when using the Qt framework for data driven testing (see the Qt docs for more detail).

void TestQgsRasterLayer::isValid()
{
  QVERIFY ( mpLayer->isValid() );
}

Normally your functional tests would cover all the range of functionality of your classes public API where feasible. With our functional tests out the way, we can look at our regression test example.

Since the issue in bug #832 is a misreported cell count, writing our test is simply a matter of using QVERIFY to check that the cell count meets the expected value:

void TestQgsRasterLayer::regression832()
{
  QVERIFY ( mpLayer->getRasterXDim() == 10 );
  QVERIFY ( mpLayer->getRasterYDim() == 10 );
  // regression check for ticket #832
  // note getRasterBandStats call is base 1
  QVERIFY ( mpLayer->getRasterBandStats(1).elementCountInt == 100 );
}

With all the unit test functions implemented, there one final thing we need to add to our test class:

QTEST_MAIN(TestQgsRasterLayer)
#include "testqgsrasterlayer.moc"

The purpose of these two lines is to signal to Qt’s moc that his is a QtTest (it will generate a main method that in turn calls each test funtion.The last line is the include for the MOC generated sources. You should replace “testqgsrasterlayer” with the name of your class in lower case.

Adding your unit test to CMakeLists.txt

Adding your unit test to the build system is simply a matter of editing the CMakeLists.txt in the test directory, cloning one of the existing test blocks, and then replacing your test class name into it. For example:

# QgsRasterLayer test
ADD_QGIS_TEST(rasterlayertest testqgsrasterlayer.cpp)

A macro ADD_QGIS_TEST explicada

Passaremos por estas linhas brevemente para explicar o que elas fazem, mas se não estiver interessado, apenas faça o passo explicado na seção e na seção acima.

MACRO (ADD_QGIS_TEST testname testsrc)
SET(qgis_${testname}_SRCS ${testsrc} ${util_SRCS})
SET(qgis_${testname}_MOC_CPPS ${testsrc})
QT4_WRAP_CPP(qgis_${testname}_MOC_SRCS ${qgis_${testname}_MOC_CPPS})
ADD_CUSTOM_TARGET(qgis_${testname}moc ALL DEPENDS ${qgis_${testname}_MOC_SRCS})
ADD_EXECUTABLE(qgis_${testname} ${qgis_${testname}_SRCS})
ADD_DEPENDENCIES(qgis_${testname} qgis_${testname}moc)
TARGET_LINK_LIBRARIES(qgis_${testname} ${QT_LIBRARIES} qgis_core)
SET_TARGET_PROPERTIES(qgis_${testname}
PROPERTIES
# skip the full RPATH for the build tree
SKIP_BUILD_RPATHTRUE
# when building, use the install RPATH already
# (so it doesn't need to relink when installing)
BUILD_WITH_INSTALL_RPATH TRUE
# the RPATH to be used when installing
INSTALL_RPATH ${QGIS_LIB_DIR}
# add the automatically determined parts of the RPATH
# which point to directories outside the build tree to the install RPATH
INSTALL_RPATH_USE_LINK_PATH true)
IF (APPLE)
# For Mac OS X, the executable must be at the root of the bundle's executable folder
INSTALL(TARGETS qgis_${testname} RUNTIME DESTINATION ${CMAKE_INSTALL_PREFIX})
ADD_TEST(qgis_${testname} ${CMAKE_INSTALL_PREFIX}/qgis_${testname})
ELSE (APPLE)
INSTALL(TARGETS qgis_${testname} RUNTIME DESTINATION ${CMAKE_INSTALL_PREFIX}/bin)
ADD_TEST(qgis_${testname} ${CMAKE_INSTALL_PREFIX}/bin/qgis_${testname})
ENDIF (APPLE)
ENDMACRO (ADD_QGIS_TEST)

Repare um pouco mais detalhadamente nas linhas individuais. Primeiro, define-se a lista de fontes para o nosso teste. Como temos apenas um ficheiro fonte (seguindo a metodologia que descrita acima, onde a declaração e a definição da classe estão no mesmo ficheiro), é uma simples declaração:

SET(qgis_${testname}_SRCS ${testsrc} ${util_SRCS})

Como a nossa classe de teste precisa ser executada através do compilador de objeto meta Qt (moc), precisamos de fornecer algumas linhas para que isso aconteça também:

SET(qgis_${testname}_MOC_CPPS ${testsrc})
QT4_WRAP_CPP(qgis_${testname}_MOC_SRCS ${qgis_${testname}_MOC_CPPS})
ADD_CUSTOM_TARGET(qgis_${testname}moc ALL DEPENDS ${qgis_${testname}_MOC_SRCS})

Em seguida, dizemos ao cmake que ele deve criar um executável a partir da classe do teste. Lembre-se, na seção anterior, sobre a última linha da implementação da classe, incluí as saídas moc diretamente na nossa classe de teste, para dar (entre outras coisas) um método main para que a classe possa ser compilada como executável:

ADD_EXECUTABLE(qgis_${testname} ${qgis_${testname}_SRCS})
ADD_DEPENDENCIES(qgis_${testname} qgis_${testname}moc)

Em seguida, precisamos de especificar as dependências da biblioteca. Neste momento, as classes foram implementadas com uma dependência QT_LIBRARIES que recolhe tudo, mas estaremos a trabalhar para substituir essas bibliotecas Qt específicas para corresponder as necessidades de cada classe. É claro que também precisa interligar às bibliotecas qgis relevantes conforme é exigido pelo seu teste de unidade.

TARGET_LINK_LIBRARIES(qgis_${testname} ${QT_LIBRARIES} qgis_core)

Seguidamente iremos indicar ao cmake para instalar os testes para o mesmo sítio como os próprios binários do qgis. Isto é algo que pretendemos remover no futuro para que os testes possam ser executados directamente dentro da source tree.

SET_TARGET_PROPERTIES(qgis_${testname}
PROPERTIES
# skip the full RPATH for the build tree
SKIP_BUILD_RPATHTRUE
# when building, use the install RPATH already
# (so it doesn't need to relink when installing)
BUILD_WITH_INSTALL_RPATH TRUE
# the RPATH to be used when installing
INSTALL_RPATH ${QGIS_LIB_DIR}
# add the automatically determined parts of the RPATH
# which point to directories outside the build tree to the install RPATH
INSTALL_RPATH_USE_LINK_PATH true)
IF (APPLE)
# For Mac OS X, the executable must be at the root of the bundle's executable folder
INSTALL(TARGETS qgis_${testname} RUNTIME DESTINATION ${CMAKE_INSTALL_PREFIX})
ADD_TEST(qgis_${testname} ${CMAKE_INSTALL_PREFIX}/qgis_${testname})
ELSE (APPLE)
INSTALL(TARGETS qgis_${testname} RUNTIME DESTINATION ${CMAKE_INSTALL_PREFIX}/bin)
ADD_TEST(qgis_${testname} ${CMAKE_INSTALL_PREFIX}/bin/qgis_${testname})
ENDIF (APPLE)

Finally the above uses ADD_TEST to register the test with cmake / ctest. Here is where the best magic happens - we register the class with ctest. If you recall in the overview I gave in the beginning of this section, we are using both QtTest and CTest together. To recap, QtTest adds a main method to your test unit and handles calling your test methods within the class. It also provides some macros like QVERIFY that you can use as to test for failure of the tests using conditions. The output from a QtTest unit test is an executable which you can run from the command line. However when you have a suite of tests and you want to run each executable in turn, and better yet integrate running tests into the build process, the CTest is what we use.

Building your unit test

To build the unit test you need only to make sure that ENABLE_TESTS=true in the cmake configuration. There are two ways to do this:

  1. Run ccmake .. ( or cmakesetup .. under windows) and interactively set the ENABLE_TESTS flag to ON.
  2. Add a command line flag to cmake e.g. cmake -DENABLE_TESTS=true ..

Other than that, just build QGIS as per normal and the tests should build too.

Corra os seus testes

A forma mais fácil para executar os testes é como um processo de compilação normal:

make && make install && make test

The make test command will invoke CTest which will run each test that was registered using the ADD_TEST CMake directive described above. Typical output from make test will look like this:

Running tests...
Start processing tests
Test project /Users/tim/dev/cpp/qgis/build
## 13 Testing qgis_applicationtest***Exception: Other
## 23 Testing qgis_filewritertest *** Passed
## 33 Testing qgis_rasterlayertest*** Passed

## 0 tests passed, 3 tests failed out of 3

The following tests FAILED:
## 1- qgis_applicationtest (OTHER_FAULT)
Errors while running CTest
make: *** [test] Error 8

If a test fails, you can use the ctest command to examine more closely why it failed. Use the -R option to specify a regex for which tests you want to run and -V to get verbose output:

$ ctest -R appl -V

Start processing tests
Test project /Users/tim/dev/cpp/qgis/build
Constructing a list of tests
Done constructing a list of tests
Changing directory into /Users/tim/dev/cpp/qgis/build/tests/src/core
## 13 Testing qgis_applicationtest
Test command: /Users/tim/dev/cpp/qgis/build/tests/src/core/qgis_applicationtest
********* Start testing of TestQgsApplication *********
Config: Using QTest library 4.3.0, Qt 4.3.0
PASS : TestQgsApplication::initTestCase()
PrefixPATH: /Users/tim/dev/cpp/qgis/build/tests/src/core/../
PluginPATH: /Users/tim/dev/cpp/qgis/build/tests/src/core/..//lib/qgis
PkgData PATH: /Users/tim/dev/cpp/qgis/build/tests/src/core/..//share/qgis
User DB PATH: /Users/tim/.qgis/qgis.db
PASS : TestQgsApplication::getPaths()
PrefixPATH: /Users/tim/dev/cpp/qgis/build/tests/src/core/../
PluginPATH: /Users/tim/dev/cpp/qgis/build/tests/src/core/..//lib/qgis
PkgData PATH: /Users/tim/dev/cpp/qgis/build/tests/src/core/..//share/qgis
User DB PATH: /Users/tim/.qgis/qgis.db
QDEBUG : TestQgsApplication::checkTheme() Checking if a theme icon exists:
QDEBUG : TestQgsApplication::checkTheme()
/Users/tim/dev/cpp/qgis/build/tests/src/core/..//share/qgis/themes/default//mIconProjectionDisabled.png
FAIL!: TestQgsApplication::checkTheme() '!myPixmap.isNull()' returned FALSE. ()
Loc: [/Users/tim/dev/cpp/qgis/tests/src/core/testqgsapplication.cpp(59)]
PASS : TestQgsApplication::cleanupTestCase()
Totals: 3 passed, 1 failed, 0 skipped
********* Finished testing of TestQgsApplication *********
-- Process completed
***Failed

## 0 tests passed, 1 tests failed out of 1

The following tests FAILED:
## 1- qgis_applicationtest (Failed)
Errors while running CTest

Well that concludes this section on writing unit tests in QGIS. We hope you will get into the habit of writing test to test new functionality and to check for regressions. Some aspects of the test system (in particular the CMakeLists.txt parts) are still being worked on so that the testing framework works in a truly platform way. I will update this document as things progress.