CMake: How to Unit-Test your own CMake Script Macros/Functions?

问题

I've written some convenience wrappers around standard CMake commands and want to unit-test this CMake script code to ensure its functionality.

I've made some progress, but there are two things I hope to get help with:

Is there some "official" way of unit-testing your own CMake script code? Something like a special mode to run CMake in? My goal is "white-box testing" (as much as possible).
How do I handle the global variables and the variable scopes issues? Inject Global variables into the test via loading the a project's cache, configure the test CMake file or pushing it via -D command line option? Simulation/Testing of variable scopes (cached vs. non-cached, macros/functions/includes, parameters passed by references)?

To start with I've looked into the CMake source code (I'm using CMake version 2.8.10) under /Tests and especially under Tests/CMakeTests. There is a huge number of varieties to be found and it looks like a lot of them are specialized on a single test case.

So I looked also into some available CMake script libraries like CMake++ to see their solution, but those - when they have unit tests - are heavily depending on their own library functions.

回答1:

Here is my current solution for unit-testing my own CMake script code.

By the assumption that using CMake Script processing mode is my best catch and that I have to mock the CMake commands that are not usable in script mode I - so far - came up with the following.

The Helper Functions

Utilizing my own global properties, I have written helper functions to store and compare function calls:

function(cmakemocks_clearlists _message_type)
    _get_property(_list_names GLOBAL PROPERTY MockLists)
    if (NOT "${_list_names}" STREQUAL "")
        foreach(_name IN ITEMS ${_list_names})
            _get_property(_list_values GLOBAL PROPERTY ${_name})
            if (NOT "${_list_values}" STREQUAL "")
                foreach(_value IN ITEMS ${_list_values})
                    _message(${_message_type} "cmakemocks_clearlists(${_name}): \"${_value}\"")
                endforeach()
            endif()
            _set_property(GLOBAL PROPERTY ${_name} "")
        endforeach()
    endif()
endfunction()

function(cmakemocks_pushcall _name _str)
    _message("cmakemocks_pushcall(${_name}): \"${_str}\"")
    _set_property(GLOBAL APPEND PROPERTY MockLists "${_name}")
    _set_property(GLOBAL APPEND PROPERTY ${_name} "${_str}")
endfunction()

function(cmakemocks_popcall _name _str)
    _get_property(_list GLOBAL PROPERTY ${_name})
    set(_idx -1)
    list(FIND _list "${_str}" _idx)
    if ((NOT "${_list}" STREQUAL "") AND (NOT ${_idx} EQUAL -1))
        _message("cmakemocks_popcall(${_name}): \"${_str}\"")
        list(REMOVE_AT _list ${_idx})
        _set_property(GLOBAL PROPERTY ${_name} ${_list})
    else()
        _message(FATAL_ERROR "cmakemocks_popcall(${_name}): No \"${_str}\"")
    endif()
endfunction()

function(cmakemocks_expectcall _name _str)
    _message("cmakemocks_expectcall(${_name}): \"${_str}\" -> \"${ARGN}\"")
    _set_property(GLOBAL APPEND PROPERTY MockLists "${_name}")
    string(REPLACE ";" "|" _value_str "${ARGN}")
    _set_property(GLOBAL APPEND PROPERTY ${_name} "${_str} <<<${_value_str}>>>")
endfunction()

function(cmakemocks_getexpect _name _str _ret)
    if(NOT DEFINED ${_ret})
        _message(SEND_ERROR "cmakemocks_getexpect: ${_ret} given as _ret parameter in not a defined variable. Please specify a proper variable name as parameter.")
    endif()

    _message("cmakemocks_getexpect(${_name}): \"${_str}\"")
    _get_property(_list_values GLOBAL PROPERTY ${_name})

    set(_value_str "")

    foreach(_value IN ITEMS ${_list_values})
        set(_idx -1)
        string(FIND "${_value}" "${_str}" _idx)
        if ((NOT "${_value}" STREQUAL "") AND (NOT ${_idx} EQUAL -1))
            list(REMOVE_ITEM _list_values "${_value}")
            _set_property(GLOBAL PROPERTY ${_name} ${_list_values})

            string(FIND "${_value}" "<<<" _start)
            string(FIND "${_value}" ">>>" _end)
            math(EXPR _start "${_start} + 3")
            math(EXPR _len "${_end} - ${_start}")
            string(SUBSTRING "${_value}" ${_start} ${_len} _value_str)
            string(REPLACE "|" ";" _value_list "${_value_str}")
            set(${_ret} "${_value_list}" PARENT_SCOPE)
            break()
        endif()
    endforeach()
endfunction()

The Mockups

By adding mockups like:

macro(add_library)
    string(REPLACE ";" " " _str "${ARGN}")
    cmakemocks_pushcall(MockLibraries "${_str}")
endmacro()

macro(get_target_property _var)
    string(REPLACE ";" " " _str "${ARGN}")
    set(${_var} "[NULL]")
    cmakemocks_getexpect(MockGetTargetProperties "${_str}" ${_var})
endmacro()

The Tests

I can write a test like this:

MyUnitTests.cmake

cmakemocks_expectcall(MockGetTargetProperties "MyLib TYPE" "STATIC_LIBRARY")
my_add_library(MyLib "src/Test1.cc")
cmakemocks_popcall(MockLibraries "MyLib src/Test1.cc")
...
cmakemocks_clearlists(STATUS)

And include it into my CMake projects with:

CMakeLists.txt

add_test(
    NAME TestMyCMake 
    COMMAND ${CMAKE_COMMAND} -P "MyUnitTests.cmake"
)

回答2:

How do I handle the global variables and the variable scopes issues? Inject Global variables into the test via loading the a project's cache, configure the test CMake file or pushing it via -D command line option?

Generally speaking all currently existing methods (via cache, via environment variables and via -D command line) are a bad choice in one or another case as involving unpredictable behaviour.

This is a least list of issues i can recall:

Which one variable could intersect/overlap another one and when?
Variables load or set can not be applied out of cmake detection stage (for example, in cmake script mode).
The same unique variable can not hold different values for different OS/compilers/configurations/architectures and so on.
Variables can not be attached to a package (not scope) term represented by system functions like Find* or add_subdirectory.

I've used variables inside cmake lists a long time and decided to write my own solution to cut off them all at once out of the cmake lists.

The idea is to write a standalone parser through a cmake script to load variables from a file or set of files and define a set of rules to enable variables set in predefined or strict order with check on collisions and overlapping.

Here a list of several features:

bool A=ON is equal to bool A=TRUE is equal to bool A=1
path B="c:\abc" is equal on the Windows to path B="C:\ABC" (explicit path variable instead of a string which is by default)
B_ROOT="c:\abc" is equal on the Windows to B_ROOT="C:\ABC" (variable's type detection by the ending of a variable name)
LIB1:WIN="c:\lib1" sets only in the Windows, when LIB1:UNIX="/lib/lib1" sets only in the Unix (a variable specialization).
LIB1:WIN=c:\lib1, LIB1:WIN:MSVC:RELEASE=$/{LIB1}\msvc_release - variables reusing via expansion and specialization

I can not said everything here, but you can take as an example the tacklelib library (https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/) to research the implementation on your own.

The example of described configuration files is stored here: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/_config/

The implementation: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/cmake/tacklelib/SetVarsFromFiles.cmake

As a mandatory the cmake list must be initialized through the configure_environment(...) macro: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/CMakeLists.txt

Read the readme file for the details around the tacklelib project: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/README_EN.txt

The entire project currently is an experimental.

PS: Write a parser script on the cmake is a tough task, read at least these issues for the start:

;-escape list implicit unescaping: https://gitlab.kitware.com/cmake/cmake/issues/18946
Not paired]or[characters breaks "file(STRINGS": https://gitlab.kitware.com/cmake/cmake/issues/19156

Is there some "official" way of unit-testing your own CMake script code? Something like a special mode to run CMake in? My goal is "white-box testing" (as much as possible).

I did my own "white-box" or a way of testing my own scripts. I have write a set of modules (which itself dependent to the library) to run test in a separate cmake process: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/cmake/tacklelib/testlib/

My tests built on it: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/cmake_tests/

The idea is to put into the tests directory a hierarchy of directories and files with tests and the runner code would just search for the tests in predefined order to execute each test in a separate cmake process:

function(tkl_testlib_enter_dir test_dir)
  # Algorithm:
  #   1. Iterate not recursively over all `*.include.cmake` files and
  #      call to `tkl_testlib_include` function on each file, then
  #      if at least one is iterated then
  #      exit the algorithm.
  #   2. Iterate non recursively over all subdirectories and
  #      call to the algorithm recursively on each subdirectory, then
  #      continue.
  #   3. Iterate not recursively over all `*.test.cmake` files and
  #      call to `tkl_testlib_test` function on each file, then
  #      exit the algorithm.
  #

, where set of functions can be used both from a runner cmake script or *.include.cmake file:

Where TestLib.cmake is designed to run cycle over creation external cmake processes with a test module - *.test.cmake and these functions should be called from a runner script or from an include module (groups other include modules - *.include.cmake or test modules - *.test.cmake):

tkl_testlib_enter_dir test_dir
tkl_testlib_include test_dir test_file_name
tkl_testlib_test test_dir test_file_name

Where TestModule.cmake automatically includes in all *.test.cmake modules in which you have to put your testing code.

After that you just use tkl_test_assert_true inside a *.test.cmake module to mark a test as succeeded or failed.

Additionally, you can use filter parameters in the runner scripts in the _scripts subdirectory to filter tests out:

--path_match_filter <[+]regex_include_match_expression> | <-regex_exclude_match_expression>[;...]
--test_case_match_filter <[+]regex_include_match_expression> | <-regex_exclude_match_expression>[;...]

Pros:

The TestModule.cmake does traverse the entire directory with tests by predefined rules, you just need to make sure the correct hierarchy and naming to order the testing.
Usage of a per directory basis separate inclusion file *.include.cmake to exclusive inclusion or to reorder the tests in the directory and its descedants.
Existence of a *.test.cmake file is the only requirement to put the test to running by default. To exclusively include or exclude the test you can start use command line flags --path_match_filter ... and --test_case_match_filter ....

Cons:

Mostly all test functions has implemented through the function keyword, which a bit reduces the functionality of several functions. For example, the tkl_test_assert_true can only mark the test is succeeded or failed. To explicitly interrupt the test you have make the branching through the call to tkl_return_if_failed macro.
All files in the directory with the tests must have the suffix, .test.cmake - for a test, and .include.cmake - for inclusion commands. All builtin search logic depends on it.
You have write you own runner to call the script RunTestLib.cmake. The example of a run all on the unix shell could be found here: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/cmake_tests/_scripts/test_all.sh

The entire project currently is an experimental.

来源：https://stackoverflow.com/questions/29818725/cmake-how-to-unit-test-your-own-cmake-script-macros-functions

标签

unit-testing

cmake

mocking

ctest