test/README.md
author Jorg Knobloch
Thu, 30 Jul 2020 22:50:42 +0200
branchENGINE-780-take2
changeset 4910 347f8b793204
parent 4654 c173428cd2c3
child 4948 d666f66e20b8
permissions -rw-r--r--
ENGINE-780, take 2: After normalizing to UTC, forget the offset.
     1 # Building and Running the Tests for the p≡p Engine
     2 
     3 Work in progress.
     4 
     5 ## Caveat, before you begin
     6 
     7 Right now, the engine tests only function on \*nix-like systems (including
     8 MacOS).
     9 
    10 *(Conversion to Windows will require, at the very least, looking at some of the
    11 file-handling code. If you want to fix this, start by looking in Engine.cc
    12 in the test/src directory!)*
    13 
    14 ## Requirements
    15 
    16 In addition to the engine requirements, you will need:
    17 
    18   * `cmake`
    19   * `python3`
    20   * `git` (for getting the `gtest-parallel` repository, unless you grab the
    21   tarball from somewhere)
    22 
    23 ## Building the prerequisites
    24 
    25 The Engine test suite now requires (at least) two additional pieces to run:
    26   * `googletest`
    27   * `gtest-parallel`
    28 
    29 How this proceeds depends on your platform and whether or not you use a packaged
    30 distribution.
    31 
    32 These instructions do this with `cmake`. If you can manage it with `bazel`
    33 instead, more power to you ;)
    34 
    35 ### Installing `googletest`
    36 
    37 #### Packaged distributions
    38 
    39 This is the currently preferred way to do this, because everyone was doing it
    40 anyway and who am I to judge?
    41 
    42 ##### Debian and Ubuntu (and derivatives)
    43 
    44 Thanks to Erik Smistad for this starting point (condensed from [Getting Started
    45 with Google Test On
    46 Ubuntu](https://www.eriksmistad.no/getting-started-with-google-test-on-ubuntu/)):
    47 
    48   1. Install the packages `cmake` and `libgtest-dev` from the repository. This
    49   will install the gtest source files to `/usr/src/gtest`. You'll still need to
    50   compile the code and link the library files to be able to use them.
    51 
    52   2. Compile the source files:
    53   ```
    54   cd /usr/src/gtest
    55   sudo cmake CMakeLists.txt
    56   sudo make
    57   ```
    58 
    59   3. Copy/symlink the libraries to the library location of your choice (here,
    60   it's `/usr/lib`, hence the `sudo`, but as long as it's in your library path,
    61   it shouldn't matter where you stick it):
    62   ```
    63   sudo cp *.a /usr/lib
    64   ```
    65 
    66 ##### MacOS
    67 
    68   1. Install package `gtest` from Macports
    69   
    70   2. Macports will build the libraries for you.
    71   
    72   3. In the next major section ("Building the Test Suite"), under 
    73   "Makefile and local.conf", set `GTEST_SRC_DIR` to 
    74   `/opt/local/src/googletest` in `local.conf` (see instructions below)
    75   
    76   4. Make sure `/opt/local/lib` is in your library path when compiling and 
    77   linking the tests.
    78 
    79 #### Downloading and compiling the source yourself
    80 
    81   1. Get the source, Fred. (Luke is tired of the source, I hear.)
    82   ```
    83   git clone https://github.com/google/googletest.git
    84   ```
    85   
    86   2. Switch into the source directory and find the directory 
    87   containing the `src` and `include` directories. Mark this directory
    88   for later. (For me, this is `./googletest/googletest`)
    89   
    90   3. Edit `CMakeLists.txt` here to contain the following line at the top:
    91   ```
    92   set (CMAKE_CXX_STANDARD 11)
    93   ```
    94   (If you don't, it won't compile, and I will shake my fist at you.)
    95   
    96   4. Execute, in this directory:
    97   ```
    98   cmake CMakeLists.txt
    99   make
   100   ```
   101   
   102   5. In the lib directory of your current directory are located the
   103   library files you'll use (`lib/*.a`). Copy or symlink them to the library 
   104   location  of your choice (make sure this is a directory that can be seen 
   105   during the test build process - i.e. one that's in one of the library paths 
   106   used in building. Mine are located in `$HOME/lib`.
   107 
   108   6. See `Makefile` and `local.conf` under "Building the test suite" below -
   109   In this scenario, I set `GTEST_SRC_DIR` as  `<clone_path>/googletest/googletest`
   110   (i.e. the absolute path of where the `src` and `include` directories were 
   111   above - for me, `/Users/krista/googletest/googletest`).
   112   
   113 ### Installing `gtest-parallel`
   114 
   115 Pick a source directory and put your `gtest-parallel` source there
   116 (e.g. via `git clone https://github.com/google/gtest-parallel.git`).
   117 
   118 We'll deal more with this when preparing to compile the test suite.
   119 
   120 ## Building the test suite
   121 
   122 ### `Makefile` and `local.conf`
   123 
   124 So `local.conf` in the top-level engine directory is where we stick all of the
   125 Makefile overrides. The test Makefile contains some defaults for relevant
   126 variables here, but if you need to override them, please either create or modify
   127 `local.conf` in the top-level engine directory as needed. The relevant variables
   128 are:
   129 
   130   * `GTEST_SRC_DIR`: This is the directory where you compiled googletest above
   131   (defaults to `/usr/src/gtest`)
   132   
   133   * `GTEST_INC_DIR`: This is where the include files for googletest are located
   134   (defaults to `$(GTEST_SRC_DIR)/include`)
   135   
   136   * `GTEST_PL`: This is the full path to the *python file* for `gtest_parallel`
   137   (default presumes you cloned it under `src` in your home directory, i.e. it is
   138   `$(HOME)/src/gtest-parallel/gtest_parallel.py`)
   139 
   140 ### Building
   141 
   142 Presuming the above works, then from the top test directory, simply run make.
   143 
   144 ## Running the test suite
   145 
   146 ### To simply run the test suite and see what tests fail...
   147 
   148 Do one of:
   149 
   150   1. `make test` OR
   151   
   152   2. `python3 <path to gtest-parallel.py> ./EngineTests`
   153 
   154 ### To run individual test suites, especially for debugging purposes
   155 
   156 Note that for some test suites, this will, if something goes dreadfully wrong,
   157 mean that one test's failure may pollute another test. This generally means you
   158 have found a dastardly bug in the engine, but it can also be a test issue.
   159 
   160 *Caveat lector*.
   161 
   162   1. To run sequentially, *in the same process*:
   163   ```
   164   ./EngineTests --gtest_filter=TestSuiteName*
   165   ```
   166   For example, for `DeleteKeyTest`:
   167   ```
   168   ./EngineTests DeleteKeyTest*
   169   ```
   170 
   171   2. To debug the same with lldb:
   172   ```
   173   lldb ./EngineTests -- --gtest_filter=TestSuiteName*
   174   ```
   175   3. To debug with gdb:
   176   ```
   177   gdb --args ./EngineTests --gtest_filter=TestSuiteName*
   178   ```
   179 
   180 ### To run and/or debug individual test cases   
   181   1. To run:
   182   ```
   183   ./EngineTests --gtest_filter=TestSuiteName.test_function_name
   184   ```
   185   For example, for `check_delete_single_pubkey` in `DeleteKeyTest`:
   186   ```
   187   ./EngineTests DeleteKeyTest.check_delete_single_pubkey
   188   ```
   189 
   190   2. To debug the same with lldb:
   191   ```
   192   lldb ./EngineTests -- --gtest_filter=TestSuiteName.test_function_name
   193   ```
   194 
   195   3. To debug with gdb:
   196   ```
   197   gdb --args ./EngineTests --gtest_filter=TestSuiteName.test_function_name
   198   ```
   199 
   200 N.B. The gtest_filter can be globbed and will run all matching tests; if you
   201 want to run every test in a test suite, be sure to use TestSuiteName*.
   202 
   203 (Different shells will require different quoting styles for this - YMMV.)
   204 
   205 When debugging a failing test, use '--gtest_break_on_failure' to have
   206 gtest automatically break into the debugger where the assertion fails.
   207 
   208 ### Output
   209 
   210 Compile tests with -DDEBUG_OUTPUT to (possibly) see output to cout. May only work
   211 in the tests run directly from ./EngineTests (with or without filter)
   212 
   213 # Creating new tests
   214 
   215 Script next on the agenda...
   216 
   217 # Known Problems
   218 
   219 The normal run of the tests in parallel eats output. Try running the individual test case as above if you need to see a test case's output.
   220