test/README.md
author Jorg Knobloch
Thu, 30 Jul 2020 22:50:42 +0200
branchENGINE-780-take2
changeset 4910 347f8b793204
parent 4654 c173428cd2c3
child 4948 d666f66e20b8
permissions -rw-r--r--
ENGINE-780, take 2: After normalizing to UTC, forget the offset.
krista@4170
     1
# Building and Running the Tests for the p≡p Engine
krista@4170
     2
krista@4176
     3
Work in progress.
krista@4024
     4
krista@4024
     5
## Caveat, before you begin
krista@4024
     6
krista@4170
     7
Right now, the engine tests only function on \*nix-like systems (including
krista@4170
     8
MacOS).
krista@4170
     9
krista@4170
    10
*(Conversion to Windows will require, at the very least, looking at some of the
krista@4170
    11
file-handling code. If you want to fix this, start by looking in Engine.cc
krista@4170
    12
in the test/src directory!)*
krista@4024
    13
krista@4024
    14
## Requirements
krista@4024
    15
krista@4024
    16
In addition to the engine requirements, you will need:
krista@4024
    17
krista@4172
    18
  * `cmake`
krista@4172
    19
  * `python3`
krista@4172
    20
  * `git` (for getting the `gtest-parallel` repository, unless you grab the
krista@4172
    21
  tarball from somewhere)
krista@4170
    22
krista@4170
    23
## Building the prerequisites
krista@4024
    24
krista@4170
    25
The Engine test suite now requires (at least) two additional pieces to run:
krista@4170
    26
  * `googletest`
krista@4170
    27
  * `gtest-parallel`
krista@4170
    28
krista@4170
    29
How this proceeds depends on your platform and whether or not you use a packaged
krista@4170
    30
distribution.
krista@4170
    31
krista@4170
    32
These instructions do this with `cmake`. If you can manage it with `bazel`
krista@4170
    33
instead, more power to you ;)
krista@4024
    34
krista@4170
    35
### Installing `googletest`
krista@4024
    36
krista@4170
    37
#### Packaged distributions
krista@4170
    38
krista@4170
    39
This is the currently preferred way to do this, because everyone was doing it
krista@4170
    40
anyway and who am I to judge?
krista@4024
    41
krista@4170
    42
##### Debian and Ubuntu (and derivatives)
krista@4024
    43
krista@4222
    44
Thanks to Erik Smistad for this starting point (condensed from [Getting Started
krista@4222
    45
with Google Test On
krista@4222
    46
Ubuntu](https://www.eriksmistad.no/getting-started-with-google-test-on-ubuntu/)):
krista@4170
    47
krista@4222
    48
  1. Install the packages `cmake` and `libgtest-dev` from the repository. This
krista@4222
    49
  will install the gtest source files to `/usr/src/gtest`. You'll still need to
krista@4222
    50
  compile the code and link the library files to be able to use them.
krista@4024
    51
krista@4222
    52
  2. Compile the source files:
krista@4222
    53
  ```
krista@4222
    54
  cd /usr/src/gtest
krista@4222
    55
  sudo cmake CMakeLists.txt
krista@4222
    56
  sudo make
krista@4222
    57
  ```
krista@4024
    58
krista@4222
    59
  3. Copy/symlink the libraries to the library location of your choice (here,
krista@4222
    60
  it's `/usr/lib`, hence the `sudo`, but as long as it's in your library path,
krista@4222
    61
  it shouldn't matter where you stick it):
krista@4222
    62
  ```
krista@4222
    63
  sudo cp *.a /usr/lib
krista@4222
    64
  ```
krista@4170
    65
krista@4170
    66
##### MacOS
krista@4024
    67
krista@4175
    68
  1. Install package `gtest` from Macports
krista@4175
    69
  
krista@4175
    70
  2. Macports will build the libraries for you.
krista@4175
    71
  
krista@4175
    72
  3. In the next major section ("Building the Test Suite"), under 
krista@4175
    73
  "Makefile and local.conf", set `GTEST_SRC_DIR` to 
krista@4175
    74
  `/opt/local/src/googletest` in `local.conf` (see instructions below)
krista@4175
    75
  
krista@4175
    76
  4. Make sure `/opt/local/lib` is in your library path when compiling and 
krista@4175
    77
  linking the tests.
krista@4024
    78
krista@4170
    79
#### Downloading and compiling the source yourself
krista@4170
    80
krista@4174
    81
  1. Get the source, Fred. (Luke is tired of the source, I hear.)
krista@4174
    82
  ```
krista@4174
    83
  git clone https://github.com/google/googletest.git
krista@4174
    84
  ```
krista@4174
    85
  
krista@4174
    86
  2. Switch into the source directory and find the directory 
krista@4174
    87
  containing the `src` and `include` directories. Mark this directory
krista@4174
    88
  for later. (For me, this is `./googletest/googletest`)
krista@4174
    89
  
krista@4174
    90
  3. Edit `CMakeLists.txt` here to contain the following line at the top:
krista@4174
    91
  ```
krista@4174
    92
  set (CMAKE_CXX_STANDARD 11)
krista@4174
    93
  ```
krista@4174
    94
  (If you don't, it won't compile, and I will shake my fist at you.)
krista@4174
    95
  
krista@4174
    96
  4. Execute, in this directory:
krista@4174
    97
  ```
krista@4174
    98
  cmake CMakeLists.txt
krista@4174
    99
  make
krista@4174
   100
  ```
krista@4174
   101
  
krista@4174
   102
  5. In the lib directory of your current directory are located the
krista@4175
   103
  library files you'll use (`lib/*.a`). Copy or symlink them to the library 
krista@4175
   104
  location  of your choice (make sure this is a directory that can be seen 
krista@4175
   105
  during the test build process - i.e. one that's in one of the library paths 
krista@4174
   106
  used in building. Mine are located in `$HOME/lib`.
krista@4024
   107
krista@4175
   108
  6. See `Makefile` and `local.conf` under "Building the test suite" below -
krista@4176
   109
  In this scenario, I set `GTEST_SRC_DIR` as  `<clone_path>/googletest/googletest`
krista@4176
   110
  (i.e. the absolute path of where the `src` and `include` directories were 
krista@4175
   111
  above - for me, `/Users/krista/googletest/googletest`).
krista@4174
   112
  
krista@4170
   113
### Installing `gtest-parallel`
krista@4170
   114
krista@4170
   115
Pick a source directory and put your `gtest-parallel` source there
krista@4170
   116
(e.g. via `git clone https://github.com/google/gtest-parallel.git`).
krista@4170
   117
krista@4170
   118
We'll deal more with this when preparing to compile the test suite.
krista@4024
   119
krista@4024
   120
## Building the test suite
krista@4024
   121
krista@4170
   122
### `Makefile` and `local.conf`
krista@4170
   123
krista@4170
   124
So `local.conf` in the top-level engine directory is where we stick all of the
krista@4170
   125
Makefile overrides. The test Makefile contains some defaults for relevant
krista@4170
   126
variables here, but if you need to override them, please either create or modify
krista@4170
   127
`local.conf` in the top-level engine directory as needed. The relevant variables
krista@4170
   128
are:
krista@4170
   129
krista@4174
   130
  * `GTEST_SRC_DIR`: This is the directory where you compiled googletest above
krista@4174
   131
  (defaults to `/usr/src/gtest`)
krista@4175
   132
  
krista@4174
   133
  * `GTEST_INC_DIR`: This is where the include files for googletest are located
krista@4170
   134
  (defaults to `$(GTEST_SRC_DIR)/include`)
krista@4175
   135
  
krista@4170
   136
  * `GTEST_PL`: This is the full path to the *python file* for `gtest_parallel`
krista@4170
   137
  (default presumes you cloned it under `src` in your home directory, i.e. it is
krista@4170
   138
  `$(HOME)/src/gtest-parallel/gtest_parallel.py`)
krista@4170
   139
krista@4170
   140
### Building
krista@4170
   141
krista@4024
   142
Presuming the above works, then from the top test directory, simply run make.
krista@4024
   143
krista@4170
   144
## Running the test suite
krista@4024
   145
krista@4170
   146
### To simply run the test suite and see what tests fail...
krista@4030
   147
krista@4030
   148
Do one of:
krista@4024
   149
krista@4170
   150
  1. `make test` OR
krista@4175
   151
  
krista@4170
   152
  2. `python3 <path to gtest-parallel.py> ./EngineTests`
krista@4170
   153
krista@4170
   154
### To run individual test suites, especially for debugging purposes
krista@4170
   155
krista@4170
   156
Note that for some test suites, this will, if something goes dreadfully wrong,
krista@4170
   157
mean that one test's failure may pollute another test. This generally means you
krista@4170
   158
have found a dastardly bug in the engine, but it can also be a test issue.
krista@4170
   159
krista@4170
   160
*Caveat lector*.
krista@4024
   161
krista@4173
   162
  1. To run sequentially, *in the same process*:
krista@4173
   163
  ```
krista@4173
   164
  ./EngineTests --gtest_filter=TestSuiteName*
krista@4173
   165
  ```
krista@4173
   166
  For example, for `DeleteKeyTest`:
krista@4173
   167
  ```
krista@4173
   168
  ./EngineTests DeleteKeyTest*
krista@4173
   169
  ```
krista@4173
   170
krista@4173
   171
  2. To debug the same with lldb:
krista@4173
   172
  ```
krista@4173
   173
  lldb ./EngineTests -- --gtest_filter=TestSuiteName*
krista@4173
   174
  ```
krista@4173
   175
  3. To debug with gdb:
krista@4173
   176
  ```
krista@4173
   177
  gdb --args ./EngineTests --gtest_filter=TestSuiteName*
krista@4173
   178
  ```
krista@4024
   179
krista@4170
   180
### To run and/or debug individual test cases   
krista@4173
   181
  1. To run:
krista@4173
   182
  ```
krista@4173
   183
  ./EngineTests --gtest_filter=TestSuiteName.test_function_name
krista@4173
   184
  ```
krista@4173
   185
  For example, for `check_delete_single_pubkey` in `DeleteKeyTest`:
krista@4173
   186
  ```
krista@4173
   187
  ./EngineTests DeleteKeyTest.check_delete_single_pubkey
krista@4173
   188
  ```
krista@4173
   189
krista@4170
   190
  2. To debug the same with lldb:
krista@4173
   191
  ```
krista@4173
   192
  lldb ./EngineTests -- --gtest_filter=TestSuiteName.test_function_name
krista@4173
   193
  ```
krista@4173
   194
krista@4170
   195
  3. To debug with gdb:
krista@4173
   196
  ```
krista@4173
   197
  gdb --args ./EngineTests --gtest_filter=TestSuiteName.test_function_name
krista@4173
   198
  ```
krista@4024
   199
krista@4654
   200
N.B. The gtest_filter can be globbed and will run all matching tests; if you
krista@4654
   201
want to run every test in a test suite, be sure to use TestSuiteName*.
krista@4654
   202
krista@4654
   203
(Different shells will require different quoting styles for this - YMMV.)
krista@4654
   204
neal@4216
   205
When debugging a failing test, use '--gtest_break_on_failure' to have
neal@4216
   206
gtest automatically break into the debugger where the assertion fails.
neal@4216
   207
krista@4477
   208
### Output
krista@4477
   209
krista@4477
   210
Compile tests with -DDEBUG_OUTPUT to (possibly) see output to cout. May only work
krista@4477
   211
in the tests run directly from ./EngineTests (with or without filter)
krista@4477
   212
krista@4024
   213
# Creating new tests
krista@4024
   214
krista@4024
   215
Script next on the agenda...
krista@4174
   216
krista@4174
   217
# Known Problems
krista@4174
   218
krista@4476
   219
The normal run of the tests in parallel eats output. Try running the individual test case as above if you need to see a test case's output.
krista@4476
   220