test/README.md
author Krista 'DarthMama' Bennett <krista@pep.foundation>
Mon, 25 Nov 2019 11:20:03 +0100
branchsync
changeset 4222 3247a7e15516
parent 4216 de70d8f412cf
child 4476 929788d2528c
permissions -rw-r--r--
Reverted out debian instructions, as 4213 fixed the issue on clean debian installs and neal's experience was from Nov. 1 - kept in the gtest break instructions, though.
krista@4170
     1
# Building and Running the Tests for the p≡p Engine
krista@4170
     2
krista@4176
     3
Work in progress.
krista@4024
     4
krista@4024
     5
## Caveat, before you begin
krista@4024
     6
krista@4170
     7
Right now, the engine tests only function on \*nix-like systems (including
krista@4170
     8
MacOS).
krista@4170
     9
krista@4170
    10
*(Conversion to Windows will require, at the very least, looking at some of the
krista@4170
    11
file-handling code. If you want to fix this, start by looking in Engine.cc
krista@4170
    12
in the test/src directory!)*
krista@4024
    13
krista@4024
    14
## Requirements
krista@4024
    15
krista@4024
    16
In addition to the engine requirements, you will need:
krista@4024
    17
krista@4172
    18
  * `cmake`
krista@4172
    19
  * `python3`
krista@4172
    20
  * `git` (for getting the `gtest-parallel` repository, unless you grab the
krista@4172
    21
  tarball from somewhere)
krista@4024
    22
krista@4170
    23
## Building the prerequisites
krista@4024
    24
krista@4170
    25
The Engine test suite now requires (at least) two additional pieces to run:
krista@4170
    26
  * `googletest`
krista@4170
    27
  * `gtest-parallel`
krista@4024
    28
krista@4170
    29
How this proceeds depends on your platform and whether or not you use a packaged
krista@4170
    30
distribution.
krista@4024
    31
krista@4170
    32
These instructions do this with `cmake`. If you can manage it with `bazel`
krista@4170
    33
instead, more power to you ;)
krista@4024
    34
krista@4170
    35
### Installing `googletest`
krista@4024
    36
krista@4170
    37
#### Packaged distributions
krista@4024
    38
krista@4170
    39
This is the currently preferred way to do this, because everyone was doing it
krista@4170
    40
anyway and who am I to judge?
krista@4024
    41
krista@4170
    42
##### Debian and Ubuntu (and derivatives)
krista@4024
    43
krista@4222
    44
Thanks to Erik Smistad for this starting point (condensed from [Getting Started
krista@4222
    45
with Google Test On
krista@4222
    46
Ubuntu](https://www.eriksmistad.no/getting-started-with-google-test-on-ubuntu/)):
krista@4024
    47
krista@4222
    48
  1. Install the packages `cmake` and `libgtest-dev` from the repository. This
krista@4222
    49
  will install the gtest source files to `/usr/src/gtest`. You'll still need to
krista@4222
    50
  compile the code and link the library files to be able to use them.
krista@4024
    51
krista@4222
    52
  2. Compile the source files:
krista@4222
    53
  ```
krista@4222
    54
  cd /usr/src/gtest
krista@4222
    55
  sudo cmake CMakeLists.txt
krista@4222
    56
  sudo make
krista@4222
    57
  ```
krista@4170
    58
krista@4222
    59
  3. Copy/symlink the libraries to the library location of your choice (here,
krista@4222
    60
  it's `/usr/lib`, hence the `sudo`, but as long as it's in your library path,
krista@4222
    61
  it shouldn't matter where you stick it):
krista@4222
    62
  ```
krista@4222
    63
  sudo cp *.a /usr/lib
krista@4222
    64
  ```
krista@4170
    65
krista@4170
    66
##### MacOS
krista@4170
    67
krista@4175
    68
  1. Install package `gtest` from Macports
krista@4175
    69
  
krista@4175
    70
  2. Macports will build the libraries for you.
krista@4175
    71
  
krista@4175
    72
  3. In the next major section ("Building the Test Suite"), under 
krista@4175
    73
  "Makefile and local.conf", set `GTEST_SRC_DIR` to 
krista@4175
    74
  `/opt/local/src/googletest` in `local.conf` (see instructions below)
krista@4175
    75
  
krista@4175
    76
  4. Make sure `/opt/local/lib` is in your library path when compiling and 
krista@4175
    77
  linking the tests.
krista@4170
    78
krista@4170
    79
#### Downloading and compiling the source yourself
krista@4170
    80
krista@4174
    81
  1. Get the source, Fred. (Luke is tired of the source, I hear.)
krista@4174
    82
  ```
krista@4174
    83
  git clone https://github.com/google/googletest.git
krista@4174
    84
  ```
krista@4174
    85
  
krista@4174
    86
  2. Switch into the source directory and find the directory 
krista@4174
    87
  containing the `src` and `include` directories. Mark this directory
krista@4174
    88
  for later. (For me, this is `./googletest/googletest`)
krista@4174
    89
  
krista@4174
    90
  3. Edit `CMakeLists.txt` here to contain the following line at the top:
krista@4174
    91
  ```
krista@4174
    92
  set (CMAKE_CXX_STANDARD 11)
krista@4174
    93
  ```
krista@4174
    94
  (If you don't, it won't compile, and I will shake my fist at you.)
krista@4174
    95
  
krista@4174
    96
  4. Execute, in this directory:
krista@4174
    97
  ```
krista@4174
    98
  cmake CMakeLists.txt
krista@4174
    99
  make
krista@4174
   100
  ```
krista@4174
   101
  
krista@4174
   102
  5. In the lib directory of your current directory are located the
krista@4175
   103
  library files you'll use (`lib/*.a`). Copy or symlink them to the library 
krista@4175
   104
  location  of your choice (make sure this is a directory that can be seen 
krista@4175
   105
  during the test build process - i.e. one that's in one of the library paths 
krista@4174
   106
  used in building. Mine are located in `$HOME/lib`.
krista@4170
   107
krista@4175
   108
  6. See `Makefile` and `local.conf` under "Building the test suite" below -
krista@4176
   109
  In this scenario, I set `GTEST_SRC_DIR` as  `<clone_path>/googletest/googletest`
krista@4176
   110
  (i.e. the absolute path of where the `src` and `include` directories were 
krista@4175
   111
  above - for me, `/Users/krista/googletest/googletest`).
krista@4174
   112
  
krista@4170
   113
### Installing `gtest-parallel`
krista@4170
   114
krista@4170
   115
Pick a source directory and put your `gtest-parallel` source there
krista@4170
   116
(e.g. via `git clone https://github.com/google/gtest-parallel.git`).
krista@4170
   117
krista@4170
   118
We'll deal more with this when preparing to compile the test suite.
krista@4024
   119
krista@4024
   120
## Building the test suite
krista@4024
   121
krista@4170
   122
### `Makefile` and `local.conf`
krista@4170
   123
krista@4170
   124
So `local.conf` in the top-level engine directory is where we stick all of the
krista@4170
   125
Makefile overrides. The test Makefile contains some defaults for relevant
krista@4170
   126
variables here, but if you need to override them, please either create or modify
krista@4170
   127
`local.conf` in the top-level engine directory as needed. The relevant variables
krista@4170
   128
are:
krista@4170
   129
krista@4174
   130
  * `GTEST_SRC_DIR`: This is the directory where you compiled googletest above
krista@4174
   131
  (defaults to `/usr/src/gtest`)
krista@4175
   132
  
krista@4174
   133
  * `GTEST_INC_DIR`: This is where the include files for googletest are located
krista@4170
   134
  (defaults to `$(GTEST_SRC_DIR)/include`)
krista@4175
   135
  
krista@4170
   136
  * `GTEST_PL`: This is the full path to the *python file* for `gtest_parallel`
krista@4170
   137
  (default presumes you cloned it under `src` in your home directory, i.e. it is
krista@4170
   138
  `$(HOME)/src/gtest-parallel/gtest_parallel.py`)
krista@4170
   139
krista@4170
   140
### Building
krista@4170
   141
krista@4024
   142
Presuming the above works, then from the top test directory, simply run make.
krista@4024
   143
krista@4170
   144
## Running the test suite
krista@4024
   145
krista@4170
   146
### To simply run the test suite and see what tests fail...
krista@4030
   147
krista@4030
   148
Do one of:
krista@4024
   149
krista@4170
   150
  1. `make test` OR
krista@4175
   151
  
krista@4170
   152
  2. `python3 <path to gtest-parallel.py> ./EngineTests`
krista@4024
   153
krista@4170
   154
### To run individual test suites, especially for debugging purposes
krista@4024
   155
krista@4170
   156
Note that for some test suites, this will, if something goes dreadfully wrong,
krista@4170
   157
mean that one test's failure may pollute another test. This generally means you
krista@4170
   158
have found a dastardly bug in the engine, but it can also be a test issue.
krista@4170
   159
krista@4170
   160
*Caveat lector*.
krista@4170
   161
krista@4173
   162
  1. To run sequentially, *in the same process*:
krista@4173
   163
  ```
krista@4173
   164
  ./EngineTests --gtest_filter=TestSuiteName*
krista@4173
   165
  ```
krista@4173
   166
  For example, for `DeleteKeyTest`:
krista@4173
   167
  ```
krista@4173
   168
  ./EngineTests DeleteKeyTest*
krista@4173
   169
  ```
krista@4173
   170
krista@4173
   171
  2. To debug the same with lldb:
krista@4173
   172
  ```
krista@4173
   173
  lldb ./EngineTests -- --gtest_filter=TestSuiteName*
krista@4173
   174
  ```
krista@4173
   175
  3. To debug with gdb:
krista@4173
   176
  ```
krista@4173
   177
  gdb --args ./EngineTests --gtest_filter=TestSuiteName*
krista@4173
   178
  ```
krista@4170
   179
krista@4170
   180
### To run and/or debug individual test cases   
krista@4173
   181
  1. To run:
krista@4173
   182
  ```
krista@4173
   183
  ./EngineTests --gtest_filter=TestSuiteName.test_function_name
krista@4173
   184
  ```
krista@4173
   185
  For example, for `check_delete_single_pubkey` in `DeleteKeyTest`:
krista@4173
   186
  ```
krista@4173
   187
  ./EngineTests DeleteKeyTest.check_delete_single_pubkey
krista@4173
   188
  ```
krista@4173
   189
krista@4170
   190
  2. To debug the same with lldb:
krista@4173
   191
  ```
krista@4173
   192
  lldb ./EngineTests -- --gtest_filter=TestSuiteName.test_function_name
krista@4173
   193
  ```
krista@4173
   194
krista@4170
   195
  3. To debug with gdb:
krista@4173
   196
  ```
krista@4173
   197
  gdb --args ./EngineTests --gtest_filter=TestSuiteName.test_function_name
krista@4173
   198
  ```
krista@4024
   199
neal@4216
   200
When debugging a failing test, use '--gtest_break_on_failure' to have
neal@4216
   201
gtest automatically break into the debugger where the assertion fails.
neal@4216
   202
krista@4024
   203
# Creating new tests
krista@4024
   204
krista@4024
   205
Script next on the agenda...
krista@4174
   206
krista@4174
   207
# Known Problems
krista@4174
   208
krista@4174
   209
There are 5 (maybe 6) tests that currently fail which are the function of
krista@4174
   210
broken tests, not the engine. They are on the agenda to fix (expired
krista@4174
   211
keys inside imported mails, etc). You'll see something like this - please
krista@4174
   212
ignore for this particular second in time:
krista@4174
   213
```
krista@4174
   214
    170 ms: ./EngineTests SubkeyRatingEvalTest.check_subkey_rating_eval_no_es
krista@4174
   215
     170 ms: ./EngineTests SubkeyRatingEvalTest.check_subkey_rating_eval_revoked_sign_no_alt
krista@4174
   216
     253 ms: ./EngineTests SubkeyRatingEvalTest.check_subkey_rating_eval_weak_e_strong_ecc_se
krista@4174
   217
    1189 ms: ./EngineTests KeyResetMessageTest.check_non_reset_receive_revoked
krista@4174
   218
    1376 ms: ./EngineTests KeyResetMessageTest.check_reset_receive_revoked
krista@4174
   219
```