README.md
author Roker <roker@pep-project.org>
Thu, 07 Jun 2018 13:53:16 +0200
branchJSON-93
changeset 541 030b53a59cab
parent 517 81d5fff0b732
child 715 fe9a0492effe
permissions -rw-r--r--
and last but not least: document the new InLength<> feature. :-)
     1 # p≡p JSON Server Adapter
     2 
     3 ## Introduction
     4 The p≡p JSON Server Adapter provides a REST-like jQuery-compatible API to
     5 connect with the p≡p engine.  It is language-independent and can be used by
     6 any client.
     7 
     8 ## Requirements
     9 In order to use the p≡p JSON Server Adapter, you need to build and run it.
    10 Currently, Linux (Debian 9, Ubuntu 16.04) and MacOS (10.11, 10.12) are
    11 supported, Windows is about to follow.  Newer versions should also work
    12 (file a bug report if not) but are not in our main focus, yet.
    13 
    14 ## Dependencies
    15 * C++ compiler: tested with g++ 4.8 and 4.9, and clang++ 2.8. Newer versions should work, too.
    16 * GNU make
    17 * libboost-thread-dev (tested with 1.58)
    18 * libboost-program-options-dev  (tested with 1.58)
    19 * libboost-filesystem-dev (tested with 1.58)
    20 * libevent-dev 2.0.21 or 2.0.22 (or build from source, see below)
    21 * [p≡p Engine](https://letsencrypt.pep.foundation/trac/wiki/Basic%20Concepts%20of%20p%E2%89%A1p%20engine)
    22   (which needs gpgme-thread, a patched libetpan, libboost-system-dev)
    23 * OSSP libuuid
    24 
    25 ## Building/Installing
    26 ### Install the dependencies
    27 Debian 9:
    28 
    29 ~~~~~
    30 apt install -y build-essential libboost1.62-dev libboost-system1.62-dev \
    31     libboost-filesystem1.62-dev libboost-program-options1.62-dev \
    32     libboost-thread1.62-dev libgpgme-dev uuid-dev googletest
    33 ~~~~~
    34 
    35 macOS 10.12:
    36 
    37 Use homebrew or macports to install the required libraries.
    38 
    39 For more explicit instructions on how to do this with macports, see the
    40 section below.
    41 
    42 Build and install the pEp Engine.  Instructions can be found here:
    43 [https://cacert.pep.foundation/dev/repos/pEpEngine/file/ef23982e4744/README.md](https://cacert.pep.foundation/dev/repos/pEpEngine/file/ef23982e4744/README.md).
    44 
    45 ### Build and install libevent
    46 ~~~~~
    47 mkdir ~/code/json-ad
    48 hg clone https://cacert.pep.foundation/dev/repos/pEpJSONServerAdapter/ ~/code/json-ad
    49 cd ~/code/json-ad/libevent-2.0.22-stable
    50 ./configure --prefix="$HOME/code/json-ad/libevent-2.0.22-stable/build/" --disable-openssl
    51 make
    52 make install
    53 ~~~~~
    54 
    55 ### Build and install the JSON server
    56 ~~~~~
    57 cd ~/code/json-ad/server
    58 ~~~~~
    59 
    60 Edit the build configuration to your needs in `./Makefile.conf`, or create a
    61 `./local.conf` that sets any of the make variables documented in
    62 `./Makefile.conf`.
    63 
    64 If a dependency is not found in your system's default include or library
    65 paths, you will have to specify the according paths in a make variable. 
    66 Typically, this has to be done at least for the pEp Engine, libetpan and
    67 libevent.
    68 
    69 Below are two sample `./local.conf` files, for orientation.
    70 
    71 macOS 10.12:
    72 
    73 ~~~~~
    74 PREFIX=$(HOME)/code/json-ad/build
    75 HTML_DIRECTORY=$(PREFIX)/share/pEp/json-adapter/html
    76 GTEST_DIR=$(HOME)/code/gtest/googletest
    77 
    78 BOOST_INC=-I$(HOME)/Cellar/boost/1.65.1/include
    79 BOOST_LIB=-L$(HOME)/Cellar/boost/1.65.1/lib
    80 
    81 ENGINE_INC=-I$(HOME)/code/engine/build/include
    82 ENGINE_LIB=-L$(HOME)/code/engine/build/lib
    83 
    84 ETPAN_INC=-I$(HOME)/code/libetpan/build/include
    85 ETPAN_LIB=-L$(HOME)/code/libetpan/build/lib
    86 
    87 EVENT_INC=-I$(HOME)/code/json-ad/libevent-2.0.22-stable/build/include
    88 EVENT_LIB=-L$(HOME)/code/json-ad/libevent-2.0.22-stable/build/lib
    89 
    90 GPGME_INC=-I$(HOME)/Cellar/gpgme/1.9.0_1/include
    91 GPGME_LIB=-L$(HOME)/Cellar/gpgme/1.9.0_1/lib
    92 
    93 UUID_INC=-I$(HOME)/Cellar/ossp-uuid/1.6.2_2/include
    94 UUID_LIB=-L$(HOME)/Cellar/ossp-uuid/1.6.2_2/lib
    95 ~~~~~
    96 
    97 Debian 9:
    98 
    99 ~~~~~
   100 PREFIX=$(HOME)/code/json-ad/build
   101 HTML_DIRECTORY=$(PREFIX)/share/pEp/json-adapter/html
   102 GTEST_DIR=/usr/src/googletest/googletest/
   103 
   104 ENGINE_INC=-I$(HOME)/code/engine/build/include
   105 ENGINE_LIB=-L$(HOME)/code/engine/build/lib
   106 
   107 ETPAN_INC=-I$(HOME)/code/libetpan/build/include
   108 ETPAN_LIB=-L$(HOME)/code/libetpan/build/lib
   109 
   110 EVENT_INC=-I$(HOME)/code/json-ad/libevent-2.0.22-stable/build/include
   111 EVENT_LIB=-L$(HOME)/code/json-ad/libevent-2.0.22-stable/build/lib
   112 ~~~~~
   113 
   114 Now, build and install the server:
   115 
   116 ~~~~~
   117 make all
   118 make install
   119 ~~~~~
   120 
   121 With `make test` you can execute the server's tests.
   122 
   123 ### Macports
   124 [Install MacPorts](https://www.macports.org/install.php) for your version of macOS.
   125 
   126 If MacPorts is already installed on your machine, but was installed by a
   127 different user, make sure your `PATH` variable is set as follows in
   128 `~/.profile`:
   129 
   130 ```
   131 export PATH="/opt/local/bin:/opt/local/sbin:$PATH"
   132 ```
   133 
   134 Install dependencies packaged with MacPorts as follows.
   135 
   136 ```
   137 sudo port install gpgme boost ossp-uuid
   138 ```
   139 
   140 ## Running the pEp JSON Adapter
   141 You can use `make run` to start the server.
   142 
   143 1. Run ./pep-json-server.  This creates a file that is readable only by the
   144    current user (~/.pEp/json-token-${USER}) and contains the address and
   145    port the JSON adapter is listening on, normally 127.0.0.1:4223 and a
   146    "security-token" that must be given in each function call to authenticate
   147    you as the valid user.
   148 
   149    ```
   150    ./pep-json-server
   151    ```
   152 
   153 2. Visit that address (normally http://127.0.0.1:4223/) in your
   154    JavaScript-enabled web browser to see the "JavaScript test client".
   155 3. Call any function ("version()" or "get_gpg_path()" should work just
   156    fine) with the correct security token.
   157 
   158 ## Using the p≡p JSON Adapter
   159 
   160 In the following section, you'll find background information on how to use
   161 the adapter and its functions.
   162 
   163 ### Server startup and shutdown
   164 
   165 The JSON Server Adapter can be started on demand.
   166 It checks automatically whether an instance for the same user on the machine
   167 is already running and if yes it ends itself gracefully.
   168 
   169 If there is no running server found the newly started server creates the
   170 server token file and forks itself into background (if not prevented via
   171 "-d" commandline switch).
   172 
   173 
   174 ### Session handling
   175 
   176 When using the p≡p engine, a session is needed to which any adapter can
   177 connect. The p≡p JSON Server Adapter automatically creates one session per
   178 HTTP client connection (and also closes that session automatically when the
   179 client connections is closed). Therefore, the client does not need to take
   180 care of the session management. However, the client has to set up a [HTTP
   181 persistent
   182 connection](https://en.wikipedia.org/wiki/HTTP_persistent_connection).
   183 
   184 ### API Principles
   185 
   186 All C data types are mapped the same way, so some day the JSON wrapper can
   187 be generated from the p≡p Engine header files (or the JSON wrapper and the
   188 p≡p engine header are both generated from a common interface description
   189 file).
   190 
   191 | C type | JSON mapping |
   192 |--|--|
   193 | `bool` | JSON boolean |
   194 | `int` | JSON decimal number |
   195 | `size_t` | JSON decimal number |
   196 | `char*` (representing a UTF-8-encoded NULL-terminated string | JSON string |
   197 | `char*` (representing a binary string | base64-encoded JSON string |
   198 | `enum` | either JSON decimal number or JSON object containing one decimal number as member |
   199 | `struct` | JSON object |
   200 | linked lists (e.g. `bloblist_t`, `stringlist_t`, `identity_list` etc.) | JSON array of their member data type (without the `next` pointer) |
   201 
   202 The parameter type PEP_SESSION is handled automatically by the JSON Server
   203 Adapter and the PEP_SESSION parameter is omitted from the JSON API.
   204 
   205 #### enum types
   206 
   207 Enum types are represented as JSON objects with one member, whose name is
   208 derived from the enum type name, holding the numeric value of the enum.
   209 
   210 Some enum types are still represented directly as JSON decimal number. It
   211 shall be changed in a future version of the JSON Adapter.
   212 
   213 #### String types
   214 
   215 The JSON Server Adapter does automatic memory management for string
   216 parameters. The current p≡p Engine's API distinguish between `const char*`
   217 parameters and `char*` parameters.  `const char*` normally means: the
   218 "ownership" of the string remains at the caller, so the JSON Adapter frees
   219 the string automatically after the call.  `char*` normally means: the
   220 "ownership" of the string goes to the Engine, so the JSON Adapter does _not_
   221 free string.
   222 
   223 If there are functions that have a different semantics the behavior of the
   224 JSON wrapper has to be changed.
   225 
   226 #### Parameter (value) restrictions
   227 
   228 Some API functions have restrictions on their parameter values.  The JSON
   229 Adapter does not know these restrictions (because it does not know the
   230 semantics of the wrapped functions at all).  So it is the client's
   231 responsibility to fulfill these parameter restrictions!  Especially when
   232 there are restrictions that are checked with assert() within the p≡p Engine,
   233 it is impossible for the JSON Adapter to catch failed assertions - the
   234 Engine and the Adapter process will be terminated immediatetely when the
   235 Engine is compiled in debug mode (= with enabled assert() checking).
   236 
   237 Currently there are no range checks for numerical parameter types (e.g. a
   238 JSON decimal number can hold a bigger value than the `int` parameter type of
   239 a certain C function).
   240 
   241 ### API Reference
   242 
   243 An complete overview with all functions that are callable from the client
   244 can be found in the [API Reference](pEp JSON Server Adapter/API Reference).
   245 
   246 That API reference is a generated file that shows the current API briefly.
   247 There is also a (currently manually written) file that holts a copy of the
   248 documentation from the Engine's header files: [API reference detail.md]
   249 
   250 Most of the callable functions are functions from the C API of the p≡p
   251 Engine.  They are described in detail, incl.  pre- and post-conditions in
   252 the appropriate C header files of the Engine.
   253 
   254 
   255 ### Authentication
   256 
   257 The JSON Server Adapter and the client have to authenticate to each other.
   258 "Authentication" in this case means "run with the same user rights". This is
   259 done by proving that each communication partner is able to read a certain
   260 file that has user-only read permissions.
   261 
   262 0. There is a common (between client & server) algorithm to create the path
   263    and filename of the "server token file", for a given user name.
   264    The token file and its directory MUST be owned by the user and MUST be
   265    readable and writable only by the user, nobody else.  Client and server
   266    check for the right ownership and access rights of the token file and its
   267    directory. (TODO: What shall be done if that check fails?)
   268 
   269 1. The server creates a "server token file" containing a "server token" (a
   270    random-generated string of printable ASCII characters) and the IP address
   271    and port where the server listens on.  This file can only be read by
   272    client programs that run with the same user rights.
   273 
   274 2. The client checks the path, reads the "server token" from the file and
   275    authenticates itself to the server in each JSON RPC call with that "server
   276    token".
   277 
   278 
   279 ## Extending / customizing
   280 
   281 If you want to extend or customize the p≡p JSON Adapter, there are several
   282 rules and definitions to take into account.
   283 
   284 ### Definitions
   285 
   286 * The `FunctionMap function` in `ev_server.cc` defines which functions
   287   are callable via the JSON-RPC interface.  The existing entries show the
   288   syntax of that map.
   289   Non-static member functions can be called, too. Thanks to `std::function<>`
   290   a member function `Foo::func(Params...)` is handled like a free-standing
   291   function `func(Foo* f, Params...)`.
   292 
   293 * For each type there must exist specializations of the template classes
   294   "In" (for input parameters) and "Out" (for output parameters).
   295   The linker will tell you, which specializations are needed.
   296 
   297 * The specializations for "generic types" are in `function_map.cc`.
   298 
   299 * The specializations for "p≡p-specific types" are in `pep-types.cc`.
   300 
   301 
   302 #### Parameter directions (In, Out, InOut)
   303 
   304 The p≡p JSON Server Adapter supports Input, Output and two ways of "In/Out"
   305 parameters.  You have to annotate the direction in the FunctionMap with
   306 `In<>` for input, `Out<>` for output and InOut<> or InOutP<> for in/out
   307 parameters.  These wrapper classes have an optional second template
   308 parameter (parameter type flag) that is explained below.
   309 
   310 Return values are always "output" parameters, so they don't have to be
   311 wrapped with `Out<>`, but this wrapper is necessary when you need
   312 non-default wrapper semantics, see below.
   313 
   314 
   315 Input parameters of fundamental or simple struct types are
   316 usually by-value parameters. Complex structs (or structs that are only
   317 forward-declared in the public API) are usually pointer
   318 parameters. Both ways are supported. You have to specialize `In<T>` or
   319 `In<T*>`, depending how your type is used.
   320 
   321 Output parameters of fundamental or simple struct types `T` are usually
   322 declared as a paremeter of type `T*`. The p≡p JSON Server Adapter manages
   323 the memory allocated by the called C function automatically and calls the
   324 appropriate de-allocating function after use.
   325 
   326 Calling a function with output parameters requires a dummy value (`null` or
   327 empty string is fine) at the JSON side for each output parameter to keep the
   328 number of parameters at the JSON side the same with the C side.
   329 
   330 For In/Out parameters there exist two calling conventions for
   331 call-by-pointer types:
   332 
   333 1. caller allocates object and fills with input values, callee can only change members.
   334 The C type of the parameter is usually `struct T*`. Use the wrapper `InOut<>`
   335 for these parameters.
   336 
   337 2. caller allocates object and fills with input values, callee might
   338 change/reallocate the whole object. The C type of the parameter is
   339 `struct T**`. Use the wrapper `InOutP<>` in these cases.
   340 
   341 `InOutP<>` is also the right wrapper for in/out parameters of fundamental or
   342 enum types due to the additional indirection in the C function call
   343 signature.
   344 
   345 
   346 #### Parameter type flags
   347 
   348 The wrapper classes might be instantiated with special "parameter type
   349 flags". If no flag is given the `DefaultFlag` is used with means the
   350 semantics described already above.
   351 
   352 At the moment there exist two parameter type flags which are interpreted as
   353 bitfield, so they can be combined:
   354 
   355 * NoInput : This flags a parameter at the C side that shall not be exposed
   356   at the JSON side. So the value cannot be specified by the client, it is
   357   provided by the JSON Server Adapter internally (e.g. for PEP_SESSION)
   358 
   359 * DontOwn : Used for pointer types who don't "own" the referred ressource,
   360   so it is not released automatically by the JSON Server Adapter after the
   361   call.
   362 
   363 More flags will be added when different semantics will be needed.
   364 
   365 
   366 ### Automatic parameter value generation
   367 
   368 For some parameters or parameter combinations the JSON Server Adapter is
   369 able to generate the values automatically either from the environment or
   370 from other parameters.
   371 
   372 These automatic parameter value generators are supported at the moment:
   373 
   374 #### InLength
   375 
   376 For functions that have a string parameter of type `const char*` followed by
   377 a `size_t` that specifies the length of the string, the JSON Adapter can
   378 calculate the value of that length parameter automatically because in the
   379 JSON API the lengths of strings are always known.
   380 
   381 Moreover, the "length" that has to be given here means the length of the
   382 string seen by the C API side, after processing of all JSON escaping
   383 mechanisms, so it might be difficult to calculate that value at client side.
   384 
   385 Example:
   386 ```
   387 // C function declaration:
   388 char* tohex(const char* input, size_t length);
   389 
   390 // API definition:
   391 // with implicit length parameter, with dummy JSON parameter
   392 FP( "tohex", new Func<char*, In<c_string>, InLength<>>( &tohex ))
   393 ```
   394 
   395 To be compatible with previous API versions the `InLength` parameter still
   396 needs a dummy placeholder in the JSON interface, but its value is no longer
   397 relevant:
   398 
   399 ```
   400 {"jsonrpc":"2.0", "id":28,
   401  "method":"tohex", "params":["some string","dummy_parameter"]
   402 }
   403 ```
   404 
   405 It is possible to specifiy `InLength<ParamFlag::NoInput>` so no
   406 parameter is exposed to the JSON API anymore:
   407 
   408 ```
   409 FP( "tohex", new Func<char*, In<c_string>, InLength<ParamFlag::NoInput>>( &tohex ))
   410 ```
   411 
   412 Now the 2nd parameter is omitted:
   413 ```
   414 {"jsonrpc":"2.0", "id":28,
   415  "method":"tohex", "params":["some string"]
   416 }
   417 ```
   418 
   419 
   420 ## TODOs
   421 
   422 The following issues are planned but not yet implemented.
   423 
   424 * More sensible unit tests
   425 
   426 * Generate all the tedious boiler plate code
   427     * the content of pep-types.cc
   428     * perhaps the FunctionMap 'function' in mt-server.cc
   429     * perhaps the JavaScript side of the HTML test page to ensure to be
   430       consistent with the server side in pep-types.cc
   431 
   432 * Adapt the "p≡p Transport API", when it is final. (either manually or by
   433   code generator, if ready)
   434 
   435 
   436 ## Appendix A: Attack scenarios on the authentication
   437 
   438 Let's discuss different attack / threat scenarios. I don't know which are
   439 realistic or possible, yet.
   440 
   441 ### General ideas / improvements
   442 
   443 Currently the JSON Server Adapter writes its server token file in a
   444 directory that is only readable & writable by the user itself.
   445 
   446 The server token file is written in $HOME/.pEp/json-token on
   447 UNIX/Linux/MacOS and %LOCALAPPDATA%/pEp/json-token on MS Windows.
   448 
   449 The JSON Server Adapter also checks whether .pEp has 0700 access rights
   450 on unixoid systems.
   451 
   452 
   453 ### Attacker with the same user rights
   454 
   455 If the attacker is able to run his malicious code with the same user
   456 rights as the JSON Server Adapter and his legitimate client, it is (and
   457 always will be) *impossible* to prevent this attack. Such an attacker also
   458 can just start a legitimate client that is under his control.
   459 
   460 The same applies to an attacker who gains root / admin access rights.
   461 
   462 ### Fake Server with different user rights
   463 
   464 ```
   465  ,----------.      ,--------.
   466  | Attacker | <==> | Client |
   467  `----------'      `--------'
   468 ```
   469 
   470 If no real JSON Adapter runs an attacker can create a fake server that
   471 pretends to be a legitimate JSON Adapter. It creates its own server token
   472 file, with different and conspicuous access rights, but a limited
   473 JavaScript client might be unable to detect the file permissions.
   474 
   475 This fake server cannot access the private key of the user but it might
   476 get sensitive plaintext data the client wants to encrypt. The fake server
   477 cannot sign the encrypted data so the fake would be conspicuous, too. But
   478 that would be too late, because the sensitive plaintext data could
   479 already be leaked by the fake server.
   480 
   481 This attack needs a user's home directory that is writable by someone else
   482 (to create a ~/.pEp/ directory) or a foreign-writable ~/.pEp/ directory.
   483 
   484 The pEpEngine creates a ~/.pEp/ directory (if not yet exists) and sets the
   485 permissions to 0700 explicitly.
   486 
   487 
   488 ### Man-in-the-middle with different user rights
   489 
   490 ```
   491  ,---------------------.      ,----------.      ,--------.
   492  | JSON Server Adapter | <==> | Attacker | <==> | Client |
   493  `---------------------'      `----------'      `--------'
   494 ```
   495 
   496 * The attacker cannot read "client token file" nor "server token file".
   497 * The server cannot check "who" connects to it, until the client
   498   authenticates itself, which might be relayed by the attacker from the
   499   original client.
   500 * The attacker has to convince the client that it is a legitimate server. It
   501   has to create a fake "server token file" to divert the client to the
   502   attacker's port. But that fake file cannot contain the right server token
   503   because the attacker does not know it.
   504   * if the server started before the attacker the "server token file"'s
   505     access rights should prevent this (no write access for the attacker, no
   506     "delete" right in common TEMP dir (sticky bit on the directory)
   507   * if the attacker starts before the server it can write a fake toke file.
   508     The server could detect it but is unable to notice the legitimate
   509     client. The client could detect it when it can check the file access
   510     rights.
   511     There might be race conditons...
   512 
   513 * Is it possible for the attacker to let the client send the right server
   514   token to him, at least temporarily (e.g. within a race condition)?
   515   * As long as the server runs, the attacker cannot bind to the same address
   516     & port. Finding and binding of the port is done by the server before the
   517     server token file is created and filled.
   518   * When the server that created the "server token file" dies, its port
   519     becomes available for the attacker, but the server token is no longer
   520     valid and no longer useful for the attacker.
   521 * there _might_ be a very _small_ chance for a race condition:
   522   1. The attacker opens a connection to the running server but does not
   523     use it. To find the server it cannot read the server configuration
   524     file, but it can ask the OS for ports that are open in "listen" mode.
   525     Normally the JSON Adapter listens on 4223 or some port numbers above
   526     that. That means: guessing the server's address is quite easy.
   527   2. when the server shuts down, the attacker immediately binds itself to
   528     that port. If a client connects just in this moment it sends the server
   529     token to the attacker, not to the server. But the attacker can use that
   530     token now to the server via the already opened TCP connection.
   531   3. To prevent this the server should call shutdown(2) on its listening
   532     socket to block any new client connection, but still block the port.
   533     (is that the case on all platforms?) Than close(2) all TCP connections
   534     to the clients (if any) and than also delete the server token file.
   535     Finally call close(2) on the listening socket.