README.md
author Roker <roker@pep-project.org>
Thu, 07 Jun 2018 12:44:11 +0200
branchJSON-93
changeset 537 ff1cdc290c32
parent 517 81d5fff0b732
child 541 030b53a59cab
permissions -rw-r--r--
JSON-94: re-implement combining of UTF-16 surrogate pairs. the 1st implementation was nuts. :-9
     1 # p≡p JSON Server Adapter
     2 
     3 ## Introduction
     4 The p≡p JSON Server Adapter provides a REST-like jQuery-compatible API to
     5 connect with the p≡p engine.  It is language-independent and can be used by
     6 any client.
     7 
     8 ## Requirements
     9 In order to use the p≡p JSON Server Adapter, you need to build and run it.
    10 Currently, Linux (Debian 9, Ubuntu 16.04) and MacOS (10.11, 10.12) are
    11 supported, Windows is about to follow.  Newer versions should also work
    12 (file a bug report if not) but are not in our main focus, yet.
    13 
    14 ## Dependencies
    15 * C++ compiler: tested with g++ 4.8 and 4.9, and clang++ 2.8. Newer versions should work, too.
    16 * GNU make
    17 * libboost-thread-dev (tested with 1.58)
    18 * libboost-program-options-dev  (tested with 1.58)
    19 * libboost-filesystem-dev (tested with 1.58)
    20 * libevent-dev 2.0.21 or 2.0.22 (or build from source, see below)
    21 * [p≡p Engine](https://letsencrypt.pep.foundation/trac/wiki/Basic%20Concepts%20of%20p%E2%89%A1p%20engine)
    22   (which needs gpgme-thread, a patched libetpan, libboost-system-dev)
    23 * OSSP libuuid
    24 
    25 ## Building/Installing
    26 ### Install the dependencies
    27 Debian 9:
    28 
    29 ~~~~~
    30 apt install -y build-essential libboost1.62-dev libboost-system1.62-dev \
    31     libboost-filesystem1.62-dev libboost-program-options1.62-dev \
    32     libboost-thread1.62-dev libgpgme-dev uuid-dev googletest
    33 ~~~~~
    34 
    35 macOS 10.12:
    36 
    37 Use homebrew or macports to install the required libraries.
    38 
    39 For more explicit instructions on how to do this with macports, see the
    40 section below.
    41 
    42 Build and install the pEp Engine.  Instructions can be found here:
    43 [https://cacert.pep.foundation/dev/repos/pEpEngine/file/ef23982e4744/README.md](https://cacert.pep.foundation/dev/repos/pEpEngine/file/ef23982e4744/README.md).
    44 
    45 ### Build and install libevent
    46 ~~~~~
    47 mkdir ~/code/json-ad
    48 hg clone https://cacert.pep.foundation/dev/repos/pEpJSONServerAdapter/ ~/code/json-ad
    49 cd ~/code/json-ad/libevent-2.0.22-stable
    50 ./configure --prefix="$HOME/code/json-ad/libevent-2.0.22-stable/build/" --disable-openssl
    51 make
    52 make install
    53 ~~~~~
    54 
    55 ### Build and install the JSON server
    56 ~~~~~
    57 cd ~/code/json-ad/server
    58 ~~~~~
    59 
    60 Edit the build configuration to your needs in `./Makefile.conf`, or create a
    61 `./local.conf` that sets any of the make variables documented in
    62 `./Makefile.conf`.
    63 
    64 If a dependency is not found in your system's default include or library
    65 paths, you will have to specify the according paths in a make variable. 
    66 Typically, this has to be done at least for the pEp Engine, libetpan and
    67 libevent.
    68 
    69 Below are two sample `./local.conf` files, for orientation.
    70 
    71 macOS 10.12:
    72 
    73 ~~~~~
    74 PREFIX=$(HOME)/code/json-ad/build
    75 HTML_DIRECTORY=$(PREFIX)/share/pEp/json-adapter/html
    76 GTEST_DIR=$(HOME)/code/gtest/googletest
    77 
    78 BOOST_INC=-I$(HOME)/Cellar/boost/1.65.1/include
    79 BOOST_LIB=-L$(HOME)/Cellar/boost/1.65.1/lib
    80 
    81 ENGINE_INC=-I$(HOME)/code/engine/build/include
    82 ENGINE_LIB=-L$(HOME)/code/engine/build/lib
    83 
    84 ETPAN_INC=-I$(HOME)/code/libetpan/build/include
    85 ETPAN_LIB=-L$(HOME)/code/libetpan/build/lib
    86 
    87 EVENT_INC=-I$(HOME)/code/json-ad/libevent-2.0.22-stable/build/include
    88 EVENT_LIB=-L$(HOME)/code/json-ad/libevent-2.0.22-stable/build/lib
    89 
    90 GPGME_INC=-I$(HOME)/Cellar/gpgme/1.9.0_1/include
    91 GPGME_LIB=-L$(HOME)/Cellar/gpgme/1.9.0_1/lib
    92 
    93 UUID_INC=-I$(HOME)/Cellar/ossp-uuid/1.6.2_2/include
    94 UUID_LIB=-L$(HOME)/Cellar/ossp-uuid/1.6.2_2/lib
    95 ~~~~~
    96 
    97 Debian 9:
    98 
    99 ~~~~~
   100 PREFIX=$(HOME)/code/json-ad/build
   101 HTML_DIRECTORY=$(PREFIX)/share/pEp/json-adapter/html
   102 GTEST_DIR=/usr/src/googletest/googletest/
   103 
   104 ENGINE_INC=-I$(HOME)/code/engine/build/include
   105 ENGINE_LIB=-L$(HOME)/code/engine/build/lib
   106 
   107 ETPAN_INC=-I$(HOME)/code/libetpan/build/include
   108 ETPAN_LIB=-L$(HOME)/code/libetpan/build/lib
   109 
   110 EVENT_INC=-I$(HOME)/code/json-ad/libevent-2.0.22-stable/build/include
   111 EVENT_LIB=-L$(HOME)/code/json-ad/libevent-2.0.22-stable/build/lib
   112 ~~~~~
   113 
   114 Now, build and install the server:
   115 
   116 ~~~~~
   117 make all
   118 make install
   119 ~~~~~
   120 
   121 With `make test` you can execute the server's tests.
   122 
   123 ### Macports
   124 [Install MacPorts](https://www.macports.org/install.php) for your version of macOS.
   125 
   126 If MacPorts is already installed on your machine, but was installed by a
   127 different user, make sure your `PATH` variable is set as follows in
   128 `~/.profile`:
   129 
   130 ```
   131 export PATH="/opt/local/bin:/opt/local/sbin:$PATH"
   132 ```
   133 
   134 Install dependencies packaged with MacPorts as follows.
   135 
   136 ```
   137 sudo port install gpgme boost ossp-uuid
   138 ```
   139 
   140 ## Running the pEp JSON Adapter
   141 You can use `make run` to start the server.
   142 
   143 1. Run ./pep-json-server.  This creates a file that is readable only by the
   144    current user (~/.pEp/json-token-${USER}) and contains the address and
   145    port the JSON adapter is listening on, normally 127.0.0.1:4223 and a
   146    "security-token" that must be given in each function call to authenticate
   147    you as the valid user.
   148 
   149    ```
   150    ./pep-json-server
   151    ```
   152 
   153 2. Visit that address (normally http://127.0.0.1:4223/) in your
   154    JavaScript-enabled web browser to see the "JavaScript test client".
   155 3. Call any function ("version()" or "get_gpg_path()" should work just
   156    fine) with the correct security token.
   157 
   158 ## Using the p≡p JSON Adapter
   159 
   160 In the following section, you'll find background information on how to use
   161 the adapter and its functions.
   162 
   163 ### Server startup and shutdown
   164 
   165 The JSON Server Adapter can be started on demand.
   166 It checks automatically whether an instance for the same user on the machine
   167 is already running and if yes it ends itself gracefully.
   168 
   169 If there is no running server found the newly started server creates the
   170 server token file and forks itself into background (if not prevented via
   171 "-d" commandline switch).
   172 
   173 
   174 ### Session handling
   175 
   176 When using the p≡p engine, a session is needed to which any adapter can
   177 connect. The p≡p JSON Server Adapter automatically creates one session per
   178 HTTP client connection (and also closes that session automatically when the
   179 client connections is closed). Therefore, the client does not need to take
   180 care of the session management. However, the client has to set up a [HTTP
   181 persistent
   182 connection](https://en.wikipedia.org/wiki/HTTP_persistent_connection).
   183 
   184 ### API Principles
   185 
   186 All C data types are mapped the same way, so some day the JSON wrapper can
   187 be generated from the p≡p Engine header files (or the JSON wrapper and the
   188 p≡p engine header are both generated from a common interface description
   189 file).
   190 
   191 | C type | JSON mapping |
   192 |--|--|
   193 | `bool` | JSON boolean |
   194 | `int` | JSON decimal number |
   195 | `size_t` | JSON decimal number |
   196 | `char*` (representing a UTF-8-encoded NULL-terminated string | JSON string |
   197 | `char*` (representing a binary string | base64-encoded JSON string |
   198 | `enum` | either JSON decimal number or JSON object containing one decimal number as member |
   199 | `struct` | JSON object |
   200 | linked lists (e.g. `bloblist_t`, `stringlist_t`, `identity_list` etc.) | JSON array of their member data type (without the `next` pointer) |
   201 
   202 The parameter type PEP_SESSION is handled automatically by the JSON Server
   203 Adapter and the PEP_SESSION parameter is omitted from the JSON API.
   204 
   205 #### enum types
   206 
   207 Enum types are represented as JSON objects with one member, whose name is
   208 derived from the enum type name, holding the numeric value of the enum.
   209 
   210 Some enum types are still represented directly as JSON decimal number. It
   211 shall be changed in a future version of the JSON Adapter.
   212 
   213 #### String types
   214 
   215 The JSON Server Adapter does automatic memory management for string
   216 parameters. The current p≡p Engine's API distinguish between `const char*`
   217 parameters and `char*` parameters.  `const char*` normally means: the
   218 "ownership" of the string remains at the caller, so the JSON Adapter frees
   219 the string automatically after the call.  `char*` normally means: the
   220 "ownership" of the string goes to the Engine, so the JSON Adapter does _not_
   221 free string.
   222 
   223 If there are functions that have a different semantics the behavior of the
   224 JSON wrapper has to be changed.
   225 
   226 #### Parameter (value) restrictions
   227 
   228 Some API functions have restrictions on their parameter values.  The JSON
   229 Adapter does not know these restrictions (because it does not know the
   230 semantics of the wrapped functions at all).  So it is the client's
   231 responsibility to fulfill these parameter restrictions!  Especially when
   232 there are restrictions that are checked with assert() within the p≡p Engine,
   233 it is impossible for the JSON Adapter to catch failed assertions - the
   234 Engine and the Adapter process will be terminated immediatetely when the
   235 Engine is compiled in debug mode (= with enabled assert() checking).
   236 
   237 Currently there are no range checks for numerical parameter types (e.g. a
   238 JSON decimal number can hold a bigger value than the `int` parameter type of
   239 a certain C function).
   240 
   241 ### API Reference
   242 
   243 An complete overview with all functions that are callable from the client
   244 can be found in the [API Reference](pEp JSON Server Adapter/API Reference).
   245 
   246 That API reference is a generated file that shows the current API briefly.
   247 There is also a (currently manually written) file that holts a copy of the
   248 documentation from the Engine's header files: [API reference detail.md]
   249 
   250 Most of the callable functions are functions from the C API of the p≡p
   251 Engine.  They are described in detail, incl.  pre- and post-conditions in
   252 the appropriate C header files of the Engine.
   253 
   254 
   255 ### Authentication
   256 
   257 The JSON Server Adapter and the client have to authenticate to each other.
   258 "Authentication" in this case means "run with the same user rights". This is
   259 done by proving that each communication partner is able to read a certain
   260 file that has user-only read permissions.
   261 
   262 0. There is a common (between client & server) algorithm to create the path
   263    and filename of the "server token file", for a given user name.
   264    The token file and its directory MUST be owned by the user and MUST be
   265    readable and writable only by the user, nobody else.  Client and server
   266    check for the right ownership and access rights of the token file and its
   267    directory. (TODO: What shall be done if that check fails?)
   268 
   269 1. The server creates a "server token file" containing a "server token" (a
   270    random-generated string of printable ASCII characters) and the IP address
   271    and port where the server listens on.  This file can only be read by
   272    client programs that run with the same user rights.
   273 
   274 2. The client checks the path, reads the "server token" from the file and
   275    authenticates itself to the server in each JSON RPC call with that "server
   276    token".
   277 
   278 
   279 ## Extending / customizing
   280 
   281 If you want to extend or customize the p≡p JSON Adapter, there are several
   282 rules and definitions to take into account.
   283 
   284 ### Definitions
   285 
   286 * The `FunctionMap function` in `ev_server.cc` defines which functions
   287   are callable via the JSON-RPC interface.  The existing entries show the
   288   syntax of that map.
   289   Non-static member functions can be called, too. Thanks to `std::function<>`
   290   a member function `Foo::func(Params...)` is handled like a free-standing
   291   function `func(Foo* f, Params...)`.
   292 
   293 * For each type there must exist specializations of the template classes
   294   "In" (for input parameters) and "Out" (for output parameters).
   295   The linker will tell you, which specializations are needed.
   296 
   297 * The specializations for "generic types" are in `function_map.cc`.
   298 
   299 * The specializations for "p≡p-specific types" are in `pep-types.cc`.
   300 
   301 
   302 #### Parameter directions (In, Out, InOut)
   303 
   304 The p≡p JSON Server Adapter supports Input, Output and two ways of "In/Out"
   305 parameters.  You have to annotate the direction in the FunctionMap with
   306 `In<>` for input, `Out<>` for output and InOut<> or InOutP<> for in/out
   307 parameters.  These wrapper classes have an optional second template
   308 parameter (parameter type flag) that is explained below.
   309 
   310 Return values are always "output" parameters, so they don't have to be
   311 wrapped with `Out<>`, but this wrapper is necessary when you need
   312 non-default wrapper semantics, see below.
   313 
   314 
   315 Input parameters of fundamental or simple struct types are
   316 usually by-value parameters. Complex structs (or structs that are only
   317 forward-declared in the public API) are usually pointer
   318 parameters. Both ways are supported. You have to specialize `In<T>` or
   319 `In<T*>`, depending how your type is used.
   320 
   321 Output parameters of fundamental or simple struct types `T` are usually
   322 declared as a paremeter of type `T*`. The p≡p JSON Server Adapter manages
   323 the memory allocated by the called C function automatically and calls the
   324 appropriate de-allocating function after use.
   325 
   326 Calling a function with output parameters requires a dummy value (`null` or
   327 empty string is fine) at the JSON side for each output parameter to keep the
   328 number of parameters at the JSON side the same with the C side.
   329 
   330 For In/Out parameters there exist two calling conventions for
   331 call-by-pointer types:
   332 
   333 1. caller allocates object and fills with input values, callee can only change members.
   334 The C type of the parameter is usually `struct T*`. Use the wrapper `InOut<>`
   335 for these parameters.
   336 
   337 2. caller allocates object and fills with input values, callee might
   338 change/reallocate the whole object. The C type of the parameter is
   339 `struct T**`. Use the wrapper `InOutP<>` in these cases.
   340 
   341 `InOutP<>` is also the right wrapper for in/out parameters of fundamental or
   342 enum types due to the additional indirection in the C function call
   343 signature.
   344 
   345 
   346 #### Parameter type flags
   347 
   348 The wrapper classes might be instantiated with special "parameter type
   349 flags". If no flag is given the `DefaultFlag` is used with means the
   350 semantics described already above.
   351 
   352 At the moment there exist two parameter type flags which are interpreted as
   353 bitfield, so they can be combined:
   354 
   355 * NoInput : This flags a parameter at the C side that shall not be exposed
   356   at the JSON side. So the value cannot be specified by the client, it is
   357   provided by the JSON Server Adapter internally (e.g. for PEP_SESSION)
   358 
   359 * DontOwn : Used for pointer types who don't "own" the referred ressource,
   360   so it is not released automatically by the JSON Server Adapter after the
   361   call.
   362 
   363 More flags will be added when different semantics will be needed.
   364 
   365 
   366 ## TODOs
   367 
   368 The following issues are planned but not yet implemented.
   369 
   370 * More sensible unit tests
   371 
   372 * Generate all the tedious boiler plate code
   373     * the content of pep-types.cc
   374     * perhaps the FunctionMap 'function' in mt-server.cc
   375     * perhaps the JavaScript side of the HTML test page to ensure to be
   376       consistent with the server side in pep-types.cc
   377 
   378 * Adapt the "p≡p Transport API", when it is final. (either manually or by
   379   code generator, if ready)
   380 
   381 
   382 ## Appendix A: Attack scenarios on the authentication
   383 
   384 Let's discuss different attack / threat scenarios. I don't know which are
   385 realistic or possible, yet.
   386 
   387 ### General ideas / improvements
   388 
   389 Currently the JSON Server Adapter writes its server token file in a
   390 directory that is only readable & writable by the user itself.
   391 
   392 The server token file is written in $HOME/.pEp/json-token on
   393 UNIX/Linux/MacOS and %LOCALAPPDATA%/pEp/json-token on MS Windows.
   394 
   395 The JSON Server Adapter also checks whether .pEp has 0700 access rights
   396 on unixoid systems.
   397 
   398 
   399 ### Attacker with the same user rights
   400 
   401 If the attacker is able to run his malicious code with the same user
   402 rights as the JSON Server Adapter and his legitimate client, it is (and
   403 always will be) *impossible* to prevent this attack. Such an attacker also
   404 can just start a legitimate client that is under his control.
   405 
   406 The same applies to an attacker who gains root / admin access rights.
   407 
   408 ### Fake Server with different user rights
   409 
   410 ```
   411  ,----------.      ,--------.
   412  | Attacker | <==> | Client |
   413  `----------'      `--------'
   414 ```
   415 
   416 If no real JSON Adapter runs an attacker can create a fake server that
   417 pretends to be a legitimate JSON Adapter. It creates its own server token
   418 file, with different and conspicuous access rights, but a limited
   419 JavaScript client might be unable to detect the file permissions.
   420 
   421 This fake server cannot access the private key of the user but it might
   422 get sensitive plaintext data the client wants to encrypt. The fake server
   423 cannot sign the encrypted data so the fake would be conspicuous, too. But
   424 that would be too late, because the sensitive plaintext data could
   425 already be leaked by the fake server.
   426 
   427 This attack needs a user's home directory that is writable by someone else
   428 (to create a ~/.pEp/ directory) or a foreign-writable ~/.pEp/ directory.
   429 
   430 The pEpEngine creates a ~/.pEp/ directory (if not yet exists) and sets the
   431 permissions to 0700 explicitly.
   432 
   433 
   434 ### Man-in-the-middle with different user rights
   435 
   436 ```
   437  ,---------------------.      ,----------.      ,--------.
   438  | JSON Server Adapter | <==> | Attacker | <==> | Client |
   439  `---------------------'      `----------'      `--------'
   440 ```
   441 
   442 * The attacker cannot read "client token file" nor "server token file".
   443 * The server cannot check "who" connects to it, until the client
   444   authenticates itself, which might be relayed by the attacker from the
   445   original client.
   446 * The attacker has to convince the client that it is a legitimate server. It
   447   has to create a fake "server token file" to divert the client to the
   448   attacker's port. But that fake file cannot contain the right server token
   449   because the attacker does not know it.
   450   * if the server started before the attacker the "server token file"'s
   451     access rights should prevent this (no write access for the attacker, no
   452     "delete" right in common TEMP dir (sticky bit on the directory)
   453   * if the attacker starts before the server it can write a fake toke file.
   454     The server could detect it but is unable to notice the legitimate
   455     client. The client could detect it when it can check the file access
   456     rights.
   457     There might be race conditons...
   458 
   459 * Is it possible for the attacker to let the client send the right server
   460   token to him, at least temporarily (e.g. within a race condition)?
   461   * As long as the server runs, the attacker cannot bind to the same address
   462     & port. Finding and binding of the port is done by the server before the
   463     server token file is created and filled.
   464   * When the server that created the "server token file" dies, its port
   465     becomes available for the attacker, but the server token is no longer
   466     valid and no longer useful for the attacker.
   467 * there _might_ be a very _small_ chance for a race condition:
   468   1. The attacker opens a connection to the running server but does not
   469     use it. To find the server it cannot read the server configuration
   470     file, but it can ask the OS for ports that are open in "listen" mode.
   471     Normally the JSON Adapter listens on 4223 or some port numbers above
   472     that. That means: guessing the server's address is quite easy.
   473   2. when the server shuts down, the attacker immediately binds itself to
   474     that port. If a client connects just in this moment it sends the server
   475     token to the attacker, not to the server. But the attacker can use that
   476     token now to the server via the already opened TCP connection.
   477   3. To prevent this the server should call shutdown(2) on its listening
   478     socket to block any new client connection, but still block the port.
   479     (is that the case on all platforms?) Than close(2) all TCP connections
   480     to the clients (if any) and than also delete the server token file.
   481     Finally call close(2) on the listening socket.