Change from pnpm to npm, add ./link.sh shortcut for npm style package linking

[build_cache.git] / README.md

# Build Cache system

An NDCODE project.

## Overview

The `build_cache` package exports a single constructor `BuildCache(diag)` which
must be called with the `new` operator. The resulting cache object is intended
to store objects of arbitrary JavaScript type, which are generated from source
files of some kind. The cache tracks the source files of each object, and makes
sure the objects are rebuilt as required if the source files change on disk.

## Calling API

Suppose one has a `BuildCache` instance named `bc`. It behaves somewhat like an
ES6 `Map` object, except that it only has the `bc.get()` function, because new
objects are added to the cache by attempting to `get` them.

The interface for the `BuildCache`-provided instance function `bc.get()` is:

`await bc.get(key, once)` — retrieves the object stored under `key`, the
`key` is a somewhat arbitrary string that usually corresponds to the on-disk
path to the main source file of the wanted object. If `key` already exists in
the cache, then the corresponding object is returned after an up-to-date check.
Otherwise, the user-provided `bc.build()` is called, to build it from sources.

If the `once` argument is specified `true`, then the cached value is atomically
set to `undefined` when the built object is returned. This is useful if the
built object is something like a constructor function, that only needs to be
called once each time it changes on-disk. Then, the cache does not store the
build result, but only the list of sources and the time last built (in order to
detect changes). Omit this argument in the case that normal caching is needed.

The function `bc.build()` must be overridden by the user in a derived class:

`await bc.build(key, result)` — user must set `result.value` to an object
to be stored in the cache, and _may_ set `result.deps` to a list of pathnames
corresponding to the build dependencies, including the main file which is
normally the same as the `key` argument to `bc.get()`. The `result.deps` field
is set to `[key]` before the callback and doesn't need to be changed in the
common case that exactly one source file compiles to one object in the cache.

## About dependencies

Usually the dependencies will be tracked during the building of an object, for
instance the main source file may contain `include` directives that bring in
further source files. Most compilers should support this feature, for instance
the programmatic API to the `Less` CSS compiler returns CSS code and a list of
dependencies. They are placed in `result.value` and `result.deps` respectively.

At the moment we do not support _optional_ source files, for instance there
might be a picture `image.jpg` and an optional metadata file `image.json` that
goes with it. To handle this case, in the future we could add a _not exists_
dependency type so that a rebuild occurs if the metadata file becomes present.

## Usage example

Here is a fairly self-contained example of how we can define a `BuildCacheLess`
object constructed by `new BuildCacheLess(root)`, whose `get()` function would
load a `Less` stylesheet from the given pathname, and compile it to CSS via the
cache. The `root` is used if the `Less` stylesheet includes further stylesheets
using absolute pathnames, often it would be the document root of the webserver.
```js
let BuildCache = require('@ndcode/build_cache')
let fs = require('fs')
let less = require('less/lib/less-node')
let path = require('path')
let util = require('util')

let fs_readFile = util.promisify(fs.readFile)

let BuildCacheLess = function(root) {
  BuildCache.prototype.call(this)
  this.root = root
}

BuildCacheLess.prototype = Object.create(BuildCache.prototype)

BuildCacheLess.prototype.build = async function(key, result) {
  let render = await less.render(
    await fs_readFile(key, {encoding: 'utf-8'}),
    {
      filename: key,
      pathnames: [path.dirname(key)],
      rootpathname: this.root
    }
  )
  result.deps = result.deps.concat(render.imports)
  result.value = Buffer.from(render.css)
}
```
We are relying on `fs_readFile()` or `less.render()` to throw exceptions if the
original stylesheet or any included stylesheet is not found or contains errors.

Also, note how much simplified the handling of asynchronicity is when using the
ES7 `async`/`await` syntax. We recommend to do this for all new code, and to do
it consistently, even if the use of `Promise` directly might give shorter code.

Code which is not already promisified, can be promisified as shown, or else we
can add specific conversions in places by code like: `await new Promise(...)`.
Note the comment `/* await */` where a `Promise` is passed through from a lower
level routine, an explicit `await` would be consistent here but less efficient.

## About repeatable builds

If different types of objects need to be cached, use several instances of
`BuildCache` so that differently-built objects do not conflict with each other.
This also applies if the build-process is parameterized, e.g. the `root` above.

## About asynchronicity

To avoid the overhead of _directory watching_, the current implementation just
does an `fs.stat()` on each source file prior to returning an object from the
cache. This means that `bc.get()` is fundamentally an asynchronous operation and
therefore returns a `Promise`, which we showed as `await bc.get()` above.

Also, the building process may be asynchronous, and so `bc.build()` is also
expected to return a `Promise`. Obviously, `bc.get()` must wait for the
`bc.build()` promise to resolve, indicating that the wanted object is safely
stored in the cache, so that it can resolve the `bc.get()` promise with the
`result.value` that is now associated with the key and wanted by the caller.

There are some rather tricky corner cases associated with this, such as what
happens when the same object is requested again while its up-to-dateness is
being checked or while it is being built. `BuildCache` correctly handles these
cases. Whilst in general the up-to-date check happens every time an object is
retrieved, it won't be overlapped with another up-to-date check or a build.

## About exceptions

Exceptions during the build process are handled by reflecting them through both
`Promise`s, and also invalidating the associated key on the way through, so that
the object is no longer cached, and a fresh rebuild will be attempted should it
be accessed again in the future. A build failure is the only way that an object
can be removed from the cache (we may add an explicit removal API).

Note that if several callers are requesting the same key simultaneously and an
exception occurs during the build or up-to-date check, each caller receives a
reference to same shared exception object, thus when the `bc.get()` `Promise`
rejects, the rejection value (exception object) should be treated as read-only.

## About deletions

Another corner case happens if source files have been deleted since building,
we handle this the same as an updated source file and attempt to rebuild it.

Note that deleting the source files does not remove an object from the cache,
since the deleted source files will only be noticed when the object is accessed
(however, the resulting rebuild will remove the cached object if it fails).

## About diagnostics

The `diag` argument to the constructor is a `bool`, which if `true` causes
messages to be printed via `console.log()` for all activities except for the
common case of retrieval when the object is already up-to-date. A `diag` value
of `undefined` is treated as `false`, thus it can be omitted in the usual case.

The `diag` output is handy for development, and can also be handy in production,
e.g. our production server is started by `systemd` which automatically routes
`stdout` output to the system log, and the cache access diagnostic acts somewhat
like an HTTP server's `access.log`, albeit up-to-date accesses are not logged.

We have not attempted to provide comprehensive logging facilities or
log-routing, because the simple expedient is to turn off the built-in
diagnostics in complex cases and just do your own. In our server we have the
built-in diagnostics enabled in some simple cases and disabled in favour of
caller-provided logging in others (we use quite a few BuildCache instances,
since there are various preprocessors, including `Less` as mentioned above).

## To be implemented

It is intended that we will shortly add a timer function (or possibly just a
function that the user should call periodically) to flush built objects from the
cache after a stale time, on the assumption that the object might not be
accessible or wanted anymore. For example, if the objects are HTML pages, the
link structure of the site may have changed to make some pages inaccessible.

## GIT repository

The development version can be cloned, downloaded, or browsed with `gitweb` at:
https://git.ndcode.org/public/build_cache.git

## License

All of our NPM packages are MIT licensed, please see LICENSE in the repository.

## Contributions

We would greatly welcome your feedback and contributions. The `build_cache` is
under active development (and is part of a larger project that is also under
development) and thus the API is considered tentative and subject to change. If
this is undesirable, you could possibly pin the version in your `package.json`.

Contact: Nick Downing <nick@ndcode.org>