--- /dev/null
+# Build Cache wrapper for JavaScript Template system
+
+An NDCODE project.
+
+## Overview
+
+The `jst_cache` package exports a single constructor
+ `JSTCache(root, args, diag)`
+which must be called with the `new` operator. The resulting cache object stores
+compiled JavaScript templates. Each template is, in general, a function that
+can be called with certain arguments, and which will generate the HTML text for
+a particular page or part of a page. The arguments can vary with each template.
+
+The `JSTCache` object takes care of loading template source code, compiling it
+using the `jst` package, and then executing the resulting JavaScript code. This
+initial execution is intended to allow the template to load any resources it
+needs, and generally set things up so that it is ready to generate HTML. The
+initial execution returns the object which is stored in the cache. As mentioned
+this is normally a function, but could be something else, e.g. any JSON object.
+
+See the `build_cache`, `disk_build` and `jst` packages for more information.
+The `JSTCache` object is essentially a wrapper object which routes the request
+between these packages, to ensure that the compiled template is retrieved from
+either RAM or disk if available, or is compiled and stored to RAM and to disk.
+
+As well as wrapping the `build_cache`, `disk_build` and `jst` functionality
+into a convenient "point and shoot" interface for template loading, `jst_cache`
+also provides a dependency resolution service for the compiled template. If the
+compiled template wants to load further templates during initialization or run-
+time, it calls back into `jst_cache`, which recursively loads the dependency.
+
+## Template examples
+
+See the `jst` package for more information about what can be in a template.
+Essentially, the templates are written in a dialect of JavaScript which allows
+HTML constructs to be specified directly, returning a string containing HTML.
+
+HTML templates occuring at expression level will be completely rendered to a
+string, and the resulting string returned as the value of the expression.
+
+HTML templates or strings (either single or double quoted strings, or template
+strings) occurring at statement level, become part of an enclosing template.
+
+For example, the JavaScript code `let text = p {'Hello, world'}` sets `text`
+to the value `<p>Hello, world</p>`. This consists of an HTML template `p { }`
+occurring at expression level, then the string `'Hello, world'` at statement
+level inside the `p { }` template (any valid JavaScript statement is allowed).
+
+Here is a complete example showing template substitution and HTML expressions:
+```
+let lang = 'en'
+let name = 'John'
+console.log(
+ html(lang=lang) {
+ body {`Hello, ${name}`}
+ }
+}
+```
+
+Running the above program will generate the output:
+```
+<html lang="en"><body>Hello, John</body></html>
+```
+
+## Template output buffer
+
+It is also possible to use statement-level HTML templates or strings to build
+up an output buffer, which can be used for whatever purpose. The output buffer
+must be called `_out` and must be a JavaScript array of string fragments. As
+each output fragment (such as an opening or closing tag or an embedded string)
+is generated, it will be sent to the output buffer by an `_out.push(...)` call.
+
+If there is dynamically generated text in the template, then it will be esacped
+by the sequence `.replace("&", "&").replace("<", "<")`, this is chosen
+so that no external dependency such as the `html-entities` package is needed,
+and so that unnecessary escaping of characters such as `'` or `>` is not done.
+Attributes are done similarly, except that `"` is replaced instead of `<`. We
+assume a `UTF-8` environment, so there is really little need for HTML entities.
+
+For example, consider a realistic template which we use on our demo server:
+```
+html(lang=lang) {
+ head {
+ link(rel="stylesheet" type="text/css" href="css/styles.css") {}
+ }
+ body {
+ p {`Hello, ${name}`}
+ }
+}
+```
+
+This compiles to the following plain JavaScript code:
+```
+_out.push("<html lang=\"" + lang.toString().replace("&", "&").replace("\"", """) + "\">");
+{
+ _out.push("<head>");
+ _out.push("<link rel=\"stylesheet\" type=\"text/css\" href=\"css/styles.css\">");
+ _out.push("</head>");
+}
+{
+ _out.push("<body>");
+ {
+ _out.push("<p>");
+ _out.push(`Hello, ${name}`.replace("&", "&").replace("<", "<"));
+ _out.push("</p>");
+ }
+ _out.push("</body>");
+}
+_out.push("</html>");
+```
+
+If invoked as an expression, the same code will be generated, but wrapped in an
+Immediately-Invoked Function Expression (IIFE) that creates the `_out` buffer,
+executes the above, then returns the buffer concatenated into a single string.
+
+## Template dependencies
+
+Templates may depend on other templates. Templates are imported imported using
+the syntax `_require(...)`, similarly to the syntax `require(...)` in CommonJS.
+
+We use `_require()` for templates instead of the normal `require()` because:
+
+(1) We do not want to interfere with other symbols in the namespace, you can
+also use `require()` in your templates to import ordinary CommonJS modules;
+
+(2) Absolute paths have a different meaning in `_require()` versus `require()`,
+for `_require()` they will be taken relative to the webserver's document root;
+
+(3) Using `_require()` says that the file is in `*.jst` format, and hence it
+should be parsed and any HTML constructs converted into regular JavaScript; and
+
+(4) The template importing is an asynchronous operation, so you normally use
+`await _require(...)`, whereas `require()` is synchronous. Because templates
+which have been edited are recompiled automatically by the running webserver,
+we don't want to stop the world, especially as templates are slow to compile.
+
+A slight difference between CommonJS exports and JavaScript template exports,
+is that where CommonJS modules set the dedicated symbol `module.exports` to
+whatever is exported, JavaScript templates instead `return` what is exported.
+
+## Template paths and variables
+
+Similarly to CommonJS, the variable `_require` is injected by `jst_cache` and
+is available to all templates. Other variables injected by `jst_cache` are
+`_pathname` which is the path to the currently executing template, and `_root`
+which is the root directory for absolute pathname references in `_require()`.
+
+Thus, paths sent to `_require()` are resolved relative to either `_root` if
+absolute (beginning with `/`) or the directory part of `_pathname` if relative
+(not beginning with `/`). Each template gets its own `_require()` function,
+which knows the `_pathname` and `_root` values of the calling template.
+
+If further variables should be made available to the template, they can be
+included as a dictionary in the `args` parameter to `JSTCache()`. For example,
+```js
+let jst_cache = new JSTCache(
+ '/home/myname/www',
+ {myvar: 'some value'}
+)
+let page = jst_cache.get('/home/myname/www/blog/page.jst')
+```
+parses and executes the template `/home/myname/www/blog/page.jst`, with the
+variables `_pathname` set to `'/home/myname/www/blog/page.jst'`, `_root` set to
+`'/home/myname/www'`, and `myvar` set to `'some value'`. Suppose `page.jst` is:
+```js
+let footer = _require('/site/footer.jst')
+```
+or
+```js
+let footer = _require('../site/footer.jst')
+```
+then the template `/home/myname/www/site/footer.jst` is parsed and executed,
+with `_pathname` set to `'/home/myname/www/site/footer.jst'`, and the other
+injected variables `_root` and `myvar` set from the original `JSTCache()` call.
+
+## Multi-part template examples
+
+### Templating with HTML text strings
+
+Here is an example of a template which takes the body of a page (as text which
+may contain HTML tags), and wraps it in the standard `html` and `body` tags to
+create a complete page. The idea is to use this to generate every page, so that
+additional commands, such as viewport commands or script tags, can be set here.
+
+Save this as `page.jst`, since it's the template for all pages in the system:
+```
+return body_text => html {
+ head {
+ link(rel="stylesheet" type="text/css" href="css/styles.css") {}
+ }
+ body {
+ _out(body_text)
+ }
+}
+```
+Save this as `index.html.jst`, as it's the template to generate `index.html`:
+```
+let page = await _require('/page.jst')
+
+return page(p {'Hello, world'})
+```
+Note that here we have somewhat hacked into the internals of the templating
+system by directly appending the body text onto the output buffer `_out`. It is
+intentional that you can access the internals for efficiency reasons, and that
+you can, for example, avoid HTML escaping of particular text where appropriate.
+
+### Templating with callbacks
+
+Another way, which is more sophisticated, is to use a callback function to
+generate the body. If doing this, we pass the output buffer `_out` into the
+callback function, so that the entire HTML of the page can be generated in an
+unbroken stream, without having to concatenate the partial page repeatedly.
+
+The page wrapper template `page.jst` which takes a body callback and calls it:
+```
+return body_callback => html {
+ head {
+ link(rel="stylesheet" type="text/css" href="css/styles.css") {}
+ }
+ body {
+ body_callback(_out)
+ }
+}
+```
+
+Then the specific page template `index.html.jst` which provides the callback:
+```
+let page = await _require('/page.jst')
+
+return html(
+ _out => {
+ p {'Hello, world'}
+ }
+)
+```
+
+Note that in the above example, the `{ }` around the callback function body is
+essential, unlike in regular JavaScript, so that `p {...}` occurs in statement
+rather than expression context. As an expression it would return a string which
+would be ignored, since the page template is expecting output to be in `_out`.
+
+## File management
+
+The `JSTCache` object a "point and shoot" interface which makes it very easy to
+manage on-disk templates. You just call `JSTCache.get()` stating a pathname to
+a template, normally a `*.jst` file, and the template is parsed, compiled to
+JavaScript, and evaluated, with the evaluation result cached for future reuse.
+
+What the `JSTCache.get()` function returns depends on the template source code.
+Template exports are similar but slightly different to CommonJS module exports,
+as noted in examples above. Usually, `JSTCache.get()` returns a JavaScript
+function, which you call to generate HTML each time the page is to be served.
+
+The HTML-generating function is re-useable in this way for efficiency reasons,
+since after the initial compilation, the cached version can be reused each time
+the page is served, but called with different arguments, resulting in different
+substitutions made on the page. For example, consider the following template:
+```
+return (lang, name) => html(lang=lang) {
+ body {
+ p {`Hello, ${name}`}
+ }
+}
+```
+Suppose the above is saved as `/hello/index.html.jst` under your document root
+`/var/www/html`. When you instantiate the `JSTCache` object giving the document
+root, and then call the `get(pathname)` instance function, you receive a
+function of two arguments `lang` and `name`, which returns the HTML string.
+
+Note that `_require()` paths beginning with `/` are taken relative to the
+document root passed into the `JSTCache()` constructor call, whereas paths
+beginning with anything else are taken relative to the `dirname` extracted from
+the current value of `_pathname` as seen by the template. Each template has its
+own `_pathname` and thus its `_require()` calls are relative to its own source.
+
+## Memory vs disk caching of templates
+
+Templates compiled using `JSTCache.get()` are cached in memory, as long as the
+same `node` interpreter is running, so that they can be retrieved using either
+`JSTCache.get()`, or equivalently `_require()` inside a template, and they will
+not be re-executed. In the above example, if you call `JSTCache.get()` twice,
+you get the same object twice (it is a 2-argument function returning `String`).
+
+As well as this, the compiled templates are also cached on disk, which requires
+the document root to be writeable. For instance if you compile `index.html.jst`
+in a `hello/` subdirectory of the document root, you get a hidden file with the
+same name prefixed by `.` in the same directory, here `hello/.index.html.jst`.
+
+The main reason for the disk caching is really to fool the node.js `require()`
+system into using correct relative paths, if the template uses `require()` as
+well as `_require()`. However, it is also handy that the templates only need to
+be re-evaluated and not recompiled if the webserver is stopped and restarted.
+
+Before using either a disk-cached or a memory-cached template, the modification
+times are checked and the template is recompiled if it is stale. Thus, you can
+edit your website while it is live, and each page will be recompiled as needed
+just before serving. Templates deleted on disk are not returned from the cache.
+
+Note: If the document root is not writeable, a simple expedient is to make sure
+that all `.*.jst` files are up to date, so no write is attempted. You could do
+this by firing up a development instance of the server in a writeable directory
+and then indexing the whole site, for example by a recursive `wget` invocation.
+
+## To be implemented
+
+It is intended that we will shortly add a timer function (or possibly just a
+function that the user should call periodically) to flush built templates from
+the cache after a stale time, on the assumption that the template might not be
+accessible or wanted anymore. For example, if the templates are HTML pages, the
+link structure of the site may have changed to make some pages inaccessible.
+
+## GIT repository
+
+The development version can be cloned, downloaded, or browsed with `gitweb` at:
+https://git.ndcode.org/public/jst_cache.git
+
+## License
+
+All of our NPM packages are MIT licensed, please see LICENSE in the repository.
+
+## Contributions
+
+The `jst` system is under active development (and is part of a larger project
+that is also under development) and thus the API is tentative. Please go ahead
+and incorporate the system into your project, or try out our example webserver
+built on the system, subject to the caution that the API could change. Please
+send us your experience and feedback, and let us know of improvements you make.
+
+Contact: Nick Downing <nick@ndcode.org>
public / jst_cache.git / commitdiff