## Overview
-The `jst` package exports a single function `jst(pathname, root, args)`,
-which loads a file `pathname` from disk, parses it for a superset of JavaScript
-(ES7) which can contain HTML-like constructs, and then generates equivalent
-plain JavaScript code which in turn, can generate the desired HTML.
+The `jst` package exports a single function
+ `jst(text, options)`
+which parses the given text in a JavaScript dialect which can contain HTML-like
+constructs, then generates equivalent plain JavaScript code which when run,
+would generate the requested HTML. As well as the HTML generation, there can be
+normal JavaScript logic to influence the generated page or substitute into it.
The system is inspired by Pug (formerly Jade) templates, but emphasizing the
JavaScript, for instance you do not need a `-` sign at the start of each line,
CommonJS conventions, for instance, `require(...)` is supported and so forth.
The embedded HTML uses a syntax similar to JavaScript function definitions,
- `function(arg1, arg2, ...) { ... }`
+ `function(arg1, arg2, ...) {...}`
except that instead of the word "function" any valid HTML (or other) tag name
is allowed, and instead of just argument names, HTML attributes of the form
`attrname`
}
```
and this would translate to the HTML file:
-```
+```htm
<html lang="en"><head></head><body></body></html>
```
HTML injection attacks), and then output as part of some enclosing template.
For example:
-```
+```js
html(lang="en") {
body {'Hello, world'}
}
```
This generates the text:
-```
+```html
<html lang="en"><body>Hello, world</body></html>
```
embedded newlines. When the entire text is held in a variable, e.g. `myvar`, a
template string such as ```${myvar}``` should be used, to convert it into a
standalone statement consisting of only a string constant, and hence into HTML.
-In the future, we may extend the parser to do this automatically for statements
-that consist of only a variable name, which would have no effect in JavaScript.
Note that ordinary single- or double-quoted string constants should be used in
preference to template strings where possible, since the constant strings will
The HTML `script` tag is treated specially, because it contains JavaScript,
which is understood directly by the template parser. The JavaScript inside the
-script tag is normalized (comments stripped, unnecessary braces and parentheses
+script tag is minified (comments stripped, unnecessary braces and parentheses
removed, semicolons added and so on), and then converted to a string constant,
which will be copied out to the HTML file whenever the template is executed.
-Code within `script` tags is output without any indentation, which reduces the
-wasted space in the HTML file quite considerably. In the future, we can add
-automatic minification and/or other transformations, such as ES5 transpilation.
-As well as browser-oriented transformations, other potential transformations
-such as TypeScript could be applied to all code rather than just `script` tags.
-
The HTML `style` tag is treated specially, because it contains CSS, which we
handle by switching temporarily to another parser that understands CSS. We use
-a slightly modified version of the `clean-css` package to do this. The CSS is
-automatically minified and then HTML-escaped at the time the template is
-compiled. Note: unlike the JST code, it currently cannot contain substitutions.
+a slightly modified version of the `clean-css` package to do this. The result
+is then minified and converted to a string constant, similar to `script` tags.
+
+Note that the `script` and `style` tags cannot contain substitutions, in the
+way that ordinary JST code can. This is because (i) the entire contents of the
+special tag are minified and stringified at template compile time, so the
+content of the special tag is the same each time the page is generated, and
+(ii) it runs elsewhere, which doesn't have access to the template variables.
### HTML class and id shorthand
Another slight limitation of the current parser is that it is more permissive
than normal in parsing regular JavaScript code, for instance commas are not
-required between function arguments, because HTML templates are basically
-parsed as function calls until we see the opening `{` that identifies it as a
+required between function arguments, because an HTML template is basically
+parsed as a function call until the opening `{` is seen to identify it as a
template. This can also likely be improved in a future version of the system.
## Expression vs statement templates
strings, in that the generated HTML will become part of an enclosing template.
Here is a complete example showing template substitution and HTML expressions:
-```
+```js
let lang = 'en'
let name = 'John'
console.log(
html(lang=lang) {
body {`Hello, ${name}`}
}
-}
+)
```
Running the above program will generate the output:
-```
+```html
<html lang="en"><body>Hello, John</body></html>
```
assume a `UTF-8` environment, so there is really little need for HTML entities.
For example, consider a realistic template which we use on our demo server:
-```
+```js
html(lang=lang) {
head {
link(rel="stylesheet" type="text/css" href="css/styles.css") {}
```
This compiles to the following plain JavaScript code:
-```
+```js
_out.push("<html lang=\"" + lang.toString().replace("&", "&").replace("\"", """) + "\">");
{
_out.push("<head>");
Immediately-Invoked Function Expression (IIFE) that creates the `_out` buffer,
executes the above, then returns the buffer concatenated into a single string.
-## Template dependencies
-
-Templates may depend on other templates. Templates are imported imported using
-the syntax `_require(...)`, similarly to the syntax `require(...)` in CommonJS.
-
-We use `_require()` for templates instead of the normal `require()` because:
-
-(1) We do not want to interfere with other symbols in the namespace, you can
-also use `require()` in your templates to import ordinary CommonJS modules;
-
-(2) Absolute paths have a different meaning in `_require()` versus `require()`,
-for `_require()` they will be taken relative to the webserver's document root;
-
-(3) Using `_require()` says that the file is in `*.jst` format, and hence it
-should be parsed and any HTML constructs converted into regular JavaScript; and
-
-(4) The template importing is an asynchronous operation, so you normally use
-`await _require(...)`, whereas `require()` is synchronous. Because templates
-which have been edited are recompiled automatically by the running webserver,
-we don't want to stop the world, especially as templates are slow to compile.
-
-A slight difference between CommonJS exports and JavaScript template exports,
-is that where CommonJS modules set the dedicated symbol `module.exports` to
-whatever is exported, JavaScript templates instead `return` what is exported.
+## Usage example
-## Template paths and variables
-
-Similarly to CommonJS, the variable `_require` is injected by the system and is
-available to all templates. Other variables injected by the system are
-`_pathname` which is the path to the currently executing template, and `_root`
-which is the root directory for absolute pathname references in `_require()`.
-
-Thus, paths sent to `_require()` are resolved relative to either `_root` if
-absolute (beginning with `/`) or the directory part of `_pathname` if relative
-(not beginning with `/`). Each template gets its own `_require()` function,
-which knows the `_pathname` and `_root` values of the calling template.
-
-If further variables should be made available to the template, they can be
-included as a dictionary in the `args` parameter to `jst()`. For example,
+Suppose we want to try out the JST template processor. Returning to the example
+from the section "Expression vs Statement templates", we could create a file
+called `mytemplate.jst` containing the example JST code to print a simple page:
```js
-let template = jst(
- '/home/myname/www/blog/page.jst',
- '/home/myname/www',
- {myvar: 'some value'}
+let lang = 'en'
+let name = 'John'
+console.log(
+ html(lang=lang) {
+ body {`Hello, ${name}`}
+ }
)
-```
-parses and executes the template `/home/myname/www/blog/page.jst`, with the
-variables `_pathname` set to `'/home/myname/www/blog/page.jst'`, `_root` set to
-`'/home/myname/www'`, and `myvar` set to `'some value'`. Suppose this contains
```js
-let footer = _require('/site/footer.jst')
-```
-or
+We could then create an example JavaScript program `example.js` to read,
+compile and then execute the template, as follows:
```js
-let footer = _require('../site/footer.jst')
-```
-then the template `/home/myname/www/site/footer.jst` is parsed and executed,
-with `_pathname` set to `'/home/myname/www/site/footer.jst'` and the other
-injected variables `_root` and `myvar` set from the original call to `jst()`.
-
-## Multi-part template examples
+let fs = require('fs')
+let jst = require('@ndcode/jst')
-### Templating with HTML text strings
-
-Here is an example of a template which takes the body of a page (as text which
-may contain HTML tags), and wraps it in the standard `html` and `body` tags to
-create a complete page. The idea is to use this to generate every page, so that
-additional commands, such as viewport commands or script tags, can be set here.
-
-Save this as `page.jst`, since it's the template for all pages in the system:
-```
-return body_text => html {
- head {
- link(rel="stylesheet" type="text/css" href="css/styles.css") {}
- }
- body {
- _out(body_text)
- }
-}
+eval(jst(fs.readFileSync('mytemplate.jst')))
```
-Save this as `index.html.jst`, as it's the template to generate `index.html`:
+To run this, we would have to run the `node` interpreter at the command line:
+```sh
+node example.js
```
-let page = await _require('/page.jst')
-
-return page(p {'Hello, world'})
+This produces the output described earlier:
+```html
+<html lang="en"><body>Hello, John</body></html>
```
-Note that here we have somewhat hacked into the internals of the templating
-system by directly appending the body text onto the output buffer `_out`. It is
-intentional that you can access the internals for efficiency reasons, and that
-you can, for example, avoid HTML escaping of particular text where appropriate.
-### Templating with callbacks
+## Command line preprocessor
-Another way, which is more sophisticated, is to use a callback function to
-generate the body. If doing this, we pass the output buffer `_out` into the
-callback function, so that the entire HTML of the page can be generated in an
-unbroken stream, without having to concatenate the partial page repeatedly.
-
-The page wrapper template `page.jst` which takes a body callback and calls it:
-```
-return body_callback => html {
- head {
- link(rel="stylesheet" type="text/css" href="css/styles.css") {}
- }
- body {
- body_callback(_out)
- }
-}
-```
+Using the programmatic interface is overkill in many situations, e.g. if we
+just want to see the generated JavaScript code, or if we want to compile all
+pages of our website to JavaScript ahead of time by a `Gruntfile` or similar.
-Then the specific page template `index.html.jst` which provides the callback:
+Hence, we provide the command line interface `jst` as a convenient way to
+convert JST to JavaScript. Using the command-line, the above example becomes:
```
-let page = await _require('/page.jst')
-
-return html(
- _out => {
- p {'Hello, world'}
- }
-)
+jst <mytemplate.jst >mytemplate.js
+node mytemplate.js
```
-
-Note that in the above example, the `{ }` around the callback function body is
-essential, unlike in regular JavaScript, so that `p {...}` occurs in statement
-rather than expression context. As an expression it would return a string which
-would be ignored, since the page template is expecting output to be in `_out`.
-
-## File management
-
-The `jst()` function gives a "point and shoot" interface which makes it
-very easy to manage on-disk templates. You essentially just state the pathname
-to the template, which is normally a `*.jst` file, and the template will be
-parsed, converted to JavaScript, evaluated, and also cached for future use.
-
-What the `jst()` function returns depends on what the template returns.
-Template exports are similar but slightly different to CommonJS module exports,
-as noted in examples above. Usually, `jst()` returns a JavaScript
-function, which you call to generate HTML each time the page is to be served.
-
-The HTML-generating function is re-useable in this way for efficiency reasons,
-since after the initial compilation, the cached version can be reused each time
-the page is served, but called with different arguments, resulting in different
-substitutions made on the page. For example, consider the following template:
+Inspecting the JavaScript file `mytemplate.js` shows how the HTML is generated:
```
-return (lang, name) => html(lang=lang) {
- body {
- p {`Hello, ${name}`}
+let lang = 'en';
+let name = 'John';
+console.log((() => {
+ let _out = [];
+ _out.push("<html lang=\"" + lang.toString().replace("&", "&").replace("\"", """) + "\">");
+ {
+ _out.push("<body>");
+ _out.push(`Hello, ${name}`.replace("&", "&").replace("<", "<"));
+ _out.push("</body>");
}
-}
-```
-Suppose the above is saved as `/hello/index.html.jst` under your document root
-`/var/www/html`. You call `jst(root, dirname, pathname)`, like this:
-```
-let template_func = jst(
- '/var/www/html',
- '/var/www/html/hello',
- 'index.jst'
-)
-```
-
-The result is a JavaScript function of 2 arguments, which you call like this:
-```
-console.log(
- template_func('en', 'world')
-)
+ _out.push("</html>");
+ return _out.join('');
+})());
```
-The generated HTML from the above example, printed on the console, would be:
-```
-<html lang="en"><body><p>Hello, world</p></body></html>
-```
-
-The reason why you provide the root and working directory separately, is that
-the template may import further dependencies using `_require()`, and these are
-taken relative to `root` or `dirname` depending on whether they begin with `/`.
-
-Note that any imported modules receive an updated `dirname`, so that relative
-imports are taken relative to the importing module, similarly to in CommonJS.
-
-## Memory vs disk caching of templates
-
-Templates compiled using `jst()` are cached in memory, as long as the
-same `node` interpreter is running, so that they can be retrieved using either
-`jst()`, or equivalently `_require()` inside a template, and they will
-not be re-executed. In the above example, if you call `jst()` twice,
-you get the same object twice (it is a 2-argument function returning `String`).
-
-As well as this, the compiled templates are also cached on disk, which requires
-the document root to be writeable. For instance if you compile `index.html.jst`
-in a `hello/` subdirectory of the document root, you get a hidden file with the
-same path and a `.js` subscript, here it would be `hello/.index.html.jst.js`.
-
-The main reason for the disk caching is really to fool the node.js `require()`
-system into using correct relative paths, if the template uses `require()` as
-well as `_require()`. However, it is also handy that the templates only need to
-be re-evaluated and not recompiled if the webserver is stopped and restarted.
-
-Before using either a disk-cached or a memory-cached template, the modification
-times are checked and the template is recompiled if it is stale. Thus you can
-edit your website while it is live, and each page will be recompiled as needed
-just before serving. Templates deleted on disk are not returned from the cache.
-
-Note: If the document root is not writeable, a simple expedient is to make sure
-that all `.*.jst.js` files are up to date, so no write is attempted. You can do
-this by firing up a development instance of the server in a writeable directory
-and then indexing the whole site, for example by a recursive `wget` invocation.
-
-## To be implemented
+Various command line options are available for setting indentation and similar.
+There is also a `--wrap` option which adds some prologue and epilogue code,
+which is part of a larger system we have designed, for allowing templates to
+embed each other and so on. See the `jst_server` and `jst_cache` modules for
+more information. If the extra complexity isn't needed, simply omit `--wrap`.
-It is intended that we will shortly add a timer function (or possibly just a
-function that the user should call periodically) to flush built templates from
-the cache after a stale time, on the assumption that the template might not be
-accessible or wanted anymore. For example, if the templates are HTML pages, the
-link structure of the site may have changed to make some pages inaccessible.
+## JSTizer program
-As mentioned throughout the above text, we will also improve the JST parser and
-translator to remove the need for some quoting or backticking workarounds, and
-to improve support for some HTML constructs such as stylesheets via a dedicated
-CSS parser, and similarly we may add various preprocessors such as TypeScript.
+In development, we often want to refer to example code from the Web, e.g. if we
+want to incorporate the Bootstrap front-end framework or if we want to create a
+web-form or some other common web-development task. The example code is usually
+written in HTML. We can drop this straight into a JST project by JSTizing it.
-We're also working on a "JSTizer" which can take ordinary HTML code and spit
-out sample JST code, which is handy during development when inserting things
-like Bootstrap templates into your page. The primary interface to the JSTizer
-would probably be a webapp, although source code would also be made available.
+See the `jstize` module for more information. Essentially the workflow is (i)
+get some HTML in a file `*.html`, it does not need to be a complete page, but
+must be a syntactically valid and complete HTML fragment, (ii) use the JSTizer
+to convert this into an equivalent `*.jst` file, (iii) use the JST system, by
+either the programmatic API or the command-line tool, to convert this into an
+equivalent `*.js` file, (iv) run the `*.js` file to recreate the original HTML.
+Of course, the `*.jst` template can be modified to add any needed extra logic.
## GIT repository
All of our NPM packages are MIT licensed, please see LICENSE in the repository.
We also gratefully acknowledge the `acorn` team for the original MIT license of
-their JavaScript parser, which we've heavily modified to produce the JST parser.
+their JavaScript parser, which we've heavily modified to create the JST parser.
## Contributions
-We would greatly welcome your feedback and contributions. The `jst` is
-under active development (and is part of a larger project that is also under
-development) and thus the API is considered tentative and subject to change. If
-this is undesirable, you could possibly pin the version in your `package.json`.
+The `jst` system is under active development (and is part of a larger project
+that is also under development) and thus the API is tentative. Please go ahead
+and incorporate the system into your project, or try out our example webserver
+built on the system, subject to the caution that the API could change. Please
+send us your experience and feedback, and let us know of improvements you make.
Contact: Nick Downing <nick@ndcode.org>
public / jst.git / commitdiff