-# Acorn
+# JavaScript Template system
-A tiny, fast JavaScript parser written in JavaScript.
+## Overview
-## Community
+The `js_template` package, consisting of the function `js_template()`, parses a
+superset of JavaScript (ES6) which can contain HTML-like constructs, and
+generates plain JavaScript code which in turn, can generate the desired HTML.
-Acorn is open source software released under an
-[MIT license](https://github.com/acornjs/acorn/blob/master/LICENSE).
+The system is inspired by Pug (formerly Jade) templates, but emphasizing the
+JavaScript, for instance you do not need a `-` sign at the start of each line,
+and the parser automatically follows the JavaScript block scope. It is also
+inspired by JSX templates, but the embedded HTML is less verbose, there are no
+closing tags since HTML is bracketed by `{ }` rather than by `<tag>...</tag>`.
-You are welcome to
-[report bugs](https://github.com/acornjs/acorn/issues) or create pull
-requests on [github](https://github.com/acornjs/acorn). For questions
-and discussion, please use the
-[Tern discussion forum](https://discuss.ternjs.net).
+## Template syntax
-## Installation
+### HTML tags in templates
-The easiest way to install acorn is from [`npm`](https://www.npmjs.com/):
+The plain JavaScript in the file is absolutely normal and is expected to use
+CommonJS conventions, for instance require(...) is supported and so forth.
-```sh
-npm install acorn
+The embedded HTML uses a syntax similar to JavaScript's function syntax,
+ `function(arg1, arg2, ...) { ... }`
+except that instead of the word "function" any valid HTML (or other) tag name
+is allowed, and instead of just argument names, HTML attributes of the form
+ `attrname`
+or
+ `attrname=value`
+are supported. In the `attr=value` syntax, the `value` is any valid JavaScript
+expression. No commas are required between attributes, so the parser has to
+automatically detect where one expression ends and another begins (similarly to
+how automatic semicolon insertion works in regular JavaScript).
+
+The `( )` around the attributes is not necessary when there are no attributes.
+Thus a simple HTML file could be written as the following JavaScript template:
+
+```
+html(lang="en") {
+ head {}
+ body {}
+}
```
+and this would translate to the HTML file:
+```
+<html lang="en"><head></head><body></body></html>
+```
+
+For certain particular HTML tags such as `img`, no closing tag is generated in
+the output HTML. However, the empty `{ }` must still be given in the template.
+This is partly for the parser's convenience, since it depends on recognizing
+the `{ }` to distinguish between an HTML template and ordinary JavaScript code.
+It is also more uniform: there is no need to remember which tags are special.
+
+### Regular text in templates
-Alternately, you can download the source and build acorn yourself:
+Regular text is placed in the generated HTML by simply quoting it. That is, if
+a statement is seen consisting of only a JavaScript string or template string,
+the string is converted to HTML by escaping it (helping to guard against any
+HTML injection attacks), and then output as part of some enclosing template.
-```sh
-git clone https://github.com/acornjs/acorn.git
-cd acorn
-npm install
+For example:
```
+html(lang="en") {
+ body {'Hello, world'}
+}
+```
+
+This generates the text:
+```
+<html lang="en"><body>Hello, world</body></html>
+```
+
+Note that in ordinary HTML, certain tags are more sensitive to whitespace than
+others, according to complicated rules about inline block elements and so on.
+This is never an issue with JavaScript templates, we can use as much indenting
+and other whitespace as we want, and only quoted whitespace will be output.
+
+### Shorthand for classes and IDs
+
+The tag name can be followed by `#name` for as shorthand for the HTML attribute
+ `id="name"`
+or by `.name` as shorthand for the HTML attribute
+ `class="name"`
+and these can be repeated as needed, the `id` attribute collects all `#name`
+separated by spaces and the `class` attribute collects all `.name` similarly.
+These must come before any ordinary attributes (before an opening parenthesis).
-## Interface
+### Parser limitations
-**parse**`(input, options)` is the main interface to the library. The
-`input` parameter is a string, `options` can be undefined or an object
-setting some of the options listed below. The return value will be an
-abstract syntax tree object as specified by the [ESTree
-spec](https://github.com/estree/estree).
+Certain tag or attribute names present difficulty since they contain "-" signs
+or other characters invalid in JavaScript identifiers, or they may be reserved
+words in JavaScript. The parser copes with this quite well (many syntax errors
+can already be re-parsed as tag or attribute names), but in difficult cases it
+could be necessary to quote the tag and/or attribute names. For example,
+ `div.class-1.class-2 {}`
+doesn't compile because "1." is a floating point number, for now we write it
+ `div.'class-1'.class-2 {}`
+although we expect that this restriction can be lifted in a future version.
-```javascript
-let acorn = require("acorn");
-console.log(acorn.parse("1 + 1"));
+Another slight limitation of the current parser is that it is more permissive
+than normal in parsing regular JavaScript code, for instance commas are not
+required between function arguments, because HTML templates are basically
+parsed as function calls until we see the opening `{` that identifies it as a
+template. This can also likely be improved in a future `js_template` version.
+
+## Expression vs statement templates
+
+HTML templates occuring at expression level will be completely rendered to a
+string, and the resulting string returned as the value of the expression.
+
+HTML templates occurring at statement level are treated somewhat like quoted
+strings, in that the generated HTML will become part of an enclosing template.
+
+Here is a complete example showing template substitution and HTML expressions:
+```
+let lang = 'en'
+let name = 'John'
+console.log(
+ html(lang=lang) {
+ body {`Hello, ${name}`}
+ }
+}
```
-When encountering a syntax error, the parser will raise a
-`SyntaxError` object with a meaningful message. The error object will
-have a `pos` property that indicates the string offset at which the
-error occurred, and a `loc` object that contains a `{line, column}`
-object referring to that same position.
+Running the above program will generate the output:
+```
+<html lang="en"><body>Hello, John</body></html>
+```
-Options can be provided by passing a second argument, which should be
-an object containing any of these fields:
+## Template output buffer
-- **ecmaVersion**: Indicates the ECMAScript version to parse. Must be
- either 3, 5, 6 (2015), 7 (2016), 8 (2017), 9 (2018) or 10 (2019, partial
- support). This influences support for strict mode, the set of
- reserved words, and support for new syntax features. Default is 7.
+It is also possible to use statement-level HTML templates or strings to build
+up an output buffer, which can be used for whatever purpose. The output buffer
+must be called _out and it must be a JavaScript array of string fragments. As
+each output fragment (such as an opening or closing tag or an embedded string)
+is generated, it will be sent to the output buffer by an `_out.push(...)` call.
- **NOTE**: Only 'stage 4' (finalized) ECMAScript features are being
- implemented by Acorn. Other proposed new features can be implemented
- through plugins.
+If there is dynamically generated text in the template, then there must be a
+function `_html_escape()` provided in the current scope, that escapes the text.
-- **sourceType**: Indicate the mode the code should be parsed in. Can be
- either `"script"` or `"module"`. This influences global strict mode
- and parsing of `import` and `export` declarations.
+For example, consider a realistic template which we use on our web server:
+```
+html(lang=lang) {
+ head {
+ link(rel="stylesheet" type="text/css" href="css/styles.css") {}
+ }
+ body {
+ p {`Hello, ${name}`}
+ }
+}
+```
-- **onInsertedSemicolon**: If given a callback, that callback will be
- called whenever a missing semicolon is inserted by the parser. The
- callback will be given the character offset of the point where the
- semicolon is inserted as argument, and if `locations` is on, also a
- `{line, column}` object representing this position.
+This compiles to the following plain JavaScript code:
+```
+_out.push("<html lang=\"" + _html_escape(lang.toString()) + "\">");
+{
+ _out.push("<head>");
+ _out.push("<link rel=\"stylesheet\" type=\"text/css\" href=\"css/styles.css\">");
+ _out.push("</head>");
+}
+{
+ _out.push("<body>");
+ {
+ _out.push("<p>");
+ _out.push(_html_escape(`Hello, ${name}`));
+ _out.push("</p>");
+ }
+ _out.push("</body>");
+}
+_out.push("</html>");
+```
-- **onTrailingComma**: Like `onInsertedSemicolon`, but for trailing
- commas.
+If invoked as an expression, the same code will be generated, but wrapped in an
+Immediately-Invoked Function Expression (IIFE) that creates the `_out` buffer,
+executes the above, then returns the buffer concatenated into a single string.
-- **allowReserved**: If `false`, using a reserved word will generate
- an error. Defaults to `true` for `ecmaVersion` 3, `false` for higher
- versions. When given the value `"never"`, reserved words and
- keywords can also not be used as property names (as in Internet
- Explorer's old parser).
+## Template dependencies
-- **allowReturnOutsideFunction**: By default, a return statement at
- the top level raises an error. Set this to `true` to accept such
- code.
+Templates may depend on other templates. Templates are imported imported using
+the syntax `_require(...)`, similarly to the syntax `require(...)` in CommonJS.
-- **allowImportExportEverywhere**: By default, `import` and `export`
- declarations can only appear at a program's top level. Setting this
- option to `true` allows them anywhere where a statement is allowed.
-
-- **allowAwaitOutsideFunction**: By default, `await` expressions can
- only appear inside `async` functions. Setting this option to
- `true` allows to have top-level `await` expressions. They are
- still not allowed in non-`async` functions, though.
+We use `_require()` for templates instead of the normal `require()` because:
-- **allowHashBang**: When this is enabled (off by default), if the
- code starts with the characters `#!` (as in a shellscript), the
- first line will be treated as a comment.
+(1) We do not want to interfere with other symbols in the namespace, you can
+also use `require()` in your templates to import ordinary CommonJS modules;
-- **locations**: When `true`, each node has a `loc` object attached
- with `start` and `end` subobjects, each of which contains the
- one-based line and zero-based column numbers in `{line, column}`
- form. Default is `false`.
+(2) Absolute paths have a different meaning in `_require()` versus `require()`,
+for `_require()` they will be taken relative to the webserver's document root;
-- **onToken**: If a function is passed for this option, each found
- token will be passed in same format as tokens returned from
- `tokenizer().getToken()`.
+(3) Using `_require()` says that the file is in `*.jst` format, and hence it
+should be parsed and any HTML constructs converted into regular JavaScript; and
- If array is passed, each found token is pushed to it.
+(4) The template importing is an asynchronous operation, so you normally use
+`await _require(...)` whereas `require()` is synchronous. Because templates can
+be changed and recompiled automatically while the webserver is serving pages,
+we don't want to stop the world, especially as templates are slow to compile.
- Note that you are not allowed to call the parser from the
- callback—that will corrupt its internal state.
+A slight difference between CommonJS exports and JavaScript template exports,
+is that where CommonJS modules set the dedicated symbol `module.exports` to
+whatever is exported, JavaScript templates instead `return` what is exported.
-- **onComment**: If a function is passed for this option, whenever a
- comment is encountered the function will be called with the
- following parameters:
+## Multi-part template examples
- - `block`: `true` if the comment is a block comment, false if it
- is a line comment.
- - `text`: The content of the comment.
- - `start`: Character offset of the start of the comment.
- - `end`: Character offset of the end of the comment.
+### Templating with HTML text strings
- When the `locations` options is on, the `{line, column}` locations
- of the comment’s start and end are passed as two additional
- parameters.
-
- If array is passed for this option, each found comment is pushed
- to it as object in Esprima format:
+Here is an example of a template which takes some text and wraps it in the
+standard HTML tags to create a page. Many templating systems work like this,
+so that you can put standard scripts, viewport commands etc, in one place.
- ```javascript
- {
- "type": "Line" | "Block",
- "value": "comment text",
- "start": Number,
- "end": Number,
- // If `locations` option is on:
- "loc": {
- "start": {line: Number, column: Number}
- "end": {line: Number, column: Number}
- },
- // If `ranges` option is on:
- "range": [Number, Number]
+Save this as `page.jst`, as it's the template for all pages in the system:
+```
+return body_text => html {
+ head {
+ link(rel="stylesheet" type="text/css" href="css/styles.css") {}
+ }
+ body {
+ _out(body_text)
}
- ```
-
- Note that you are not allowed to call the parser from the
- callback—that will corrupt its internal state.
-
-- **ranges**: Nodes have their start and end characters offsets
- recorded in `start` and `end` properties (directly on the node,
- rather than the `loc` object, which holds line/column data. To also
- add a
- [semi-standardized](https://bugzilla.mozilla.org/show_bug.cgi?id=745678)
- `range` property holding a `[start, end]` array with the same
- numbers, set the `ranges` option to `true`.
-
-- **program**: It is possible to parse multiple files into a single
- AST by passing the tree produced by parsing the first file as the
- `program` option in subsequent parses. This will add the toplevel
- forms of the parsed file to the "Program" (top) node of an existing
- parse tree.
-
-- **sourceFile**: When the `locations` option is `true`, you can pass
- this option to add a `source` attribute in every node’s `loc`
- object. Note that the contents of this option are not examined or
- processed in any way; you are free to use whatever format you
- choose.
-
-- **directSourceFile**: Like `sourceFile`, but a `sourceFile` property
- will be added (regardless of the `location` option) directly to the
- nodes, rather than the `loc` object.
-
-- **preserveParens**: If this option is `true`, parenthesized expressions
- are represented by (non-standard) `ParenthesizedExpression` nodes
- that have a single `expression` property containing the expression
- inside parentheses.
-
-**parseExpressionAt**`(input, offset, options)` will parse a single
-expression in a string, and return its AST. It will not complain if
-there is more of the string left after the expression.
-
-**tokenizer**`(input, options)` returns an object with a `getToken`
-method that can be called repeatedly to get the next token, a `{start,
-end, type, value}` object (with added `loc` property when the
-`locations` option is enabled and `range` property when the `ranges`
-option is enabled). When the token's type is `tokTypes.eof`, you
-should stop calling the method, since it will keep returning that same
-token forever.
-
-In ES6 environment, returned result can be used as any other
-protocol-compliant iterable:
-
-```javascript
-for (let token of acorn.tokenizer(str)) {
- // iterate over the tokens
}
-
-// transform code to array of tokens:
-var tokens = [...acorn.tokenizer(str)];
```
+Save this as `index.html.jst`, as it's the template to generate `index.html`:
+```
+let page = await _require('/page.jst')
-**tokTypes** holds an object mapping names to the token type objects
-that end up in the `type` properties of tokens.
-
-**getLineInfo**`(input, offset)` can be used to get a `{line,
-column}` object for a given program string and offset.
+return page(p {'Hello, world'})
+```
+Note that here we have somewhat hacked into the internals of the templating
+system by directly appending the body text onto the output buffer `_out`. It is
+intentional that you can access the internals for efficiency reasons, and that
+you can, for example, avoid HTML escaping of particular text where appropriate.
-### The `Parser` class
+### Templating with callbacks
-Instances of the **`Parser`** class contain all the state and logic
-that drives a parse. It has static methods `parse`,
-`parseExpressionAt`, and `tokenizer` that match the top-level
-functions by the same name.
+Another way, which is more sophisticated, is to use a callback function to
+generate the body. If doing this, we pass the output buffer _out into the
+callback function, so that the entire HTML of the page can be generated in an
+unbroken stream, without having to concatenate the partial page repeatedly:
-When extending the parser with plugins, you need to call these methods
-on the extended version of the class. To extend a parser with plugins,
-you can use its static `extend` method.
+The page wrapper template `page.jst` which takes a body callback and calls it:
+```
+return body_callback => html {
+ head {
+ link(rel="stylesheet" type="text/css" href="css/styles.css") {}
+ }
+ body {
+ body_callback(_out)
+ }
+}
+```
+Then the specific page template `index.html.jst` which provides the callback:
+```
+let page = await _require('/page.jst')
-```javascript
-var acorn = require("acorn");
-var jsx = require("acorn-jsx");
-var JSXParser = acorn.Parser.extend(jsx());
-JSXParser.parse("foo(<bar/>)");
+return html(
+ _out => {
+ p {'Hello, world'}
+ }
+)
```
-The `extend` method takes any number of plugin values, and returns a
-new `Parser` class that includes the extra parser logic provided by
-the plugins.
+Note that in the above example, the `{ }` around the callback function body is
+essential (unlike in regular JavaScript), to make `p {...}` occur in statement
+rather than expression context. As an expression it would return a string which
+would be ignored, since the page template is expecting output to be in `_out`.
+
+## File management
+
+The `js_template()` function gives a "point and shoot" interface which makes it
+very easy to manage on-disk templates. You essentially just state the pathname
+to the template, which is normally a `*.jst` file, and the template will be
+parsed, converted to JavaScript, evaluated, and also cached for future use.
+
+What the `js_template()` function returns depends on what the template returns.
+Template exports are similar but slightly different to CommonJS module exports,
+as noted in examples above. Usually, `js_template()` returns a JavaScript
+function, which you call to generate HTML each time the page is to be served.
+
+The HTML-generating function is re-useable in this way for efficiency reasons,
+since after the initial compilation, the cached version can be reused each time
+the page is served, but called with different arguments, resulting in different
+substitutions made on the page. For example, consider the following template:
+```
+return (lang, name) => html(lang=lang) {
+ body {
+ p {`Hello, ${name}`}
+ }
+}
+```
+Suppose the above is saved as `/hello/index.html.jst` under your document root,
+`/var/www/html`. You call `js_template(root, dirname, pathname)`, like this:
+```
+let template_func = js_template(
+ '/var/www/html',
+ '/var/www/html/hello',
+ 'index.jst'
+)
+```
-## Command line interface
+The result is a JavaScript function of 2 arguments, which you call like this:
+```
+console.log(
+ template_func('en', 'world')
+)
+```
-The `bin/acorn` utility can be used to parse a file from the command
-line. It accepts as arguments its input file and the following
-options:
+The generated HTML from the above example, printed on the console, would be:
+```
+<html lang="en"><body><p>Hello, world</p></body></html>
+```
-- `--ecma3|--ecma5|--ecma6|--ecma7|--ecma8|--ecma9|--ecma10`: Sets the ECMAScript version
- to parse. Default is version 9.
+The reason why you provide the root and working directory separately, is that
+the template may import further dependencies using `_require()`, and these are
+taken relative to `root` or `dirname` depending on whether they begin with `/`.
-- `--module`: Sets the parsing mode to `"module"`. Is set to `"script"` otherwise.
+Note that any imported modules receive an updated `dirname`, so that relative
+imports are taken relative to the importing module, similarly to in CommonJS.
-- `--locations`: Attaches a "loc" object to each node with "start" and
- "end" subobjects, each of which contains the one-based line and
- zero-based column numbers in `{line, column}` form.
+## Memory vs disk caching of templates
-- `--allow-hash-bang`: If the code starts with the characters #! (as
- in a shellscript), the first line will be treated as a comment.
+Templates compiled using `js_template()` are cached in memory, as long as the
+same `node` interpreter is running, so that they can be retrieved using either
+`js_template()`, or equivalently `_require()` inside a template, and they will
+not be re-executed. In the above example, if you call `js_template()` twice,
+you get the same object twice (it is a 2-argument function returning `String`).
-- `--compact`: No whitespace is used in the AST output.
+As well as this, the compiled templates are also cached on disk, which requires
+the document root to be writeable. For instance if you compile `index.html.jst`
+in a `hello/` subdirectory of the document root, you get a hidden file with the
+same name and a `.js` subscript, here it would be `hello/.index.html.jst.js`.
-- `--silent`: Do not output the AST, just return the exit status.
+The main reason for the disk caching is really to fool the node.js `require()`
+system into using correct relative paths, if the template uses `require()` as
+well as `_require()`. However, it is also handy that the templates only need to
+be re-evaluated and not recompiled when the webserver is stopped and restarted.
-- `--help`: Print the usage information and quit.
+Before using either a disk-cached or a memory-cached template, the modification
+times are checked and the template is recompiled if it is stale. Thus you can
+edit your website while it is live, and each page will be recompiled as needed
+each time it is served. Templates deleted on disk are deleted from the cache.
-The utility spits out the syntax tree as JSON data.
+Note: deletion of stale pages from the memory cache is not implemented yet (it
+will use a long timeout such as one week), so the webserver should be restarted
+occasionally. This will be fixed if it becomes an issue in practice. Also, if
+the document root is not writeable, there is a simple expedient of making sure
+that all the needed `.jst.js` files are up to date, so no write is attempted.
-## Existing plugins
+## Conclusions
- - [`acorn-jsx`](https://github.com/RReverser/acorn-jsx): Parse [Facebook JSX syntax extensions](https://github.com/facebook/jsx)
-
-Plugins for ECMAScript proposals:
-
- - [`acorn-stage3`](https://github.com/acornjs/acorn-stage3): Parse most stage 3 proposals, bundling:
- - [`acorn-async-iteration`](https://github.com/acornjs/acorn-async-iteration): Parse [async iteration proposal](https://github.com/tc39/proposal-async-iteration)
- - [`acorn-bigint`](https://github.com/acornjs/acorn-bigint): Parse [BigInt proposal](https://github.com/tc39/proposal-bigint)
- - [`acorn-class-fields`](https://github.com/acornjs/acorn-class-fields): Parse [class fields proposal](https://github.com/tc39/proposal-class-fields)
- - [`acorn-dynamic-import`](https://github.com/kesne/acorn-dynamic-import): Parse [import() proposal](https://github.com/tc39/proposal-dynamic-import)
- - [`acorn-import-meta`](https://github.com/acornjs/acorn-import-meta): Parse [import.meta proposal](https://github.com/tc39/proposal-import-meta)
- - [`acorn-numeric-separator`](https://github.com/acornjs/acorn-numeric-separator): Parse [numeric separator proposal](https://github.com/tc39/proposal-numeric-separator)
- - [`acorn-private-methods`](https://github.com/acornjs/acorn-private-methods): parse [private methods, getters and setters proposal](https://github.com/tc39/proposal-private-methods)n
+Further detailed documentation and tutorials will be added to the project in
+due course. For the present, the software is in active development, so it would
+be premature to document everything exactly as it is now. Please do contact the
+author Nick Downing <nick@ndcode.org>, should you have questions or feedback.
public / jst.git / commitdiff