return async env => {
- let post = await _require('/_lib/post.jst')
+ let blog_post = await _require('/_lib/blog_post.jst')
- await post(
+ await blog_post(
env,
// head
async _out => {},
// body
async _out => {
- p {'Normally I would write a blog post coincident with the release of a new version of the public website you are reading; in this case there is some heavy refactoring going on and so I have decided to just document the process of the work and let it be viewed and interacted with later when ready for release.'}
+ p {
+ 'Normally I would write a blog post coincident with the release of a new version of the public website you are reading; in this case there is some heavy refactoring going on and so I have decided to just document the process of the work and let it be viewed and interacted with later when ready for release.'
+ }
h4 {'NoSQL discussion'}
- p {'One of the early decisions I made when teaching myself web development was that I would not be using SQL for the backend. I simply believe the disadvantages outweigh the advantages for what I am trying to do. And I’m probably not the only person with this view, because the NoSQL movement has been gaining traction.'}
+ p {
+ 'One of the early decisions I made when teaching myself web development was that I would not be using SQL for the backend. I simply believe the disadvantages outweigh the advantages for what I am trying to do. And I’m probably not the only person with this view, because the NoSQL movement has been gaining traction.'
+ }
p {
'SQL pros:'
}
}
- p {'I want my application to be as self-contained and self-configuring as possible, so basically I would just check out the website’s repository on the intended Virtual Private Server (VPS) and launch it. In a traditional configuration you have a long list of other services to configure first, such as PHP, PHP-FPM, MySQL, etc, etc, and setting up or migrating to a new server could be days of work with tedious troubleshooting. There are of course cases where these activities have a payoff (such as if you require a highly scalable website), but my case is not one of them.'}
+ p {
+ 'I want my application to be as self-contained and self-configuring as possible, so basically I would just check out the website’s repository on the intended Virtual Private Server (VPS) and launch it. In a traditional configuration you have a long list of other services to configure first, such as PHP, PHP-FPM, MySQL, etc, etc, and setting up or migrating to a new server could be days of work with tedious troubleshooting. There are of course cases where these activities have a payoff (such as if you require a highly scalable website), but my case is not one of them.'
+ }
h4 {'First try—all JSON approach'}
', and I would then have access to complex queries which I do not with JSON, but I felt the added complexity was not worthwhile. If I need to do complex queries, I will simply write them in Javascript, including the maintenance of any indices needed to carry out the queries efficiently. This will give me complete control, rather than simply hoping the SQL server optimizes the indices and queries well.'
}
- p {'My JSON database was sufficient for building the prototype blog and online store that you can interact with now, but it did suffer from a serious problem which is that upon successful launch, there could be thousands of customers and/or orders and the database could potentially grow to a huge size. Then it would not be feasible to hold the entire file in memory or write it every 5 seconds.'}
+ p {
+ 'My JSON database was sufficient for building the prototype blog and online store that you can interact with now, but it did suffer from a serious problem which is that upon successful launch, there could be thousands of customers and/or orders and the database could potentially grow to a huge size. Then it would not be feasible to hold the entire file in memory or write it every 5 seconds.'
+ }
h4 {'Second try—log-structured JSON approach'}
- p {'To solve the scalability issue, my plan has always been to allow the JSON tree to be read into memory lazily as required (and cached, with some kind of cache policy implemented), and for modifications to be written to the end of the file. Thus the file would be log-structured and simply keep growing forever, except for the intervention of a daily cleaning routine which would copy all the live (still reachable) parts of the log into a new log to reduce the size.'}
+ p {
+ 'To solve the scalability issue, my plan has always been to allow the JSON tree to be read into memory lazily as required (and cached, with some kind of cache policy implemented), and for modifications to be written to the end of the file. Thus the file would be log-structured and simply keep growing forever, except for the intervention of a daily cleaning routine which would copy all the live (still reachable) parts of the log into a new log to reduce the size.'
+ }
p {
'The database I created with this approach is called '
', is to allocate new objects contiguously from a region of spare memory instead of scanning for a free spot every time, and perform a periodic garbage collection to reclaim free space. It was controversial at the time because of garbage collection pauses, but if you can perform garbage collection concurrently with normal processing, there should be no significant impact.'
}
- p {'For my application, it seemed simpler to implement a daily copy from old to new log than to deal with free-space management. And there are certain other benefits, such as having a series of snapshots of the database to aid in trouble-shoooting. I also liked the idea of the log being human-readable (traditional for web developers, who like ASCII formats such as JSON rather than binary formats), and this would have complicated the free-space management considerably.'}
+ p {
+ 'For my application, it seemed simpler to implement a daily copy from old to new log than to deal with free-space management. And there are certain other benefits, such as having a series of snapshots of the database to aid in trouble-shoooting. I also liked the idea of the log being human-readable (traditional for web developers, who like ASCII formats such as JSON rather than binary formats), and this would have complicated the free-space management considerably.'
+ }
h4 {
tt {'LogJSON'}
', containing the file pointer and length of the previously written JSON.'
}
- p {'So in the above example, it’s possible to open the file, synchronize to the root element using the angle brackets, and then read the root element as a reference to an unread portion of the file, without actually having to read the contents of the root element until they are required.'}
+ p {
+ 'So in the above example, it’s possible to open the file, synchronize to the root element using the angle brackets, and then read the root element as a reference to an unread portion of the file, without actually having to read the contents of the root element until they are required.'
+ }
h5 {'Nested objects'}
`
}
- p {'And the encoded version in LogJSON is:'}
+ p {
+ 'And the encoded version in LogJSON is:'
+ }
pre {
`{
`
}
- p {'Finally, we need to rewrite the root object to point to the newly written subtree:'}
+ p {
+ 'Finally, we need to rewrite the root object to point to the newly written subtree:'
+ }
pre {
`<[
`
}
- p {'And of course the file will not be considered modified (the old root will be active) until the new root has been fully written.'}
+ p {
+ 'And of course the file will not be considered modified (the old root will be active) until the new root has been fully written.'
+ }
h5 {'Modified tree‒full example'}
h4 {'Garbage collection'}
- p {'I have implemented the garbage collection in a rather sophisticated way, taking advantage of the immutability of the log file once written in order that garbage collection can proceed in the background without disturbing the operation of the website.'}
+ p {
+ 'I have implemented the garbage collection in a rather sophisticated way, taking advantage of the immutability of the log file once written in order that garbage collection can proceed in the background without disturbing the operation of the website.'
+ }
- p {'A garbage collection is triggered every midnight in Universal Time Coordinated (UTC), that is, when the date rolls over. A new log file is constructed by copying all the live data from the old log file into it. And then transaction processing is briefly paused whilst we atomically rename the old log file out of the way to a dated name (contains the year, month and day in UTC) and rename the new log file over the old one. If the system crashes during renaming it will recover at the next restart.'}
+ p {
+ 'A garbage collection is triggered every midnight in Universal Time Coordinated (UTC), that is, when the date rolls over. A new log file is constructed by copying all the live data from the old log file into it. And then transaction processing is briefly paused whilst we atomically rename the old log file out of the way to a dated name (contains the year, month and day in UTC) and rename the new log file over the old one. If the system crashes during renaming it will recover at the next restart.'
+ }
- p {'The synchronization magic happens by atomically grabbing the current root object and copying everything reachable from it into the new log file, but allowing more transactions to be written into the old log file simultaneously. It then atomically grabs the updated root and copies that as well, and continues to do so until both files are up-to-date and it can perform the atomic replacement.'}
+ p {
+ 'The synchronization magic happens by atomically grabbing the current root object and copying everything reachable from it into the new log file, but allowing more transactions to be written into the old log file simultaneously. It then atomically grabs the updated root and copies that as well, and continues to do so until both files are up-to-date and it can perform the atomic replacement.'
+ }
h4 {'Concurrency'}
' as not conflicting.'
}
- p {'Currently there is a bit of a risk that if I do not implement good try/catch handling in my code, then a webserver request will crash, leaving the webserver running but the mutex in ‘held’ state. This would hang the webserver, requiring a manual restart. I could implement some sort of disaster recovery, but there is probably no point at this time, I will simply use try/catch and make sure transactions are rolled back if not completed.'}
+ p {
+ 'Currently there is a bit of a risk that if I do not implement good try/catch handling in my code, then a webserver request will crash, leaving the webserver running but the mutex in ‘held’ state. This would hang the webserver, requiring a manual restart. I could implement some sort of disaster recovery, but there is probably no point at this time, I will simply use try/catch and make sure transactions are rolled back if not completed.'
+ }
h4 {'Conclusion'}
- p {'The log-structured format provides a useful way of maintaining huge JSON files. It remains to complete the documentation and release the source code and NPM package. It also remains to implement a stress test of my website, once up and running, to simulate thousands of user accounts and multiple logins to the same account all performing transactions simultaneously. Therefore, things are still experimental at this stage. I enjoy writing this blog to update on those experiments.'}
+ p {
+ 'The log-structured format provides a useful way of maintaining huge JSON files. It remains to complete the documentation and release the source code and NPM package. It also remains to implement a stress test of my website, once up and running, to simulate thousands of user accounts and multiple logins to the same account all performing transactions simultaneously. Therefore, things are still experimental at this stage. I enjoy writing this blog to update on those experiments.'
+ }
},
// scripts
async _out => {}
public / ndcode_site.git / blobdiff