2020-01-22 | CB | Cruft Buster Log Format
Cruft Buster data is a tree that is modified locally, and includes symbolic links that make the tree a graph. Logs can be thought of in multiple ways, one of which is observing the incremental changes to a system. Log entries are sometimes stored in a database and sometimes stored in a flat file. For the purposes of Cruft Buster, though, it makes sense to store the log nestled in the leaves as well as in a traditional flat file. The format of a log entry in the leaves looks like this:
This is a UTC time, username, hostname, and changed item, delimited by ^^.
To make the system as robust as possible, this is generated by relatively stock Unison, with some small modifications to get the format standard. The modified source can be compiled easily on most systems, and is available from here and here. The MD5 hash is daa5b23a143d7a2c6b5d19bb412e897a. Changes are replicated to a central hub using this command:
unison -terse -auto -batch
And this configuration file:
knoppix@thick:~/.unison$ cat default.prf ignore = Name .* backupnot = Name Z*^^*^^*^^* backup = BelowPath source backupcurrnot = Name * backuplocation = local maxbackups = 200 ui = text perms = 0 rsrc = false include root knoppix@thick:~/.unison$ cat root.prf root = /home/knoppix/websites root = ssh://email@example.com//home/knoppix/websites
Additionally, the modified Unison creates log entries like this:
L1G3Rmod-UTC-2020-05-02T16:12:52.587Z[END] Deleting source/trb/0/1/1/20200403T212543.787Z^^knoppix^^thick^^narrative.md L1G3Rmod-UTC-2020-05-02T16:12:52.587Z[END] Deleting source/trb/0/1/1/20200403T212559.421Z^^knoppix^^thick^^narrative.md L1G3Rmod-UTC-2020-05-02T16:12:52.587Z[END] Deleting source/trb/0/1/1/20200406T023419.977Z^^knoppix^^backroom^^narrative.md L1G3Rmod-UTC-2020-05-02T16:12:52.587Z[END] Deleting source/trb/0/1/1/has_tag/logging.md L1G3Rmod-UTC-2020-05-02T16:12:52.587Z[END] Deleting source/trb/0/2.title.txt L1G3Rmod-UTC-2020-05-02T16:12:52.587Z[END] Deleting source/trb/0/3 L1G3Rmod-UTC-2020-05-02T16:12:52.587Z[END] Deleting source/trb/0/3.title.txt L1G3Rmod-UTC-2020-05-02T16:12:52.587Z[END] Deleting source/trb/0/tags/logging.md Synchronization complete at 09:12:52 (296 items transferred, 0 skipped, 0 failed) knoppix@thick:~/websites/site/log/files$
These can be parsed and fed to other more traditional log ingestion and analysis programs. Note that the modified Unison logs here stick everything on one line instead of breaking up the entries on multiple lines like stock Unison.
Generally, nothing gets pushed up unless it is done intentionally, since rsync is using the -n option. This makes it easier to code in stages if some of the other rsync flags need to be added or dealt with. Also, the source code for all components will be in a live state on Tributary Software.