cyclog — safely log standard input to status files and log directories
cyclog [--max-file-size max-file-size] [--max-total-size max-total-size] [--margin margin] {directories...}
cyclog is a simple logging utility that logs everything that it receives on its standard input to a set of log file directories named as its command arguments. It automatically timestamps logs, produces time-sortable log file names, rotates log files, and caps log directory sizes.
At startup, cyclog attempts to open all of the directories.
If they are directories (or symbolic links to the same), it treats them as log directories.
Otherwise, if they don't exist or are other types of file such as regular files, device files, or FIFOs, it aborts startup without reading from standard input.
It keeps a file descriptor open to each log directory for its lifetime, and accesses files relative to that open file descriptor. Log directories can thus be renamed as it is running, and it will continue to use the original directory, wherever it is renamed to.
A log directory is compatible with the log directories employed by daemontools' and daemontools-encore's multilog(1), s6's s6-log(1), and runit's svlog(1). It contains:
a lock file (compatible with setlock(1)) that prevents two logging processes from sharing an active log directory;
a set of old log files, named as @ followed by a TAI64N timestamp in external form (16 hexadecimal digits of seconds and 8 hexadecimal digits of nanoseconds) and either .s (safely written to disc) or .u (aborted whilst writing to disc and possibly incomplete);
and
a current file where incoming log data are currently being appended.
An active current file, or an old log file or a current file that was not written to disc by cyclog properly shutting down, has permissions 0644.
A current file or an old log file where cyclog properly shut down has permissions 0744.
(See the Compatibility section.)
cyclog shuts down when either it sees EOF on its standard input, or it receives one of the "good" termination signals (SIGTERM, SIGPIPE, SIGINT, or SIGHUP).
At shut down, cyclog writes all pending data that it has already read from standard input, flushes current to disc, and changes current's permissions.
At start up, it checks current's permissions, creating the file with permissions 0744 if it does not already exist.
If they indicate proper shutdown, as they will also do if the file was created because it did not exist, it carries on.
If they indicate that the file was not properly written at shutdown, it performs automatic log rotation to prevent appending to a potentially corrupt file (that may be sparse or contain ranges of zeroed blocks).
In the normal case, the standard input of cyclog will be a pipe; and both ends of the the pipe will be open in a long-lived supervisor process such as service-manager(1). cyclog, as aforementioned, ensures that it writes all data read from its standard input, even if it is terminated by a (good) signal.
Thus no log data are lost, even if cyclog is shut down and restarted.
cyclog performs automatic log rotation when it writes a linefeed within margin bytes of the maximum size (max-file-size) of the current file, when it is about to exceed that size, when it receives a SIGALRM, or when it finds an improperly written to disc current file at startup.
When recovering from an improperly finalized current, it simply renames it to a timestamped .u name.
Otherwise, it renames it to a timestamped .u name, flushes it to disc, changes its permissions, and then renames it to a timestamped .s name.
In both cases, it then creates a new current file.
The TAI64N timestamp of an old log file is the timestamp of when cyclog rotated current to that file.
At log rotation, and also at startup, it checks to ensure that the total size of all log files in the directory does not exceed the maximum total size (max-total-size).
(It only totals the sizes of current and old log files.
Other files, not managed or created by cyclog, are ignored.)
If the total exceeds that maximum, it deletes each old log file with the numerically lowest name until either the total is less than the maximum or there is only the current file left.
Thus the maximum size of all log files at any time is max-total-size (the total size after the last rotation) plus max-file-size (the data written since that rotation).
The default max-file-size is 16MiB and the default max-total-size is 1GiB.
cyclog totals file sizes rather than space allocated on disc. The amount of space allocated to all files may be, depending from the filesystem type and the maxima chosen, higher or lower than the space usage calculated by cyclog.
cyclog writes a timestamp at the beginning of every line written to current, which is the time when it processes the beginning of a line.
This is slightly after it has read the beginning of that line.
The timestamp is in TAI64N external form (16 hexadecimal digits of seconds and 8 hexadecimal digits of nanoseconds), which can be converted to human-readable form using tai64nlocal(1).
cyclog uses the CLOCK_REALTIME clock of the clock_gettime(2) system call.
On many systems this has nanosecond resolution.
cyclog attempts to generate real TAI timestamps. On a Linux system where it detects an Olson "right" timezone currently in use, it knows that the system clock is TAI seconds since the Epoch and performs a simple conversion. On other Linux systems, and on BSDs, it assumes that the system clock is UTC seconds since the Epoch and attempts to correct for (known) UTC leap seconds in order to determine TAI.
A cyclog(1) utility was part of daemontools version 0.53. This utility is a workalike, with some minor additions to automatic log rotation and changes to log directory semantics.
daemontools later replaced cyclog(1) with multilog(1), which treats its command arguments as a script and has the abilities to do pattern matching and run external commands. cyclog is intended for the commonest, simple, use cases of logging services where there is just one plain logging directory with no bells and whistles. As such, it does not implement a script syntax, does not have pattern matching, and does not invoke other programs at all.
The daemontools version 0.53 cyclog(1) used permissions 0644 and 0444 to flag files that had not been written to disc. cyclog adopts the convention employed by the later multilog(1), and thence s6-log(1) and svlog(1). This convention changed to using the owner execute permission bit rather than the owner write permission bit as the flag. cyclog sets and clears the execution bit as appropriate for interoperability with those tools, and uses it itself for the same purpose. multilog(1) altered the convention in order to cope with scenarios where it would end up trying to open a read-only file for writing.
The daemontools version 0.53 cyclog(1) used the TAI64 timestamp of when a log file was started as its timestamp. Again cyclog adopts the convention employed by later tools.
daemontools' and daemontools-encore's multilog(1) uses a timer with only microsecond resolution, and for the same events will thus produce different timestamps to cyclog.
See the timestamps section of the nosh Guide for detailed information on the differences from daemontools, daemontools-encore, and other toolsets in how cyclog handles TAI.