Analog CE 6.0: Cache files

For most people, the cache file will not be needed: compressing the logfile using a standard compression utility such as gzip will be sufficient. Compressing a logfile is very efficient owing to the large number of repeated strings: I find about 12 times compression in practice. That in itself may solve your filespace problems, without needing to throw away any information.

The cache file is also not the best format for post-processing the data or feeding it into a spreadsheet. For that you should use the computer-readable output style.

Many people have trouble using the cache file, and end up accidentally recording corrupt data. You do need to understand what you're doing before you throw away your logfiles. See the discussion on Procedures below.

If you are going to use the cache file feature, it is also very important that you understand what is and what is not recorded. The summary is that all INCLUDE and EXCLUDE commands, including FROM and TO, and any ALIASes and LOGTIMEOFFSETs, must be applied when you create the cache file, not when you read it later. If you want different sets of options, you must create several cache files from the same logfile.

The reason for this is that it is not possible to reconstruct everything of interest in the logfile from the cache file. The cache file does contain information about the total number of requests for each host and each file, but not about, for example, which files were read by which hosts. (To do so would take up as much disk space as the compressed logfile.) So you cannot later look at only one file and see which hosts read that file.

Another way to look at this: if you do, for example, a HOSTEXCLUDE when reading the cache file, you are not doing a genuine HOSTEXCLUDE because files that that host read will still be included. You are only excluding those hosts from the Host Report, Organisation Report and Domain Report. This is why you must do all the inclusions and exclusions you want when you create the cache file.

When analog reads in a cache file, it does not apply any more aliases to the items. This is to avoid double-aliasing. So you must do any aliases you want at the time you create the cache file. Similarly, it does not obey the LOGTIMEOFFSET variable, to avoid double-offsetting, so any offset you want must be applied at cache-creation time too.

Also, the cache file does not contain data about the number of requests for each item in the last seven days: it can't, because the figures will be different by the time they are wanted.

Finally, times are only recorded to five-minute resolution.

CACHEOUTFILE none

You can read in a previously-made cache file with the CACHEFILE command, or with the +U command line option. This works exactly the same as the LOGFILE command, so you can use commas and wildcards to read in several cache files, and read compressed cache files using the UNCOMPRESS mechanism.

If the name of the CACHEFILE or the CACHEOUTFILE doesn't include a directory, it will be looked for, or written to, wherever analog expects to find its cache files. (This location is built in when the program is compiled.) For example, on Windows it would be in the same folder as the analog executable. But a cache file specified in a +U command line option is within the current directory.

It is possible (and useful) to analyse some CACHEFILEs and some LOGFILEs together. LOGFILE and CACHEFILE commands are basically cumulative, except that any logfiles and cache files in the mandatory configuration file or configuration files loaded from there override any on the command line or in configuration files specified on the command line, which themselves override any in the default configuration file or configuration files loaded from there, which in turn override compile-time options. Usually you don't need to worry about this, and it will do what you expect.

Sometimes you don't want to record all the types of item in the cache file. You might want to forget about which hosts had accessed your web site, for example, and only remember how many times each file was requested. You can choose not to include one type of item in the cache file by setting its LOWMEM to 3; for example, specify

HOSTLOWMEM 3

Procedures

Here is one way to use the cache files correctly. It's not the only correct way, but I think it's conceptually the simplest. The idea is that whenever you start a new logfile, you make a cache file out of the old logfile. So each cache file contains all the data from one, and only one, logfile. You never use old cache files to make new ones: so you never have a CACHEFILE and a CACHEOUTFILE command in the same configuration file.

Here is the procedure.

1. Rotate your logs: that means, archive the old logfile, and restart the server with a fresh logfile. (There are several standard tools to do this: or see your server documentation.)
2. Make both a cache file and an ordinary report from the old logfile. You can do this simultaneously by using one LOGFILE command, one OUTFILE command, and one CACHEOUTFILE command.
3. Make a test report from the cache file (using CACHEFILE and OUTFILE but no LOGFILE) and compare it against the report from the logfile to check it works. (This step really is worth doing!)
4. Now you can throw away the old logfile, if you've really understood what data you're losing by doing so. (But please remember that I can take no responsibility if something goes wrong: see the licence.)
5. When you want to make the main report, you can now use all your cache files and the current (not-yet-cached) logfile.