Copyright (C) 2007 Mauricio Fernandez Fast implementations of the Tim Bray's "Wide Finder" web log analyzer, see http://www.tbray.org/ongoing/When/200x/2007/09/20/Wide-Finder Building -------- The executables can be built with $ make If you only want the single-core executables, or you don't have a working JoCaml installation, do $ make single-core Note that JoCaml must have OCaml 3.10.0 as its "companion OCaml" since some libraries from the latter are required by wf-mmap-multicore. Essentially, you have to install JoCaml 3.10.0 after OCaml 3.10.0. Some more details about how to build JoCaml correctly are shown below. Running ------- Baseline, basic analyzer based on regexps (1X, all other speeds referred to this): $ ./wf-kig Basic analyzer with sublinear string search (1.30X): $ ./wf Analyzer with sublinear string search, processing by blocks (1.8X): $ ./wf-block Analyzer with sublinear string search, using mmap'ed areas (3.06X) $ ./wf-mmap Analyzer with sublinear string search, using mmap'ed areas and a custom pattern matcher instead of a regexp (4X) $ ./wf-mmap-nore Analyzer with sublinear string search, using mmap'ed areas and multiple cores (2.8X on a single core): $ ./wf-mmap-multicore [num processes] The number of processes can be specified with the optional second argument. If not given, 16 processes will be used. The best performance should be obtained when as many processes as cores in the system are used. On a two-core machine, using 16 processes instead of 2 makes the analysis 20% slower. Analyzer with sublinear string search, using mmap'ed areas and multiple cores, with custom pattern matcher: $ ./wf-mmap-multicore-nore [num processes] License ------- The program is distributed under the terms of the GNU Library General Public License version 2.1 (found in LICENSE). As a special exception to the GNU Lesser General Public License, you may link, statically or dynamically, a "work that uses the Library" with a publicly distributed version of the Library to produce an executable file containing portions of the Library, and distribute that executable file under terms of your choice, without any of the additional requirements listed in clause 6 of the GNU Lesser General Public License. By "a publicly distributed version of the Library", we mean either the unmodified Library as distributed by the author, or a modified version of the Library that is distributed under the conditions defined in clause 2 of the GNU Lesser General Public License. This exception does not however invalidate any other reasons why the executable file might be covered by the GNU Lesser General Public License. Building JoCaml with a "companion OCaml" ---------------------------------------- JoCaml must be installed after OCaml 3.10.0. If the OCaml interpreter is installed as "ocamlc" (the default name), JoCaml's configure script should detect your OCaml installation and everything should work fine after you build normally (scroll down for a quick refresher of how to build JoCaml). Otherwise, you might have to use one these options from the configure script: JoCaml may have a 'companion' Objective Caml, so as to allow object level compatibility beetween JoCaml and OCaml. There are severe restructions: the version numbers of both systems should match. -ocamlc If is executable, then the script runs it, to check version number and, when it matches with JoCaml version number, to extract the location of the Objective Caml library. By default, this option is activated with value 'ocamlc' -ocamllib The Objective Caml library is . No version check is performed. Use this option at your own risk. When the configure script is done, you must see something like ** Configuration summary ** Directories where JoCaml will be installed: binaries.................. /usr/local/bin standard library.......... /usr/local/lib/jocaml Companion ocaml: version: 3.10.0, library: /usr/lib/ocaml/3.10.0. ... Then $ make world | alternatively: $ make opt | $ make opt.opt | $ make world.opt $ make install If all goes well, you should get something like this: $ jocamlopt -v The JoCaml native-code compiler, version 3.10.0 Standard library directory: /usr/local/lib/jocaml Companion OCaml library directory: /usr/lib/ocaml/3.10.0 Once JoCaml has got a companion OCaml, you can compile with jocamlopt and use OCaml's library: $ jocamlopt -unsafe -c bigstring.ml $ jocamlopt -unsafe -c wf-mmap-multicore.ml $ jocamlopt -o wf-mmap-multicore bigarray.cmxa str.cmxa bigstring.cmx wf-mmap-multicore.cmx -- Mauricio Fernandez