mgpp_passes(1)

NAME
mgpp_passes - builds mgpp databases

SYNOPSIS
mgpp_passes [ -J doc-tag ] [ -K level-tag ]
[ -L index-level ] [ -m invf-mem-buffer ]
[ -T1 ] [ -T2 ] [ -I1 ] [ -I2 ] [ -S ] [ -C ]
[ -h ] [ -d directory ] -f name [ filename(s) ]

DESCRIPTION
mgpp_passes is the program that does most of the work when building mgpp database systems. The input documents can come from either stdin or from a list of files on the command line. In general, mgpp_passes must be run twice to build a database, first with the -T1 and -I1 options, and second with the -T2 and -I2 options. Several other programs must be run in order to get an mgpp database. The EXAMPLE section below gives an example of how to build a complete mgpp database.

OPTIONS
Options may appear in any order, but the filename(s), if specified, must be last.

-J doc-tag
    Specifies the SGML tag that encloses each document. Text appearing outside this tag is ignored. The document tag defines the highest level document that can be queried and printed. The default document tag is 'Document'.

-K level-tag
    Specifies the SGML tag of a sub document level. A level tag must enclose all text enclosed by the document tag. Levels can be queried and printed as if they were separate documents. Multiple document levels can be specified (the document tag is always added as a document level).

-L index-level
    Specifies the SGML tag enclosing the smallest indexed element. The index level should be no larger than the smallest document level. An empty string can be used to specify a word level index (which is the default).

-m invf-mem-buffer
    Maximum amount of memory to use for the pass-2 file inversion in megabytes. This option is only useful when used in conjunction with the option -I1. The larger this value, the faster the pass-2 inversion will proceed. The default value is 5 MB.

-T1 Generate the *.text.stats file.

-T2 Generate the *.text, *.text.idx, *.text.level, and possibly the *.text.dict.aux files. Using this option requires that the *.text.dict file be present.

-I1 Generate the *.invf.dict, *.invf.level, *.invf.chunk, and *.invf.chunk.trans files.

-I2 Generate the *.invf and *.invf.idx files. Using this option requires that the *.invf.dict.hash, *.invf.level, *.invf.chunk, and *.invf.chunk.trans files be present. The *.invf.dict.hash file is generated by mgpp_perf_hash_build(1) from the *.invf.dict file.

-S  This option causes a special pass to be executed. It is up to a user to modify mg.special.c in the source code to do something with the documents it is given.

-C  This activates the compatibility parsing mode. When using this mode documents are separated by control-B and paragraphs are separated by control-C. Internally these are converted to documents surrounded by 'Document' tags and paragraphs surrounded by 'Paragraph' tags.

-h  This displays a usage line on stderr.

-d directory
    This specifies the directory where the document collection is to be written.

-f name
    This specifies the base name of the document collection that will be created.

filename(s)
    This specifies the source text. If this is not specified, then the program expects the source text from stdin.

EXAMPLE
What follows is a UNIX csh(1) script as an example of how to build an mgpp document collection.

#! /bin/csh
# The first argument on the command line specifies the
#   source of the text
set source = ($1)

# The second argument is the name of the collection
set text = ($2)

# Create *.text.stats, *.invf.dict, *.invf.level
#   *.invf.chunk and *.invf.chunks.trans
${source} | mgpp_passes -T1 -I1 -f ${text}

# Create *.text.dict
mgpp_compression_dict -f ${text}

# Create *.invf.dict.hash
mgpp_perf_hash_build -f ${text}

# Create *.text, *.text.idx, *.text.level
#   *.invf and *.invf.idx
${source} | mgpp_passes -T2 -I2 -f ${text}

# Create *.text.weight and *.weight.approx
mgpp_weights_build -f ${text}

# Create *.invf.dict.blocked
mgpp_invf_dict -f ${text}

# Create *.invf.dict.blocked.1
mgpp_stem_idx -s 1 -f ${text}

# Create *.invf.dict.blocked.2
mgpp_stem_idx -s 2 -f ${text}

# Create *.invf.dict.blocked.3
mgpp_stem_idx -s 3 -f ${text}

# Create *.text.dict.fast
mgpp_fast_comp_dict -f ${text}

ENVIRONMENT

MGDATA
    If this environment variable exists, then its value is used as the default directory where the mgpp collection files are. If this variable does not exist, then the directory “.” is used by default. The command line option -d directory overrides the directory in MGDATA.

FILES

*.invf
    Inverted file.

*.invf.chunk
    Inverted file chunk descriptor file. When the inverted file is created it is created in chunks that use no more than a set amount of memory. This file describes those chunks.

*.invf.chunk.trans
    Word-occurrence-order to lexical-order translation file. The *.invf.chunk file is written in word-occurrence order but is required by -I2 to be in lexical order.

*.invf.dict
    Compressed stemmed dictionary.

*.invf.dict.blocked
    Compressed stemmed dictionary with index into the dictionary.

*.invf.dict.blocked.n
    Transformation dictionary from words stemmed with method n to unstemmed words.

*.invf.dict.hash
    Data for an order-preserving perfect hash function.

*.invf.idx
    The index into the inverted file.

*.invf.level
    Information about the document levels needed for querying.

*.text
    Compressed text.

*.text.dict
    Compressed compression dictionary.

*.text.dict.fast
    A fast loading version of the compressed compression dictionary.

*.text.idx
    Index into the compressed documents.

*.text.level
    Information about the document levels needed for text decompression.

*.text.stats
    Statistics about the text.

*.weight
    The exact weights file.

*.weight.approx
    The approximate weights file.

SEE ALSO
mgpp_compression_dict(1), mgpp_fast_comp_dict(1), mgpp_invf_dict(1), mgpp_perf_hash_build(1), mgpp_stem_idx(1), mgpp_weights_build(1)