htdig

ht://Dig Copyright © 1995-2000 The ht://Dig Group
Please see the file COPYING for license information.


Synopsis

htdig [options]

Description

Htdig retrieves HTML documents using the HTTP protocol and gathers information from these documents which can later be used to search these documents. This program can be referred to as the search robot.

Options

-
Get the list of URLs to start indexing from the STDIN. This will override the default start_url and the file supplied to -m [url_file].
-a
Use alternate work files. Tells htdig to append .work to database files, causing a second copy of the database to be built. This allows the original files to be used by htsearch during the indexing run. When used without the "-i" flag for an update dig, htdig will use any existing .work files for the databases to update.
-c configfile
Use the specified configfile file instead of the default.
-h maxhops
Restrict the dig to documents that are at most maxhops links away from the starting document.
-i
Initial. Do not use any old databases. This is accomplished by first erasing the databases.
-l
Stop and restart. Reads in the progress of any previous interrupted digs from the log file and write the progress out if interrupted by a signal.
-m [url_file]
Minimal. Only index the URLs in the file provided and no others.
-s
Print statistics about the dig after completion.
-t
Create an ASCII version of the document database. This database is easy to parse with other programs so that information can be extracted from it for purposes other than searching. One could gather some interesting statistics from this database.

Each line in the file starts with the document id followed by a list of \tfieldname:value. The fields always appear in the order listed bellow:

fieldnamevalue
uURL
tTitle
aState (0 = normal, 1 = not found, 2 = not indexed, 3 = obsolete)
mLast modification time as reported by the server
sSize in bytes
HExcerpt
hMeta description
lTime of last retrieval
LCount of the links in the document (outgoing links)
bCount of the links to the document (incoming links or backlinks)
cHopCount of this document
gSignature of the document used for duplicate-detection
eE-mail address to use for a notification message from htnotify
nDate to send out a notification e-mail message
SSubject for a notification e-mail message
dThe text of links pointing to this document. (e.g. <a href="docURL">description</a>)
AAnchors in the document (i.e. <A NAME=...)
-u username:password
Tells htdig to send the supplied username and password with each HTTP request. The credentials will be encoded using the 'Basic' authentication scheme. There HAS to be a colon (:) between the username and password.
-v
Verbose mode. This increases the verbosity of the program. Using more than 2 is probably only useful for debugging purposes. The default verbose mode (using only one -v) gives a nice progress report while digging.

Files

CONFIG_DIR/htdig.conf
The default configuration file.

See Also

htmerge, htsearch, Configuration file format, and A Standard for Robot Exclusion.

Last modified: $Date: 2000/04/11 03:18:01 $