1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html>
<head>
<title>
ht://Dig: htdig
</title>
</head>
<body bgcolor="#eef7ff">
<h1>
htdig
</h1>
<p>
ht://Dig Copyright © 1995-2004 <a href="THANKS.html">The ht://Dig Group</a><br>
Please see the file <a href="COPYING">COPYING</a> for
license information.
</p>
<hr size="4" noshade>
<dl>
<dd>
<h2>
Synopsis
</h2>
</dd>
<dd>
htdig [<em>options</em>] [<em>start_url_file</em>]
</dd>
</dl>
<dl>
<dd>
<h2>
Description
</h2>
</dd>
<dd>
Htdig retrieves HTML documents using the HTTP protocol and
gathers information from these documents which can later be
used to search these documents. This program can be
referred to as the search robot.
</dd>
</dl>
<dl>
<dd>
<h2>
Options
</h2>
</dd>
<dd>
<dl compact>
<dt>
-a
</dt>
<dd>
Use alternate work files. Tells htdig to append <em>
.work</em> to database files, causing a second copy of
the database to be built. This allows the original
files to be used by htsearch during the indexing run. When
used without the "-i" flag for an update dig, htdig will
use any existing .work files for the databases to update.
</dd>
<dt>
-c <em>configfile</em>
</dt>
<dd>
Use the specified <em>configfile</em> file instead of the
default.
</dd>
<dt>
-h <em>maxhops</em>
</dt>
<dd>
Restrict the dig to documents that are at most <em>
maxhops</em> links away from the starting document.
</dd>
<dt>
-i
</dt>
<dd>
Initial. Do not use any old databases. This is
accomplished by first erasing the databases.
</dd>
<dt>
-m <em>url_file</em>
</dt>
<dd>
Minimal. Index only the URLs listed in
<em>url_file</em> and no others.
A file name of "-" reads from STDIN.
See also the <em>start_url_file</em> argument.
</dd>
<dt>
-s
</dt>
<dd>
Print statistics about the dig after completion.
</dd>
<dt>
-t
</dt>
<dd>
Create an ASCII version of the document database. This
database is easy to parse with other programs so that
information can be extracted from it for purposes other
than searching. One could gather some interesting
statistics from this database.
<p>Each line in the file starts with the document id
followed by a list of
<strong>\t<em>fieldname</em>:<em>value</em></strong>.
The fields always appear in the order listed below:
</p>
<table border=0>
<tr> <th>fieldname</th><th>value</th></tr>
<tr> <td>u</td><td>URL</td></tr>
<tr> <td>t</td><td>Title</td></tr>
<tr> <td>a</td><td>State (0 = normal, 1 = not found, 2
= not indexed, 3 = obsolete)</td></tr>
<tr> <td>m</td><td>Last modification time as reported
by the server</td></tr>
<tr> <td>s</td><td>Size in bytes</td></tr>
<tr> <td>H</td><td>Excerpt</td></tr>
<tr> <td>h</td><td>Meta description</td></tr>
<tr> <td>l</td><td>Time of last retrieval</td></tr>
<tr> <td>L</td><td>Count of the links in the document
(outgoing links)</td></tr>
<tr> <td>b</td><td>Count of the links to the document
(incoming links or backlinks)</td></tr>
<tr> <td>c</td><td>HopCount of this document</td></tr>
<tr> <td>g</td><td>Signature of the document used for
duplicate-detection</td></tr>
<tr> <td>e</td><td>E-mail address to use for a
notification message from htnotify</td></tr>
<tr> <td>n</td><td>Date to send out a notification
e-mail message</td></tr>
<tr> <td>S</td><td>Subject for a notification e-mail
message</td></tr>
<tr> <td>d</td><td>The text of links pointing to this
document. (e.g. <a
href="docURL">description</a>)</td></tr>
<tr> <td>A</td><td>Anchors in the document (i.e. <A
NAME=...)</td></tr>
</table>
</dd>
<dt>
-u <em>username:password</em>
</dt>
<dd>
Tells htdig to send the supplied username and password
with each HTTP request. The credentials will be encoded
using the 'Basic' authentication scheme. There <strong>
HAS</strong> to be a colon (:) between the username and
password.
</dd>
<dt>
-v
</dt>
<dd>
Verbose mode. This increases the verbosity of the
program. Using more than 2 is probably only useful for
debugging purposes. The default verbose mode (using
only one -v) gives a nice progress report while
digging. This progress report can be a bit
cryptic, so here is a brief explanation. A line
is shown for each URL, with 3 numbers before the
URL and some symbols after the URL. The first
number is the number of documents parsed so
far, the second is the DocID for this document,
and the third is the hop count of the document
(number of hops from one of the start_url
documents). After the URL, it shows a "*" for
a link in the document that it already visited,
a "+" for a new link it just queued, and a "-"
for a link it rejected for any of a number of
reasons. To find out what those reasons are,
you need to run htdig with at least 3 -v options,
i.e. -vvv. If there are no "*", "+" or "-" symbols
after the URL, it doesn't mean the document was
not parsed or was empty, but only that no links
to other documents were found within it. With
more verbose output, these symbols will get
interspersed in several lines of debugging output.
</dd>
<dt>
<em>start_url_file</em>
</dt>
<dd>
A file containing a list of URLs to start indexing
from, or "-" for STDIN. This will augment the default
<a href="attrs.html#start_url">start_url</a>
and override the file supplied to
[-m <em>url_file</em>].
</dd>
</dl>
</dd>
</dl>
<dl>
<dd>
<h2>
Files
</h2>
</dd>
<dd>
<dl>
<dt>
<a href="attrs.html#config_dir">CONFIG_DIR</a>/htdig.conf
</dt>
<dd>
The default configuration file.
</dd>
</dl>
<dl>
<dt>
<a href="attrs.html#database_dir">DATABASE_DIR</a>/db.docdb
</dt>
<dd>
Stores data about each document (title, url, etc.).
</dd>
</dl>
<dl>
<dt>
<a href="attrs.html#database_dir">DATABASE_DIR</a>/db.words.db,
<a href="attrs.html#database_dir">DATABASE_DIR</a>/db.words.db_weakcmpr
</dt>
<dd>
Record which documents each word occurs in.
</dd>
</dl>
<dl>
<dt>
<a href="attrs.html#database_dir">DATABASE_DIR</a>/db.excerpts
</dt>
<dd>
Stores start of each document to show context of
matches.
</dd>
</dl>
</dd>
</dl>
<dl>
<dd>
<h2>
See Also
</h2>
</dd>
<dd>
<a href="htmerge.html">htmerge</a>,
<a href="htsearch.html" target="_top">htsearch</a>,
<a href="attrs.html">Configuration file format</a>, and
<a href="http://www.robotstxt.org/wc/norobots.html">
A Standard for Robot Exclusion</a>.
</dd>
</dl>
<hr size="4" noshade>
Last modified: $Date: 2004/06/12 13:39:13 $
</body>
</html>
|
