Changeset 33565 for gs3-extensions/maori-lang-detection/MoreReading/crawling-Nutch.txt

Timestamp:

2019-10-14T21:04:58+13:00 (5 years ago)

Author:

ak19

Message:

CCWETProcessor: domain url now goes in as a seedURL after the individual seedURLs, after Dr Bainbridge explained why the original ordering didn't make sense. 2. conf: we inspected the first site to be crawled. It was a non-top site, but we still wanted to control the crawling of it in the same way we control topsites. 3. Documented use of the nutch command for testing which urls pass and fail the existing regex-urlfilter checks.

File:

• gs3-extensions/maori-lang-detection/MoreReading/crawling-Nutch.txt (modified) (1 diff)

gs3-extensions/maori-lang-detection/MoreReading/crawling-Nutch.txt

@@ -293 +293 @@
 ---
 
+----------------------------------------------------------------------
+    Testing URLFilters: testing a URL to see if it's accepted
+----------------------------------------------------------------------
+Use the command
+    ./bin/nutch org.apache.nutch.net.URLFilterChecker -allCombined 
+(mentioned at https://lucene.472066.n3.nabble.com/Correct-syntax-for-regex-urlfilter-txt-trying-to-exclude-single-path-results-td3600376.html)
+
+Use as follows:
+
+    cd apache-nutch-2.3.1/runtime/local
+
+    ./bin/nutch org.apache.nutch.net.URLFilterChecker -allCombined 
+
+Then paste the URL you want to test, press Enter.
+    A + in front of response means accepted
+    A - in front of response means rejected.
+Can continue pasting URLs to test against filters until you send Ctrl-D to terminate input.
+
+
+
+
+