Changeset 33565 for gs3-extensions/maori-lang-detection/conf/sites-too-big-to-exhaustively-crawl.txt

Timestamp:

2019-10-14T21:04:58+13:00 (5 years ago)

Author:

ak19

Message:

CCWETProcessor: domain url now goes in as a seedURL after the individual seedURLs, after Dr Bainbridge explained why the original ordering didn't make sense. 2. conf: we inspected the first site to be crawled. It was a non-top site, but we still wanted to control the crawling of it in the same way we control topsites. 3. Documented use of the nutch command for testing which urls pass and fail the existing regex-urlfilter checks.

File:

• gs3-extensions/maori-lang-detection/conf/sites-too-big-to-exhaustively-crawl.txt (modified) (1 diff)

gs3-extensions/maori-lang-detection/conf/sites-too-big-to-exhaustively-crawl.txt

@@ -50 +50 @@
 #   column 3: whether nutch should do fetch all or not
 #   column 4: number of crawl iterations
+
+
+# NOT TOP SITES, BUT SITES WE INSPECTED AND WANT TO CONTROL SIMILARLY TO TOP SITES
+00.gs,SINGLEPAGE
+
+
+# TOP SITES
 
 # docs.google.com is a special case: not all pages are public and any interlinking is likely to