Changeset 33558 for gs3-extensions

Timestamp:

2019-10-10T23:41:36+13:00 (5 years ago)

Author:

ak19

Message:

Committing cumulative changes since last commit.

File:

• gs3-extensions/maori-lang-detection/MoreReading/crawling-Nutch.txt (modified) (5 diffs)

gs3-extensions/maori-lang-detection/MoreReading/crawling-Nutch.txt

@@ -4 +4 @@
 https://www.quora.com/What-are-some-Web-crawler-tips-to-avoid-crawler-traps
 
-
+https://cwiki.apache.org/confluence/display/nutch/
+https://cwiki.apache.org/confluence/display/NUTCH/Nutch2Crawling
+https://cwiki.apache.org/confluence/display/nutch/ReaddbOptions
+
+https://moz.com/top500
 -----------
 NUTCH
@@ -43 +47 @@
 
 
+Nutch doesn't work with spark (yet):
+https://stackoverflow.com/questions/29950299/distributed-web-crawling-using-apache-spark-is-it-possible
+
 SOLR:
 * Query syntax: http://www.solrtutorial.com/solr-query-syntax.html
 * Deleting a core: https://factorpad.com/tech/solr/reference/solr-delete.html
 
+
+* If you change a nutch 2 configuration, https://stackoverflow.com/questions/16401667/java-lang-classnotfoundexception-org-apache-gora-hbase-store-hbasestore
+explains you can rebuild nutch with:
+     cd <apache-nutch>
+     ant clean
+     ant runtime
 ----------------------------------
 Apache Nutch 2 with newer HBase
@@ -71 +84 @@
 * https://waue0920.wordpress.com/2016/08/25/nutch-2-3-1-hbase-0-98-hadoop-2-5-solr-4-10-3/
 
+The older but recommended hbase 0.98.21 for hadoop 2 can be downloaded from https://archive.apache.org/dist/hbase/0.98.21/
 
 -----
@@ -77 +91 @@
 https://learnhbase.net/2013/03/02/hbase-shell-commands/
 http://dwgeek.com/read-hbase-table-using-hbase-shell-get-command.html/
+dropping tables: https://www.tutorialspoint.com/hbase/hbase_drop_table.htm
 
 > list
@@ -184 +199 @@
 Dump output on local filesystem:
     rm -rf /tmp/bla
-    ./bin/nutch readdb -dump /tmp/bla
+    ./bin/nutch readdb -dump /tmp/bla [-crawlId ID -text]
     less /tmp/bla/part-r-00000