Changeset 33425 for gs3-extensions/maori-lang-detection/MoreReading/CommonCrawl.txt

Timestamp:

2019-08-16T22:15:40+12:00 (4 years ago)

Author:

ak19

Message:

A few more links now that I got past getting the vagrant VM with spark and hadoop working.

File:

• gs3-extensions/maori-lang-detection/MoreReading/CommonCrawl.txt (modified) (3 diffs)

gs3-extensions/maori-lang-detection/MoreReading/CommonCrawl.txt

@@ +1 @@
+sudo apt-get install maven
+(or sudo apt update
+sudo apt install maven)
+git clone https://github.com/commoncrawl/cc-index-table.git
+cd cc-index-table
+mvn package
+vagrant@node1:~/cc-index-table$ ./src/script/convert_url_index.sh https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-30/indexes/cdx-00000.gz hdfs:///user/vagrant/cc-index-table
+
+
+
+
+spark:
+https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-shell.html
+
+============
 Dr Bainbridge found the following vagrant file that will set up hadoop and spark presumably for cluster computing:
 
@@ -6 +21 @@
     * Guide: https://www.vagrantup.com/intro/getting-started/index.html
     * Common cmds: https://blog.ipswitch.com/5-vagrant-commands-you-need-to-know
+    * vagrant reload = vagrant halt + vagrant up https://www.vagrantup.com/docs/cli/reload.html
+    * https://stackoverflow.com/questions/46903623/how-to-use-firefox-ui-in-vagrant-box
+    * https://stackoverflow.com/questions/22651399/how-to-install-firefox-in-precise64-vagrant-box
+      sudo apt-get -y install firefox
+    * vagrant install emacs: https://medium.com/@AnnaJS15/getting-started-with-virtualbox-and-vagrant-8d98aa271d2a
+
+    * hadoop conf: sudo vi /usr/local/hadoop-2.7.6/etc/hadoop/mapred-site.xml
+    * https://data-flair.training/forums/topic/mkdir-cannot-create-directory-data-name-node-is-in-safe-mode/
+---
+==> node1: Forwarding ports...
+    node1: 8080 (guest) => 8081 (host) (adapter 1)
+    node1: 8088 (guest) => 8089 (host) (adapter 1)
+    node1: 9083 (guest) => 9084 (host) (adapter 1)
+    node1: 4040 (guest) => 4041 (host) (adapter 1)
+    node1: 18888 (guest) => 18889 (host) (adapter 1)
+    node1: 16010 (guest) => 16011 (host) (adapter 1)
+    node1: 22 (guest) => 2200 (host) (adapter 1)
+==> node1: Running 'pre-boot' VM customizations...
+
+
+==> node1: Checking for guest additions in VM...
+    node1: The guest additions on this VM do not match the installed version of
+    node1: VirtualBox! In most cases this is fine, but in rare cases it can
+    node1: prevent things such as shared folders from working properly. If you see
+    node1: shared folder errors, please make sure the guest additions within the
+    node1: virtual machine match the version of VirtualBox you have installed on
+    node1: your host and reload your VM.
+    node1: 
+    node1: Guest Additions Version: 5.1.38
+    node1: VirtualBox Version: 5.2
+
 ------------
 
@@ -43 +89 @@
 SPARK (Spark SQL): https://github.com/commoncrawl/cc-index-table
     with example on selecting languages
-
+https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-30/indexes/cluster.idx
+
+./convert_url_index.sh https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-30/indexes/cdx-00000.gz hdfs:///user/vagrant/cc-index-table
 ---