Changeset 33428

Timestamp:

2019-08-19T20:31:23+12:00 (5 years ago)

Author:

ak19

Message:

Working commoncrawl cc-warc-examples' WET wordcount example using Hadoop. And some more links.

File:

• gs3-extensions/maori-lang-detection/MoreReading/CommonCrawl.txt (modified) (1 diff)

gs3-extensions/maori-lang-detection/MoreReading/CommonCrawl.txt

@@ +1 @@
+To run firefox/anything graphical inside the VM run by vagrant, have to ssh -Y onto both analytics and then to the vagrant VM from analytics:
+1. ssh analytics -Y
+2. [anupama@analytics vagrant-hadoop-hive-spark]$ vagrant ssh -- -Y
+or
+vagrant ssh -- -Y node1
+(the -- flag tells the vagrant command that the subsequent -Y flag should be passed to the ssh cmd that vagrant runs)
+
+Only once ssh-ed with vagrant into the VM whose hostname is "node1", do you have access to node1's assigned IP: 10.211.55.101
+- Connecting machines, like analytics, must access node1 or use port forwarding to view the VM's servers on localhost. For example, on analytics, can view Yarn pages at http://localhost:8088/
+- If firefox is launched inside the VM (so inside node1), then can access pages off their respective ports at any of localhost|10.211.55.101|node1.
+
+
+
+
+WET example from https://github.com/commoncrawl/cc-warc-examples
+
+vagrant@node1:~/cc-warc-examples$ hdfs dfs -mkdir /user/vagrant/data
+vagrant@node1:~/cc-warc-examples$ hdfs dfs -put data/CC-MAIN-20190715175205-20190715200159-00000.warc.wet.gz hdfs:///user/vagrant/data/.
+vagrant@node1:~/cc-warc-examples$ hdfs dfs -ls data
+Found 1 items
+-rw-r--r--   1 vagrant supergroup  154823265 2019-08-19 08:23 data/CC-MAIN-20190715175205-20190715200159-00000.warc.wet.gz
+vagrant@node1:~/cc-warc-examples$ hadoop jar target/cc-warc-examples-0.3-SNAPSHOT-jar-with-dependencies.jar org.commoncrawl.examples.mapreduce.WETWordCount
+
+<ONCE FINISHED:>
+
+vagrant@node1:~/cc-warc-examples$ hdfs dfs -cat /tmp/cc/part*
+
+
+
+INFO ON HADOOP/HDFS:
+https://www.bluedata.com/blog/2016/08/hadoop-hdfs-upgrades-painful/
+
+---------------
+More examples to try:
+https://github.com/commoncrawl/cc-warc-examples
+
+
+A bit outdated?
+https://www.journaldev.com/20342/apache-spark-example-word-count-program-java
+https://www.journaldev.com/20261/apache-spark
+
+--------
+
 sudo apt-get install maven
 (or sudo apt update