Changeset 33423 for gs3-extensions

Timestamp:

2019-08-15T20:07:04+12:00 (5 years ago)

Author:

ak19

Message:

Adding in the link to the vagrant VM with Hadoop, Spark for cluster machines (not standalone) that Dr Bainbridge found, as well as links to getting started with vagrant and basic cmds

File:

• gs3-extensions/maori-lang-detection/MoreReading/CommonCrawl.txt (modified) (2 diffs)

gs3-extensions/maori-lang-detection/MoreReading/CommonCrawl.txt

@@ +1 @@
+Dr Bainbridge found the following vagrant file that will set up hadoop and spark presumably for cluster computing:
+
+https://github.com/martinprobson/vagrant-hadoop-hive-spark
+
+Vagrant:
+    * Guide: https://www.vagrantup.com/intro/getting-started/index.html
+    * Common cmds: https://blog.ipswitch.com/5-vagrant-commands-you-need-to-know
+------------
+
+At http://commoncrawl.org/2018/10/september-2018-crawl-archive-now-available/, it says
 "The September crawl contains 500 million new URLs, not contained in any crawl archive before. New URLs stem from
 
@@ -9 +19 @@
 Mixnode turns the web into a database that you can run queries against. Say goodbye to web crawling, forget about web scraping, never run a spider again: get all the web data that you need using simple SQL queries."
 
-
+--------------
 https://commoncrawl.github.io/cc-crawl-statistics/plots/languages
 http://commoncrawl.org/2018/08/august-2018-crawl-archive-now-available/