Changeset 30913 for other-projects/hathitrust/vagrant-spark-hdfs-cluster/trunk/README.txt

Timestamp:

2016-10-25T10:02:58+13:00 (7 years ago)

Author:

davidb

Message:

Renaming to better represent what the cluster is designed for

Location:

other-projects/hathitrust/vagrant-spark-hdfs-cluster

Files:

• . (moved) (moved from other-projects/hathitrust/vagrant-hadoop-cluster )
• trunk/README.txt (modified) (2 diffs)

other-projects/hathitrust/vagrant-spark-hdfs-cluster/trunk/README.txt

@@ -1 +1 @@
 
-Vargrant provisioning files to spin up a modest (4 node) Hadoop
-cluster for experiments processing HTRC Extracted Feature JSON files
-suitable for ingesting into Solr.
+Vargrant provisioning files to spin up a modest Spark cluster (master
++ 3 slaves + backup) for experiments processing HTRC Extracted Feature
+JSON files suitable for ingesting into Solr.
+
+To aid parallelism, code is designed to read JSON files from HDFS, so
+the provision of the cluster includes Hadoop core in addition to Spark
 
 
-Top-level code Apache Spark, processing HDFS stored JSON files, hence
-the need for an underlying Hadoop cluster.
+Provisioning uses Puppet scripting, based on the following on-line
+resources, but updated to use newer versions of Ubuntu, Java,
+and Hadoop.  Spark is then added in on top of that.
 
-Provisioning based on the following online resources, but updated to
-use newer versions of Ubuntu, Java, and Hadoop.
 
   http://cscarioni.blogspot.co.nz/2012/09/setting-up-hadoop-virtual-cluster-with.html
 
   https://github.com/calo81/vagrant-hadoop-cluster
+
+To get everything setup, type:
+
+  vargrant up
+
+Then log in to the master node, and swithc to 'ubuntu' user
+
+  vargrant ssh master
+  sudo su - ubuntu
+
+If the first time, you need to format an HDFS area to use:
+  hdfs namenode -format
+
+Otherwise start up HDFS and Spark deamon processes:
+
+  start-dfs.sh
+  spark-start-all.sh
+
+You can visit the Spark cluster monitoring page at:
+
+  http://10.10.0.52:8080/
+
 
 
@@ -60 +84 @@
 http://kvz.io/blog/2013/01/16/vagrant-tip-keep-virtualbox-guest-additions-in-sync/
 
+----
+SecondaryNode
+----
+
+http://stackoverflow.com/questions/23581425/hadoop-how-to-start-secondary-namenode-on-other-node
+
+<property>
+  <name>dfs.namenode.secondary.http-address</name>
+  <value>ec2-54-187-222-213.us-west-2.compute.amazonaws.com:50090</value>
+</property>
+
+----
+Spark Cluster
+----
+
+http://spark.apache.org/docs/latest/spark-standalone.html