Changeset 33538 for gs3-extensions

Timestamp:

2019-10-01T21:36:06+13:00 (5 years ago)

Author:

ak19

Message:

Some additions to the setup.sh script to query commoncrawl for MRI data on hadoop before I commit what I've done to crawl with Autistici's crawl software.

Location:

gs3-extensions/maori-lang-detection/hdfs-cc-work

Files:

• Readme.txt (modified) (2 diffs)
• scripts/setup.sh (modified) (1 diff)

gs3-extensions/maori-lang-detection/hdfs-cc-work/Readme.txt

@@ -5 +5 @@
 B.  Create IAM role on Amazon AWS to use S3a
 C.  Configure Spark on your vagrant VM with the AWS authentication details
+---
+Script scripts/setup.sh now is automated to do the steps in D-F below
+and prints out the main instruction for G.
+---
 D.  OPTIONAL? Further configuration for Hadoop to work with Amazon AWS
 E.  Setup cc-index-table git project
@@ -152 +156 @@
 ]
 
+----------------------------------------------------------------------
+NOTE:
+Script scripts/setup.sh now is automated to do the steps in D-F below
+and prints out the main instruction for G.
+
 
 ----------------------------------------------------------------------

gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/setup.sh

@@ -71 +71 @@
 fi
 
-echo "Done compiling and setting up."
+echo "Done compiling and automated parts of setting up."
+echo "NEXT STEP:"
+echo "Ensure you have sudo edited $SPARK_HOME/conf/spark-defaults.conf"
+echo "  (/usr/local/spark-2.3.0-bin-hadoop2.7/conf/spark-defaults.conf)"
+echo "to contain the following 3 lines with YOUR Amazon AWS IAM Role access and secret keys:"
+echo "   spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem"
+echo "   spark.hadoop.fs.s3a.access.key=YOUR_AWS_IAM-ROLE_ACCESSKEY_HERE"
+echo "   spark.hadoop.fs.s3a.secret.key=YOUR_AWS_IAM-ROLE_SECRETKEY_HERE"
+echo "Consult GS_README.TXT section B (and C) for instructions on setting up an AWS IAM role."
+echo "Only when that's done will you be ready to run the following script."
+echo ""
+echo "THEN:"
 echo "To get MRI warc to wet for a particular crawl timestamp, cd into cc-index-table and RUN:"
 echo "./get_maori_WET_records_for_crawl.sh CC-MAIN-<YYYY-##>"