- Timestamp:
- 2019-10-01T21:36:06+13:00 (5 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
gs3-extensions/maori-lang-detection/hdfs-cc-work/scripts/setup.sh
r33535 r33538 71 71 fi 72 72 73 echo "Done compiling and setting up." 73 echo "Done compiling and automated parts of setting up." 74 echo "NEXT STEP:" 75 echo "Ensure you have sudo edited $SPARK_HOME/conf/spark-defaults.conf" 76 echo " (/usr/local/spark-2.3.0-bin-hadoop2.7/conf/spark-defaults.conf)" 77 echo "to contain the following 3 lines with YOUR Amazon AWS IAM Role access and secret keys:" 78 echo " spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" 79 echo " spark.hadoop.fs.s3a.access.key=YOUR_AWS_IAM-ROLE_ACCESSKEY_HERE" 80 echo " spark.hadoop.fs.s3a.secret.key=YOUR_AWS_IAM-ROLE_SECRETKEY_HERE" 81 echo "Consult GS_README.TXT section B (and C) for instructions on setting up an AWS IAM role." 82 echo "Only when that's done will you be ready to run the following script." 83 echo "" 84 echo "THEN:" 74 85 echo "To get MRI warc to wet for a particular crawl timestamp, cd into cc-index-table and RUN:" 75 86 echo "./get_maori_WET_records_for_crawl.sh CC-MAIN-<YYYY-##>"
Note:
See TracChangeset
for help on using the changeset viewer.