source: other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-DOWNLOAD-EVERY-N.sh@ 31053

Last change on this file since 31053 was 31053, checked in by davidb, 6 years ago

Addition of second argument, optional, for where to save the files

  • Property svn:executable set to *
File size: 656 bytes
Line 
1#!/bin/bash
2
3argc=$#
4
5if [ "$argc" != "1" ] && [ "$argc" != "2" ] ; then
6 echo "Usage FULL-DOWNLOAD-EVERY-N.sh <num> [output-dir]" >&2
7 exit 1
8fi
9
10num=$1
11
12if [ "$argc" = "2" ] ; then
13 output_dir="$2"
14else
15 output_dir="full-ef-json-files"
16fi
17
18# Remove any trailing /.
19output_dir="${output_dir%/\.}"
20output_dir="${output_dir%/}"
21
22if [ ! -d "$output_dir" ] ; then
23 echo ""
24 echo "****"
25 echo "* Creating $output_dir"
26 echo "****"
27 echo ""
28 mkdir "$output_dir"
29fi
30
31# Now ensure a trailing /. is present!
32output_dir="${output_dir}/."
33
34
35rsync -pav --files-from=full-listing-step${num}.txt \
36 data.analytics.hathitrust.org::features/ $output_dir
Note: See TracBrowser for help on using the repository browser.