source: other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/PAIRTREE-FOREACH-HDFS-PUT.sh@ 31128

Last change on this file since 31128 was 31128, checked in by davidb, 5 years ago

Some scripts to help with pushing and monitoring the progress of the put into the HDFS file area

  • Property svn:executable set to *
File size: 362 bytes
Line 
1#!/bin/bash
2
3if [ "$#" != "1" ] ; then
4 echo "Usage: $0 top-level-dir"
5 exit
6fi
7
8toplevel_dir=$1
9
10
11for d in /data/features/ef-full/$toplevel_dir/*/* ; do
12 sub_dir=${d##/data/features/ef-full/}
13
14 echo "Processing: $sub_dir"
15 hdfs dfs -mkdir -p "full-ef-json-files/$sub_dir"
16
17 into_dir=${sub_dir%/*}
18 hdfs dfs -put $d "full-ef-json-files/$into_dir/."
19done
20
Note: See TracBrowser for help on using the repository browser.