|
Last change
on this file since 31128 was 31128, checked in by davidb, 7 years ago |
|
Some scripts to help with pushing and monitoring the progress of the put into the HDFS file area
|
-
Property svn:executable
set to
*
|
|
File size:
362 bytes
|
| Line | |
|---|
| 1 | #!/bin/bash
|
|---|
| 2 |
|
|---|
| 3 | if [ "$#" != "1" ] ; then
|
|---|
| 4 | echo "Usage: $0 top-level-dir"
|
|---|
| 5 | exit
|
|---|
| 6 | fi
|
|---|
| 7 |
|
|---|
| 8 | toplevel_dir=$1
|
|---|
| 9 |
|
|---|
| 10 |
|
|---|
| 11 | for d in /data/features/ef-full/$toplevel_dir/*/* ; do
|
|---|
| 12 | sub_dir=${d##/data/features/ef-full/}
|
|---|
| 13 |
|
|---|
| 14 | echo "Processing: $sub_dir"
|
|---|
| 15 | hdfs dfs -mkdir -p "full-ef-json-files/$sub_dir"
|
|---|
| 16 |
|
|---|
| 17 | into_dir=${sub_dir%/*}
|
|---|
| 18 | hdfs dfs -put $d "full-ef-json-files/$into_dir/."
|
|---|
| 19 | done
|
|---|
| 20 |
|
|---|
Note:
See
TracBrowser
for help on using the repository browser.
