Ignore:
Timestamp:
08/13/13 21:13:25 (7 years ago)
Author:
ak19
Message:

The script to generate the MODEL collections now also handles the svn rm and svn add if passed the svndelete flag, and just the copying over of the rebuilt archives and index into the model-collect (svn) folder if passed in the svnupdate flag.

File:
1 edited

Legend:

Unmodified
Added
Removed
  • other-projects/nightly-tasks/diffcol/trunk/gen-model-colls.sh

    r28027 r28037  
    1212
    1313# USAGE
    14 # Put this file in the toplevel of the Greenstone 2 binary/SVN installation
     14# Put this file in the toplevel of the Greenstone 2 binary/compiled SVN installation
    1515# that you want to generate the model collections with.
    16 # Call with 0 args to process all model collections
    17 # Call with n args, where each is a collection name in the model-collections
    18 # list
     16# You can provide a list of collection names or none, in which case all the collections
     17# are processed.
     18
     19# Pass in --svnupdate to copy across the contents of archives and index in the
     20# rebuilt collection, overwriting their equivalents in the svn model collection,
     21# but not removing any extraneous HASH folders already present.
     22
     23# Pass in --svndelete to remove what's in archives and index from svn and replace
     24# the contents of archives and index with the matching contents from the rebuild
     25# collection. Useful for when the HASH directory naming has changed and everything
     26# in archives and index has to be wiped out and moved back in from the rebuilt col.
     27
     28# If neither is passed in, then the collections are rebuilt but the svn model-collect
     29# is not updated and the repository is not updated.
     30
     31# This program leaves you to do the final commit on the (svn) model-collect folder!
     32
     33# Examples of usage:
     34# ./gen-model-colls.sh
     35# ./gen-model-colls.sh --svndelete
     36# ./gen-model-colls.sh --svnupdate Tudor-Basic Tudor-Enhanced
     37
     38# The first just rebuilds all the collections in a new folder called collect and stops there
     39# The second rebuilds all the collections and then removes all contents of the archives and the
     40# index folders from the svn checked-out model-collect and removes them from svn. Then it copies
     41# across all the contents of the rebuilt archives and index into model-collect and svn adds them.
     42# The third example checks out all the model-collections again, but rebuilds only the 3 collections
     43# specified in the new collect folder. Then it copies across the contents of the archives and index
     44# folders of those 3 collections into their model-collect equivalents.
     45
     46# You then still have to do the final svn commit on the model-collect folder.
     47
     48# Also valid examples:
     49# ./gen-model-colls.sh Tudor-Basic Tudor-Enhanced
     50# ./gen-model-colls.sh --svndelete Tudor-Basic Tudor-Enhanced
     51# ./gen-model-colls.sh --svnupdate
    1952
    2053# PSEUDOCODE
     
    2356# Makes a copy
    2457# In the copy: gets rid of their .svn folders, and builds each collection in turn, moving building to index once done
    25 
    26 
    27 
     58# If --svndelete was passed in: svn removes model-collect/archives/* and model-collect/index/*, copies over index/*
     59# and archives/* from collect into model-collect and svn adds model-collect/archives/* and model-collect/index/*
     60# If --svnupdate was passed in: copies collect/archives/* into model-collect/archives/*, and copies collect/index/*
     61# into model-collect/index/*, overwriting files that already existed but have now been updated upon rebuild. However,
     62# --svnupdate will leave untouched any files and folders unique to model-collect.
     63
     64# svn update/delete a single collection
     65function update_single_collection () {
     66    collection=$1
     67    mode=$2
     68
     69    #escape the filename (in case of space)
     70    collection=`echo $collection | sed 's@ @\\\ @g'`
     71   
     72    #get just the basename
     73    collection=`basename $collection`
     74
     75    if [ ! -e model-collect/$collection ]; then
     76    echo "$collection does not exist in model-collect, commit separately"
     77    return;
     78    fi
     79
     80    # diff the svn model and rebuilt model collections
     81    diff_result=`diff -rq model-collect/$collection collect/$collection | grep -v ".svn"`
     82#    echo "Diff result for collection $collection: $diff_result"
     83
     84    # if no differences in the current collection, then we're done
     85    if [ "x$diff_result" == "x" ]; then
     86    echo "No differences in collection $collection"
     87    return;
     88    fi
     89
     90    # check that none of the lines mention files outside the archives or index folders
     91    # http://en.gibney.org/tell_the_bash_to_split_by_newline_charac
     92    # http://forums.gentoo.org/viewtopic-p-3130541.html
     93
     94    # http://wi-fizzle.com/article/276
     95    # http://stackoverflow.com/questions/918886/how-do-i-split-a-string-on-a-delimiter-in-bash
     96    # http://www.linuxquestions.org/questions/programming-9/split-a-string-on-newlines-bash-313206/
     97    # http://unix.stackexchange.com/questions/39473/command-substitution-splitting-on-newline-but-not-space
     98
     99    # store backup of Internal Field Separator value, then set IFS to newline for splitting on newline
     100
     101    IFS_BAK=$IFS
     102#    IFS='\n' # splits on all whitespace
     103IFS='
     104'
     105    # in the lines returned from the diff, test for archives or newline
     106    # http://stackoverflow.com/questions/229551/string-contains-in-bash
     107    for line in `diff -rq model-collect/$collection collect/$collection | grep -v ".svn"`; do
     108    # echo "LINE: $line"   
     109    if [[ "$line" != *archives* && "$line" != *index* ]]; then
     110        # the file that is different is neither in index nor in archives, send this diffline to the report
     111        echo $line >> report.txt
     112    fi
     113    done
     114 
     115    IFS=$IFS_BAK
     116    IFS_BAK=
     117
     118    # Now svn remove what's unique to model-collect and svn add what's been rebuilt in index and archives
     119    # see # http://stackoverflow.com/questions/7502261/delete-folder-content-and-remove-from-version-control
     120 
     121    if [ "$mode" == "svndelete" ]; then
     122    svn rm model-collect/$collection/archives/*
     123    svn rm model-collect/$collection/index/*
     124    fi
     125
     126    # copy across the contents of the rebuilt model-collection's index and archives to the svn model-collect
     127    cp -r collect/$collection/archives/* model-collect/$collection/archives/.
     128    cp -r collect/$collection/index/* model-collect/$collection/index/.
     129
     130    if [ "$mode" == "svndelete" ]; then
     131    svn add model-collect/$collection/archives/*
     132    svn add model-collect/$collection/index/*
     133    fi
     134
     135    # To undo the changes made by svndelete, run the following manually
     136    # svn revert --depth infinity <model-collect/$collection/archives/*
     137    # svn revert --depth infinity <model-collect/$collection/archives/*
     138    # then remove both the local archives and index, and do an svn up to get original checkout back
     139
     140    # if etc/collect.cfg is different, copy it across too?
     141
     142
     143}
     144
     145# re-build a single collection in "collect" which is a copy of model-collect
    28146function single_collection () {
    29147    collection=$1   
     
    47165}
    48166
    49 
     167# unused
     168# re-build all collections in "collect" which is a copy of model-collect
    50169function all_collections() {
    51170
     
    53172    for collection in collect/*; do
    54173    single_collection $collection;
     174    #update_single_collection $collection;
    55175    done
    56176}
    57177
     178# http://stackoverflow.com/questions/16483119/example-of-how-to-use-getopt-in-bash
     179function usage() {
     180# usage() { echo "Usage: $0 [-s <45|90>] [-p <string>]" 1>&2; exit 1; }
     181
     182    echo "*******************************************"
     183    echo "Usage: $0 [--svnupdate|--svndelete] [col1, col2, col3,...]";
     184    echo "If no collections are provided, all collections will be processed.";
     185    echo "If neither svnupdate nor svndelete are provided, svnupdate is assumed.";
     186    echo "*******************************************"
     187    exit 1;
     188}
     189
    58190
    59191# The program starts here
     192
     193# process optional command line arguments
     194# http://blog.onetechnical.com/2012/07/16/bash-getopt-versus-getopts/
     195# Execute getopt
     196ARGS=$(getopt -o ud -l "svnupdate,svndelete" -n "$0" -- "$@");
     197
     198#Bad arguments
     199if [ $? -ne 0 ];then
     200    usage
     201    exit 1
     202fi
     203
     204eval set -- "$ARGS";
     205
     206# mode can be svndelete or svnupdate
     207mode=
     208
     209# -n: http://tldp.org/LDP/abs/html/testconstructs.html
     210while true; do
     211  case "$1" in
     212    -d|--svndelete)
     213      shift;
     214      if [ "x$mode" == "xsvnupdate" ]; then
     215      echo
     216      echo "Can't use both svndelete and svnupdate"
     217      usage
     218      exit 1
     219      else
     220      mode=svndelete
     221      fi
     222      ;;
     223    -u|--svnupdate)
     224      shift;
     225      if [ "x$mode" == "xsvndelete" ]; then
     226      echo
     227      echo "Can't use both svndelete and svnupdate"
     228      usage
     229      exit 1
     230      else
     231      mode=svnupdate
     232      fi
     233      ;;
     234    --)
     235      shift;
     236      break;
     237      ;;
     238  esac
     239done
     240
     241# If no mode provided (svndelete|svnupdate) as cmd line arg, then don't modify
     242# the svn model-collect folder. We leave it at rebuilding its copy in collect
     243
     244#http://www.cyberciti.biz/faq/linux-unix-sleep-bash-scripting/
     245#if [ "x$mode" == "x" ]; then
     246#    echo
     247#    echo "***Mode svndelete or svnupdate not provided. Defaulting to svnupdate... 3s to change to your mind"
     248#    echo
     249#    sleep 3s
     250#    mode=svnupdate
     251#fi
     252
     253# the remaining arguments to the script are assumed to be collections
     254
     255# debugging
     256#for collection in "$@"; do
     257#    collection=collect/$collection
     258#    echo "Collection: $collection"
     259#done
     260
     261# finished processing arguments
     262
     263
     264# report will contain the output of the diff for
     265if [ -f report.txt ]; then
     266    rm report.txt
     267fi
    60268
    61269# Need pdfbox for the PDFBox tutorial
     
    93301#find . -name ".svn" -type d -exec rm -rf {} \;
    94302#cd ..
    95 find collect -name ".svn" -type d -exec rm -rf {} \;
     303find collect -name ".svn" -type d -exec rm -rf {} \; #2>&1 > /dev/null
    96304
    97305
     
    104312
    105313if [ "$1" == "" ]; then
    106     all_collections
     314    # all_collections   
     315    #for each collection, import, build, move building to index
     316    for collection in collect/*; do
     317    single_collection $collection;
     318    if [ "x$mode" != "x" ]; then
     319        update_single_collection $collection $mode;
     320    fi
     321    done
    107322else
    108323    # Command-line args are a list of collections,
    109324    # process each command-line arg, after confirming such a collection exists
    110325
    111     for collection in collect/"$@"; do
     326    for collection in "$@"; do
     327    collection=collect/$collection
    112328    if test -e $collection; then   
    113329        single_collection $collection;
     330        if [ "x$mode" != "x" ]; then
     331        update_single_collection $collection $mode;
     332        fi
    114333    else
    115334        echo "Can't find collection $collection. Skipping."
     
    117336    done
    118337fi
     338
     339
     340echo
     341echo "*****************************************"
     342# if we were svn updating/deleting collections, then mode was set
     343# if in that case a report was generated with additional differences, point the user to it
     344if [ -f report.txt ] && [ "x$mode" != "x" ]; then
     345    echo "Some files outside of archives and index folders were different. See report.txt"
     346    echo
     347fi
     348
     349echo "The original collect directory has been left renamed as collect_orig"
     350echo
     351if [ "x$mode" == "x" ]; then
     352    echo "The model-collect folder has not been altered. Changes have only been made to collect"
     353else
     354    echo "You still need to run svn status and svn commit on the model-collect folder"
     355fi
     356echo "*****************************************"
     357echo
     358
     359
Note: See TracChangeset for help on using the changeset viewer.