Index: /main/trunk/model-sites-dev/waikato-edu/TT4.02_Science Resources_Crates/science-activities/DATA-PREPARATION/README-science.txt
===================================================================
--- /main/trunk/model-sites-dev/waikato-edu/TT4.02_Science Resources_Crates/science-activities/DATA-PREPARATION/README-science.txt	(revision 33689)
+++ /main/trunk/model-sites-dev/waikato-edu/TT4.02_Science Resources_Crates/science-activities/DATA-PREPARATION/README-science.txt	(revision 33689)
@@ -0,0 +1,78 @@
+--------------------
+  README
+--------------------
+
+Obtain the TT4.02 Science Resources Crates content from the Uni of Waikato Education Department. We do not have the permission to distribute it or make it public.
+But assuming the Education Dept. has given this content you in a zip format, follow the instructions below.
+
+1. Extract the TT4.02_Science Resources_Crates.zip, which will create a folder called TT4.02_Science Resources_Crates.zip
+For example, extract it on the Desktop and your Windows desktop will now have a new folder TT4.02_Science Resources_Crates.
+
+2. Next, move the files procCrates.bat and all 14 of the .csv files (A-1 to F-2) into the new TT4.02_Science Resources_Crates folder
+
+3. Move the "sciencea.gs3coll" folder into your Greenstone 3.09rc1 installation's "web\sites\localsite\collect" folder.
+
+4. Move the siteConfig.xml file  into your Greenstone 3.09rc1 installation's "web\sites\localsite" folder, thereby replacing the existing file there.
+
+5. Start your Greenstone's installation's "Greenstone Server" application. Pressing this application's "Enter Library" button should open your default browser on your Greenstone DL home page. Visit your new Science Activities collection, as it's prebuilt.
+
+--------------------
+TO REGENERATE THE MATERIALS FOR YOUR COLLECTION AND TO REBUILD THAT COLLECTION:
+4. ONLY if you ever want to REGENERATE the folders for ingesting into GS3 your crate folders, PDF/Word files and metadata, then:
+
+In our example, your extracted TT4.02_Science Resources_Crates was on the Desktop, e.g. C:\Liang\Desktop\TT4.02_Science Resources_Crates
+Whatever the actual location, remember it and use it in the following. I will stick with the example of C:\Liang\Desktop\TT4.02_Science Resources_Crates
+
+a. Open a DOS prompt:
+	first hold down the Windows key on the keyboard (located between Alt and Ctrl keys) and press the R key (R for run);
+	then a little dialog will pop up. Type "cmd" into its edit field and press enter and a DOS prompt will appear.
+
+In the DOS prompt, type:
+	cd C:\Liang\Desktop\TT4.02_Science Resources_Crates
+(i.e. "cd" followed by the full path to the actual location of your extracted folder)
+
+b. type:
+	procCrates.bat
+	
+c. Wait until it finishes. It should have created a folder called "import" in C:\Liang\Desktop\TT4.02_Science Resources_Crates.
+
+d. ONLY if you now want to reingest and rebuild your collection with these new folders, use a file browser to do the following:
+- Delete all but the top-level metadata.xml file in your Greenstone 3.09 (rc1) installation's "web\sites\localsite\collect\sciencea\import" folder.
+- Go into the "C:\Liang\Desktop\TT4.02_Science Resources_Crates\import" folder. You should see subfolders A-1 to F-2.
+- MOVE all these into your Greenstone 3.09 (rc1) installation's "web\sites\localsite\collect\sciencea\import" folder, where you should already have the old solitary metadata.xml file at this level.
+
+e. Run the GLI/Greenstone Librarian Interface application of your installed Greenstone.
+In GLI, select File > Open. Choose your "Science Activities (sciencea)" collection.
+When this finishes opening, click on the Create tab in GLI.
+Find the Build Collection button at the bottom and press it. This can take a long time, maybe 15 mins or more. Eventually it will finish building and you can press the Preview button also at the bottom of the Create pane to have it open a browser for you and show you the finished collection.
+
+--------------------
+INFORMATION:
+- I exported each crate's .csv file individually from Excel (by default it exports in ASCII encoding format, so I left it as such and let procCrates.bat below deal with this).
+I did this by opening the "Crates (TT4.02).xlsx" file that was in the TT4.02_Science Resources_Crates.zip downloaded from your Google drive. Then I clicked on each sheet and did Save As > *.csv (the most basic CSV option).
+
+- the procCrates.bat now processes each crate's csv file and outputs additional metadata from before.
+In processing each csv file, it simplifies the Activity Name (handles &, ", ', etc) to make it more compatible as metadata for Greenstone.
+NOTE: I have had to change / (forward slash) to ; (semi-colon) as the / is used as a hierarchy separator by the browsing classifier code.
+
+- I've modified the Education Resources metadata set to have an extra field called CrateHierarchy, which makes it easier to produce the book nodes.
+The educationresources.mds (metadata set file) lives inside your web\sites\localsite\collect\sciencea\metadata folder. You can copy it into gli\metadata to re-use it for other collections.
+- I've modified web\sites\localsite\siteConfig.xml with a single additional line to have a word icon for docx files too (not just for the old .doc files). The additional line is:
+	<replace macro="_icondocx_" scope="metadata" text="&lt;img src='interfaces/default/images/imsword.gif' border='0'/&gt;" resolve="false"/>
+
+- In GLI's Design tab, I configured the plugins used to build your collection, and set up the search indexes and browsing classifiers that should be available:
+	only one search Index on ActivityName, as you requested
+	and one browsing classifier that works with the newly introduced er.CrateHierarchy metadata
+- I used GLI's Format tab to set up the display of the search results and browsing classifier content.
+Basically, when you BROWSE to an activity, under the activity name you will see its document(s). For each document: you get an icon linked to the HTML version of each document (produced by Greenstone), an appropriate icon linking to the original document format (PDF or docx) and the FILENAME of the document, since the activity name of the browser is what the filename is found under.
+When you SEARCH for an activity name or part of an activity name, you once more get the HTML doc icon and source doc icon but this is then further followed by the ACTIVITY NAME and then the filename in brackets.
+
+Based on the changes made to GLI's Design and Format panels, a customised collectionConfig.xml is generated for this collection. When you create future collections through GLI, you can use "Base collection on" option in GLI's New Collection dialog to base those future collections on the design and formatting of this collection (make sure you assign the same metadata). Making this choice in GLI will automatically reuse the same collectionConfig.xml, inheriting all its configurations. The new collection thus created will also get to use the same metadata sets, including the custom one we prepared, "educationresources.mds".
+
+So all the hard work here went into procCrates.bat and into setting up the collection's configuration, which produced collectionConfig.xml.
+
+--------------------
+BEWARE:
+Beware not to change the encoding of procCrates.bat from ASCII to UTF-8 or other encodings. If you wish to edit this file, open it in Notepad++ and leave it's encoding in ASCII.
+
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Index: /main/trunk/model-sites-dev/waikato-edu/TT4.02_Science Resources_Crates/science-activities/DATA-PREPARATION/procCrates.bat
===================================================================
--- /main/trunk/model-sites-dev/waikato-edu/TT4.02_Science Resources_Crates/science-activities/DATA-PREPARATION/procCrates.bat	(revision 33689)
+++ /main/trunk/model-sites-dev/waikato-edu/TT4.02_Science Resources_Crates/science-activities/DATA-PREPARATION/procCrates.bat	(revision 33689)
@@ -0,0 +1,270 @@
+@ECHO OFF
+SETLOCAL enabledelayedexpansion
+
+:: THIS SCRIPT FILE NEEDS TO BE IN ASCII FOR THE CHARACTER REPLACEMENTS TO WORK
+:: If working with utf8 chars to be replaced, set this script file to be in utf-8 and change the codepage, see
+:: https://stackoverflow.com/questions/5273937/how-to-replace-substrings-in-windows-batch-file
+::chcp 65001
+
+set origdir=%CD%
+
+set odir=import
+if not exist %odir% mkdir %odir%
+
+
+:: Process each CSV file. Each CSV file is assumed by procCrate to be in ASCII not UTF-8!
+call :procCrate "Crates A-1.csv" "A-1" "Crate A Science - Primary"
+call :procCrate "Crates A-2.csv" "A-2" "Crate A Science - Primary"
+
+call :procCrate "Crates B-1.csv" "B-1" "Crate B Science - Primary"
+call :procCrate "Crates B-2.csv" "B-2" "Crate B Science - Primary"
+
+call :procCrate "Crates C-1.csv" "C-1" "Crate C Biology - Secondary"
+call :procCrate "Crates C-2.csv" "C-2" "Crate C Biology - Secondary"
+call :procCrate "Crates C-3.csv" "C-3" "Crate C Biology - Secondary"
+
+call :procCrate "Crates D-1.csv" "D-1" "Crate D Science - Secondary"
+call :procCrate "Crates D-2.csv" "D-2" "Crate D Science - Secondary"
+
+call :procCrate "Crates E-1.csv" "E-1" "Crate E - Generic"
+call :procCrate "Crates E-2.csv" "E-2" "Crate E - Generic"
+call :procCrate "Crates E-3.csv" "E-3" "Crate E - Generic"
+
+call :procCrate "Crates F-1.csv" "F-1" "Crate F - Chemistry"
+call :procCrate "Crates F-2.csv" "F-2" "Crate F - Chemistry"
+
+:: UNUSED
+rem call :procCrate "origcsv_Crates.csv" "A-1" "Crate A Science - Primary"
+
+
+:end
+ENDLOCAL
+GOTO :EOF
+
+
+:procCrate
+rem IF [%1]==[] (
+rem 	echo.
+rem 	echo ************************************************************************
+rem 	echo.
+rem 	echo This script must reside directly inside your TT4.02_Science Resources_Crates folder which must furthermore contain your CSV files for each crate.
+rem 	echo.
+rem 	echo Usage: %0 "csv-file" "crate" "cratename"
+rem 	echo		Example: %0 "C:\PATH\TO\Crates (TT4.02) A-1.csv" "A-1" "Crate A Science - Primary"
+rem 	echo.
+rem 	echo ************************************************************************
+rem 	echo.
+rem 	goto end
+rem )
+
+
+:: store params stripped of any double quotes
+:: https://stackoverflow.com/questions/1964192/removing-double-quotes-from-variables-in-batch-file-creates-problems-with-cmd-en
+set csvfile=%1
+set csvfile=%csvfile:"=%
+
+set crate=%2
+set crate=%crate:"=%
+
+set crateName=%3
+set crateName=%crateName:"=%
+
+echo.
+echo PROCESSING:
+echo 	CSVfile: %csvfile%
+echo 	Crate: [%crate%] --- Crate Name: %crateName%
+echo ***********
+echo.
+
+FOR /F "usebackq tokens=1,2 delims=," %%G in ("%csvfile%") do (
+
+	rem echo %%G
+	rem echo(%%H
+
+	:: skip rows in the CSV file that aren't of the form: activity,activityName
+	rem https://stackoverflow.com/questions/36355490/continue-equivalent-command-in-nested-loop-in-windows-batch
+	:: only if we have an activityName column for this row, do we process this row
+	if [%%H]==[] (
+		echo ^(not an activity row^)
+	) else (
+		set activity=%%G
+		
+		rem set activity=%activity:"=%
+		
+		rem set activityName=%%H inside for loop will leave activityName empty without doing setlocal enabledelayedexpansion above
+		rem https://stackoverflow.com/questions/23525681/how-to-print-variable-inside-loop-in-batch-script
+		rem Didn't seem to make a difference for me: https://stackoverflow.com/questions/4333754/problem-with-search-and-replace-batch-file	
+		
+		
+		set activityName=%%H
+		
+		rem echo Activity Name (before cleaning): !activityName!
+		
+		rem clean the activity name to deal with ampersands, and the stylised double quotes, hyphens and ellipses,
+		rem single quotes, and any others (e.g. 3/4 fraction) that change in Notepad++'s Encoding > Encode in UTF-8 file-view
+		set activityName=!activityName:
+=...!
+		set activityName=!activityName:=-!
+		set activityName=!activityName:^&=and!
+		set activityName=!activityName:/=;!
+		rem finally, remove the regular double quotes around parameter before converting stylised double quotes to ASCII double quotes
+		REM set activityName=!activityName:"=!
+		set activityName=!activityName:="!
+		set activityName=!activityName:="!
+		set activityName=!activityName:='!
+		set activityName=!activityName:='!
+		set activityName=!activityName:Ÿ=3/4!
+		
+		echo Activity: [!activity!] --- ^(cleaned^) Activity Name: [!activityName!]
+		
+		if not exist %odir%\%crate% mkdir %odir%\%crate%
+		pushd %odir%\%crate%
+		
+		rem NOTE: setdelayedexpansion to use exclamation marks requires double caret escaping to echo the exclamation mark
+		echo ^<?xml version="1.0" encoding="UTF-8"?^> > metadata.xml
+		echo ^<^^!DOCTYPE DirectoryMetadata SYSTEM "http://greenstone.org/dtd/DirectoryMetadata/1.0/DirectoryMetadata.dtd"^> >> metadata.xml
+		echo ^<DirectoryMetadata^> >> metadata.xml
+		echo 	^<FileSet^> >> metadata.xml
+		echo 		^<FileName^>.*^</FileName^> >> metadata.xml
+		echo 		^<Description^> >> metadata.xml
+		echo				^<Metadata mode="accumulate" name="er.Crate"^>%crate%^</Metadata^> >> metadata.xml
+		echo				^<Metadata mode="accumulate" name="er.CrateName"^>%crateName%^</Metadata^> >> metadata.xml
+		echo 		^</Description^> >> metadata.xml
+		echo 	^</FileSet^> >> metadata.xml
+		echo ^</DirectoryMetadata^> >> metadata.xml
+		
+		popd
+		
+		if not exist %odir%\%crate%\!activity! mkdir %odir%\%crate%\!activity!
+		pushd %odir%\%crate%\!activity!
+
+		echo ^<?xml version="1.0" encoding="UTF-8"?^> > metadata.xml
+		echo ^<^^!DOCTYPE DirectoryMetadata SYSTEM "http://greenstone.org/dtd/DirectoryMetadata/1.0/DirectoryMetadata.dtd"^> >> metadata.xml
+		echo ^<DirectoryMetadata^> >> metadata.xml
+		echo 	^<FileSet^> >> metadata.xml
+		echo 		^<FileName^>.*^</FileName^> >> metadata.xml
+		echo 		^<Description^> >> metadata.xml
+	rem Crate|Activity
+	rem	echo				^<Metadata mode="accumulate" name="er.CrateHierarchy"^>%crate%^|!activity!^</Metadata^> >> metadata.xml
+	rem Crate: CrateName|Activity: ActivityName
+		echo				^<Metadata mode="accumulate" name="er.CrateHierarchy"^>%crate%: %crateName%^|!activity!: !activityName!^</Metadata^> >> metadata.xml
+		echo				^<Metadata mode="accumulate" name="er.Activity"^>!activity!^</Metadata^> >> metadata.xml
+		echo				^<Metadata mode="accumulate" name="er.ActivityName"^>!activityName!^</Metadata^> >> metadata.xml
+		echo 		^</Description^> >> metadata.xml
+		echo 	^</FileSet^> >> metadata.xml
+		echo ^</DirectoryMetadata^> >> metadata.xml
+		
+		if exist "%origdir%\Crate %crate%\!activity!*" copy "%origdir%\Crate %crate%\!activity!.*" .		
+		if exist "%origdir%\Crate %crate%\!activity!_*" copy "%origdir%\Crate %crate%\!activity!_*" .
+		if exist "%origdir%\Crate %crate%\!activity!-*" copy "%origdir%\Crate %crate%\!activity!-*" .
+		if exist "%origdir%\Crate %crate%\!activity!^(*" copy "%origdir%\Crate %crate%\!activity!^(*" .
+		
+		if exist "%origdir%\Crate %crate%\Extra*" copy "%origdir%\Crate %crate%\Extra*" ..\.
+		
+		rem check for dir https://stackoverflow.com/questions/138981/how-to-test-if-a-file-is-a-directory-in-a-batch-script
+		rem xcopy for recursively copying dirs: https://stackoverflow.com/questions/13314433/batch-file-to-copy-directories-recursively
+		rem For loop to call xcopy on file glob *directory* name match https://ss64.com/nt/for.html
+		rem For each source directory of files, copy it into the destination too		
+		FOR /D %%G IN ("%origdir%\Crate %crate%\!activity!*") do (xcopy "%%G" . /s /e /i)		
+		popd
+	
+	)
+)
+echo ********************************
+goto :eof	
+
+:: UNUSED
+:skipLine
+	echo ^(not a row^)
+	goto :eof
+
+:: UNUSED Call procLine with double-quoted %%G/activity and %%H/activityName (first 2 columns of csv files)
+:procLine	
+	set activity=%1
+	set activity=%activity:"=%
+	
+	:: set activityName=%%H inside for loop will leave activityName empty without doing setlocal enabledelayedexpansion above
+	:: https://stackoverflow.com/questions/23525681/how-to-print-variable-inside-loop-in-batch-script
+	:: Didn't seem to make a difference for me: https://stackoverflow.com/questions/4333754/problem-with-search-and-replace-batch-file	
+	
+	
+	set activityName=%2
+	
+	rem echo Activity Name (before cleaning): !activityName!
+	
+	:: clean the activity name to deal with ampersands, and the stylised double quotes, hyphens and ellipses, etc.	
+	set activityName=!activityName:
+=...!
+	set activityName=!activityName:=-!
+	set activityName=!activityName:='!
+	set activityName=!activityName:='!
+	set activityName=!activityName:Ÿ=3/4!
+	set activityName=!activityName:^&=and!
+	rem finally, remove the regular double quotes around parameter before converting stylised double quotes to ASCII double quotes	
+	set activityName=!activityName:"=!
+	set activityName=!activityName:="!
+	set activityName=!activityName:="!	
+		
+	echo Activity: [!activity!] --- (cleaned) Activity Name: [!activityName!]
+	
+ 
+	if not exist %odir%\%crate% mkdir %odir%\%crate%
+	pushd %odir%\%crate%
+	
+	echo ^<?xml version="1.0" encoding="UTF-8"?^> > metadata.xml
+	echo ^<!DOCTYPE DirectoryMetadata SYSTEM "http://greenstone.org/dtd/DirectoryMetadata/1.0/DirectoryMetadata.dtd"^> >> metadata.xml
+	echo ^<DirectoryMetadata^> >> metadata.xml
+	echo 	^<FileSet^> >> metadata.xml
+	echo 		^<FileName^>.*^</FileName^> >> metadata.xml
+	echo 		^<Description^> >> metadata.xml
+	echo				^<Metadata mode="accumulate" name="er.Crate"^>%crate%^</Metadata^> >> metadata.xml
+	echo				^<Metadata mode="accumulate" name="er.CrateName"^>%crateName%^</Metadata^> >> metadata.xml
+	echo 		^</Description^> >> metadata.xml
+	echo 	^</FileSet^> >> metadata.xml
+	echo ^</DirectoryMetadata^> >> metadata.xml
+	
+	popd
+	
+	if not exist %odir%\%crate%\!activity! mkdir %odir%\%crate%\!activity!
+	pushd %odir%\%crate%\!activity!
+
+	echo ^<?xml version="1.0" encoding="UTF-8"?^> > metadata.xml
+	echo ^<!DOCTYPE DirectoryMetadata SYSTEM "http://greenstone.org/dtd/DirectoryMetadata/1.0/DirectoryMetadata.dtd"^> >> metadata.xml
+	echo ^<DirectoryMetadata^> >> metadata.xml
+	echo 	^<FileSet^> >> metadata.xml
+	echo 		^<FileName^>.*^</FileName^> >> metadata.xml
+	echo 		^<Description^> >> metadata.xml
+:: Crate|Activity
+rem	echo				^<Metadata mode="accumulate" name="er.CrateHierarchy"^>%crate%^|!activity!^</Metadata^> >> metadata.xml
+:: Crate: CrateName|Activity: ActivityName
+	echo				^<Metadata mode="accumulate" name="er.CrateHierarchy"^>%crate%: %crateName%^|!activity!: !activityName!^</Metadata^> >> metadata.xml
+	echo				^<Metadata mode="accumulate" name="er.Activity"^>!activity!^</Metadata^> >> metadata.xml
+	echo				^<Metadata mode="accumulate" name="er.ActivityName"^>!activityName!^</Metadata^> >> metadata.xml
+	echo 		^</Description^> >> metadata.xml
+	echo 	^</FileSet^> >> metadata.xml
+	echo ^</DirectoryMetadata^> >> metadata.xml
+	
+	
+	if exist "%origdir%\Crate %crate%\!activity!*" copy "%origdir%\Crate %crate%\!activity!.*" .
+	if exist "%origdir%\Crate %crate%\!activity!_*" copy "%origdir%\Crate %crate%\!activity!_*" .
+	if exist "%origdir%\Crate %crate%\!activity!-*" copy "%origdir%\Crate %crate%\!activity!-*" .
+	if exist "%origdir%\Crate %crate%\!activity!^(*" copy "%origdir%\Crate %crate%\!activity!^(*" .
+		
+	if exist "%origdir%\Crate %crate%\Extra*" copy "%origdir%\Crate %crate%\Extra*" ..\.
+	
+	rem https://stackoverflow.com/questions/138981/how-to-test-if-a-file-is-a-directory-in-a-batch-script
+	rem https://stackoverflow.com/questions/13314433/batch-file-to-copy-directories-recursively
+	rem if exist "%origdir%\Crate %crate%\!activity!*\NUL" xcopy "%origdir%\Crate %crate%\!activity!*" . /s /e /i
+	rem For folders of docs for an activity, the following XXcreates a target subfolder named activity and into it,XX copies the files from the source folder
+	rem https://ss64.com/nt/for.html
+	for /D %%G in ("%origdir%\Crate %crate%\!activity!*") do (xcopy "%%G" . /s /e /i)
+
+ 
+	::if exist "%origdir%\Crate %crate%\Extra*" copy "%origdir%\Crate %crate%\Extra*" .\..
+	popd
+	endlocal
+	
+)
+GOTO :EOF
+
+
