source: gs3-extensions/fedora/README@ 26380

********************************************************************************************************************
                            CONTENTS
********************************************************************************************************************

A. To install your GS3 Fedora extension from fedora3-gs3ext.tar.gz

B. How to manually build fedora collections

C. The customisations that have been made for the extension

D. How to manually set up Fedora 3.6.1 with Fedora GSearch 2.5 from scratch
    Installing Fedora
    Installing Fedora GSearch
    Debugging


Version:
Greenstone 3.05
Fedora 3.6.1
Fedora GSearch 2.5

Web services pages once installed: 
http://localhost:8383/fedora/services
http://localhost:8383/fedoragsearch/services

-- ak19, 2012



********************************************************************************************************************
    A. To install your Fedora extension for Greenstone 3 from the fedora3-gs3ext.tar.gz distribution file
********************************************************************************************************************

(Tested to work on Linux machines.)

0. You will need Greenstone 3.05 or a fresh checkout from SVN, already installed.

1. Download the Fedora Extension for Greenstone 3: http://trac.greenstone.org/browser/gs3-extensions/fedora/fedora3-gs3ext.tar.gz

2. Save into your Greenstone3/ext folder

3. Untar and decompress it:
    $ tar -xvzf fedora3-gs3ext.tar.gz

4. This will create a fedora3 folder in your Greenstone3/ext folder. 

5. Open Greenstone3/build.properties and find the section:

    ##These need to be uncommented if using Fedora and Fedora GSearch with Greenstone's tomcat
    fedora.home=#${basedir}/ext/fedora3
    #fedora.maxpermsize=-XX:MaxPermSize=128m
    #fedora.password=pounamu

Uncomment all these properties and change the value for the fedora password if necessary:

    ##These need to be uncommented if using Fedora and Fedora GSearch with Greenstone's tomcat
    fedora.home=${basedir}/ext/fedora3
    fedora.maxpermsize=-XX:MaxPermSize=128m
    fedora.password=pounamu     


6. Use a terminal to go into this folder:
    $ cd Greenstone3/ext/fedora3

7. If you type `ant` in the x-term, it will print the Usage message which will display the list of commands available and the order in which to execute them for proper installation. It's a 5 step process (explained in detail further below):

    $ ant -Dgsdl3.home=<type-full-path-to-your-gs3> start-fedora-install

    $ ant start

    $ ant -Dgsdl3src.home=<type-full-path-to-your-gs3> continue-fedora-install

    $ ant stop

    $ ant -Dgsdl3src.home=<type-full-path-to-your-gs3> finish-fedora-install


That's it. 
Now you can start tomcat again and visit http://localhost:8383/fedora and http://localhost:8383/fedoragsearch/rest to check the pages display.

8. You can start creating Greenstone Fedora collections with FLI. Run FLI from the GS3 toplevel folder by typing:
    $ ./gli/fli.sh



DETAILS ON STEP 7:
a. From your Greenstone3/ext/fedora3 folder, run:
    $ ant -Dgsdl3.home=<type-full-path-to-your-gs3> start-fedora-install

This step prepares the fedora and gsearch war files for deployment (by customising template files) and moves them into Greenstone 3's tomcat.

b. Start tomcat by going into your GS3 toplevel folder and running: 
    $ ant start

This step gets Greenstone's tomcat to deploy the fedora and fedoragsearch war files.
You may want to check your Greenstone library is working by visiting http://localhost:8383/greenstone3 in your browser

c. Go back into Greenstone3/ext/fedora3 and run:
    $ ant -Dgsdl3src.home=<type-full-path-to-your-gs3> continue-fedora-install

This target configures fedora-gsearch now that it has been deployed. It runs a fedora-gsearch ant build file to customise properties files that are internal to fedora-gsearch.

d. Stop tomcat by going back into your GS3 toplevel folder and running: 
    $ ant stop

You may want to check java instances running tomcat have indeed stopped:
    $ ps aux | grep tomcat

You will need to kill the process if any tomcat is still running at this stage (this can happen if you accidentally start tomcat several times in succession without stopping it).
    $ kill -9 <process-id> 

e. Now that tomcat has stopped, go back into Greenstone3/ext/fedora3 and run:
    ant -Dgsdl3src.home=<Full-Path-To-GS3> finish-fedora-install

This will clean up the extension's war files in tomcat/packages/webapps now that they have already been deployed. (Do not perform this final step when tomcat is still running, else your fedora extension webapps will become undeployed.)



When installation is completed, there will be a fedora3 folder in Greenstone3/ext, which you should not delete, since the Fedora Digital objects will be stored here, and the FedoraGSearch index will be created here. 

Further, the Fedora extension installation process would have made tomcat deploy several webapps in packages/tomcat/webapps. The resulting folders are:
- fedora
- fedora-demo
- fedoragsearch
- fop
- imagemanip
- saxon

(Note that the installation process will have removed the Fedora extension war files it put into packages/tomcat/webapps during installation.)



********************************************************************************************************************
                    B. Manual building of fedora collections
********************************************************************************************************************

You can use FLI to create, build and preview Fedora collections using Greenstone.
 
The same can be done manually by calling the g2f perl scripts from the command line.

1. Fedora needs to be running, so ensure the Greenstone tomcat is running:
    ant start

2. Set up the GS3 environment:
    source gs3-setup.bash

3. Create a new collection with mkcol.pl. Call it "fedora1" for example 
    mkcol.pl -collectdir /<GS3>/web/sites/localsite/collect fedora1

4. Run import and build to ingest the new collection

- First put the documents you want into the import directory of your new collection.

- g2f-import.pl -hostname localhost -port 8383 -password pounamu -removeold -collectdir /<GS3>/web/sites/localsite/collect fedora1
(password may be optional at this stage, by include it for convenience)

- g2f-buildcol.pl -hostname localhost -port 8383 -password pounamu -removeold -collectdir /<GS3>/web/sites/localsite/collect fedora1


5. If building the GS3 demo collection as a fedora collection: 
- use FLI to transfer dls to dc metadata upon Gathering the documents.
- then before building, turn on the description_tags in the HTMLPlugin.

6. If you ran FLI, exit it. It should stop the Greenstone server.

With fedora installed, it's always good to check that the java process that launched tomcat has indeed stopped:
    ps aux | grep "tomcat"

(Since things will fail if multiple instances of this same tomcat are running, kill any java processes that are referring to tomcat.)

7. Create an index folder in the new fedora collection folder. Create a buildConfig.xml file in it containing the following:

<buildConfig> 
  <metadataList/>
  <serviceRackList>                                                            
    <serviceRack name="FedoraServiceProxy" />
  </serviceRackList>
</buildConfig>


8. Restart the GS3 server.
    ant start

9. Visit the collection from the Greenstone collections page



********************************************************************************************************************
            C. The customisations that have been made for the extension
********************************************************************************************************************

This section is of use when Fedora or GSearch is updated or if you want to update the fedora3-gs3ext.tar.gz distribution file with further customisations.

The first version of the Fedora extension for Greenstone uses Fedora 3.6.1 and the GSearch 2.5 (which goes with Fedora 3.6.1).

Some template files were added to Fedora and Fedora Gsearch's distribution files in order to customise these for installing them within a Greenstone 3 installation. These files are committed to SVN (without directory structure) at http://trac.greenstone.org/browser/gs3-extensions/fedora/fedora-files and http://trac.greenstone.org/browser/gs3-extensions/fedoragsearch-files

They can be edited there if and when necessary, but they would then need to be included in the extension's distribution file fedora3-gs3ext.tar.gz to update its own existing copies.

That's because these template files are included in the Fedora Extension for GS3 (fedora3-gs3ext.tar.gz). The extension's build.xml modifies these template files when installing the Fedora Extension into Greenstone 3. The template files contain placeholder strings that get updated when ant targets are run over the build.xml file.


1. The customisation files for Fedora 3.6.1 are:

- ./install/install.properties.in
- ./server/config/spring/akubra-llstore.xml.in
- ./server/config/fedora.fcfg.in
- ./server/config/fedora-users.xml.in

All these files get converted to filenames without the ".in" suffix upon installation, and placeholder strings in these template files get replaced. The replacements are on the strings 
- @GSDL3SRCHOME@, 
- @FEDORA_HOME@ (set to gsdl3srchome/ext/fedora3), 
- @tomcatserver@, 
- @tomcatport@, 
- @tomcatshutdownport@, 
- @fedorapassw@. 
Except for the first, which needs to be set when running the Fedora extension's ant targets, all the remaining ones can be specified 


2. The customisation files for Fedora GSearch 2.6 are located in the "adjust_war_files" subfolder of fedora3-gs3ext.tar.gz file. This custom folder contains the official unpacked fedoragsearch folder, but with the following customisation files:

- adjust_war_files/fedoragsearch/WEB-INF/web.xml
- adjust_war_files/fedoragsearch/FgsConfig/fgsconfig-basic.properties.in 
- adjust_war_files/fedoragsearch/FgsConfig/FgsConfigIndexTemplate/Lucene/foxmlToLucene.xslt 
- adjust_war_files/fedoragsearch/FgsConfig/FgsConfigIndexTemplate/Lucene/foxmlToLuceneGenerated.xslt
- adjust_war_files/fedoragsearch/client/runRESTClient.sh
- adjust_war_files/fedoragsearch/client/runSOAPClient.sh
- adjust_war_files/fedoragsearch/client/runSOAPClient.bat

All these files will get copied into the same locations within <GS3>/packages/tomcat/webapps/fedoragsearch. The file fgsconfig-basic.properties.in will get copied as fgsconfig-basic.properties but with the previously-listed placeholder strings replaced.


3. There's also the fedora.xml.in inside the "adjust_war_files" folder of the unpacked extension. This template file will be copied over as <GS3>/packages/tomcat/conf/Catalina/localhost/fedora.xml during the extension installation process, also with placeholder strings replaced.


DETAILS TO STEP 2 (Fedora GSearch customisation files):

a. The changes to web.xml are that the authorisation filters are commented out:

<!--    <filter-mapping>
      <filter-name>EnforceAuthnFilter</filter-name>
      <servlet-name>AxisServlet</servlet-name>
    </filter-mapping>
    <filter-mapping>
      <filter-name>EnforceAuthnFilter</filter-name>
      <servlet-name>GenericSearchREST</servlet-name>
    </filter-mapping>
    <filter-mapping>
      <filter-name>EnforceAuthnFilter</filter-name>
      <url-pattern>/index.html</url-pattern>
    </filter-mapping>
    <filter-mapping>
      <filter-name>EnforceAuthnFilter</filter-name>
      <url-pattern>/rest</url-pattern>
    </filter-mapping>
-->


b. The 3 client scripts runRESTClient.bat, runRESTClient.sh and runSOAPClient.sh have been modified to be runnable from any directory, as they will be called by GS3's g2f perl scripts. 

- The bash files have been made executable on extraction of the GS3 Fedora extension and the following has been added to the top of the files:

    # Need to run this script from its own directory instead of whichever directory it may be called from
    thisdir="`dirname \"$0\"`"
    thisdir="`cd \"$thisdir\" && pwd`"
    cd "$thisdir"


- Additions to the bat scripts are at the top and bottom:

    @echo off

    ::pushd "%CD%"
    set startdir=%CD%
    CD /D "%~dp0"

    ...

    :: popd
    cd "%startdir%"
    set startdir=


c. The changes to fedoragsearch/FgsConfig/FgsConfigIndexTemplate/Lucene's foxmlToLucene.xslt and foxmlToLuceneGenerated.xslt are identical. The following xslt has been added:

- Add the following namespaces to the namespace declarations at the top:
    xmlns:ex="http://www.greenstone.org/namespace/fake/ex"
    xmlns:dls="http://www.greenstone.org/namespace/fake/dls"

- Add custom indexing for EX and DLS datastreams below the comment on datastreams:

    <!-- a datastream is fetched, if its mimetype 
         can be handled, the text becomes the value of the field. 
         This is the version using PDFBox,
         below is the new version using Apache Tika. -->

    <xsl:for-each select="foxml:datastream[starts-with(@ID,'EX')]/foxml:datastreamVersion[last()]/foxml:xmlContent/ex:ex/ex:metadata">
      <IndexField index="TOKENIZED" store="YES" termVector="YES">
        <xsl:attribute name="IFname">
          <xsl:value-of select="concat('ex.', @name)"/>
        </xsl:attribute>
        <xsl:value-of select="text()"/>
      </IndexField>
    </xsl:for-each>
    
    <xsl:for-each select="foxml:datastream[starts-with(@ID,'DLS')]/foxml:datastreamVersion[last()]/foxml:xmlContent/dls:dls/dls:metadata">
      <IndexField index="TOKENIZED" store="YES" termVector="YES">
        <xsl:attribute name="IFname">
          <xsl:value-of select="concat('dls.', @name)"/>
        </xsl:attribute>
        <xsl:value-of select="text()"/>
      </IndexField>
    </xsl:for-each>

- Near the end of the XSLT files, just after index for the "foxml.all.text" field, allow just the full text of the documents to be indexed by adding an index for the ds.fulltext field:

    <IndexField IFname="ds.fulltext" index="TOKENIZED" store="YES" termVector="YES">
      <xsl:for-each select="//foxml:datastream[@CONTROL_GROUP='M' or @CONTROL_GROUP='E' or @CONTROL_GROUP='R']">
        <xsl:value-of select="exts:getDatastreamText($PID, $REPOSITORYNAME, @ID, $FEDORASOAP, $FEDORAUSER, $FEDORAPASS, $TRUSTSTOREPATH, $TRUSTSTOREPASS)"/>
        <xsl:text>Â </xsl:text>
      </xsl:for-each>
    </IndexField>


Further datastreams and metadata sets can be indexed by adding similar namespace declarations and xml elements to these 2 XSLT files. 
If the changes are not meant to apply for all Greenstone users, individual Greenstone users can add fields for indexing by making such changes to just the file fedoragsearch/FgsConfig/FgsConfigIndexTemplate/Lucene/foxmlToLuceneGenerated.xslt



********************************************************************************************************************
            D. To manually set up Fedora with Fedora GSearch 2.5 from scratch
********************************************************************************************************************

When installing Fedora, you can choose to install it outside Greenstone, and also instruct it to use its own tomcat. However, in the following, /GS3/ext/fedora3 is still assumed to be FEDORA_HOME and hence Fedora's installation location and it's assumed to be installed to use Greenstone's tomcat. 

The Fedora GSearch war file will need to be unpacked in the tomcat/webapps folder of whichever tomcat Fedora is using. In the following, this is still Greenstone's tomcat.

Wherever Fedora is installed, that's where the digital objects and their datastreams will be stored, and that's where Fedora GSearch will create its GSearch index.


********************************
INSTALLING FEDORA
********************************
1. Set FEDORA_HOME=/<GS3>/ext/fedora3 in .profile (emacs ~/.profile)
source ~/.profile

2. Run fedora installer:
    java -jar fcrepo-installer-3.6.1.jar

Install by specifying to use the existing tomcat (existingTomcat), and provide the Greenstone3 tomcat stop and start ports (usually 8383 and 8305).
Then set the existingTomcat to the Greenstone3 one: /<GS3>/packages/tomcat
Turn on messaging, as this is necessary for Fedora GSearch.

- Installation options:

custom
FEDORA_HOME
pwd: pounamu
host: <full-tomcat.server-host-name>
def server context: fedora
default false for user authentication for APIA-A
false for SSL availability
existingTomcat
path to tomcat: </full-path-to-GS3>/packages/tomcat
tomcat listen port: 8383
tomcat shutdown port:8305
derby:included
upstream HTTP authentication: (default) false
FeSL AuthZ: (default) false
XACML policy enforcement enabled: false
Low Level Storage: (default) akubra-fs 
Resource Index: true
Enable Messaging: true
Messaging Provider URI: [default is vm:(broker:(tcp://localhost:61616))]
Deploy local services and demos: (default) true


- The Fedora installation options used are to be found (after installation) in fedora3/install/install.properties:

#Install Options
#Fri Sep 21 15:29:29 NZST 2012
ri.enabled=true
messaging.enabled=false
apia.auth.required=false
database.jdbcDriverClass=org.apache.derby.jdbc.EmbeddedDriver
upstream.auth.enabled=false
ssl.available=false
database.jdbcURL=jdbc\:derby\:/<GS3>/ext/fedora3/derby/fedora3;create\=true
database.password=fedoraAdmin
database.username=fedoraAdmin
fesl.authz.enabled=false
tomcat.shutdown.port=8305
deploy.local.services=true
xacml.enabled=false
tomcat.http.port=8383
fedora.serverHost=<full-tomcat.server-host-name>
database=included
database.driver=included
fedora.serverContext=fedora
llstore.type=akubra-fs
tomcat.home=/<GS3>/packages/tomcat
fedora.home=/<GS3>/ext/fedora3
install.type=custom
servlet.engine=existingTomcat
fedora.admin.pass=pounamu


- If trying to turn a custom installation into one for distribution, you would need to modify the following 4 files by inserting placeholder strings where applicable for the tomcatserver name, tomcat listen port and shutdown port, fedorapassword, GSLD3SRCHOME and FEDORA_HOME:

fedora3/install/install.properties
fedora3/server/config/spring/akubra-llstore.xml
fedora3/server/config/fedora.fcfg
fedora3/server/config/fedora-users.xml

Then you would rename these files with the suffix .in 


3. Just to confirm there are no differences between the server.xml fedora has generated for fedora, and the server.xml of GS3's tomcat, run a diff:

    diff -w /<GS3>/packages/tomcat/conf/server.xml </GS3/ext/>fedora3/install/server.xml

4. Copy the fedora war files from fedora3/install into packages/tomcat/webapps
fedora.war, fedora-demo.war, fop.war, imagemanip.war, saxon.war

5. Copy the jar files xalan.jar, serializer.jar (and xsltc.jar) from Greenstone3's web/WEB-INF/lib into Greenstone3's packages/tomcat/lib so that fedora has access to the xalan version of the TransformerFactoryImpl class.

There's also a xalan.jar in /<GS3>/packages/tomcat/webapps/fop/WEB-INF/lib
So this can be copied into /<GS3>/packages/tomcat/lib/. instead of the Greenstone version.

6. Create the file /<GS3>/packages/tomcat/conf/Catalina/localhost/fedora.xml
containing:

<?xml version="1.0" encoding="UTF-8"?>
<Context>
  <Parameter name="fedora.home" value="/<GS3>/ext/fedora3" />
</Context>


7. Before running the Greenstone server, make a copy of the fedora3 folder. Because once you start up tomcat, and visit the fedora home page, it will create a lot of files customised to the location of the current installation.

8. Visit http://localhost:8383/greenstone3 and 
    http://localhost:8383/fedora 

to confirm both work.


********************************
INSTALLING FEDORA GSEARCH
********************************
1. Download Fedora GSearch 2.5

2. Stop tomcat. Copy the fedoragsearch.war files into /<GS3>/packages/tomcat/webapps

If Fedora was not installed with messaging turned on, then turn it on in now /<GS3>/ext/fedora3/server/config/fedora.fcfg:

 <module role="org.fcrepo.server.messaging.Messaging" class="org.fcrepo.server.messaging.MessagingModule">
    <comment>Fedora's Java Messaging Service (JMS) Module</comment>
    <param name="enabled" value="true"/>
 ...

3. Update the following properties in /<GS3>/packages/tomcat/webapps/fedoragsearch/FgsConfig/fgsconfig-basic.properties

gsearchBase=http://<tomcat.server>:8383
gsearchUser=fedoraAdmin
gsearchPass=<fedora.password>
local.FEDORA_HOME=/<GS3>/ext/fedora3
finalConfigPath=/<GS3>/packages/tomcat/webapps/fedoragsearch/WEB-INF/classes

fedoraBase=http://<tomcat.server>:8383
fedoraPass=<fedora.password>

4. Add the edited foxmlToLucene.xslt & foxmlToLuceneGenerated.xslt files to /<GS3>/packages/tomcat/webapps/fedoragsearch/FgsConfig/FgsConfigIndexTemplate/Lucene

5. Add the edited runRESTClient.sh (and runSOAPClient.sh, runRESTClient.bat) to /<GS3>/packages/tomcat/webapps/fedoragsearch/client
And give them execute permissions, unless they already have it.

6. In a text editor, open up /<GS3>/packages/tomcat/webapps/fedoragsearch/WEB-INF/web.xml
and comment out the authentication filters (one or two of these is probably all that is necessary):

<!--
    <filter-mapping>
      <filter-name>EnforceAuthnFilter</filter-name>
      <servlet-name>AxisServlet</servlet-name>
    </filter-mapping>
    <filter-mapping>
      <filter-name>EnforceAuthnFilter</filter-name>
      <servlet-name>GenericSearchREST</servlet-name>
    </filter-mapping>
    <filter-mapping>
      <filter-name>EnforceAuthnFilter</filter-name>
      <url-pattern>/index.html</url-pattern>
    </filter-mapping>
    <filter-mapping>
      <filter-name>EnforceAuthnFilter</filter-name>
      <url-pattern>/rest</url-pattern>
    </filter-mapping>
-->

Maybe what I really need to comment out is:
    <filter>
      <filter-name>EnforceAuthnFilter</filter-name>
      <filter-class>org.fcrepo.server.security.servletfilters.FilterEnforceAuthn</filter-class>
    </filter>

7. Start tomcat to have it deploy fedoragsearch.
This will unpack fedoragsearch.

8. With tomcat running, configure fedoragsearch, which will adjust various config files. You will need to go into the deployed fedoragsearch webapps folder's FgsConfig folder first to be able to run the configuration target:

<GS3>/packages/tomcat/webapps/fedoragsearch/FgsConfig>ant -f fgsconfig-basic.xml

9. Run tomcat and check fedoragsearch has been properly installed by visiting:
http://localhost:8383/fedoragsearch/rest

10. If you wish to remove the various war files from GS3's tomcat webapps folder, you will need to stop tomcat first before deleting the war files, because doing so when tomcat is running will undeploy those webapps.


********************************
DEBUGGING
********************************

Logs to consult when debugging:

1. Check /<GS3>/packages/tomcat/logs/catalina.out

2. </GS3/ext/>fedora3/server/logs/fedora.log for Fedora error logging.

3. To turn on FedoraGSearch's logging of debug statements (Huge files, can reach 500 Mb in a day. Do this only when debugging):
- /<GS3>/packages/tomcat/webapps/fedoragsearch/WEB-INF/classes/log4j.xml is already set to output DEBUG statements and higher logging levels
- So open </GS3/ext/>fedora3/server/logs/fedoragsearch.daily.log to look at debug messages in case fedoragsearch doesn't work as expected.


********************************************************************************************************************