Changeset 34132 for main

Timestamp:

2020-05-30T16:01:47+12:00 (4 years ago)

Author:

ak19

Message:

Committing the commoncrawl site of Nutch recrawls of our CC data where content-language = MRI. 1. Contains the collection configuration files, but also the keep-urls *.txt files in the etc folder, used by NutchTextDumpPlugin to filter URLs of interest. 2. The import_nutchDumpTxtsOfcrawledMRICC.tar.gz file needs to decompressed into any of the collections that need to be rebuilt. This contains just the Nutch dump.txt files (in their siteID folders) as I've removed the binary files. 3. The script moveDumpTxtFilesIntoImport.sh can be used to generate such cut down versions of the Nutch crawled folders that contain only the dump.txt files within their siteID folders. 4. In the next commit, I'll try to add svn externals to get the import_nutchDumpTxtsOfcrawledMRICC.tar.gz from sitelevel into the collection folders for the 2 current collections in this site.

Location:

main/trunk/model-sites-dev/commoncrawl

Files:

• . (added)
• collect (added)
• collect/allIsMRIForDomainShortlist (added)
• collect/allIsMRIForDomainShortlist/etc (added)
• collect/allIsMRIForDomainShortlist/etc/all_isMRIPages_forManual_containsMRIDomainListing.txt (added)
• collect/allIsMRIForDomainShortlist/etc/collectionConfig.bak (added)
• collect/allIsMRIForDomainShortlist/etc/collectionConfig.xml (added)
• collect/allIsMRIForDomainShortlist/etc/oai-inf.jdb (added)
• collect/allIsMRIForDomainShortlist/etc/oai-inf.jdb.bak (added)
• collect/allIsMRIForDomainShortlist/etc/oai-inf.lg (added)
• collect/allIsMRIForDomainShortlist/images (added)
• collect/allIsMRIForDomainShortlist/metadata (added)
• collect/allIsMRIForDomainShortlist/metadata/ex.mds (added)
• collect/allIsMRIForDomainShortlist/metadata/greenstone.mds (added)
• collect/allIsMRIForDomainShortlist/metadata/profile.xml (added)
• collect/allIsMRIForDomainShortlist/script (added)
• collect/allIsMRIForDomainShortlist/style (added)
• collect/allismri (added)
• collect/allismri/etc (added)
• collect/allismri/etc/collectionConfig.bak (added)
• collect/allismri/etc/collectionConfig.xml (added)
• collect/allismri/etc/isMRI_urls.txt (added)
• collect/allismri/etc/oai-inf.jdb (added)
• collect/allismri/etc/oai-inf.jdb.bak (added)
• collect/allismri/etc/oai-inf.lg (added)
• collect/allismri/images (added)
• collect/allismri/metadata (added)
• collect/allismri/metadata/ex.mds (added)
• collect/allismri/metadata/greenstone.mds (added)
• collect/allismri/metadata/profile.xml (added)
• collect/allismri/script (added)
• collect/allismri/style (added)
• etc (added)
• import_nutchDumpTxtsOfcrawledMRICC.tar.gz (added)
• moveDumpTxtFilesIntoImport.sh (added)
• resources (added)
• resources/siteConfig.properties (added)
• siteConfig.xml (added)