Index: /gs3-extensions/maori-lang-detection/MoreReading/CommonCrawl.txt
===================================================================
--- /gs3-extensions/maori-lang-detection/MoreReading/CommonCrawl.txt	(revision 33376)
+++ /gs3-extensions/maori-lang-detection/MoreReading/CommonCrawl.txt	(revision 33376)
@@ -0,0 +1,114 @@
+http://webdatacommons.org/
+
+https://dzone.com/articles/need-billions-of-web-pages-dont-bother-crawling
+
+	Ran Geva 2017-04-09
+	Like (0)
+
+	Excellent article! CommonCrawl is an amazing resourouce. You should also check out webdatacommons.org that is using their data and extract structured data (using RDFa, Microdata..)
+
+	If I may add a shameless plug here and tell you about Webhose.io [PAYWARE/SERVICES]. We provide an API to structured web data. The idea is the same as the one you presented. Instead of crawling the web, we already crawl millions of sites, download the data, structure and organize it so anyone can easily consume it and plug into their own system. 
+	Reply
+
+
+https://stackoverflow.com/questions/12097848/finding-all-domains-of-a-country
+
+	-> http://urlsearch.commoncrawl.org/
+	-> http://index.commoncrawl.org/
+	-> INSTRUCTIONS: https://groups.google.com/forum/#!msg/common-crawl/3QmQjFA_3y4/vTbhGqIBBQAJ
+
+
+Go to: http://index.commoncrawl.org/
+Grab the newest gzipped archive file.
+Then open it and find the cluster.idx file listed in it.
+Copy its relative URL, prefix with https://commoncrawl.s3.amazonaws.com/
+
+THEN:
+
+wharariki:[101]/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT>wget https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-26/indexes/cluster.idx
+	--2019-07-29 17:40:45--  https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-26/indexes/cluster.idx
+	Resolving commoncrawl.s3.amazonaws.com (commoncrawl.s3.amazonaws.com)... 52.216.8.171
+	Connecting to commoncrawl.s3.amazonaws.com (commoncrawl.s3.amazonaws.com)|52.216.8.171|:443... connected.
+	HTTP request sent, awaiting response... 200 OK
+	Length: 125059234 (119M) [binary/octet-stream]
+	Saving to: âcluster.idxâ
+
+	cluster.idx                       100%[============================================================>] 119.27M  8.51MB/s    in 15s     
+
+	2019-07-29 17:41:01 (7.83 MB/s) - âcluster.idxâ saved [125059234/125059234]
+
+wharariki:[102]/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT>grep '^nz,' cluster.idx | cut -f2 | uniq
+cdx-00237.gz
+cdx-00238.gz
+
+Prefix "https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-26/indexes/" to the listed gz files and wget them:
+
+	https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-26/indexes/cdx-00237.gz
+	https://commoncrawl.s3.amazonaws.com/cc-index/collections/CC-MAIN-2019-26/indexes/cc-index/collections/CC-MAIN-2019-26/indexes/cdx-00238.gz
+
+
+
+Unzip those, and we have all URLs with TLD .nz:
+	wharariki:[131]/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT>gunzip cdx-00237.gz 
+	wharariki:[132]/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT>gunzip cdx-00238.gz 
+
+
+The first of these files includes Norwegian TLDs (start with "no,") and the second gz file includes TLDs that start with "org,".
+So extract just those that start with "^nz," [https://www.unix.com/shell-programming-and-scripting/176608-how-copy-lines-starts-either-3-4-into-new-file.html].
+
+	wharariki:[107]/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT>egrep "^nz," cdx-00238 > nz-only-TLDs-from-237-238.txt
+	wharariki:[108]/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT>egrep "^nz," cdx-00238 >> nz-only-TLDs-from-237-238.txt
+
+
+Checking the abacusinstitute.ac.nz is also in the current June 2019 list:
+	egrep "ac,abacusinstitute" nz-only-TLDs-from-237-238.txt
+
+
+OTHER:
+https://www.tutorialspoint.com/hadoop/hadoop_mapreduce
+http://stormcrawler.net/
+http://storm.apache.org/getting-help.html
+
+
+https://dzone.com/articles/need-billions-of-web-pages-dont-bother-crawling
+Basically, each release is split into 100 segments. Each segment has three types of files: WARC, WAT, and WET. As explained on the Get Started page:
+
+    WARC files store the raw crawl data.
+    WAT files store computed metadata for the data stored in the WARC.
+    WET files store extracted plaintext from the data stored in the WARC.
+
+Note that WAT and WET are in the WARC format too! In fact, the WARC format is nothing more than an envelope with metadata and content. In the case of the WARC files, that content is the HTTP requests and responses, whereas, for the WET files, it is simply the plain text extracted from the WARCs. The WAT files contain a JSON representation of metadata extracted from the WARCs, e.g. title, links etc.
+
+
+Resources
+
+The Get Started page on the CommonCrawl website contains useful pointers to libraries and code in various programming languages to process the datasets. There is also a list of tutorials and presentations.
+
+It is also worth noting that CommonCrawl provides an index per release, allowing you to search for URLs (including wildcards) and retrieve the segment and offset therein where the content of the URL is stored, e.g.:
+
+	{ "urlkey": "org,apache)/", "timestamp": "20170220105827", "status": "200", "url": "http://apache.org/", "filename": "crawl-data/CC-MAIN-2017-09/segments/1487501170521.30/warc/CC-MAIN-20170219104610-00206-ip-10-171-10-108.ec2.internal.warc.gz", "length": "13315", "mime": "text/html", "offset": "14131184", "digest": "KJREISJSKKGH6UX5FXGW46KROTC6MBEM" }
+
+
+This is useful but only if you are interested in a limited number of URLs which you know in advance. In many cases, what you know in advance is what you want to extract, not where it will be extracted from. For situations such as these, you will need distributed batch-processing using MapReduce in Apache Hadoop or Apache Spark.
+
+
+https://www.forbes.com/sites/kalevleetaru/2017/09/28/common-crawl-and-unlocking-web-archives-for-research/#7067d4313b83
+One large web archive has bucked this trend and stood alone among its peers: Common Crawl. Similar to other large web archiving initiatives like the Internet Archive, Common Crawl conducts regular web wide crawls of the open web and preserves all of the content it downloads in the standard WARC file format. Unlike many other archives, it focuses primarily on preserving HTML web pages and does not archive images, videos, JavaScript files, CSS stylesheets, etc. Its goal is not to preserve the exact look and feel of a website on a given snapshot in time, but rather to collect a vast cross section of HTML web pages from across the web in a single place to enable large-scale data mining at web scale.
+...
+The project excludes sites which have robots.txt exclusion policies, following the historical policy of many other web archives, though it is worth noting that the Internet Archive earlier this year began slowly phasing out its reliance on such files due to their detrimental effect on preservation completeness. Common Crawl also allows sites to request removal from their index. Other than these cases, Common Crawl attempts to crawl as much of the remaining web as possible, aiming for a representative sample of the open web.
+...
+Ms. Crouse [Director of Common Crawl] noted the risk adverse nature of the web archiving community as a whole (historically many adhered and still adhere to a strict âopt inâ policy requiring prior approval before crawling a site) and the unwillingness of many archives to modernize their thinking on copyright and to engage more closely with the legal community in ways that could help them expand fair use horizons. In particular, she noted âsince we [in the US] are beholden to the Copyright Act, while living in a digital age, many well-intentioned organizations devoted to web science, archiving, and information provision may benefit from a stronger understanding of how copyright is interpreted in present day, and its hard boundariesâ and that âmany talented legal advisers and groups are interested in the precedent-setting nature of this topic; some are willing to work Pro Bono.â
+...
+Returning to the difference between Common Crawlâs datasets and traditional preservation-focused web archiving, Ms. Crouse emphasized that they capture only HTML pages and exclude multimedia content like images, video and other dynamic content.
+
+She noted that a key aspect of their approach to fair use is that web pages are intended for consumption by human beings one at a time using a web browser, while Common Crawl concatenates billions of pages together in the specialized WARC file format designed for machine data mining. Specifically, âCommon Crawl does not offer separate/individual web pages for easy consumption. The three data formats that are provided include text, metadata, and raw data, and the data is concatenatedâ and âthe format of the output is not a downloaded web page. The output is in WARC file format which contains the components of a page that are beneficial to machine-level analysis and make for space- efficient archiving (essentially: header, text, and some metadata).â
+
+As Ms. Crouse put it, âthis is big data intended for machine learning/readability. Further, our intention for its use is for public benefit i.e. to encourage research and innovation, not direct consumption.â She noted that âfrom the laypersonâs perspective, it is not at all trivial at present to extract a specific websiteâs content (that is, text) from a Common Crawl dataset. This task generally requires one to know how to install and run a Hadoop cluster, among other things. This is not structured data. Further it is likely that not all pages of that website will be included (depending on the parameters for depth set for the specific crawl).â This means that âthe bulk of [Common Crawlâs] users are from the noncommercial, educational, and research sectors. At a higher level, itâs important to note that we provide a broad and representative sample of the web, in the form of web crawl data, each month. No one really knows how big the web is, and at present, we limit our monthly data publication to approximately 3 billion pages.â
+
+
+Common Crawl believes it addresses this through the fact that its archive represents only a sample of each website crawled, rather than striving for 100% coverage. Specifically, Ms. Crouse noted that âat present, [crawls are] in monthly increments that are discontinuous month-to-month. We do only what is reasonable, necessary, and economical to achieve a representative sample. For instance, we limit the number of pages crawled from any given domain so, for large content owners, it is highly probable that their content, if included in a certain monthâs crawl data, is not wholly represented and thus not ideal for mining for comprehensive results âŠ if the content owner is not a large site, or in a niche market, their URL is less likely to be included in the seeds in the frontier, and, since we limit depth (# of links followed) for the sake of both economy and broader representative web coverage, 'niche' content may not even appear in a given monthâs dataset.â
+
+To put it another way, Common Crawlâs mission is to create a ârepresentative sampleâ of the web at large by crawling a sampling of pages and limiting the number of pages from each site they capture. Thus, their capture of any given site will represent a discontinuous sampling of pages that can change from month to month. A researcher wishing to analyze a single web site in its entirety would therefore not be able to turn to Common Crawl and would instead have to conduct their own crawl of the site or turn to a commercial aggregator that partners with the content holder to license the complete contents of the site.
+
+In Common Crawlâs view this is a critical distinction that sets it apart from both traditional web archiving and the commercial content aggregators that generate data mining revenue for content owners. By focusing on creating a ârepresentative sampleâ of the web at large, rather than attempting to capture a single site in its entirety (and in fact ensuring that it does not include more than a certain number of pages per site), the crawl self-limits itself to being applicable only to macro-level research examining web scale questions. Such âweb scaleâ questions cannot be answered through any existing open dataset and by incorporating specific design features Common Crawl ensures that more traditional research questions, like data mining the entirety of a single site, which might be viewed as redistribution of that site or competing with its ownerâs ability to license its content for data mining, is simply not possible.
+------------
Index: /gs3-extensions/maori-lang-detection/MoreReading/Heritrix-and-WCT.txt
===================================================================
--- /gs3-extensions/maori-lang-detection/MoreReading/Heritrix-and-WCT.txt	(revision 33376)
+++ /gs3-extensions/maori-lang-detection/MoreReading/Heritrix-and-WCT.txt	(revision 33376)
@@ -0,0 +1,369 @@
+
+
+RUN:
+export HERITRIX_HOME=/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT
+$HERITRIX_HOME/bin/heritrix -a admin:admin
+Visit: https://localhost:8443/engine
+
+MORE READING:
+http://open-s.com/en/content/heritrix-configuration
+http://open-s.com/en/content/wget
+	https://superuser.com/questions/655346/wget-execute-script-after-download
+https://www.gnu.org/software/wget/manual/wget.html
+	â--execute command
+
+    	Execute command as if it were a part of .wgetrc (see Startup File). A command thus invoked will be executed after the commands in .wgetrc, thus taking precedence over them.
+	If you need to specify more than one wgetrc command, use multiple instances of â-eâ.
+
+"crawler TRAPS": https://www.contentkingapp.com/academy/crawler-traps/
+	https://www.billhartzer.com/internet-marketing/crawl-thousands-urls/
+
+
+
+-----------------------------------------
+ Scope DecideReject Configuration rules
+-----------------------------------------
+Issues in H3.docx
+https://sbforge.org/download/attachments/.../Issues%20in%20H3.docx?version=1...
+
+By using maxTransHops and maxSpeculativeHops we thought that we could manage how long our discovery path 'X' should be, but we see different results and ...
+DOCX: https://sbforge.org/download/attachments/21856421/Issues%20in%20H3.docx?version=1&modificationDate=1465912557659&api=v2&usg=AOvVaw03EWO6Xy0XiMRITvFwR2v0
+[ https://webcache.googleusercontent.com/search?q=cache:bEM1AdjQR2cJ:https://sbforge.org/download/attachments/21856421/Issues%2520in%2520H3.docx%3Fversion%3D1%26modificationDate%3D1465912557659%26api%3Dv2+&cd=10&hl=en&ct=clnk&gl=nz&client=ubuntu ]
+
+
+
+    Speculative Hops
+
+        No really effect using maxTransHops or maxSpeculativeHops
+
+    By using maxTransHops and maxSpeculativeHops we thought that we could manage how long our discovery path âXâ should be, but we see different results and we still harvest several âXXâ or more.
+    To accomplish this we use HopsPathMatchesDecideRule but we havenât found a specific path that we certainly can say we want excluded from our harvest. We tried to make regex with R, E and X. Does anyone have experience with this?
+	
+
+	
+    Path seeds
+
+        Harvesting specific URI paths that doesnât end with slashes
+
+    When adding path seeds to only harvest from a specific place on a domain we sometimes have problems with redirections.
+    We always end path seeds with a slash, but sometimes we are redirected e.g. HTTP 301 http://www.bt.dk/plus/ â http://www.bt.dk/plus.
+    If we have following as a seed http://www.bt.dk/plus we will harvest the whole site because H3 harvest from the last slash in the seed?
+    Any experience with path seeds?
+
+
+--------------------------------------------------------------------------------
+
+TODO:
+
+https://www.stat.auckland.ac.nz/~paul/Reports/maori/maori.html
+ The macron is the only accent required for written MÄori and the accent can only be applied to vowels, so the full set of accented characters are:
+lower case a, with macron 	Ä 		upper case A, with macron 	Ä
+lower case e, with macron 	Ä 		upper case E, with macron 	Ä
+lower case i, with macron 	Ä« 		upper case I, with macron 	Äª
+lower case o, with macron 	Å 		upper case O, with macron 	Å
+lower case u, with macron 	Å« 		upper case U, with macron 	Åª 
+
+http://emacs.1067599.n8.nabble.com/Entering-vowels-with-macrons-td72136.html
+	Ctrl+\
+	type rfc1345 (and enter)
+	type &a- to get a-macron
+	Then Ctrl+\ to toggle back to default input
+	(Can thereafter toggle with Ctrl+\ to get back to rfc1345 input method)
+
+
+
+https://sachachua.com/blog/2011/04/writing-macrons-linux-latin-pronunciation/
+To add macrons: Ctrl-\ "latin-alt-postfix". But doesn't have all macronised vowels as used in te reo.
+Then Ctrl-\ to get default input editor back.
+https://www.gnu.org/software/emacs/manual/html_node/emacs/Select-Input-Method.html
+
+
+DOWNLOAD HERITRIX:
+BINARY: http://builds.archive.org/maven2/org/archive/heritrix/heritrix/3.4.0-SNAPSHOT/
+CODE, STATIC: https://github.com/internetarchive/heritrix3
+https://github.com/internetarchive/heritrix3/wiki/How%20To%20Crawl
+
+WEB CURATOR TOOL:
+http://dia-nz.github.io/webcurator/
+https://webcuratortool.readthedocs.io/en/latest/guides/quick-start-guide.html
+https://webcuratortool.readthedocs.io/en/latest/guides/overview-history.html
+
+
+Crawl as much of the nz domains as we can
+and run the language detection for Maori on the pages that come through
+and only save those pages and configure (somehow) that it knows that it doesn't need to redownload pages already *inspected* for language (not just that it detects it doesn't need to redownload the *stored* pages, since we only store mri language pages and not all pages inspected)
+
+Then break up the pages by sentences using our SentenceDetector model.
+
+
+SURT urls:
+http://crawler.archive.org/apidocs/org/archive/util/SURT.html
+
+WARC
+https://en.wikipedia.org/wiki/Web_ARChive
+
+Q:
+"The harvested material is captured in ARC/WARC format which has strong storage and archiving characteristics." at https://webcuratortool.readthedocs.io/en/latest/guides/user-manual.html
+
+https://blogs.loc.gov/thesignal/2013/11/anatomy-of-a-web-archive/
+
+ Nicholas Taylor
+November 13, 2013 at 2:49 pm
+
+Hi Ross, thanks for the comment. The tools for personal archiving of web pages and websites to WARC format are getting better, with the capture side further along than the replay side. Archive Ready (http://archiveready.com/) and WARCreate (http://warcreate.com/) can both be used to create a WARC containing all of the objects that make up an individual web page. GNU Wget 1.14+ (http://www.archiveteam.org/index.php?title=Wget_with_WARC_output) and WAIL (http://matkelly.com/wail/) can both be used to capture entire websites to WARC. WAIL also bundles a standalone Wayback Machine that runs locally, which is the easiest way I know of for users to view the content theyâve collected in WARC format.
+
+https://webcuratortool.readthedocs.io/en/latest/guides/user-manual.html
+Doesn't mention https
+
+"How targets work
+
+Targets consist of several important elements, including a name and description for internal use; a set of Seed URLs, ******a web harvester profile that controls the behaviour of the web crawler during the harvest******, one or more schedules that specify when the Target will be harvested, and (optionally) a set of descriptive metadata for the Target."
+
+Harvestor Configuration section:
+"The remaining tabs Pre-fetchers, Fetchers, Extractors, Writers, and Post-Processors are a series of processors that a URI passes through when it is crawled."
+
+
+http://crawler.archive.org/articles/user_manual/config.html
+Look for post-process*. Found under:
+6.1.3. Processing Chains
+
+6.1.2. Frontier
+
+The Frontier is a pluggable module that maintains the internal state of the crawl. What URIs have been discovered, crawled etc. As such its selection greatly effects, for instance, the order in which discovered URIs are crawled.
+
+There is only one Frontier per crawl job.
+
+Multiple Frontiers are provided with Heritrix, each of a particular character.
+6.1.2.1. BdbFrontier
+
+The default Frontier in Heritrix as of 1.4.0 and later is the BdbFrontier(Previously, the default was the Section 6.1.2.2, âHostQueuesFrontierâ). The BdbFrontier visits URIs and sites discovered in a generally breadth-first manner, it offers configuration options controlling how it throttles its activity against particular hosts, and whether it has a bias towards finishing hosts in progress ('site-first' crawling) or cycling among all hosts with pending URIs.
+
+Discovered URIs are only crawled once, except that robots.txt and DNS information can be configured so that it is refreshed at specified intervals for each host.
+
+The main difference between the BdbFrontier and its precursor, Section 6.1.2.2, âHostQueuesFrontierâ, is that BdbFrontier uses BerkeleyDB Java Edition to shift more running Frontier state to disk.
+
+6.1.2.2. HostQueuesFrontier
+
+The forerunner of the Section 6.1.2.1, âBdbFrontierâ. Now deprecated mostly because its custom disk-based data structures could not move as much Frontier state out of main memory as the BerkeleyDB Java Edition approach. Has same general characteristics as the Section 6.1.2.1, âBdbFrontierâ.
+
+
+https://webcuratortool.readthedocs.io/en/latest/guides/quick-start-guide.html
+You can use OpenWayback to view harvests from within WCT, see the wiki on the WCT Github page: https://github.com/DIA-NZ/webcurator/wiki/Wayback-Integration
+
+https://webarchive.jira.com/wiki/spaces/Heritrix/overview
+https://github.com/internetarchive/heritrix3/wiki/Heritrix3
+
+	"Unlike with previous releases, the web control interface is only made available via secure-socket HTTPS, and corresponding to this change the default port has changed to 8443. Additionally, unless you supply a compatible keystore via the new optional '-s' command-line switch, an 'ad-hoc' keystore with a new locally-generated SSL-capable certificate will be created (and then reused on future launches).
+
+	To then contact the web interface from a browser running on the same machine, visit the URL:
+
+	https://localhost:8443/
+	"
+
+	LOCALHOST WITH HTTPS is possible??? vs https://letsencrypt.org/docs/certificates-for-localhost/
+
+	"For local development
+
+	If youâre developing a web app, itâs useful to run a local web server like Apache or Nginx, and access it via http://localhost:8000/ in your web browser. However, web browsers behave in subtly different ways on HTTP vs HTTPS pages. The main difference: On an HTTPS page, any requests to load JavaScript from an HTTP URL will be blocked. So if youâre developing locally using HTTP, you might add a script tag that works fine on your development machine, but breaks when you deploy to your HTTPS production site. To catch this kind of problem, itâs useful to set up HTTPS on your local web server. However, you donât want to see certificate warnings all the time. How do you get the green lock locally?
+
+	The best option: Generate your own certificate, either self-signed or signed by a local root, and trust it in your operating systemâs trust store. Then use that certificate in your local web server. See below for details."
+
+	[Googled: https certificate localhost
+		https://www.freecodecamp.org/news/how-to-get-https-working-on-your-local-development-environment-in-5-minutes-7af615770eec/
+	]
+
+
+https://github.com/internetarchive/heritrix3/wiki/Heritrix%20Output#HeritrixOutput-WARCfiles
+* source-report.txt
+
+This report contains a line item for each host, which includes the seed from which the host was reached.
+Note
+
+    The sourceTagSeeds property of the TextSeedModule bean must be set to true for this report to be generated. 
+
+* WARC files
+
+Assuming you are using the WARC writer that comes with Heritrix, a number of WARC files will be generated containing crawled content.
+
+You can specify the storage location of WARC files by setting the directory value of the WARCWriterProcessor bean.
+
+
+https://github.com/internetarchive/heritrix3/wiki/Archiving%20Rich-Media%20Content
+Large File Sizes
+
+Rich-media content, such as Flash and video, is usually much larger than standard text/html pages.  Crawling such content requires large investments in storage and bandwidth.  To mitigate these issues, deduplication is recommended for rich-media crawls.  Deduplication detects previously collected content that is redundant and skips the download of such content.  Pointers to the duplicate content allow it to appear in subsequent crawls.  For details see Configuring Heritrix for Deduplication.
+
+
+Excessive Memory and CPU Usage
+
+Downloading rich-media content can often cause excessive load to be placed on the crawling computers memory and CPU.  For example, extracting links from Flash and other rich-media resources requires extensive data parsing, which is CPU intensive.   Atypical input patterns can also cause excessive CPU usage when regular expressions used by Heritrix are run.  It is therefore recommended that rich-media crawls be allocated more memory and CPU than "normal" crawls.  The memory allocated to Heritrix is set from the command line.  The following example shows the command line option to allocate 1 GB of memory to Heritrix, which should be sufficient for most rich-media crawls.
+
+export JAVA_OPTS=-Xmx1024M
+
+Multi-core processors are also recommended for rich-media crawls.
+
+
+Streaming media
+and
+Social Networking Sites
+Many social networking sites make use of rich-media to enhance their user-experience.  For specific guidelines on archiving social media sites see Archiving Social Networking Sites with Archive-It .  These instructions apply to the Archive-It application, which is built on top of Heritrix.
+
+
+Q: https://github.com/internetarchive/heritrix3/wiki/Avoiding%20Too%20Much%20Dynamic%20Content
+"To allow both foo.org and www.foo.org to be captured, you could add two seeds: http://www.foo.org/ and http://foo.org/. To allow every subdomain of foo.org to be crawled, you can add the seed http://foo.org.  Note the absence of a trailing slash."
+(Does the latter encompass both of the former?)
+
+
+Delete the TranclusionDecideRule, since this rule has the potential to lead Heritrix onto another host.  For example, if a URI returns a 301 response code (move permanently) or 302 (found) response code as well as a URI that contains a different host name than the seeds, Heritrix would accept this URI using the TransclusionDecideRule.  Removing this rule will keep Heritrix from straying off of our www.foo.org host.
+...
+Alternately, you can add the MatchesFilePatternDecideRule.  Set usePresetPattern to CUSTOM and set the regexp to something like: .foo\.org(?!/calendar).|.*foo\.org/calendar?year=200[56].*
+
+
+https://github.com/internetarchive/heritrix3/wiki/Mirroring%20HTML%20Files%20Only
+Mirroring HTML Files Only
+Alex Osborne edited this page on Jul 4, 2018 Â· 2 revisions
+
+Suppose you only want to crawl URIs that match http://foo.org/bar/\*.html.  Also, you would like to save the crawled files in a file/directory format instead of saving them in WARC files.  Also, assume the web server is case-sensitive.  For example, http://foo.org/bar/abc.html and http://foo.org/bar/ABC.HTML are pointing to two different resources.
+
+
+!! [If Heritrix needs to] be configured to differentiate between abc.html and ABC.HTML.  Do this by removing the LowercaseRule from the canonicalizationPolicy bean.
+
+https://github.com/internetarchive/heritrix3/wiki/Only%20Store%20Successful%20HTML%20Pages
+
+
+https://github.com/internetarchive/heritrix3/wiki/Jobs
+[multiple URLs to crawl can be specified. But what is the separator]
+
+Look up: Spring framework, Spring beans
+
+
+https://localhost:8443/engine/job/pinky/jobdir/crawler-beans.cxml?format=textedit
+ <!-- 
+   PROCESSING CHAINS
+    Much of the crawler's work is specified by the sequential 
+    application of swappable Processor modules. These Processors
+    are collected into three 'chains'. The CandidateChain is applied 
+    to URIs being considered for inclusion, before a URI is enqueued
+    for collection. The FetchChain is applied to URIs when their 
+    turn for collection comes up. The DispositionChain is applied 
+    after a URI is fetched and analyzed/link-extracted.
+  -->
+
+
+https://github.com/internetarchive/heritrix3/wiki/Fetch%20Chain%20Processors
+fetchHttp
+	
+
+This processor fetches HTTP URIs.  As of Heritrix 3.1, the crawler will now properly decode 'chunked' Transfer-Encoding -- even if encountered when it should not be used, as in a response to an HTTP/1.0 request. Additionally, the fetchHttp processor now includes the parameter 'useHTTP11', which if true, will cause Heritrix to report its requests as 'HTTP/1.1'.  This allows sites to use the 'chunked' Transfer-Encoding. (The default for this parameter is false for now, and Heritrix still does not reuse a persistent connection for more than one request to a site.)
+fetchHttp also includes the parameter 'acceptCompression', which if true, will cause Heritrix requests to include an "Accept-Encoding: gzip,deflate" header, which offers to receive compressed responses. (The default for this parameter is false for now.)
+
+extractorHttp
+	
+
+This processor extracts outlinks from HTTP headers.  As of Heritrix 3.1, the extractorHttp processor now considers any URI on a hostname to imply that the '/favicon.ico' from the same host should be fetched.  Also, as of Heritrix 3.1, the "inferRootPage" property has been added to the extractorHttp bean.  If this property is "true", Heritrix infers the '/' root page from any other URI on the same hostname.  The default for this setting is "false", which means the pre-3.1 behavior of only fetching the root page if it is a seed or otherwise discovered and in-scope remains in effect. Discovery via these new heuristics is considered to be a new 'I' (inferred) hop-type, and is treated the same in scoping/transclusion decisions as an 'E' (embed).
+
+
+https://github.com/internetarchive/heritrix3/wiki/Processor%20Settings
+
+fetchHttp:
+
+timeoutSeconds - This setting determines how long an HTTP request will wait for a resource to respond.  This setting should be set to a high value.
+ defaultEncoding - The character encoding to use for files that do not have one specified in the HTTP response headers. The default is ISO-8859â-1.
+
+soTimeoutMs - If the socket is unresponsive for this number of milliseconds, the request is cancelled.  Setting the value to zero (no timeout) is not recommended as it could hang a thread on an unresponsive server. This timeout is used to time out socket opens and socket reads. Make sure this value is less than timeoutSeconds for optimal configuration.  This ensures at least one retry read.
+
+sendIfModifiedSince - Send If-Modified-Since header, if previous Last-Modified fetch history information is available in URI history.
+
+sendIfNoneMatch - Send If-None-Match header, if previous Etag fetch history information is available in URI history.
+
+sendConnectionClose - Send Connection: close header with every request.
+    w3.org connection header documentation
+
+sendRange- Send the Range header when there is a limit on the retrieved document size.  This is for politeness purposes.  The Range header states
+that only the first n bytes are of interest.  It is only pertinent if maxLengthBytes is greater than zero.  Sending the Range header results in a
+206 Partial Content status response, which is better than cutting the response mid-download. On rare occasion, sending the Range header will
+generate 416 Request Range Not Satisfiable response.
+
+
+acceptHeaders - Accept Headers to include in each request. Each must be the complete header, e.g., Accept-Language: en.
+(mi is the 2 letter code for Maori, see https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)
+	https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept
+
+
+
+ExtractorHtml:
+
+    extractJavascript - If true, in-page Javascript is scanned for strings that appear to be URIs. This typically finds both valid and invalid URIs.  Attempts to fetch the invalid URIs can generate webmaster concern over odd crawler behavior. Default is true.
+
+extractValueAttributes- If true, strings that look like URIs found in unusual places (such as form VALUE attributes) will be extracted. This typically finds both valid and invalid URIs.  Attempts to fetch the invalid URIs may generate webmaster concerns over odd crawler behavior. Default is true.
+
+
+ ignoreFormActionUrls - If true, URIs appearing as the ACTION attribute in HTML FORMs are ignored. Default is false.
+extractOnlyFormGets - If true, only ACTION URIs with a METHOD of GET (explicit or implied) are extracted. Default is true. 
+
+
+     
+
+candidates
+
+    seedsRedirectNewSeeds - If enabled, any URI found because a seed redirected to it (original seed returned 301 or 302), will also be treated as a seed.
+
+
+https://github.com/internetarchive/heritrix3/wiki/Statistics%20Tracking
+Statistics Tracking
+Alex Osborne edited this page on Jul 4, 2018 Â· 2 revisions
+
+Any number of statistics tracking modules can be attached to a crawl.  Currently only one is provided with Heritrix.  The statisticsTracker Spring bean that comes with Heritrix creates the progress-statistics.log file and provides the WUI with data to display progress information about the crawl.  It is strongly recommended that any crawl run through the WUI use this bean.
+
+
+
+https://github.com/internetarchive/heritrix3/wiki/Configuring-Crawl-Scope-Using-DecideRules
+
+------------
+REST API: https://heritrix.readthedocs.io/en/latest/api.html
+	Execute Script in Job
+
+	POST https://(heritrixhost):8443/engine/job/(jobname)/script
+
+	    Executes a script. The script can be written as Beanshell, ECMAScript, Groovy, or AppleScript.
+
+
+https://github.com/beanshell/beanshell
+https://github.com/internetarchive/heritrix3/wiki/BeanShell%20Script%20For%20Downloading%20Video
+https://github.com/internetarchive/heritrix3/wiki/Heritrix3-Useful-Scripts
+
+-----------
+
+LOGGING
+
+https://github.com/internetarchive/heritrix3/wiki/Configuring%20Crawl%20Scope%20Using%20DecideRules
+
+"DecideRuleSequence Logging
+
+Enable FINEST logging on the class org.archive.crawler.deciderules.DecideRuleSequence to watch each DecideRule's evaluation of the processed URI. This can be done in the logging.properties file
+
+logging.properties
+
+	org.archive.modules.deciderules.DecideRuleSequence.level = FINEST
+
+in conjunction with the -Dsysprop VM argument
+	-Djava.util.logging.config.file=/path/to/heritrix3/dist/src/main/conf/logging.properties
+
+"
+
+I couldn't get the above logging instructions to work, but here's what I did. 
+
+a. I modified conf/logging.properties by adding:
+
+# PINKY
+# DecideRuleSequence Logging
+# https://github.com/internetarchive/heritrix3/wiki/Configuring-Crawl-Scope-Using-DecideRules
+org.archive.modules.deciderules.DecideRuleSequence.level = FINEST
+
+b. I opened the bin/heritrix script and edited in the following into the 2 locations using $JAVACMD:
+
+	CLASSPATH=${CP} nohup $JAVACMD -Djava.util.logging.config.file=/Scratch/ak19/heritrix/heritrix-3.4.0-SNAPSHOT/conf/logging.properties -Dheritrix.home=${HERITRIX_HOME}
+
+
+
+
Index: /gs3-extensions/maori-lang-detection/MoreReading/other.txt
===================================================================
--- /gs3-extensions/maori-lang-detection/MoreReading/other.txt	(revision 33376)
+++ /gs3-extensions/maori-lang-detection/MoreReading/other.txt	(revision 33376)
@@ -0,0 +1,4 @@
+https://codereview.stackexchange.com/questions/198343/crawl-and-gather-all-the-urls-recursively-in-a-domain
+http://lucene.472066.n3.nabble.com/Using-nutch-just-for-the-crawler-fetcher-td611918.html
+
+https://www.quora.com/What-are-some-Web-crawler-tips-to-avoid-crawler-traps
