Index: /other-projects/maori-lang-detection/journal-paper/MRI_slideNotes.txt
===================================================================
--- /other-projects/maori-lang-detection/journal-paper/MRI_slideNotes.txt	(revision 33903)
+++ /other-projects/maori-lang-detection/journal-paper/MRI_slideNotes.txt	(revision 33903)
@@ -0,0 +1,398 @@
+1. Where on the web can Maori text be found?
+2 letter-langcode: MI
+3 letter-langcode: MRI
+
+
+2. General limitations:
+- only TEXT in Maori, not audio, video, etc
+- can't get at the deep web
+e.g. sites not linked up with rest of web,
+large digital repositories where there's no
+direct links to individual pages
+but which are found only by searching
+
+
+3. Initial consideration: 
+Do the exploratory Crawl ourselves.
+
+* unimpeded internet-wide crawl
+* crawl just NZ (AU, UK) sites: limit TLD
+
+In both cases, start off with known NZ sites
+acting as seed URLs for exploratory search
+via all linked sites.
+Seed URls could include NZ govt,
+language resource sites, digital library sites,
+Maori language blogs, community resource sites
+
+
+4. Things to think about:
+* web traps:
+stuck crawling one or more pages forever.
+
+Some crawling software deal with this
+better than others, but problems remain
+
+* disk space
+In the early 2000s, Internet Archive's
+regular web wide crawl was already in the petabytes.
+
+To save space, we could analyse each site
+once crawled and throw away unpromising ones
+before crawling further
+
+* when would we know we have enough data
+to finally start analysing?
+
+
+5. Alternative approaches to doing the
+web-wide crawl ourselves:
+
+Discovery of Ready-Made Crawl Data:
+- payware site that offers access to
+(query) its web-wide crawl data for money
+- free web crawl data offered by Common Crawl,
+which encourages individuals, businesses,
+institutions to use its crawl data,
+so researchers won't burden the internet
+with countless crawls for individual ends.
+
+6. Common Crawl (CC) - limitations
+- not exhaustive
+	* crawls focus on breadth (representing
+	a wide cross-section of web), not full-depth
+	crawl of sites for copyright reasons a.o.
+	So need to recrawl sites of interest
+	at greater depth.
+	* crawls done monthly, trying to minimise
+	overlaps. So a month's crawl is not of
+	the entire known web.
+- needed Amazon s3 (paid account).	
+- distributed CC data needs distributed
+system to access/query.
+- Big data: still takes some time chugging away.
+
+
+7. Advantages of using CC:
+* Ready-made crawl data enriched with 
+metadata fields stored in distributed DB.
+that you can run (distributed) queries against.
+e.g. get all .nz TLD sites of a CC crawl.
+* BETTER: Aug 2018 introduction of "content-language"
+metadata field, stores top few detected languages of
+each web page in descending order.
+Since Sep 2018, this field can be queried too!
+
+
+8. Plan:
+1. Query for MRI (Maori) as content-language 
+2. Pool results of multiple contiguous months worth of
+crawl data, to construct completer cross-section of web
+3. re-crawl each *site* (domain) found at greater depth
+to hopefully crawl more sites fully than CC did. 
+(At least still not an exploratory search
+of the entire internet.)
+4. Run Apache Open NLP language detection over
+both downloaded web pages
+AND individual sentences (ideally paragraphs...)
+5. CC's language detector software wasn't Apache
+OpenNLP, so still worth re-running over recrawls.
+
+
+9. * Initial testing effectively queried each CC crawl:
+	get all webpages where
+	content-language 'contains' MRI
+But low-quality results!
+e.g. Single-word pages that weren't actually Maori.
+* Ended up querying:
+	content-language = MRI
+	(not just primary language detected, but the
+	sole language detected)
+Still some disappointing results, but far less common.
+
+
+10. We were in July/Aug of 2018 when we began.
+Queried Sep 2018 - Aug 2019 (12 months) CC data.
+
+Next, need to prepare data for crawling locally:
+- ensure unique domains across CC crawl results,
+- remove low-quality sites and process special sites
+- create seed URLs, regex filters for each site
+to recrawl at depth 10 with Apache Nutch
+
+
+11. Low quality data
+Countless auto-translated sites like adult
+and product sites:
+- Blacklisted adult sites
+- Greylisted obvious product sites providing (auto)
+translations in countless languages of the globe.
+But too many to go through.
+Left this issue for "later" in the process pipeline.
+
+
+12. Special handling regex list for certain sites
+e.g. large sites.
+Don't want to crawl all of blogspot or docs.google
+or wikipedia, etc.
+Instead crawl mi.wikipedia; <blogname>.blogspot;
+docs.google/<individual-seed-page-id>
+
+
+13. <PROCESS FLOW CHART>
+
+
+14. Stripping html stripped paragraph info,
+so had to deal with sentences as units.
+But Apache OpenNLP language detection
+prefers to work on >= 2 sentences at a time.
+
+Still, in testing this, OpenNLP returned MRI as
+primary language for single sentences
+as often as it did for 2 contiguous sentences.
+But lower confidence level.
+
+
+15. MongoDB Webpage level meta:
+* URL,
+* full page text of downloaded webpage,
+* "sentences" array (trained basic Apache
+Open NLP sentence model for MRI)
+* isMRI? - whether openNLP detected MRI to be
+the primary language of overall page content
+* containsMRI? - whether openNLP detected MRI as
+primary language of any sentence on the page
+
+
+16. MongoDB Website level meta: 
+* domain,
+* geo-location of site's server,
+* numPagesInMRI,
+* numPagesContainingMRI,
+* did_nutch_finish_crawling_site_fully?
+
+
+17. Querying MongoDB:
+Simple queries: 
+* How many webSITES crawled?
+CC said these sites had MRI page(s)
+* How many webPAGES crawled?
+* How many PAGES with isMRI = true (openNLP)
+* How many PAGES with containsMRI = true
+* How many SITES where numPagesInMRI > 0
+* How many SITES where numPagesContainingMRI > 0
+(= sites with at least 1 webpage with at least
+sentence that openNLP detected as MRI)
+
+
+	After blacklisting, 1462 sites to crawl with Nutch, but a few were obvious product sites, so removed before crawling
+   	or while crawling other sites.
+
+	After crawling, 
+	# Num websites in MongoDB
+	1445 
+
+	# Num webpages
+	117496
+
+	# The number of web SITES that contain 1 or more pages detected as being in Maori (num sites with a positive numPagesInMRI)
+	361
+
+	# Number of web SITES containing at least one page with at least one sentence for which OpenNLP detected the best language = MRI
+	# (Num sites with a positive numPagesContainingMRI)
+	868
+
+	# The number of web PAGES that are deemed to be overall in MRI (pages where isMRI=true)
+	7818
+
+	# Number of web PAGES that contain any number of MRI sentences
+	20371
+
+	# Number of web SITES with crawled web pages that have any URLs containing /mi(/) OR http(s)://mi.*
+	670
+
+	# Number of web SITES that are outside NZ that contain /mi(/) OR http(s)://mi.*
+	# in any of its crawled webpage urls
+	656
+
+	# 14 sites with page URLs containing /mi(/) OR http(s)://mi.* that are in NZ
+	14
+
+	# ATTEMPT TO FILTER OUT LIKELY AUTO-TRANSLATED SITES
+	# Get a count of all non-NZ (or .nz TLD) sites that don't have /mi(/) or http(s)://mi.*
+	# in the URL path of any crawled web pages of the site
+	220
+
+
+	# Count of websites that have at least 1 page containing at least one sentence detected as MRI
+	# AND which websites have mi in a webpage's URL path:
+	491
+
+
+	# The websites that have some MRI detected AND which are either in NZ or with NZ TLD
+	# or (so if they're from overseas) don't contain /mi or mi.* in a page's URL path:
+	396
+
+	# Include Australia, to get the valid "kiwiproperty.com" website included in the result list:
+	397
+
+
+	# counts of pages by country code excluding NZ related sites and AU sites 
+	# that are detected as containing at least one Maori sentence:
+
+	221 websites
+
+
+	# But to produce the tentative non-product sites, we also want the aggregate for all NZ sites (from NZ or with .nz tld):
+	176
+
+	(Total is 221+176 = 397, which adds up).
+
+
+	# Manually inspected shortlist of the 221 non-NZ websites to weed out those that aren't MRI (weeding out those misdetected as MRI, autotranslated or just contain placenames etc), and adding the 176 NZ on top:
+
+	MANUAL INSPECTION: TOTAL COUNT BY COUNTRY OF SITES WITH AT LEAST ONE PAGE CONTAINING ONE SENTENCE OF MRI CONTENT (numPagesContainingMRI > 0):
+	NZ: 126
+	US: 25+4
+	AU: 2
+	DE: 2
+	DK: 2
+	BG: 1
+	CZ: 1
+	ES: 1
+	FR: 1
+	IE: 1
+	TOTAL: 166
+
+18. More complex MongoDB queries:
+Count of SITES by site geolocation
+where
+- numPagesInMRI > 0
+- numPagesContainingMRI > 0
+(- AND miInURLPath for overseas sites = false)
+
+Also: do the counts grouping NZ origin sites
+with ".nz" TLD sites (regardless of server
+geo-origin) under NZ.
+
+
+19. Detected results can turn out low-quality:
+- misdetection, e.g. Tongan, Kiribati, etc
+(not in OpenNLP language model)
+or ENG sentences with MRI words
+detected as MRI sentences
+- Maori personal and place names in 
+references and gallery photo captions
+suffice to return sentences
+and single-sentence pages as MRI
+- autotranslated sites!!!!
+
+
+20. Auto-translated content = UNWANTED
+
+Don't want automatically translated sites
+when building a corpus of high quality Maori
+language text for researchers to work with.
+
+Also, it can be polluting:
+auto-translated content can't serve as 
+proper training data set to inform better
+automatic translation in future either.
+
+
+21. Heuristics for some detection
+of auto-translated sites
+
+Dr Dave Nichols suggested:
+Find non-NZ sites that have /mi or mi* in URL
+(2 letter code for Maori) and remove them
+as they're more likely to be product sites.
+
+In practice: Still had to wade through list of all
+overseas sites with page URLs containing "mi"
+for the occasional exception.
+And reverse: some NZ sites with "mi" in any
+web page's URL could be auto-translated product sites.
+
+
+22. Bigger problem:
+Even if overseas sites with mi in page URLs
+were filtered out, a large set of auto-translated
+sites never use mi in the URL path.
+
+PROBLEM: can't detect auto-translated sites
+automatically. Confirmed by Dr Stephen Joe,
+Mr Bill Rogers, Dr Bainbridge.
+Human, manual intervention needed
+to weed them out.
+
+
+23. So manually went through MongoDB result list of
+	all websites with numPagesContainingMRI > 0
+to shortlist just those websites which had any 
+webpage that truly contained at least one sentence
+in MRI.
+
+(Not even website[x].numPagesInMRI > 0)
+
+
+24. Results
+Results at website level (not webpage level).
+<TABLES AND GEO-JSON MAPS>
+
+
+25. Recommendation
+There's a case to be made for WWW standards
+to make it compulsory, including on legacy
+sites, to include some indicator on each
+webpage or even at paragraph level
+(HTML markup tag comparable to "lang"?)
+to denote whether the text content is formulated
+by a human or auto-translated. 
+
+Or a processing sequence,
+e.g. content-source="human, ocr, bot-translation"
+for an automatic translation of a digitised book
+by a human auteur.
+
+
+26. Working on the final stages
+- Code generates random sample of webpage URLs
+for sitelisting for which we can make 90% 
+confidence with 5% margin of error predictions.
+
+Then need to go over each sample webpage URL 
+produced from manually pruned webSITE listing,
+and manually verify whether in cases where a
+webPAGE isMRI=true, the page's
+genuinely largely in Maori or not.
+
+- Finish writing code to automatically run the
+mongodb queries I've manually run, to
+summarise the results for generating tables
+and geojson maps.
+
+
+27. Future work
+- Knowing the site-level results,
+can fully recrawl those promising sites
+that weren't fully crawled before
+- Maybe retrain OpenNLP language model
+for Maori using high quality web pages found?
+
+
+28. Wider Applicability
+Repeating the process for other languages
+not in wide use:
+
+- CC prefers not to be burdened by data
+requests for very common languages, but
+low-resource languages are fine
+- Check if Apache OpenNLP supports language
+else need to train and add model.
+- MongoDB queries need to be adjusted.
+At present specific to Maori, e.g. its unique
+geographic distribution: NZ + .nz TLD
+treated specially vs overseas.
+But for the French language, France, Canada,
+New Caledonia etc TLDs need to be considered.
