Changeset 33890

Timestamp:

2020-02-03T20:31:33+13:00 (4 years ago)

Author:

ak19

Message:

Finished going through NZ sites listing of numPagesContainingMRI > 0 and manually determining which of these sites really contained at least one webpage containing at least one sentence inMRI.

File:

• other-projects/maori-lang-detection/mongodb-data/6table_nonProductSites1_manualShortlist.json (modified) (6 diffs)

other-projects/maori-lang-detection/mongodb-data/6table_nonProductSites1_manualShortlist.json

@@ -199 +199 @@
         "http://pukapuka.nz", 3/3 1/4 [lorem ipsum used on first 3 pages]
         "http://community.nzdl.org", 3/3 0/3 [containsMRI has detected Te Taka Keegan as MRI sentence]
-!!        "http://kmpmusic.co.nz", 0-4/4? [but CD listing of some MRI song titles] 0 [no other pages containsMRI]
-        "http://maori.livingheritage.org.nz", 2/2 2/2
+X!!        "http://kmpmusic.co.nz", 0-4/4? [but CD listing of some MRI album and song titles] 0 [no other pages containsMRI]
+        "http://maori.livingheritage.org.nz", 2/2 2/2 {includes: http://www.livingheritage.org.nz}
         "http://pukoro.co.nz", 2/2 0/2
-        "https://register.tpota.org.nz", 0/1 [form] 0/2
-X        "https://cdn.tehiku.nz" => DOMAIN: "tehiku.nz",  0/4, 1/3 [but audio content may be in MRI]
+X        "https://register.tpota.org.nz", 0/1 [form] 0/2
++        "https://cdn.tehiku.nz" => DOMAIN: "tehiku.nz",  0/4, 1/3 [but audio content may be in MRI] But there are pages containing MRI to be found by non-random sampling of tehiku.nz, e.g. https://tehiku.nz/te-hiku-radio/te-tangihanga-ki-a-erima-henare/ contains MRI sentences
 !!        "http://www.runanga.co.nz", 3/3 0 [no containsMRI outside isMRI pages]
 !        "http://kuraaiwi.maori.nz", 2/4 [navigation only downloaded. But site content checked] 2/3
@@ -211 +211 @@
         "http://teaohou.natlib.govt.nz", 4/4, 2/4
         "http://www.tuwharetoa.iwi.nz", 2/3 0/3
-X        "http://auturoa.nz", 0/4 0/3 [lots of MRI terms among English] - COMMUNITY
++        "http://auturoa.nz", 0/4 0/3 [lots of MRI terms among English] - COMMUNITY (But there are pages inMRI to be found by non-random sampling, e.g. http://auturoa.nz/KarakiaMoKuaToRangiTeRaa.html)
         "https://www.terito.school.nz", 3/3, 0/2 total
         "https://ttw1.cwp.govt.nz", 3/3 3/3
@@ -228 +228 @@
 
         "http://anglicanprayerbook.nz", 3/3 3/3
-        "http://arataua.nz", 4/4, 2/3
-        "http://blog.teara.govt.nz", 3/3, 0/3 [AS: teara.govt.nz]
+        "http://arataua.nz", 4/4, 2/3        
         "http://maori.tki.org.nz", 3/3 3/3
 DONE (with/out www):        "http://www.firstworldwar.tki.org.nz", 
@@ -236 +235 @@
         "https://haereheikaiako.co.nz", 1/1, 0 [no containsMRI outside isMRI pages]
         "https://curriculumtool.education.govt.nz", 4/4, 3/3
-        "http://kurakokiri.maori.nz", 3/3, 3/3 [same nav menus on each page]
-        "http://kete.wcl.govt.nz", 2/5 [first 3 misdetected: Tokelauan (American Samoa), Kiribati, Tongan], 0/3
+        "http://kurakokiri.maori.nz", 3/3, 3/3 [same nav menus on each page] {includes: "http://www.kurakokiri.maori.nz"}   
         "http://www.kkmmaungarongo.co.nz", 3/3, 3/3
         "http://www.heartland.co.nz", 3/3, 1/1 total
         "http://oilcrash.com", 2/2 total, 0/3
-        "http://www.kura-porirua.school.nz", 4/4, 2/3
-        "http://videos.e-agent.nz", [AT: e-agent.nz] 3/3, 3/3 [repeated nav]
+        "http://www.kura-porirua.school.nz", 4/4, 2/3       
         "https://www.sporty.co.nz", 3/3, 0 [no containsMRI outside isMRI pages]
         "https://www.tematawai.maori.nz", 3/3, 3/3
 
-        "https://www.terakipaewhenua.school.nz", 
-        "http://www.tetaurawhiri.govt.nz", 
-        "http://archive.stats.govt.nz", 
-        "http://tiritiowaitangi.govt.nz", 
-        "http://www.waiata.maori.nz", 
-        "http://hana.co.nz", 
-        "http://kaupare.co.nz", 
-        "http://www.tereowrap.nz", 
-        "https://www.e-agent.nz", 
-        "http://www.hrc.co.nz", 
-        "http://ngatiporoukiponeke.org.nz", 
-        "http://rurued.school.nz", 
-        "http://www.twtop.school.nz", 
-        "https://www.infinite-electronic.nz", 
-        "http://www.huri-translations.pf", 
-        "https://admin.teara.govt.nz", 
-        "https://tiritiowaitangi.govt.nz", 
-        "http://www.tmoa.tki.org.nz", 
-        "https://www.komako.org.nz", 
-        "http://www.wcl.govt.nz", 
-        "https://office.e-agent.nz", 
-        "http://punareo.co.nz", 
-        "http://www.kurakokiri.maori.nz", 
-        "https://rapuatearatika.education.govt.nz", 
-        "http://tmmkkm.school.nz", 
-        "https://www.components-mart.nz", 
-        "http://www.cs.waikato.ac.nz", 
-        "http://www.kupengahao.co.nz", 
-        "https://www.hapuhauora.health.nz", 
-        "https://www.lcds-display.nz", 
-        "http://waiata.maori.nz", 
-        "http://cms.sunsmartschools.co.nz", 
-        "http://www.livingheritage.org.nz", 
-        "http://kuraproductions.co.nz", 
-        "https://keepourmoneyclean.govt.nz", 
-        "http://www.tekura.school.nz", 
-        "http://www.tkkmmokopuna.school.nz", 
-        "http://hangaraumatihiko.tki.org.nz", 
-        "http://www.pakanae.maori.nz"
++        "https://www.terakipaewhenua.school.nz", 
++        "http://www.tetaurawhiri.govt.nz", 
++        "http://archive.stats.govt.nz", (1 page isMRI)
++        "http://tiritiowaitangi.govt.nz", 
++!!      "http://www.waiata.maori.nz", {includes: "http://waiata.maori.nz"}
++        "http://hana.co.nz", [crawled version of page contains MRI sentences, but current page is labelled for mobiles whereas in browser it just has a picture]
++        "http://kaupare.co.nz", 
++        "http://www.tereowrap.nz", 
+?X        "https://www.e-agent.nz", [autotranslated? SEO related site in NZ, Chinese, English and MRI] {includes: "https://office.e-agent.nz"}
+                 { included: "http://videos.e-agent.nz", [AT: e-agent.nz] 3/3, 3/3 [repeated nav] }             
++        "http://www.hrc.co.nz", 
++        "http://ngatiporoukiponeke.org.nz", 
+
++        "http://rurued.school.nz", 
++        "http://www.twtop.school.nz", 
+X        "https://www.infinite-electronic.nz", [autotranslated product site]
++!!      "http://www.huri-translations.pf", 
++        "https://admin.teara.govt.nz", {included: "http://blog.teara.govt.nz", 3/3, 0/3 [AS: teara.govt.nz]}
++!!        "https://tiritiowaitangi.govt.nz", 
++        "http://www.tmoa.tki.org.nz", 
++        "https://www.komako.org.nz", [no longer available, but crawled page had a paragraph in Maori with presumably a translation thereafter]
++        "http://www.wcl.govt.nz", {included: "http://kete.wcl.govt.nz" as wcl.govt.nz; 2/5 [first 3 misdetected: Tokelauan (American Samoa), Kiribati, Tongan], 0/3}        
++!!      "http://punareo.co.nz", [waiata]        
+
++        "https://rapuatearatika.education.govt.nz", 
++        "http://tmmkkm.school.nz", 
+X        "https://www.components-mart.nz",  [autotranslated product site]
++        "http://www.cs.waikato.ac.nz", [Te Taka's pages!]
++!!!        "http://www.kupengahao.co.nz", [MRI language books and resources]
++        "https://www.hapuhauora.health.nz", [Smokefree site. The one isMRI page has a proper sentence.]
+X        "https://www.lcds-display.nz",  [autotranslated product site]        
++        "http://cms.sunsmartschools.co.nz", [as sunsmartschools.co.nz, e.g. http://sunsmartschools.co.nz/kura/how-to-become.html]     
++        "http://kuraproductions.co.nz", 
++        "https://keepourmoneyclean.govt.nz", [1 page]
+
++!!      "http://www.tekura.school.nz", 
++        "http://www.tkkmmokopuna.school.nz", [e.g. http://www.tkkmmokopuna.school.nz/newsletter_sets/newsletters/13-wahanga-4-wiki-5-he-puna-korero]
++        "http://hangaraumatihiko.tki.org.nz", [govt website. e.g. http://hangaraumatihiko.tki.org.nz/te-tupuranga-tangata-me-te-rorohiko/whakatupuranga-3-te-whakatau-tikanga-kawe-i-nga-mahi-whakaoti-hopanga/]
++        "http://www.pakanae.maori.nz"
     ],
     "numPagesInMRICount" : 4360,
@@ -290 +287 @@
 }
 
+
+96 sites detected as having isMRI pages - 7 sites/subdomains already included in the existing 96 = 89 sites.
+
+-2.5* product sites -2 non-MRI sites with songlistings or forms etc
+    *0.5 for e-agent.nz site
+= 84.5 sites total that at least contain MRI, most have pages inMRI.
 ----------------------------
 
@@ -474 +477 @@
 
 
+----------------------------
+
+The remainder: 80 NZ sites detected as not containing pages InMRI, but with positive number of pages detected as containsMRI:
+
+db.Websites.aggregate([
+    {
+        $match: {
+            $and: [
+                {numPagesContainingMRI: {$gt: 0}},
+                {numPagesInMRI: {$eq: 0}},
+                {$or: [{geoLocationCountryCode:"NZ"},{domain: /\.nz/}]}
+            ]
+        }
+    },
+    { $unwind: "$geoLocationCountryCode" },
+    {
+        $group: {
+            _id: "nz",
+            count: { $sum: 1 },
+            domain: { $addToSet: '$domain' },
+            numPagesInMRICount: { $sum: '$numPagesInMRI' },
+            numPagesContainingMRICount: { $sum: '$numPagesContainingMRI' }
+        }
+    },
+    { $sort : { count : -1} }
+]);
+
+
+Find pages for testing with:
+    db.getCollection('Webpages').find({URL:/ashtangatauranga\.co\.nz/, containsMRI: true, mriSentenceCount: {$gt: 0}})
+
+
+/* 1 */
+{
+    "_id" : "nz",
+    "count" : 80.0,
+    "domain" : [ 
+X        "http://www.zoomin.co.nz", [map site, so placenames]
+X        "http://www.biketorqueyamaha.co.nz", [placenames] {includes "http://biketorqueyamaha.co.nz"}
+X        "http://archerpix.com", [photo captions containing placenames]
+X        "http://philipbeadle.co.nz", [art captions containing placenames]
+X        "https://2019.nethui.nz", [Just MRI words in ENG sentences]
+X        "http://crimson.co.nz", [address]
++        "http://holyspirit.nz", (e.g. https://holyspirit.nz/wp-content/uploads/02-25-2018-Newsletter.pdf)
+X        "https://www.wingspan.co.nz", [1 page, 1 conjoined word, looks like placename]
+X        "http://nzpostcard.co.nz", [postcards with placenames]
++        "https://www.ngamanawainc.co.nz", (e.g. https://www.ngamanawainc.co.nz/history/id/60) {includes "http://www.ngamanawainc.co.nz"}
+
++        "http://www.finlaysonpark.school.nz", [e.g. http://www.finlaysonpark.school.nz/58/pages/61-team-overview; Some Tongan language pages and Samoan pages]
+X        "http://artizani.co.nz", [address]
++        "http://www.w3vietnam.org.nz", [e.g. http://www.w3vietnam.org.nz/w3%20mihi%20ki%20ka%20hoia.htm] (includes "http://w3vietnam.org.nz")
+X        "https://sooty.nz", [names, war death notices, place names]
+X?        "http://rakaumanga.school.nz", [School has a longer Maori title name, but that's it] {includes "http://www.rakaumanga.school.nz"}
+X        "http://www.rotoruanz.com", [e.g. https://www.rotoruanz.com/RNZ/media/Media-Library/SOI-Board-version-08-June-2018-final-published-on-website.pdf]
+X        "https://www.cruisetourstauranga.co.nz", [English with Tauranga being MRI placename]
+X        "http://www.jeremybaker.nz", [one word, HOkio]
+
+X        "https://liveresults.co.nz", [canoe sports team names]
+X        "http://rexedra.gen.nz", [ENG sentence with MRI words]
++        "https://www.takitimu.ac.nz", [school of performing arts, example MRI sentence at https://www.takitimu.ac.nz/about-us]
+X        "http://www.electionresults.govt.nz", [placenames and misdetection of Return to Home Page] {includes https://www.electionresults.org.nz which seems to be an alias for the same} {includes "http://archive.electionresults.govt.nz"}
++        "https://kotahimiriona.co.nz", e.g. (https://kotahimiriona.co.nz/about-the-app/)
++        "https://rehuamarae.co.nz", (e.g. https://rehuamarae.co.nz/)
++        "http://reoora.co.nz", (e.g. https://reoora.co.nz/about/)
+
+X        "http://otorohanga.directorybusiness.co.nz", [placenames]
+X        "http://waitarahistory.org.nz", [placenames and misdetection of sentences of the form "31, No 262." as MRI]
++        "https://manawatuheritage.pncc.govt.nz", (e.g. https://manawatuheritage.pncc.govt.nz/about)
++        "http://rsnz.natlib.govt.nz", [e.g. http://rsnz.natlib.govt.nz/volume/rsnz_26/rsnz_26_00_004560.html] NOTE: site appears to use Greenstone
+X        "https://www.rotorua-rafting.co.nz", [placenames]
++        "https://www.taitokerautrust.org.nz", (e.g. https://www.taitokerautrust.org.nz/)
++        "http://tewikiotereomaori.nz", (e.g. https://tewikiotereomaori.nz/about/)
++        "https://www.korokikahukura.co.nz", (e.g. https://www.korokikahukura.co.nz/ko-wai-m257tou.html about the connection to the Waikato River)
+
+X        "https://www.puhaandpakeha.co.nz", [ENG sentences with MRI words]
+X        "http://myfathersworld.net.nz", [placenames]
+X        "https://www.ashtangatauranga.co.nz", [misdetection]
++        "https://www.pinterest.nz", (e.g. sentence "E tu Whare Rangi, e tu Whare Pataka, kia ora koutou nga Whare Tipuna o te Ao Maori." at https://www.pinterest.nz/amp/marylizw03/maori/)
++        "https://www.rereahu.maori.nz", (e.g. https://www.rereahu.maori.nz/uploads/72717/files/168860/Te_Maru_o_Rereahu_Newsletter_No_5_February_2011.pdf)
++        "http://givealittle.co.nz", (e.g. https://givealittle.co.nz/cause/rebuild-tapu-te-ranga-marae which contains greetings phrases and sentence ""Nā te ringa tangata i hanga te whare Nā te tuarā o te whare i whakatipu i te tangata")
+X        "http://www.gans.co.nz", [placenames]
++        "https://kaiiwicamp.nz", [placenames] {includes "http://kaiiwicamp.nz"}
++        "http://ngarauhuia.ngatiapakiterato.iwi.nz", (e.g. http://ngarauhuia.ngatiapakiterato.iwi.nz/pdf/Nga-Rau-Huia_Raumati_2015_Issue1.pdf)
++        "https://m.wairarapatv.co.nz", (e.g. https://m.wairarapatv.co.nz/archive/i/JGmJqCZCNK8/te-ara-whanui-kura-kaupapa-maori-o-nga-khanga-reo-o-te-awa-kairangi)
+
+X        "http://www.methodist.org.nz", [ENG sentence with MRI words]
++        "http://avonside.net", (e.g. http://avonside.net/Nicky/indexmain.htm)
+X        "http://www.ruralfind.co.nz", [placenames]
++        "http://www.maoriinvestments.co.nz", (e.g. Vision, Mission & Values tab of https://maoriinvestments.co.nz/about/organisation)
++        "http://conference.tpwt.maori.nz", (e.g. https://www.tpwt.maori.nz/te-mahe-matauranga/)
++        "https://www.puau.school.nz", (e.g. https://www.puau.school.nz/k%C4%81inga-home)
++?        "http://ngatiwhakaue.iwi.nz", (e.g. greetings at http://ngatiwhakaue.iwi.nz/)
+X        "http://www.nzpcn.org.nz", [lots of misdetections and one sentence in English with MRI words, e.g. "Kaori is a Taonga to Maori".] {includes "http://dev.nzpcn.org.nz"}
++?        "https://interactives.stuff.co.nz", [2 x page mostly empty except for the title "TE WIKI O TE REO MĀORI Māori"]
++        "http://tehauora.org.nz", (e.g. http://tehauora.org.nz/ and proverb at bottom of http://tehauora.org.nz/about-us)
+
++        "http://temahurehure.maori.nz", (e.g. greeting http://temahurehure.maori.nz/site/file/temp/hirebook.pdf)
+X        "http://pukekohe.directorybusiness.co.nz", [placenames]
++!!      "http://www.temarareo.org", (dictionary with sample sentences. e.g. http://www.temarareo.org/PAPAKUPU/dictionary-searchresults/A-2.htm)
+X        "https://www.tasteofplenty.co.nz", [Tauranga placename and misdetection of "No one went away hungry] {includes "http://www.tasteofplenty.co.nz"}
+      
++        "http://www.tetaumuturunanga.iwi.nz", (e.g. http://www.tetaumuturunanga.iwi.nz/wp-content/uploads/2016/04/Malvern-Cultural-Narrative-Draft.pdf)
+        
+        
+X        "https://www.blushandbrows.nz", [misdetection of "Makeup..."]
+X        "http://talkingtothecan.com", [misdetection of things like ""22, no." and mistaking ENG sentences with MRI words]
+
++?       "http://whatonga.school.nz", [school title]
++?       "https://player.vimeo.com", [Video titles contain MRI sentences and MRI titles interspersed in ENG sentence. Video's audio content may be in MRI]
++        "http://www.writersfestival.co.nz", (e.g. http://www.writersfestival.co.nz/news/Page1/urgently-relevant-novel-wins-countrys-richest-literary-award/)
++?       "http://southerntribes.co.nz", [brazilian jiu jitsu site with a greeting in MRI on main page]
++        "http://www.kmk.maori.nz", (e.g. http://www.kmk.maori.nz/kmk-events)
++        "https://www.stats.govt.nz", (e.g. http://archive.stats.govt.nz/Census/2001-census-data/2001-census-pacific-profiles/cook-island-maori-people-in-new-zealand.aspx)
+X        "http://www.eventcinemas.co.nz", [placenames and misdetection of ENG "Atura Hotels"]
+X        "https://www.zenbu.co.nz" [misdetection and NZ school addresses]
+    ],
+    "numPagesInMRICount" : 0,
+    "numPagesContainingMRICount" : 1673
+}