/* For sites originating in NZ or with nz TLD, none of the URLs are manually inspected and all URLs are accepted. For all but NZ, get final column results with: db.getCollection('Websites').find({domain:/coggle\.it/}) And can check for URLs with: db.getCollection('Webpages').find({URL: /coggle\.it/, isMRI: true}) NOTES: 1. DE: "de","2.0","0+1","9+35 misdetected", http://www.cartogiraffe.com, https://www.cartogiraffe.com, Ought to be 2+2 numPagesInMRICount and 9+2 numPagesContainingMRICount: - both cartogiraffe.com pages were identical and had mostly MRI sentences with one name not being MRI. So isMRI should have been true for both pages. - Only one of the 2 MRI translations of the universal declaration of human rights at http://www.udhr.de got downloaded. A total of 75 pages were downloaded, but more translated pages appeared to be on the webpage. Not sure why the crawl had a _SUCCESS file to indicate completed download. - Then http://www.udhr.de had 35-1 non-MRI language translations of the universal declaration of human rights where one or more sentences were misdetected as MRI. With the additional MRI page that didn't get downloaded, should have 9+2 = 11 pages containing MRI. So instead of "de","2.0","1","44", http://www.cartogiraffe.com, https://www.cartogiraffe.com, http://www.udhr.de "de","2.0","4","11", http://www.cartogiraffe.com, https://www.cartogiraffe.com, http://www.udhr.de "au","3.0",7+0+1,83+1+3,https://www.kiwiproperty.com, https://infogram.com/te-marautanga-o-aotearoa-moe-pld-allocations-2012-1go502ygvn562jd,https://koreromaori.com 2. US: aclhokiangarocks.blogspot.com contains at least a page with MRI paragraphs. See http://aclhokiangarocks.blogspot.com/feeds/posts/default under section "Nga Tuhinga o tatou Tupuna" Although this page has been crawled by Nutch, the contents were presented in the blog in a complex way and therefore the text wasn't retrieved here. See also the dedicated page this text should have been in http://aclhokiangarocks.blogspot.com/2012/05/nga-tuhinga-o-tatou-tupuna.html "_id","siteCount","numPagesInMRICount","numPagesContainingMRICount","URLs of pages detected as inMRI" "nz","176.0" containsMRI vs 96 pages inMRI,"4360","9641" in 176 containsMRI pages vs 7968 in isMRI pages "us","29.0", 1+2+0+0+4+166+0+39 +257+2+21+12+25+13+53+0+1+0+1+11 +32+37+4 +0+0+0 = 681, 31+2+2+20+58+166+3+91 +258+2+25+12+66+22+53+6+1+1+2+10 +58+54+6 +1+2+1 = 953, anglicanhistory.org,unicode.org,static-promote.weebly.com,aclhokiangarocks.blogspot.com,bahaiprayers.net,biblehub.com,muhammad.com,godrules.net,m.biblepub.com, krassotkin.ru,gotquestions.org, maorinews.com,maaori.com,kiaorahola.blogspot.com,kjohnsonnz.blogspot.com,pumanawawhangara.blogspot.com,dannykahei.tripod.com,burkekm001.tripod.com,tkkpipipaopao.blogspot.com, manateina.blogspot.com, tatai09.blogspot.com,twttoa.com,tuhua2010.blogspot.com, breaker.audio,drive.google.com/file/d/1NwuzafjddaP8gxI7O_Zapts5bM7mrtwn/preview,in.pinterest.com/pin/317363104978423418/ "au","2.0","8","86", https://www.kiwiproperty.com, https://koreromaori.com "de","2.0","4","11", http://www.cartogiraffe.com, https://www.cartogiraffe.com, http://www.udhr.de "dk","2.0","4","7", *.ngapuhitelevision.com, *.ngapuhiradio.com "bg","1.0","2","2", http://anitra.net/activism/humanrights/UDHR/mbf_print.htm, http://anitra.net/activism/humanrights/UDHR/rrt_print.htm "cz","1.0","0","1", http://www.henryklahola.nazory.cz/094.Maori.htm, http://henryklahola.nazory.cz/094.Maori.htm "es","1.0","1","1", https://www.uv.es/~pla/red.net/intmaori.html "fr","1.0","1","1", http://chantsdeluttes.free.fr/versionsinter/page%20maori.html "ie","1.0","1","3", https://coggle.it/diagram/WSYB0mLA2QABD5BH/t/ko-au-ko-koe -------------- https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/find-sample-size/#CI1 https://stats.stackexchange.com/questions/207584/sample-size-choice-with-binary-outcome https://www.statisticshowto.datasciencecentral.com/z-alpha2-za2/ N (NZ pages where isMRI comes out true) = 4360 solving for n, the sample size confidence level = 90% m, margin of error = 5% From the "z alpha/2" table, for 90% confidence, we get a z alpha/2 value of 1.6449 (or 1.645). Then the sample size, n, we need is = 1.6449^2 * 4360 / ( 1.6449^2 + (4 * 4359) * 0.05^2) = 255 (rounded up) For N = 681, sample size n is = 1.6449^2 * 681 / ( 1.6449^2 + (4 * 680) * 0.05^2) = 194 (rounded up) sample size for NZ: 255 (90% confidence with 5% margine of error, Including a finite correction factor) sample size for US: 194 */ "_id","siteCount","numPagesInMRICount","numPagesContainingMRICount","URLs of pages detected as inMRI" "nz","176.0","4360","9641" "us","29.0","681","953" "au","2.0","8","86" "de","2.0","4","11" "dk","2.0","4","7" "bg","1.0","2","2" "cz","1.0","0","1" "es","1.0","1","1" "fr","1.0","1","1" "ie","1.0","1","3" Total sites containing MRI: 216 Total pages detected as being in MRI: 5062 Total pages detected as containing MRI sentences: 10706 NZ - sample 255 pages from: /* db.Websites.aggregate([ { $match: { $and: [ {numPagesContainingMRI: {$gt: 0}}, {$or: [{geoLocationCountryCode:"NZ"},{domain: /\.nz/}]} ] } }, { $unwind: "$geoLocationCountryCode" }, { $group: { _id: "nz", count: { $sum: 1 }, domain: { $addToSet: '$domain' }, numPagesInMRICount: { $sum: '$numPagesInMRI' }, numPagesContainingMRICount: { $sum: '$numPagesContainingMRI' } } }, { $sort : { count : -1} } ]); OR is this better: db.Websites.aggregate([ { $match: { $and: [ {numPagesInMRI: {$gt: 0}}, {$or: [{geoLocationCountryCode:"NZ"},{domain: /\.nz/}]} ] } }, { $unwind: "$geoLocationCountryCode" }, { $group: { _id: "nz", count: { $sum: 1 }, domain: { $addToSet: '$domain' }, numPagesInMRICount: { $sum: '$numPagesInMRI' }, numPagesContainingMRICount: { $sum: '$numPagesContainingMRI' } } }, { $sort : { count : -1} } ]); */ num NZ sites with > 0 isMRI pages = 96 Total numPagesInMRI in NZ sites = 4360 Total numPagesContainingMRI in NZ sites = 7968 Using the results you get a list of domains that matched. 171 nz domains, though it should be 176? -1 Copy each domain (up to 255 of them) and look for the first 1 or 2 max that matches isMRI: 1. db.getCollection('Webpages').find({URL:/pukekohe.directorybusiness.co.nz/, isMRI: true}) - check it contains a positive number of pages in MRI and check the first 1-2 pages to make sure they are indeed in MRI. Note down the ratio of MRI finds. e.g. 2/2. 2. Find those pages that containsMRI but not isMRI and check if there are indeed sentences in MRI. Note down the ratio for the first 2 pages. db.getCollection('Webpages').find({URL:/maori.livingheritage.org.nz/, isMRI: false, containsMRI: true}) /* 1 */ { "_id" : "nz", "count" : 96.0, "domain" : [ "http://www.teipukarea.maori.nz", 3/3 1/3 "http://ngatipahauwera.co.nz", 2/2, 2/2 "http://www.oag.govt.nz", 2/2 0/2 "https://sexualviolence.victimsinfo.govt.nz", 3/3 0/3 "http://tmoa.tki.org.nz", 3/3 3/3 "http://www.tewhanake.maori.nz", 3/3 2/3 "http://www.matarikifestival.org.nz", 4/4 0/3 "http://www.otepoti.school.nz", 3/3 0/4 !! "https://www.maoritelevision.com", 3/4, 0 [no containsMRI outside isMRI pages] "http://pukapuka.nz", 3/3 1/4 [lorem ipsum used on first 3 pages] "http://community.nzdl.org", 3/3 0/3 [containsMRI has detected Te Taka Keegan as MRI sentence] !! "http://kmpmusic.co.nz", 0-4/4? [but CD listing of some MRI song titles] 0 [no other pages containsMRI] "http://maori.livingheritage.org.nz", 2/2 2/2 "http://pukoro.co.nz", 2/2 0/2 "https://register.tpota.org.nz", 0/1 [form] 0/2 X "https://cdn.tehiku.nz" => DOMAIN: "tehiku.nz", 0/4, 1/3 [but audio content may be in MRI] !! "http://www.runanga.co.nz", 3/3 0 [no containsMRI outside isMRI pages] ! "http://kuraaiwi.maori.nz", 2/4 [navigation only downloaded. But site content checked] 2/3 "http://kurataiao.tki.org.nz", 3/3, 1/total 3 !! "http://satellites.co.nz", 3/3 [kpop], 0 [no containsMRI outside isMRI pages] "http://teaohou.natlib.govt.nz", 4/4, 2/4 "http://www.tuwharetoa.iwi.nz", 2/3 0/3 X "http://auturoa.nz", 0/4 0/3 [lots of MRI terms among English] - COMMUNITY "https://www.terito.school.nz", 3/3, 0/2 total "https://ttw1.cwp.govt.nz", 3/3 3/3 "https://www.whanau-tahi.school.nz", 4/4, 1/2 total "https://e-ako-pangarau.nzmaths.co.nz", 3/3 total, 1/1 total "https://teaomaori.news", 3/3, 0/1 total "http://tetaurawhiri.govt.nz", 3/3 /3/3 [Māori Language Commission site] "https://www.tuiatematangi.ac.nz", 4/4 3/3 "http://animations.tewhanake.maori.nz", 3/3 3/3 !! "https://www.dnc.org.nz", 1/1 total, 0 [no containsMRI outside isMRI pages] !! "http://firstworldwar.tki.org.nz", 3/3, 0 [no containsMRI outside isMRI pages] "http://www.28maoribattalion.org.nz", 3/3, 1/3 "http://www.tewikiotereomaori.co.nz", 1/1 total, 3/3 "http://www.brettgraham.co.nz", 1/1 total, 0/3 !! "https://hepatakakupu.nz", 3/3, 0 [no containsMRI outside isMRI pages] "http://anglicanprayerbook.nz", 3/3 3/3 "http://arataua.nz", 4/4, 2/3 "http://blog.teara.govt.nz", 3/3, 0/3 [AS: teara.govt.nz] "http://maori.tki.org.nz", 3/3 3/3 DONE (with/out www): "http://www.firstworldwar.tki.org.nz", X "http://www.topomap.co.nz", 0/2 [all placenames], 0 [no containsMRI outside isMRI pages] "https://paekupu.co.nz", 4/4, 0 [no containsMRI outside isMRI pages] "https://haereheikaiako.co.nz", 1/1, 0 [no containsMRI outside isMRI pages] "https://curriculumtool.education.govt.nz", 4/4, 3/3 "http://kurakokiri.maori.nz", 3/3, 3/3 [same nav menus on each page] "http://kete.wcl.govt.nz", 2/5 [first 3 misdetected: Tokelauan (American Samoa), Kiribati, Tongan], 0/3 "http://www.kkmmaungarongo.co.nz", 3/3, 3/3 "http://www.heartland.co.nz", 3/3, 1/1 total "http://oilcrash.com", 2/2 total, 0/3 "http://www.kura-porirua.school.nz", 4/4, 2/3 "http://videos.e-agent.nz", [AT: e-agent.nz] 3/3, 3/3 [repeated nav] "https://www.sporty.co.nz", 3/3, 0 [no containsMRI outside isMRI pages] "https://www.tematawai.maori.nz", 3/3, 3/3 "https://www.terakipaewhenua.school.nz", "http://www.tetaurawhiri.govt.nz", "http://archive.stats.govt.nz", "http://tiritiowaitangi.govt.nz", "http://www.waiata.maori.nz", "http://hana.co.nz", "http://kaupare.co.nz", "http://www.tereowrap.nz", "https://www.e-agent.nz", "http://www.hrc.co.nz", "http://ngatiporoukiponeke.org.nz", "http://rurued.school.nz", "http://www.twtop.school.nz", "https://www.infinite-electronic.nz", "http://www.huri-translations.pf", "https://admin.teara.govt.nz", "https://tiritiowaitangi.govt.nz", "http://www.tmoa.tki.org.nz", "https://www.komako.org.nz", "http://www.wcl.govt.nz", "https://office.e-agent.nz", "http://punareo.co.nz", "http://www.kurakokiri.maori.nz", "https://rapuatearatika.education.govt.nz", "http://tmmkkm.school.nz", "https://www.components-mart.nz", "http://www.cs.waikato.ac.nz", "http://www.kupengahao.co.nz", "https://www.hapuhauora.health.nz", "https://www.lcds-display.nz", "http://waiata.maori.nz", "http://cms.sunsmartschools.co.nz", "http://www.livingheritage.org.nz", "http://kuraproductions.co.nz", "https://keepourmoneyclean.govt.nz", "http://www.tekura.school.nz", "http://www.tkkmmokopuna.school.nz", "http://hangaraumatihiko.tki.org.nz", "http://www.pakanae.maori.nz" ], "numPagesInMRICount" : 4360, "numPagesContainingMRICount" : 7968 } ---------------------------- /* 1 */ { "_id" : "nz", "count" : 176.0, "domain" : [ !! "http://pukekohe.directorybusiness.co.nz", 0/2, 0/2, isMRI = 0!! "http://maori.livingheritage.org.nz", 2/2 2/2 "http://pukoro.co.nz", 2/2 0/2 "http://www.rakaumanga.school.nz", 0/4 0/4 "http://www.ngamanawainc.co.nz", 0/2 0/2 "https://office.e-agent.nz", "https://www.components-mart.nz", "http://tmmkkm.school.nz", "http://www.rotoruanz.com", "http://www.huri-translations.pf", "https://admin.teara.govt.nz", "http://hangaraumatihiko.tki.org.nz", "https://sexualviolence.victimsinfo.govt.nz", "http://www.tekura.school.nz", "http://philipbeadle.co.nz", "http://www.cs.waikato.ac.nz", "https://www.hapuhauora.health.nz", "http://cms.sunsmartschools.co.nz", "https://keepourmoneyclean.govt.nz", "http://www.kura-porirua.school.nz", "http://waitarahistory.org.nz", "http://oilcrash.com", "http://videos.e-agent.nz", "https://manawatuheritage.pncc.govt.nz", "https://www.terakipaewhenua.school.nz", "http://dev.nzpcn.org.nz", "https://kotahimiriona.co.nz", "http://kurakokiri.maori.nz", "https://www.sporty.co.nz", "http://kaupare.co.nz", "http://ngatiporoukiponeke.org.nz", "https://www.takitimu.ac.nz", "http://www.tetaurawhiri.govt.nz", "http://www.waiata.maori.nz", "http://conference.tpwt.maori.nz", "http://ngatiwhakaue.iwi.nz", "http://www.nzpcn.org.nz", "http://www.ruralfind.co.nz", "https://www.dnc.org.nz", "https://www.puau.school.nz", "https://kaiiwicamp.nz", "https://www.terito.school.nz", "https://www.pinterest.nz", "https://e-ako-pangarau.nzmaths.co.nz", "http://givealittle.co.nz", "https://teaomaori.news", "https://www.korokikahukura.co.nz", "http://myfathersworld.net.nz", "http://www.firstworldwar.tki.org.nz", "https://www.ashtangatauranga.co.nz", "http://biketorqueyamaha.co.nz", "https://www.rereahu.maori.nz", "http://www.tewikiotereomaori.co.nz", "http://www.brettgraham.co.nz", "http://tewikiotereomaori.nz", "http://anglicanprayerbook.nz", "http://arataua.nz", "http://blog.teara.govt.nz", "http://www.otepoti.school.nz", "http://www.kmk.maori.nz", "http://www.eventcinemas.co.nz", "https://www.stats.govt.nz", "http://www.oag.govt.nz", 2/2 0/2 "http://whatonga.school.nz", "http://www.tewhanake.maori.nz", "https://www.maoritelevision.com", "http://kuraaiwi.maori.nz", "http://kurataiao.tki.org.nz", "http://teaohou.natlib.govt.nz", "http://www.tetaumuturunanga.iwi.nz", "http://www.tasteofplenty.co.nz", "http://community.nzdl.org", "https://www.blushandbrows.nz", "https://register.tpota.org.nz", "https://cdn.tehiku.nz", "http://www.wcl.govt.nz", "http://www.jeremybaker.nz", "http://punareo.co.nz", "https://rapuatearatika.education.govt.nz", "http://www.kurakokiri.maori.nz", "https://www.cruisetourstauranga.co.nz", "https://sooty.nz", "http://rakaumanga.school.nz", "https://tiritiowaitangi.govt.nz", "http://www.tmoa.tki.org.nz", "http://www.w3vietnam.org.nz", "https://www.infinite-electronic.nz", "https://www.komako.org.nz", "http://nzpostcard.co.nz", "http://artizani.co.nz", "http://www.finlaysonpark.school.nz", "http://crimson.co.nz", "http://holyspirit.nz", "http://www.tkkmmokopuna.school.nz", "http://www.pakanae.maori.nz", "http://www.teipukarea.maori.nz", "http://archerpix.com", "https://2019.nethui.nz", "http://www.kupengahao.co.nz", "https://www.lcds-display.nz", "http://waiata.maori.nz", "http://kuraproductions.co.nz", "http://www.biketorqueyamaha.co.nz", "http://www.livingheritage.org.nz", "http://www.zoomin.co.nz", "http://rsnz.natlib.govt.nz", "http://otorohanga.directorybusiness.co.nz", "http://reoora.co.nz", "http://w3vietnam.org.nz", "https://rehuamarae.co.nz", "https://www.electionresults.org.nz", "https://www.ngamanawainc.co.nz", "https://www.rotorua-rafting.co.nz", "https://www.taitokerautrust.org.nz", "https://www.wingspan.co.nz", "http://www.kkmmaungarongo.co.nz", "http://kete.wcl.govt.nz", "http://www.heartland.co.nz", "http://www.electionresults.govt.nz", "https://www.tematawai.maori.nz", "http://hana.co.nz", "http://www.tereowrap.nz", "http://rurued.school.nz", "http://www.twtop.school.nz", "http://rexedra.gen.nz", "http://archive.stats.govt.nz", "https://liveresults.co.nz", "https://www.e-agent.nz", "http://tiritiowaitangi.govt.nz", "http://www.hrc.co.nz", "http://animations.tewhanake.maori.nz", "https://interactives.stuff.co.nz", "http://avonside.net", "http://www.methodist.org.nz", "https://www.tasteofplenty.co.nz", "http://www.maoriinvestments.co.nz", "https://m.wairarapatv.co.nz", "http://www.gans.co.nz", "https://ttw1.cwp.govt.nz", "http://ngarauhuia.ngatiapakiterato.iwi.nz", "https://www.tuiatematangi.ac.nz", "http://tetaurawhiri.govt.nz", "http://maori.tki.org.nz", "http://www.topomap.co.nz", "https://www.puhaandpakeha.co.nz", "https://haereheikaiako.co.nz", "https://paekupu.co.nz", "https://curriculumtool.education.govt.nz", "http://firstworldwar.tki.org.nz", "http://www.28maoribattalion.org.nz", "https://hepatakakupu.nz", "https://www.zenbu.co.nz", "http://www.matarikifestival.org.nz", "http://pukapuka.nz", "http://ngatipahauwera.co.nz", 2/2 2/2 "http://southerntribes.co.nz", "https://player.vimeo.com", "http://tmoa.tki.org.nz", "http://www.writersfestival.co.nz", "http://talkingtothecan.com", "https://www.whanau-tahi.school.nz", "http://satellites.co.nz", "http://auturoa.nz", "http://www.tuwharetoa.iwi.nz", "http://kmpmusic.co.nz", "http://www.temarareo.org", "http://archive.electionresults.govt.nz", "http://kaiiwicamp.nz", "http://tehauora.org.nz", "http://temahurehure.maori.nz", "http://www.runanga.co.nz" ], "numPagesInMRICount" : 4360, "numPagesContainingMRICount" : 9641 }