Changeset 33816

Timestamp:

2019-12-19T22:33:08+13:00 (4 years ago)

Author:

ak19

Message:

Finished manually going through the sites that I couldn't easily filter out as probably autotranslated. I listed the ones that appeared to have genuine content in the Maori language and crossed out the ones that were misdetected or otherwise irrelevant. Some are question marked.

File:

• other-projects/maori-lang-detection/MoreReading/mongodb.txt (modified) (6 diffs)

other-projects/maori-lang-detection/MoreReading/mongodb.txt

@@ -1025 +1025 @@
      BUT: it's auto-translated (e.g. Dutch is clearly auto-translated), MRI not in default or any visible drop down list, and the domain changes once you view it in Dutch to https://nl.admission.nz/
 
-* FR: 35 sites from FR
-    http://blueheavenisland.com - French Polynesia
+* FR: 16 sites from FR
+    http://blueheavenisland.com, http://www.blueheavenisland.com - misdetection. French Polynesia
     https://www.lexilogos.com/ -> takes me to NZ website MaoriDictionary.co.nz etc for translating words anyway
     http://kihikihi.fr/ -> travel (blog?). Appears to be Hawaiian related and not Maori.
 !!  http://chantsdeluttes.free.fr/versionsinter/page%20maori.html -> Seems it may be a proper translation or composition, as Dutch and Flemish (and Groningense) versions are different songs by individual translators/composers
     http://splaf.free.fr/pfurb.html - Tahiti, French Polynesian, ... island names
-    http://mi.fitnessrebates.com - Uses https://wordpress.org/plugins/weglot/ wordpress-compatible multilingual plugin, which ensures translated pages get indexed by google - exactly what we want to avoid
-*
-
-
+X    http://mi.fitnessrebates.com - Uses https://wordpress.org/plugins/weglot/ wordpress-compatible multilingual plugin, which ensures translated pages get indexed by google - exactly what we want to avoid
+   http://mahajana.net - misdetected a Japanese Zen Buddhist chant as MRI
+   http://rapanui.fr - Rapa Nui Easter Island. Misdetected.
+   http://www.gif.ovh - autotranslated pages. Supposedly a GIF repository
+   http://baladeornithologique.com - misdetection of the word "Retour"
+   http://www.gaudry.be - misdetection of Japanese hiragana etc, and French "faire", as MRI
+   http://www.gototahiti.net - probably misdetection, see title
+   http://www.maraamusurfskirace.com - Bora Bora, French Polynesia. Misdetected.
+   http://www.rongo-rongo.com - appears to be related to Easter Island. Just 1 sentence however.
+   http://pt.city-usa.net - misdetection. Hawaii.
+   https://www.manualscat.com - Misdetection. Appears to be in German. Manuals pages. 
+NL:
+!!! - http://www.gouvernante.info and http://gouvernante.info - radio links to NZ websites not found by commoncrawl and which potentially have Maori language content. For example, http://irirangi.net/, https://www.atiawatoafm.com, www.maori.org.nz
+- https://www.arrowhead.eu, https://arrowheadproject.azurewebsites.net, arrowhead.eu - misidentification of URL
+- tonhut.nl - misidentication
+? http://nielsonboutique.co.uk, http://longhornlaw.net, http://tetsubo.org, http://hidsonphoto.com, http://wearehomework.com/- Feels autotranslated, but no language options visible. All SEO related
+- diverosa.com - Rapa Nui, Easter Island
+- nonlinear.demon.nl - misidentified
+- encyclo.co.uk - misidentification
+- henrifloor.nl - misidentification
+- http://skimap.info/ - maps, NZ placenames in PDF
+DK:
+!! -  http://akona.ngapuhitelevision.com, http://waiatarangatiratanga.ngapuhitelevision.com, 
+http://jazz.ngapuhitelevision.com, http://ngapuhitelevision.com, http://ngapuhiradio.com,
+http://powhiri.ngapuhitelevision.com, http://komisch.ngapuhitelevision.com
+- http://www.rennertweb.de - a photogallery page mentioning NZ placenames
+CA:
+- http://bcmarina.com AND http://bckayak.com - photos with Canadian placenames
+- http://www.myrasplace.net - pagse of photos, captions involving NZ placenames
+~ http://00.gs/Maniapoto;Uriwera;Moriori;Hivaoa;Kumulipo.htm - Maori-Polynesian comparative dictionary words listing
+- aguadilla.airport-authority.com - misidentification
+- https://articles.imperialtometric.com - misidentification
+- http://daandehn.com - no more than 1 sentence over multiple files. Appears to be photo captions of NZ placenames
 DE:
-http://etymologie.info/~e/n_/nz-___reg.html - placenames, not meaningful
-!! https://www.cartogiraffe.com/ - some genuine pages (Rarotongan), but one page is in Czech that had a single word misindentified as MRI
+- http://etymologie.info/~e/n_/nz-___reg.html - placenames, not meaningful
+!! https://www.cartogiraffe.com/ and https://www.cartogiraffe.com - some genuine pages (Rarotongan), but one page is in Czech that had a single word misindentified as MRI
 ~ http://svenkirsten.com/ - one page mentions "tiki" but the rest is in English. The other is an (English) caption of "Book of Tiki A Maori Maiden"
 - herocity - autotranslated
@@ -1045 +1074 @@
 ~ http://klaaskoehne.de/galleries/nzl-taranaki/index.html - mentions NZ mountain names
 - http://www.nierstrasz.org/deGrauwRegister.rtf - misdetected European (Dutch) names as MRI
-- https://afrikhepri.org/mi/ - autotranslated
+X https://afrikhepri.org/mi/ - autotranslated
 - https://www.tvteile.de - pure German pages, misdetected "Automatik" as a Maori language word
 - etoile-de-lune.net - 5 pages containing 1 sentence each but none with 2 sentences detected
-
+- https://www.you-fly.com - misdetection of German "Warum?" as MRI
+- http://vulkane.ch - misdetected pages on Hawaiian volcanoes.
+- http://www.stephe.de - photos from NZ captioned with NZ placenames
+- http://insecta.pro - misdetection
+- http://m.distanta.1km.net - NZ placenames. Lots of distances mentioning Waitangi. Nothing detected as containing more than 1 sentence.
+- https://ersatzteile-fachversand.de - German misdetected as Maori.
+- https://laskar02cinta.page.tl/Info.htm - seems like a junk site with a random sentence autotranslated into many different languages. So one sentence possibly in Maori, but may not make sense.
+- http://www.behlig.de - misdetection. Photos from Hawaii.
+!! http://www.udhr.de - Universal Declaration of Human Rights. (Also on a Bulgarian site). Multiple translations available.
 - ITALY:
   http://oipaz.net/IMG/GalleriaAotearoa/ - NZ photogallery with each photo captioned by placename
@@ -1062 +1099 @@
 - RUSSIA: https://www.gismeteo.lv - misidentification of an email address
 - JAPAN: http://yutaka.it-n.jp - many pages of scientific names of (plants?) which are often misdetected as MRI
-!! Ireland, ie: https://coggle.it
+!! - Ireland, ie: https://coggle.it
 - IRAN: https://www.dideo.ir/v/yt/d6cgya0ze-E - video title from MaoriTelevision website
-? - CZECH republic: https://www.fipojobs.com/new-zealand/jobs-work-p-1 - NZ job position title in MRI but rest in English
-- SPAIN: http://www.info-hoteles.com/nz/2/hotels_lake_rotoiti.asp - 2 uses of the same placename
+- CZECH republic:
+?  https://www.fipojobs.com/new-zealand/jobs-work-p-1 - NZ job position title in MRI but rest in English
+!! http://www.henryklahola.nazory.cz/094.Maori.htm and http://henryklahola.nazory.cz variant
+  http://about.ilikeyou.com - dating site. Misidentification.
+- SPAIN:
+!! https://www.uv.es/~pla/red.net/intmaori.html
+  https://www.reclamaciondevuelos.com - 2 occurrences of the word "kiwi"
+  http://www.info-hoteles.com/nz/2/hotels_lake_rotoiti.asp - 2 uses of the same placename
+  http://www.cruceros-princess.mx/princessMX/Oferta_Cruzeiros_Polinesia.html - Polynesian placenames
 - SINGAPORE: https://omg-solutions.com - autotranslated
 - TURKEY: https://www.elitedeluxe.com.tr/mi/yatak-odasi-takimlari - autotranslated
@@ -1122 +1166 @@
     http://mikestephens.co.uk/ - photo captions containing NZ placenames
     http://www.woolrych.org/nzholiday2004/ - photogallery captioned with NZ placenames
+    
 --------------
 
@@ -1194 +1239 @@
                 {domain: {$not: /\.nz/}},
                 {numPagesContainingMRI: {$gt: 0}},
-                {$or: [{geoLocationCountryCode: "AU"}, {urlContainsLangCodeInPath: false}]}            
+                {$or: [{geoLocationCountryCode: "AU"}, {urlContainsLangCodeInPath: false}]}
             ]}).count()
 
@@ -1253 +1298 @@
     { $sort : { count : -1} }
 ]);
+
+
+-----------------------
+Done: manually inspected 68/117 sites
+
+DEFINITELY:
++ http://anglicanhistory.org, 
++ http://www.unicode.org, [Universal declaration of Human Rights]
++ https://static-promote.weebly.com,
++ http://aclhokiangarocks.blogspot.com, [often English, but COMMUNITY]
+
+BIBLE/MOHAMMED/BAHAI TRANSLATIONS probably not auto translations:
++ http://bahaiprayers.net, [Dutch seems to be properly translated, not auto-translated, so maybe MRI too]
++ https://biblehub.com, 
++ http://www.muhammad.com, [possibly not autotranslated]
++ http://www.godrules.net, [possibly not autotranslated] 
++ http://m.biblepub.com,
++ http://www.krassotkin.ru, [probably real translations, as there are multiple Dutch translations from different sources provided]
++ http://www.gotquestions.org, [doesn't appear autotranslated]
+X https://ebible.org, [Hiri Motu, PNG language misdetected. Doesn't seem to have Maori]
+X https://www.bible.com, doesn't have Maori translation. Misdetected.
+X https://wol.jw.org, - doesn't have Maori translations. Instead, Rongo-rongo, Kiribati (Micronesian) etc misdetected
+X https://png.bible, [misdetected, Papua New Guinea]
+X http://www.precious-testimonies.com, http://precious-testimonies.com/JesusDidItTranslations/JesusDidItMaoriTranslation.htm may be autotranslated as the Dutch page looks more like Danish or some Scandinavian language and the French page is missing accented characters.
+
+PROBABLY:
+!! https://maorinews.com, 
+!! http://maaori.com, 
+!!+ http://kiaorahola.blogspot.com/
++ https://kjohnsonnz.blogspot.com, 
++ http://pumanawawhangara.blogspot.com, 
++ http://dannykahei.tripod.com,
++ http://burkekm001.tripod.com
++ http://tkkpipipaopao.blogspot.com, 
++ http://manateina.blogspot.com, 
+? tkkpipipaopao.blogspot.com? http://rangiwewehi.com, [English, but community]
+? https://www.terakau.org, [COMMUNITY, but English]
+? https://www.pipirikiapapatuanuku.org, [COMMUNITY?, in English, environment site]
+~ http://georgegi.tripod.com, 
+~ http://ngarangatahi.tripod.com, [1 page, image caption, Maori language warden position title with English sentence for appointment as warden]
+X http://fhr.kiwicelts.com, 
+X http://tkrow.tripod.com, [English, background of NZ place]
+X http://www.mkiwi.com,  - placenames
+X http://www.waimate.com, [English, NZ place]
+
+MAYBE, INSPECT:
+? https://www.natekore2018.com, [lots of English, but COMMUNITY, CULTURE]
++ http://tatai09.blogspot.com, 
++ http://www.twttoa.com, 
++ http://tuhua2010.blogspot.com, 
+X http://www.huapala.org, [misdetected, Hawaiian]
+X https://www.vaihaunui.net, [misdetected, Tahiti]
+X https://www.kaifineart.com, [art site by different artists. A Chinese and another (possibly Japanese) name were misdetected]
+X http://mahoraroom8.blogspot.com,  [NZ school, but main page mostly in English. No pages with > 1 senteced detected as MRI
++ http://piripi.blogspot.com, 
+X http://www.hiroa.pf,  [misdetected. Crawled content appears Polynesian not Maori]
+X http://korora.econ.yale.edu, [NZ place photo caption]
+X https://www.poehalisnami.ua, [mostly Cyrillic, with some NZ or Polynesian names misdetected]
+X http://hannas-reiseblog.blogspot.com - one page contained NZ placenames, another had a word misdetected
+
+
++ https://www.breaker.audio, [audio, with occasional English.]
+? https://livestream.com, [video and audio, seems in English, but maybe CULTURAL/COMMUNITY?]
+
+X https://docs.google.com, timetable with occasional Maori language word
++ https://drive.google.com, https://drive.google.com/file/d/1NwuzafjddaP8gxI7O_Zapts5bM7mrtwn/preview is an image of Maori number names. But other page on drive.google.com is a NZ certificate or ID (in English) of a person's position.
+http://ritusehji.blogspot.com - no page with more than 1 sentence detected. But short string of actual MRI content. Educator blog with pictures and English language content.
+
+
+PINTEREST
++ https://in.pinterest.com/pin/317363104978423418/
+  "karakia mo te moana - Google Search | Te Reo Maori Resources | Moana, Powerpoint tips, Google"
+? https://za.pinterest.com/pin/524669425310419500/
+  Maori Moko | Image | Moko Maori Tattoo & Portraits | TA MOKO | Maori tribe, Maori people, Maori art [COMMUNITY, CULTURE]
+[The other pinterest detected as numPagesContainingMRI > 0 was misdetected]
+
+https://nl.pinterest.com, 
+https://www.pinterest.jp, 
+https://www.pinterest.it, 
+https://www.pinterest.co.uk, 
+https://www.pinterest.ca, 
+https://za.pinterest.com, 
+https://www.pinterest.fr, 
+https://in.pinterest.com, 
+
+MORE BLOGSPOTS
+X http://word-dialect.blogspot.com,  [Indonesian, misdetected]
+~ http://atopeconlostopes.blogspot.com, [title on page appears to be in MRI, but content appears to be in English and South/Central American. Internationally focussed content.]
+X http://lianzaconference2012.blogspot.com, [NZ placename or institution]
+? http://mrshamiltonskoolkidz.blogspot.com, [te reo Maori related school activities. Described in English.]
+X http://capsuraotearoa.blogspot.com, [blog in French, photo captions contain NZ placenames]
+X http://blogdepasopor.blogspot.com, [blog in French, Rapa Nui/Easter Island related content, misdetected.]
+
+
+UNLIKELY
+?? http://naturalfatburner.net, http://naturalfatburner.net/NoNonsenseTed/fatloss-mao/ feels like it's autotranslated, an image of text appears, but the text is in MRI [advertising for some weight loss gimmick]
+
+
+BLACKLIST:
+X http://ww25.milfsplease.com, 
+X http://www.the-naked.com
+
+OTHER:
+X http://seapixonline.com, https://www.seapixonline.com, [photo captions of ships. Sometimes misdetected Japanese words as MRI.]
+X http://www.code-postal.com, https://www.code-postal.com, [not more than 1 sentence detected as in MRI]
+X https://www.dbnames.net, [Name database, lots misdetected]
+
+STILL TO DO LIST:
+
+X https://www.myadsclassified.com, [misdetected 3 short English sentences as MRI]
+X http://www.whoisthatr.com, [misdetected short English sentence as MRI]
+X https://www.oemsec.com, [autotranslated product site]
+X http://svenskadress.net, [linkfarm like site of related junk links, contained URLs misdetected as MRI]
+
+X https://www.webwiki.com, [contains URLs. URLs containing Aotearoa as substring detected as MRI. But no proper sentence content. ]
+X http://mikebonnice.com, [Hawaiian and Tahiti related content misdetected]
+X http://www.hudl.com, [misdetected short English sentence as MRI]
+X http://www.wikitree.com,  [misdetected short English sentence as MRI]
+X http://shuttersportnelson.photoshelter.com, [image captions of "Wairua Warrior"]
+
+X http://niken8media.logdown.com, [Poker website? Looks autotranslated or Lorem Ipsum type of meaningless sentences.]
+X https://www.podrozeady.com, Looks Polish or other East-European language. The NZ page https://www.podrozeady.com/NZ/4/ had placenames detected.
+
+X http://www.thesalmons.org, [detection and misdetection of author names of papers hosted]
+
+X http://linkvip.top, [.rar and media file links misdetected as MRI]
+
+
+X http://www.lunar-occultations.com, [NZ place names for astronomical phenomena]
+X http://shangrilapress.net, [NZ placenames]
+X http://malecek.com, [misdetection CD title]
+X https://www.blue-frontiers.com, [Tahitian, Reo Tahiti, misdetected as MRI]
+X http://www.whoisentry.com, [URL names, looked at several which were probably misdetected as MRI]
+X http://loquevendra318.com, [uses Google translate for auto-translation]
+
+
+?? http://www.forensicfashion.com, [historical information, useful for CULTURE? e.g. http://www.forensicfashion.com/1807MaoriChief.html]
+
+X http://www.eyecontactsite.com, [Lots of names. And a few short sentences or words possibly in comments.]
+X http://eartheum.com, [Rapa Nui, Easter Island related content. Misdetected]
+X http://www.steve-wheeler.co.uk, [Blogspot. Title of a single page is in Maori. "Aotearoa ... kei te aroha au ki a koe"]
+X https://chromium.googlesource.com, [some source code related to languages' two letter codes]
+
+X http://www.roadsmile.com, [Lots of misdetection based on word Kia.]
+?? https://www.knowatom.com, https://phet.colorado.edu [Similar looking science web sites for children. Uses auto-translation?]
+
+X https://www.indexmundi.com, [place names. Pages about Solomon Islands. Misdetection of placenames.]
+
+
+
+X http://wowwars.net, [Has a page on Kia Kaha meaning, but URL redirects to a different low quality site with bad formatting and adverts. ]
+?? https://www.hidroponia.org.mx, [Not sure if https://www.hidroponia.org.mx/index.php/idiomas/284-hydroponics-te-ahurea-wai-maori is autotranslated or not. Can't easily locate existence of Dutch or German translated pages. There's Tamil-Singapore, but no other Tamil. So maybe translations based on target buyer audience?]
+X http://www.v3whois.com, [URLs are misdetected as MRI]
+X http://rhymebrain.com, [appears to misdetected a short phrase of 2 words, Kai Kaia, besides phrase words from other languages]
+
+
+X SINGLE SENTENCE DETECTED (NO MORE AND NOT PAGE:)
+  http://frontrowphotos.com, 
+  http://www.pressreader.com, 
+  https://www.nccri.ie, 
+  http://takethatvacation.com, 
+  http://worldradiomap.com, 
+  http://www.namesdir.com, 
+
+  X http://www.frogsonline.com, [NZ hotels, placenames]
+  X http://www.geni.com, [Single sentence misdetection]
+  X http://wikiedit.org, [just a list of lots of words, possibly placenames. Some misdetected, e.g. Rapa Nui] 
+
+