Changeset 33838

Timestamp:

2020-01-16T17:56:50+13:00 (4 years ago)

Author:

ak19

Message:

Updated after checking non-NZ and non-nz TLD sites with mi in URL path

File:

• other-projects/maori-lang-detection/MoreReading/mongodb.txt (modified) (1 diff)

other-projects/maori-lang-detection/MoreReading/mongodb.txt

@@ -1500 +1500 @@
 
 
+------------------------------
+
+Need to inspect all those URLs with mi in URL path (mi.* or */mi) that are not sites with nz TLD or originating in NZ:
+
+db.getCollection('Websites').find({$and: [{numPagesContainingMRI: {$gt: 0}},{urlContainsLangCodeInPath: true}, {domain: {$not: /.nz$/}}, {geoLocationCountryCode: {$ne: "NZ"}}]}).count()
+472
+
+(vs:
+db.getCollection('Websites').find({$and: [{numPagesInMRI: {$gt: 0}},{urlContainsLangCodeInPath: true}, {domain: {$not: /.nz$/}}, {geoLocationCountryCode: {$ne: "NZ"}}]}).count()
+209)
+
+
+db.Websites.aggregate([
+    {
+            $match: {
+                $and: [{numPagesContainingMRI: {$gt: 0}},{urlContainsLangCodeInPath: true}, {domain: {$not: /.nz$/}}, {geoLocationCountryCode: {$ne: "NZ"}}]
+            }
+        },
+    {$group: {_id: "$geoLocationCountryCode", count: {$sum: 1}, domain: { $addToSet: '$domain' }}},
+        { $sort : { count : -1} }
+])
+
+
+Of interest or possible interest:
+US: 
+!! http://indigenousblogs.com [15/18 blogs work]
+X https://biblia.gospelprime.com.br - misdetection (containsMRI)
+X ?https://follow3rs.com - seems dodgy and possibly auto-translated. Can't spell account, misspelled as accout
+!! https://mi.m.wikipedia.org, https://mi.wikipedia.org
+X https://usahello.org - autotranslated
+X http://church-of-christ.org, http://www.church-of-christ.org - I think autotranslated, because "HET kerken van Christus" at https://church-of-christ.org/nl/ i.p.v. meervoud
+X https://www.livehoster.com
+X http://www.americasportsfloor.com, - product store. Misdetected
+!! http://csunplugged.org, https://www.csunplugged.org - University of Canterbury NZ and site only available in EN, MI, DE, ES, CN
+X https://mi.lawyers.cafe - autotranslated
+    X https://mi.centr-zashity.ru - same as lawyers.cafe above: autotranslated
+! https://policies.oclc.org - not completely translated. Copyright page, privacy statement and cookie statement pages appear to be in Maori. Not sure if autotranslated since other pages aren't available in MI. Dutch equivalent pages seem human translated.
+X http://jobdescriptionsample.org - autotranslated
+X http://mi.broadcastbeat.com - autotranslated product site
+X http://www.samewe.net - autotranslated product site
+X https://mi.kidspicturedictionary.com - autotranslated, but MAY BE USEFUL
+X https://www.rikoooo.com - autotranslated
+
+CN: -
+
+FR:
+? https://mi.phcoker.com - product site "Shangke Chemical Rapu + 86 (1812) 4514114 [email protected]"
+X http://www.gpedia.com - dodgy copy of wikipedia, see http://www.gpedia.com/nl/gpedia/Hoofdpagina
+
+NL:
+X http://www.martinvrijland.nl - wordpress, autotranslated
+
+CA:
+X https://www.wikiplanet.click (seems like a dodgy copy of wikipedia)
+X cloudsfeed.com - wordpress admin page
+
+
+db.getCollection('Webpages').find({$and: [{isMRI: true}, {URL: /indigenousblogs\.com/}]})
+=> http://indigenousblogs.com/mi/