Index: /other-projects/maori-lang-detection/MoreReading/mongodb.txt
===================================================================
--- /other-projects/maori-lang-detection/MoreReading/mongodb.txt	(revision 33806)
+++ /other-projects/maori-lang-detection/MoreReading/mongodb.txt	(revision 33807)
@@ -856,6 +856,89 @@
 - http://www.hzhinew.com/
 
+https://culturesconnection.com/manual-or-automatic-translation/
+Manual Or Automatic Translation?
+
+Automatic translation continues to improve day by day. However, it is still unable to reach perfect levels of accuracy and lacks a natural feel. Will it ever replace human translation?
 
 --------------
+Mr Bill Rogers' suggestions for beginnings of trying to sieve out the auto-translated sites:
+- skip .com. .co.<tld>. But .co.nz is also used for non-commercial sites or sites that nevertheless have high quality Maori language content.
+- change cut-off value of OpenNLP language prediction? But for sentences and overlapping sentences, we're not using the cut-off value, we're just checking the best predicted language regardless of confidence level for this.
+
+- Code for (a range of) loading of language options in auto-translated sites?
+
+====================
+
+# https://stackoverflow.com/questions/20175122/how-can-i-use-not-like-operator-in-mongodb
+
+Info on the sites with Maori language content that are either from NZ or have .nz domain (TLD):
+
+       db.getCollection('Websites').find({$and: [{numPagesContainingMRI: {$gt: 0}}, {$or:[{geoLocationCountryCode: "NZ"}, {domain: /.nz$/}]}]})
+       
+       db.getCollection('Websites').find({$and: [{numPagesContainingMRI: {$gt: 0}}, {$or:[{geoLocationCountryCode: "NZ"}, {domain: /.nz$/}]}]}).count()
+       183
+
+Inverse: the sites detected as containing at least 1 Maori language sentence that are NOT from NZ NOR have .nz domain ending (TLD):
+	 db.getCollection('Websites').find({$and: [{numPagesContainingMRI: {$gt: 0}}, {geoLocationCountryCode: {$ne: "NZ"}}, {domain: {$not: /.nz$/}}]}).count()
+	 685
+
+The above two figures correctly add up to a total of 868 sites, which is the number of sites detected as containing at least 1 sentence in MRI:
+	db.getCollection('Websites').find({numPagesContainingMRI: {$gt: 0}}).count()
+	868
+
+Without those with /mi in path:
+	db.getCollection('Websites').find({$and: [{numPagesContainingMRI: {$gt: 0}}, {geoLocationCountryCode: {$ne: "NZ"}}, {domain: {$not: /.nz$/}}, {urlContainsLangCodeInPath: false}]}).count()
+
+Now let's get a listing of all 685 sites to be manually inspected to determine whether they're auto-translated:
+
+/*
+db.Websites.aggregate([
+    {
+        $match: {
+            $and: [{numPagesContainingMRI: {$gt: 0}}, {geoLocationCountryCode: {$ne: "NZ"}}, {domain: {$not: /.nz$/}}, {urlContainsLangCodeInPath: false}]
+        }
+    },
+    { $unwind: "$geoLocationCountryCode" },
+    {
+        $group: {
+            _id: {$toLower: '$geoLocationCountryCode'},
+            count: { $sum: 1 },
+            domain: { $addToSet: '$domain' }
+        }
+    },
+    { $sort : { count : -1} }
+]);
+*/
+db.Websites.aggregate([
+    {
+        $match: {
+            $and: [{numPagesContainingMRI: {$gt: 0}}, {geoLocationCountryCode: {$ne: "NZ"}}, {domain: {$not: /.nz$/}}, {urlContainsLangCodeInPath: {$ne: true}}, ]
+        }
+    },
+    { $unwind: "$geoLocationCountryCode" },
+    {
+        $group: {
+            _id: {$toLower: '$geoLocationCountryCode'},
+            count: { $sum: 1 },
+            domain: { $addToSet: '$domain' }
+        }
+    },
+    { $sort : { count : -1} }
+]);
+
+* CN: Only 1/113 sites from CN stood out as being of interest: http://kiwi2china.com/
+     BUT: it's auto-translated (e.g. Dutch is clearly auto-translated), MRI not in default or any visible drop down list, and the domain changes once you view it in Dutch to https://nl.admission.nz/
+
+* FR: 35 sites from FR
+    http://blueheavenisland.com - French Polynesia
+    https://www.lexilogos.com/ -> takes me to NZ website MaoriDictionary.co.nz etc for translating words anyway
+    http://kihikihi.fr/ -> travel (blog?). Appears to be Hawaiian related and not Maori.
+!!  http://chantsdeluttes.free.fr/versionsinter/page%20maori.html -> Seems it may be a proper translation or composition, as Dutch and Flemish (and Groningense) versions are different songs by individual translators/composers
+    http://splaf.free.fr/pfurb.html - Tahiti, French Polynesian, ... island names
+    http://mi.fitnessrebates.com - Uses https://wordpress.org/plugins/weglot/ wordpress-compatible multilingual plugin, which ensures translated pages get indexed by google - exactly what we want to avoid
+* 
+
+--------------
+
 GETTING TABLE DATA OUT OF MONGO DB:
 
@@ -876,486 +959,2 @@
 
 ---------------------
-
-/* 1 */
-{
-    "_id" : "US",
-    "count" : 93.0,
-	-95.8,40.33
-}
-
-/* 2 */
-{
-    "_id" : "AU",
-    "count" : 7.0,
-	135.8,-25.33
-}
-
-/* 3 */
-{
-    "_id" : "CN",
-    "count" : 7.0,
- 100.8,
-      32.33
-}
-
-/* 4 */
-{
-    "_id" : "NZ",
-    "count" : 5.0,
-175.8,
-      -40.33
-}
-
-/* 5 */
-{
-    "_id" : "DE",
-    "count" : 5.0,
-10.8,
-      50.33
-}
-
-/* 6 */
-{
-    "_id" : "HK",
-    "count" : 5.0,
-114,
-      22.33
-}
-
-/* 7 */
-{
-    "_id" : "RU",
-    "count" : 4.0,
-38.4,
-      55.5
-}
-
-/* 8 */
-{
-    "_id" : "JP",
-    "count" : 3.0,
-      137.8,
-      36
-}
-
-/* 9 */
-{
-    "_id" : "GB",
-    "count" : 3.0,
--2,
-      53.33
-}
-
-/* 10 */
-{
-    "_id" : "CA",
-    "count" : 2.0,
-      -105.8,
-      55.33
-}
-
-/* 11 */
-{
-    "_id" : "FR",
-    "count" : 2.0,
-      3,
-      47.33
-}
-
-/* 12 */
-{
-    "_id" : "DK",
-    "count" : 2.0,
- 9.5,
-      55.33
-}
-
-/* 13 British Virgin Islands */
-{
-    "_id" : "VG",
-    "count" : 2.0,
- -64.8,
-      18.35
-}
-
-/* 14 Ukraine */
-{
-    "_id" : "UA",
-    "count" : 1.0,
-      31.5,
-      48.5
-}
-
-/* 15 */
-{
-    "_id" : "CZ",
-    "count" : 1.0,
-      16.2,
-      49.7
-}
-
-/* 16 Switzerland */
-{
-    "_id" : "CH",
-    "count" : 1.0,
-      8.5,
-      47
-}
-
-/* 17 Zuid-Afrika */
-{
-    "_id" : "ZA",
-    "count" : 1.0,
-      24.2,
-      -30.7
-}
-
-/* 18 */
-{
-    "_id" : "NL",
-    "count" : 1.0,
-5.8,
-      52.33
-}
-
-/* 19 */
-{
-    "_id" : "KR",
-    "count" : 1.0,
-      127.8,
-      36.8
-}
-
-
-/** http://geojson.tools/
-
-
-{
-  "type": "MultiPoint",
-  "coordinates": [
-    [
-      -95.8,
-      40.33
-    ],
-    [
-      135.8,
-      -25.33
-    ],
-    [
-      100.8,
-      32.33
-    ],
-    [
-      175.8,
-      -40.33
-    ],
-    [
-      10.8,
-      50.33
-    ],
-    [
-      10.8,
-      50.33
-    ],
-    [
-      114,
-      22.33
-    ],
-    [
-      38.4,
-      55.5
-    ],
-    [
-      -2,
-      53.33
-    ],
-    [
-      137.8,
-      36
-    ],
-    [
-      -105.8,
-      55.33
-    ],
-    [
-      3,
-      47.33
-    ],
-    [
-      9.5,
-      55.33
-    ],
-    [
-      -64.8,
-      18.35
-    ],
-    [
-      31.5,
-      48.5
-    ],
-    [
-      16.2,
-      49.7
-    ],
-    [
-      8.5,
-      47
-    ],
-    [
-      24.2,
-      -30.7
-    ],
-    [
-      5.8,
-      52.33
-    ],
-    [
-      127.8,
-      36.8
-    ]
-  ]
-}
-
-*/
-
-/* 1 */
-{
-    "_id" : "US",
-    "count" : 93.0,
-	-95.8,40.33
-}
-
-/* 2 */
-{
-    "_id" : "AU",
-    "count" : 7.0,
-	135.8,-25.33
-}
-
-/* 3 */
-{
-    "_id" : "CN",
-    "count" : 7.0,
- 100.8,
-      32.33
-}
-
-/* 4 */
-{
-    "_id" : "NZ",
-    "count" : 5.0,
-175.8,
-      -40.33
-}
-
-/* 5 */
-{
-    "_id" : "DE",
-    "count" : 5.0,
-10.8,
-      50.33
-}
-
-/* 6 */
-{
-    "_id" : "HK",
-    "count" : 5.0,
-114,
-      22.33
-}
-
-/* 7 */
-{
-    "_id" : "RU",
-    "count" : 4.0,
-38.4,
-      55.5
-}
-
-/* 8 */
-{
-    "_id" : "JP",
-    "count" : 3.0,
-      137.8,
-      36
-}
-
-/* 9 */
-{
-    "_id" : "GB",
-    "count" : 3.0,
--2,
-      53.33
-}
-
-/* 10 */
-{
-    "_id" : "CA",
-    "count" : 2.0,
-      -105.8,
-      55.33
-}
-
-/* 11 */
-{
-    "_id" : "FR",
-    "count" : 2.0,
-      3,
-      47.33
-}
-
-/* 12 */
-{
-    "_id" : "DK",
-    "count" : 2.0,
- 9.5,
-      55.33
-}
-
-/* 13 British Virgin Islands */
-{
-    "_id" : "VG",
-    "count" : 2.0,
- -64.8,
-      18.35
-}
-
-/* 14 Ukraine */
-{
-    "_id" : "UA",
-    "count" : 1.0,
-      31.5,
-      48.5
-}
-
-/* 15 */
-{
-    "_id" : "CZ",
-    "count" : 1.0,
-      16.2,
-      49.7
-}
-
-/* 16 Switzerland */
-{
-    "_id" : "CH",
-    "count" : 1.0,
-      8.5,
-      47
-}
-
-/* 17 Zuid-Afrika */
-{
-    "_id" : "ZA",
-    "count" : 1.0,
-      24.2,
-      -30.7
-}
-
-/* 18 */
-{
-    "_id" : "NL",
-    "count" : 1.0,
-5.8,
-      52.33
-}
-
-/* 19 */
-{
-    "_id" : "KR",
-    "count" : 1.0,
-      127.8,
-      36.8
-}
-
-
-/** http://geojson.tools/
-
-
-{
-  "type": "MultiPoint",
-  "coordinates": [
-    [
-      -95.8,
-      40.33
-    ],
-    [
-      135.8,
-      -25.33
-    ],
-    [
-      100.8,
-      32.33
-    ],
-    [
-      175.8,
-      -40.33
-    ],
-    [
-      10.8,
-      50.33
-    ],
-    [
-      10.8,
-      50.33
-    ],
-    [
-      114,
-      22.33
-    ],
-    [
-      38.4,
-      55.5
-    ],
-    [
-      -2,
-      53.33
-    ],
-    [
-      137.8,
-      36
-    ],
-    [
-      -105.8,
-      55.33
-    ],
-    [
-      3,
-      47.33
-    ],
-    [
-      9.5,
-      55.33
-    ],
-    [
-      -64.8,
-      18.35
-    ],
-    [
-      31.5,
-      48.5
-    ],
-    [
-      16.2,
-      49.7
-    ],
-    [
-      8.5,
-      47
-    ],
-    [
-      24.2,
-      -30.7
-    ],
-    [
-      5.8,
-      52.33
-    ],
-    [
-      127.8,
-      36.8
-    ]
-  ]
-}
-
-*/
