Changeset 33847 for other-projects/maori-lang-detection/MoreReading

Timestamp:

2020-01-17T19:32:16+13:00 (4 years ago)

Author:

ak19

Message:

indigenousblogs.com did have one page actually in Maori (an XML feed). So adding 1 to the table of counts for US sites with mi in the URL path that contained actual MRI.

File:

• other-projects/maori-lang-detection/MoreReading/mongodb.txt (modified) (4 diffs)

other-projects/maori-lang-detection/MoreReading/mongodb.txt

@@ -1488 +1488 @@
 TIDIED:
 NZ: 176
-US: 25+3 from US with mi in URL path = 28
+US: 25+4 from US with mi in URL path = 29
 AU: 3
 DE: 2
@@ -1497 +1497 @@
 FR: 1
 IE: 1
-TOTAL: 213+3 from US with mi in URL path = 216
+TOTAL: 213+4 from US with mi in URL path = 217
 
 
@@ -1525 +1525 @@
 Of interest or possible interest:
 US: 
-!! http://indigenousblogs.com [15/18 blogs work]
+!! http://indigenousblogs.com [15/18 blogs work] - has one page in Maori (http://indigenousblogs.com/feeds/mi.xml)
 X https://biblia.gospelprime.com.br - misdetection (containsMRI)
 X ?https://follow3rs.com - seems dodgy and possibly auto-translated. Can't spell account, misspelled as accout
@@ -1559 +1559 @@
 db.getCollection('Webpages').find({$and: [{isMRI: true}, {URL: /indigenousblogs\.com/}]})
 => http://indigenousblogs.com/mi/
+
+--------------------------
+
+
+db.Websites.aggregate([
+    {
+        $match: {
+            $and: [
+                {geoLocationCountryCode: {$ne: "NZ"}},
+                {domain: {$not: /\.nz/}},
+                {numPagesContainingMRI: {$gt: 0}},
+                {$or: [{geoLocationCountryCode: "AU"}, {urlContainsLangCodeInPath: false}]}            
+            ]
+        }
+    },
+    { $unwind: "$geoLocationCountryCode" },
+    {
+        $group: {
+            _id: {$toLower: '$geoLocationCountryCode'},
+            count: { $sum: 1 },
+            domain: { $addToSet: '$domain' },
+            numPagesInMRI: { $addToSet: '$numPagesInMRI' },
+            numPagesContainingMRI: { $addToSet: '$numPagesContainingMRI' },
+            numPagesInMRICount: { $sum: '$numPagesInMRI' },
+            numPagesContainingMRICount: { $sum: '$numPagesContainingMRI' }
+        }
+    },
+    { $sort : { count : -1} }
+]);
+
+
+To convert json to csv
+In gedit replace
+\/\*\s*\d+\s*\*\/ => ,