Changeset 33872 for other-projects/maori-lang-detection/mongodb-data/6table_nonProductSites1_manualShortlist.json

Timestamp:

2020-01-24T21:44:04+13:00 (4 years ago)

Author:

ak19

Message:

1. Added the file containing the 255 random NZ page URLs to sample. 2. Minor updates to 2 existing counts files. 3. Recorded isMRI aggregate command used for selecting NZ domains to sample from - for NZ sites did not use containsMRI to generate samples.

File:

• other-projects/maori-lang-detection/mongodb-data/6table_nonProductSites1_manualShortlist.json (modified) (2 diffs)

other-projects/maori-lang-detection/mongodb-data/6table_nonProductSites1_manualShortlist.json

@@ -1 +1 @@
 /*
+
+db.Websites.aggregate([
+    {
+        $match: {
+            $and: [
+                {numPagesInMRI: {$gt: 0}},
+                {$or: [{geoLocationCountryCode:"NZ"},{domain: /\.nz/}]}
+            ]
+        }
+    },
+    { $unwind: "$geoLocationCountryCode" },
+    {
+        $group: {
+            _id: "nz",
+            count: { $sum: 1 },
+            domain: { $addToSet: '$domain' },
+            numPagesInMRICount: { $sum: '$numPagesInMRI' },
+            numPagesContainingMRICount: { $sum: '$numPagesContainingMRI' }
+        }
+    },
+    { $sort : { count : -1} }
+]);
+
 For sites originating in NZ or with nz TLD, none of the URLs are manually inspected and all URLs are accepted.
 
@@ -118 +141 @@
 
 
-OR is this better:
+OR is this better (only numPagesINMRI):
 
 db.Websites.aggregate([