source: other-projects/maori-lang-detection/MoreReading/mongodb.txt@ 33800

Last change on this file since 33800 was 33800, checked in by ak19, 4 years ago

Removed an adult site from crawled contents and added its url to blacklist conf file (for if ever anyone crawls our MRI set of common crawl sites again)
File size: 30.1 KB
Line 
1MongoDB
2Installation:
3 https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
4 https://docs.mongodb.com/manual/administration/install-on-linux/
5 https://hevodata.com/blog/install-mongodb-on-ubuntu/
6 https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04
7 CENTOS (Analytics): https://tecadmin.net/install-mongodb-on-centos/
8 FROM SOURCE: https://github.com/mongodb/mongo/wiki/Build-Mongodb-From-Source
9GUI:
10 https://robomongo.org/
11 Robomongo is Robo 3T now
12
13https://www.tutorialspoint.com/mongodb/mongodb_java.htm
14JAR FILE:
15 http://central.maven.org/maven2/org/mongodb/mongo-java-driver/
16 https://mongodb.github.io/mongo-java-driver/
17
18
19
20https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
21http://www.programmersought.com/article/6500308940/
22
23 52 sudo apt-get install mongodb-clients
24 53 mongo 'mongodb://mongodb.cms.waikato.ac.nz:27017' -u anupama -p
25
26Failed with
27 Error: HostAndPort: host is empty at src/mongo/shell/mongo.js:148
28 exception: connect failed
29
30This is due to a version incompatibility between Client and mongodb Server.
31The solution is to follow instructions at http://www.programmersought.com/article/6500308940/
32and then https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
33as below:
34
35 54 sudo apt-get purge mongodb-clients
36 55 sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
37 56 echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
38 57 sudo apt-get update
39 58 sudo apt-get install mongodb-clients
40 59 mongo 'mongodb://mongodb.cms.waikato.ac.nz:27017' -u anupama -p
41(still doesn't work)
42 60 sudo apt-get install -y mongodb-org
43The above ensures an up to date mongo client but installs the mongodb server too. Maybe this is the only step that is needed to install up-to-date mongo client and mongodb server?
44 72 sudo service mongod status
45
46 103 sudo service mongod start
47"mongod" stands for mongo-daemon. This runs the mongo db server listening for client connections
48 104 sudo service mongod status
49 88 sudo service mongod stop
50
51
52DETAILS:
53
54wharariki:[879]/Scratch/ak19/gs3-extensions/maori-lang-detection>mongo 'mongodb://mongodb.cms.waikato.ac.nz:27017' -u anupama -p
55
56didn't work with the pwd. Failed with:
57
58 MongoDB shell version: 2.6.10
59 Enter password:
60 connecting to: mongodb://mongodb.cms.waikato.ac.nz:27017
61 2019-11-04T20:02:47.970+1300 Assertion: 13110:HostAndPort: host is empty
62 2019-11-04T20:02:47.970+1300 0x6b75c9 0x659e9f 0x636f69 0x4fa55c 0x501249 0x4fa7f1 0x6006fd 0x5eb869 0x7f7bfbd47d76 0x1f3c10d06362
63 mongo(_ZN5mongo15printStackTraceERSo+0x39) [0x6b75c9]
64 mongo(_ZN5mongo10logContextEPKc+0x21f) [0x659e9f]
65 mongo(_ZN5mongo11msgassertedEiPKc+0xd9) [0x636f69]
66 mongo(_ZN5mongo16ConnectionString12_fillServersENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x50c) [0x4fa55c]
67 mongo(_ZN5mongo16ConnectionStringC1ENS0_14ConnectionTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x99) [0x501249]
68 mongo(_ZN5mongo16ConnectionString5parseERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERS6_+0x201) [0x4fa7f1]
69 mongo(_ZN5mongo17mongoConsExternalEPNS_7V8ScopeERKN2v89ArgumentsE+0x11d) [0x6006fd]
70 mongo(_ZN5mongo7V8Scope10v8CallbackERKN2v89ArgumentsE+0xa9) [0x5eb869]
71 /usr/lib/libv8.so.3.14.5(+0x99d76) [0x7f7bfbd47d76]
72 [0x1f3c10d06362]
73 2019-11-04T20:02:47.971+1300 Error: HostAndPort: host is empty at src/mongo/shell/mongo.js:148
74 exception: connect failed
75
76
77This is due to a version incompatibility between Client and mongodb Server.
78Can find client version above. (2.6.10)
79Server version can be found by running the mongo client shell. Doing so without loading a db:
80
81
82 wharariki:[880]/Scratch/ak19/gs3-extensions/maori-lang-detection>mongo --shell -nodb
83 MongoDB shell version: 2.6.10 <<<<<<<<<-------------------<<<< MONGO CLIENT VERSION
84 type "help" for help
85 > help
86 db.help() help on db methods
87 db.mycoll.help() help on collection methods
88 sh.help() sharding helpers
89 rs.help() replica set helpers
90 help admin administrative help
91 help connect connecting to a db help
92 help keys key shortcuts
93 help misc misc things to know
94 help mr mapreduce
95
96 show dbs show database names
97 show collections show collections in current database
98 show users show users in current database
99 show profile show most recent system.profile entries with time >= 1ms
100 show logs show the accessible logger names
101 show log [name] prints out the last segment of log in memory, 'global' is default
102 use <db_name> set current database
103 db.foo.find() list objects in collection foo
104 db.foo.find( { a : 1 } ) list objects in foo where a == 1
105 it result of the last line evaluated; use to further iterate
106 DBQuery.shellBatchSize = x set default number of items to display on shell
107 exit quit the mongo shell
108
109 > help connect
110
111 Normally one specifies the server on the mongo shell command line. Run mongo --help to see those options.
112 Additional connections may be opened:
113
114 var x = new Mongo('host[:port]');
115 var mydb = x.getDB('mydb');
116 or
117 var mydb = connect('host[:port]/mydb');
118
119 Note: the REPL prompt only auto-reports getLastError() for the shell command line connection.
120
121 Getting help on connect options:
122
123 > var x = new Mongo('mongodb.cms.waikato.ac.nz:27017');
124 > var mydb = x.getDB('anupama');
125
126 > mydb.connect.help()
127 DBCollection help
128 db.connect.find().help() - show DBCursor help
129 db.connect.count()
130 db.connect.copyTo(newColl) - duplicates collection by copying all documents to newColl; no indexes are copied.
131 db.connect.convertToCapped(maxBytes) - calls {convertToCapped:'connect', size:maxBytes}} command
132 db.connect.dataSize()
133 db.connect.distinct( key ) - e.g. db.connect.distinct( 'x' )
134 db.connect.drop() drop the collection
135 db.connect.dropIndex(index) - e.g. db.connect.dropIndex( "indexName" ) or db.connect.dropIndex( { "indexKey" : 1 } )
136 db.connect.dropIndexes()
137 db.connect.ensureIndex(keypattern[,options]) - options is an object with these possible fields: name, unique, dropDups
138 db.connect.reIndex()
139 db.connect.find([query],[fields]) - query is an optional query filter. fields is optional set of fields to return.
140 e.g. db.connect.find( {x:77} , {name:1, x:1} )
141 db.connect.find(...).count()
142 db.connect.find(...).limit(n)
143 db.connect.find(...).skip(n)
144 db.connect.find(...).sort(...)
145 db.connect.findOne([query])
146 db.connect.findAndModify( { update : ... , remove : bool [, query: {}, sort: {}, 'new': false] } )
147 db.connect.getDB() get DB object associated with collection
148 db.connect.getPlanCache() get query plan cache associated with collection
149 db.connect.getIndexes()
150 db.connect.group( { key : ..., initial: ..., reduce : ...[, cond: ...] } )
151 db.connect.insert(obj)
152 db.connect.mapReduce( mapFunction , reduceFunction , <optional params> )
153 db.connect.aggregate( [pipeline], <optional params> ) - performs an aggregation on a collection; returns a cursor
154 db.connect.remove(query)
155 db.connect.renameCollection( newName , <dropTarget> ) renames the collection.
156 db.connect.runCommand( name , <options> ) runs a db command with the given name where the first param is the collection name
157 db.connect.save(obj)
158 db.connect.stats()
159 db.connect.storageSize() - includes free space allocated to this collection
160 db.connect.totalIndexSize() - size in bytes of all the indexes
161 db.connect.totalSize() - storage allocated for all data and indexes
162 db.connect.update(query, object[, upsert_bool, multi_bool]) - instead of two flags, you can pass an object with fields: upsert, multi
163 db.connect.validate( <full> ) - SLOW
164 db.connect.getShardVersion() - only for use with sharding
165 db.connect.getShardDistribution() - prints statistics about data distribution in the cluster
166 db.connect.getSplitKeysForChunks( <maxChunkSize> ) - calculates split points over all chunks and returns splitter function
167 db.connect.getWriteConcern() - returns the write concern used for any operations on this collection, inherited from server/db if set
168 db.connect.setWriteConcern( <write concern doc> ) - sets the write concern for writes to the collection
169 db.connect.unsetWriteConcern( <write concern doc> ) - unsets the write concern for writes to the collection
170 > mydb.version()
171 4.0.13 <<<<<<<<<-------------------<<<< MONGODB SERVER VERSION
172
173(Check Mongo server version: https://stackoverflow.com/questions/38160412/how-to-find-the-exact-version-of-installed-mongodb)
174
175Finally we now know the mongodb server version: 4.0.13
176This version doesn't work with our mongo client (shell) version of 2.6.10.
177
178
179DETAILS OF INSTALLING MONGO-CLIENT AND UPDATING IT, AND INSTALLING MONGODB SERVER:
180
181
182 54 sudo apt-get purge mongodb-clients
183 55 sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
184 56 echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
185 57 sudo apt-get update
186 58 sudo apt-get install mongodb-clients
187 59 mongo 'mongodb://mongodb.cms.waikato.ac.nz:27017' -u anupama -p
188 60 sudo apt-get install -y mongodb-org
189 61 mongo 'mongodb://mongodb.cms.waikato.ac.nz:27017' -u anupama -p
190 62 sudo service apache2 status
191 63 sudo service sshd status
192 64 sudo service mongodb status
193 65 sudo service mongo status
194 66 mongod
195 67 mongod --help
196 68 mongod --help | less
197 69 mongod -f /etc/mongod.conf
198 70 sudo mongod -f /etc/mongod.conf
199 71 less /etc/mongod.conf
200 72 sudo service mongod status
201 73 sudo service mongod start
202 74 sudo service mongod status
203 75 ls -l /var/log/mongodb/mongod.log
204 76 sudo rm /var/log/mongodb/mongod.log
205 77 sudo service mongod status
206 78 sudo service mongod start
207 79 sudo service mongod status
208 80 sudo service mongod stop
209 81 ps auxww | grep mongo
210 82 sudo service mongod start
211 83 sudo service mongod status
212 84 ps auxww | grep mongo
213 85 sudo dmsg
214 86 sudo dmesg
215 87 sudo service mongod status
216 88 sudo service mongod stop
217 89 sudo service mongod start
218 90 sudo dmesg
219 91 sudo less /var/log/mongodb/mongod.log
220 92 ls /var/lib/
221 93 ls -ld /var/lib/
222 94 ls -l /var/log/mongodb/mongod.log
223 95 ls -ld /var/lib/
224 96 groups mongodb
225 97 less /etc/mongod.conf
226 98 sudo less /var/log/mongodb/mongod.log
227 99 less /etc/mongod.conf
228 100 ls -l /var/lib/mongodb/
229 101 sudo chown -R mongodb /var/lib/mongodb/
230 102 sudo chgrp -R mongodb /var/lib/mongodb/
231 103 sudo service mongod start
232 104 sudo service mongod status
233 105 history
234
235
236
237MONGO DB ROBO 3T
2381. Download "Double Pack" from https://robomongo.org/
2392. Untar its contents. Then untar the tarball in that.
2403. Run:
241 wharariki:[110]~/Downloads/robo3t-1.3.1-linux-x86_64-7419c406>./bin/robo3t
242
243===================
244On analytics, vagrant node1, we've installed the mongodb server and client.
245We're able to successfully create collections on here.
246
247
248vagrant@node1:~$ mongo
249MongoDB shell version v4.2.1
250connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
251Implicit session: session { "id" : UUID("87bb585c-4685-47f6-bf89-a93801daeb2d") }
252MongoDB server version: 4.2.1
253Server has startup warnings:
2542019-11-04T07:48:14.197+0000 I STORAGE [initandlisten]
2552019-11-04T07:48:14.198+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2562019-11-04T07:48:14.198+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2572019-11-04T07:48:14.624+0000 I CONTROL [initandlisten]
2582019-11-04T07:48:14.624+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2592019-11-04T07:48:14.624+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2602019-11-04T07:48:14.624+0000 I CONTROL [initandlisten]
261---
262Enable MongoDB's free cloud-based monitoring service, which will then receive and display
263metrics about your deployment (disk utilization, CPU, operation statistics, etc).
264
265The monitoring data will be available on a MongoDB website with a unique URL accessible to you
266and anyone you share the URL with. MongoDB may use this information to make product
267improvements and to suggest MongoDB products and deployment options to you.
268
269To enable free monitoring, run the following command: db.enableFreeMonitoring()
270To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
271---
272
273> show dbs
274admin 0.000GB
275config 0.000GB
276local 0.000GB
277> use db ateacrawldata
2782019-11-05T05:24:20.155+0000 E QUERY [js] Error: [db ateacrawldata] is not a valid database name :
279Mongo.prototype.getDB@src/mongo/shell/mongo.js:51:12
280getDatabase@src/mongo/shell/session.js:913:28
281DB.prototype.getSiblingDB@src/mongo/shell/db.js:22:12
282shellHelper.use@src/mongo/shell/utils.js:803:10
283shellHelper@src/mongo/shell/utils.js:790:15
284@(shellhelp2):1:1
285> db.createCollection('webpages');
286{ "ok" : 1 }
287> db.webpages.drop();
288... ^C
289
290> db.webpages.drop();
291true
292> use ateacrawldata
293switched to db ateacrawldata
294> db.createCollection('webpages');
295{ "ok" : 1 }
296> show collections
297webpages
298> db.createCollection('websites');
299{ "ok" : 1 }
300>
301
302------------------------
303
304Ask Clint to rename "anupama" database to "ateacrawldata" database following the instructions at:
305 https://stackoverflow.com/questions/9201832/how-do-you-rename-a-mongodb-database
306I don't have permissions to do this.
307Nor do I have permissions to create Mongo collections within a new database that I create, like ateacrawldata.
308I only seem to have rights to the "anupama" database.
309
310
311
312-----------------------
313Vagrant virtual machine Node1 has the mongodb installed.
314
315After doing "vagrant up" on node1 to start node1:
316
317 [anupama@analytics vagrant-hadoop-hive-spark]$ vagrant ssh
318 vagrant@node1:~$ mongo
319 MongoDB shell version v4.2.1
320 connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
321 2019-11-13T09:22:46.996+0000 E QUERY [js] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused :
322 connect@src/mongo/shell/mongo.js:341:17
323 @(connect):2:6
324 2019-11-13T09:22:46.999+0000 F - [main] exception: connect failed
325 2019-11-13T09:22:46.999+0000 E - [main] exiting with code 1
326 vagrant@node1:~$ sudo service mongod status
327 ● mongod.service - MongoDB Database Server
328 Loaded: loaded (/lib/systemd/system/mongod.service; disabled; vendor preset: enabled)
329 Active: inactive (dead)
330 Docs: https://docs.mongodb.org/manual
331 vagrant@node1:~$ sudo service mongod start
332 vagrant@node1:~$ sudo service mongod status
333 ● mongod.service - MongoDB Database Server
334 Loaded: loaded (/lib/systemd/system/mongod.service; disabled; vendor preset: enabled)
335 Active: active (running) since Wed 2019-11-13 09:24:07 UTC; 2s ago
336 Docs: https://docs.mongodb.org/manual
337 Main PID: 4383 (mongod)
338 Tasks: 32
339 Memory: 199.3M
340 CPU: 754ms
341 CGroup: /system.slice/mongod.service
342 └─4383 /usr/bin/mongod --config /etc/mongod.conf
343
344 Nov 13 09:24:07 node1 systemd[1]: Started MongoDB Database Server.
345 vagrant@node1:~$
346
347
348So now mongodb is running on node1 on localhost:27017.
349
350Next, in another x-term connected to analytics' node1 Vagrant VM, port forward node1's localhost:27017 to analytics' localhost:27017:
351 vagrant ssh -- -L 27017:localhost:27017
352
353
354
355Finally, in another x-term, port-forward from analytics:27017 to current machine's 27017:
356 ssh -L 27017:localhost:27017 analytics
357
358
359Now can connect Robo-3T running on current machine to localhost:27017.
360
361Then in a new x-term, can use the client mongo shell to connect (by default to localhost:27017):
362
363 wharariki:[122]/Scratch/ak19/GS309>mongo --shell
364 MongoDB shell version v4.0.13
365 connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
366 ...
367 > show dbs
368 admin 0.000GB
369 ateacrawldata 1.532GB
370 config 0.000GB
371 local 0.000GB
372 > use ateacrawldata
373
374 > show collections
375 Webpages
376 Websites
377 oldwebpages
378 oldwebsites
379-------------------
380
381Country code to geolocation CSV file found by Dr Bainbridge:
382https://developers.google.com/public-data/docs/canonical/countries_csv
383
384Import into mongodb with:
385https://stackoverflow.com/questions/4686500/how-to-use-mongoimport-to-import-csv
386
387
388
389NOTE: mongoimport is a commandline utility and not a command to be run from the mongo shell. See https://jira.mongodb.org/browse/DOCS-11072
390This means, in an x-term, DON'T RUN MONGO SHELL/client first. Instead, directly from x-term, run the following to import the countrycodes.csv file:
391
392
393 mongoimport -d ateacrawldata -c countrylocations --type csv --file /Scratch/ak19/maori-lang-detection/MoreReading/countrycodes.csv --headerline
394
395
396-------------------------
397
398MONGODB QUERIES:
399
400db.getCollection('webpages').find({"isMRI": true, "singleSentences.langCode": "mri"})
401db.getCollection('webpages').find({"singleSentences": { $elemMatch: {"langCode":"mri"} } }, {"singleSentences.$": "mri"})
402db.getCollection('Webpages').find({"isMRI": true, "singleSentences": { $elemMatch: {"langCode":"eng"} } }, {"singleSentences.$": "eng"}) [single English lang sentence]
403db.getCollection('Webpages').find({"containsMRI": true, "singleSentences": { $elemMatch: {"langCode":"mri"} } }, {"singleSentences.$": "mri"}) [gets 1st sentence of docs which have sentences containing MRI]
404
405
406READING
407
408mongodb java convert class
409https://www.quora.com/What-are-the-ways-of-converting-a-Java-object-to-a-MongoDB-document-and-vice-versa
410https://stackoverflow.com/questions/39320825/pojo-to-org-bson-document-and-vice-versa
411X https://mongodb.github.io/morphia/
412https://stackoverflow.com/questions/10170506/inserting-java-object-to-mongodb-collection-using-java
413X https://www.google.com/search?q=morphia+example&oq=morphia+example&aqs=chrome.0.0l6.4223j0j9&sourceid=chrome&ie=UTF-8
414https://www.baeldung.com/mongodb-morphia
415X https://web.archive.org/web/20171117121335/http://mongodb.github.io/morphia/1.3/getting-started/
416=> https://morphia.dev/1.4/getting-started/quick-tour/
417https://github.com/MorphiaOrg/morphia/tree/master/docs/reference
418
419
420mongodb querying
421https://docs.mongodb.com/manual/tutorial/query-embedded-documents/
422https://docs.mongodb.com/manual/tutorial/query-arrays/
423https://www.google.com/search?q=mongodb+find+subdocument&oq=mongodb+find+&aqs=chrome.0.69i59j69i57j0l4.7607j1j8&sourceid=chrome&ie=UTF-8
424https://stackoverflow.com/questions/25586901/how-to-find-document-and-single-subdocument-matching-given-criterias-in-mongodb
425https://stackoverflow.com/questions/21113543/mongodb-get-subdocument
426https://stackoverflow.com/questions/36948856/find-subdocuments-in-mongo
427https://docs.mongodb.com/v3.0/reference/operator/projection/positional/#proj._S_
428https://www.google.com/search?q=mongodb+query+tutorial&oq=mongodb+query+tutorial&aqs=chrome..69i57j0l2j69i60l3.4719j0j7&sourceid=chrome&ie=UTF-8
429https://blog.exploratory.io/an-introduction-to-mongodb-query-for-beginners-bd463319aa4c
430https://docs.mongodb.com/manual/reference/method/db.collection.find/
431https://docs.mongodb.com/manual/reference/method/db.collection.find/#find-projection
432https://stackoverflow.com/questions/39641925/mongodb-aggregation-framework-to-get-frequencies-of-fields-values
433
434https://exploratory.io/note/kanaugust/0961813761939766
435https://docs.mongodb.com/manual/tutorial/project-fields-from-query-results/
436https://docs.mongodb.com/manual/aggregation/
437
438
439Mongo Studio 3T documentation:
440https://studio3t.com/download/ (also has uninstall information)
441https://studio3t.com/download-thank-you/?OS=x64
442
443Google: MongoDB visualization
444MongoDB visualization map
445MongoDB Charts
446 (Open source visualisation tools)
447
448json map visualizer
449 geojson.tools
450-------------------
451
452Some queries with results:
453
454# Num websites
455db.getCollection('Websites').find({}).count()
4561446
457
458# Num webpages
459db.getCollection('Webpages').find({}).count()
460X75139
461117496
462
463# Find number of websites who have 1 or more pages in Maori (a positive numPagesInMRI)
464db.getCollection('Websites').find({numPagesInMRI: { $gt: 0}}).count()
465361
466
467# Find number of webpages that are deemed to be overall in MRI (pages where isMRI=true)
468db.getCollection('Webpages').find({isMRI:true}).count()
469X5224
470X5215
471db.getCollection('Webpages').find({isMRI:true}).count()
4727818
473
474# Number of pages that contain any number of MRI sentences
475db.getCollection('Webpages').find({containsMRI: true}).count()
476X12858
47720371
478
479
480# Number of sites with URLs containing /mi(/)
481db.getCollection('Websites').find({urlContainsLangCodeInPath:true}).count()
482153
483
484# Number of websites that are outside NZ that contain /mi(/) in any of its sub-urls
485db.getCollection('Websites').find({urlContainsLangCodeInPath:true, geoLocationCountryCode: {$ne : "NZ"} }).count()
486148
487
488# 5 sites with URLs containing /mi(/) that are in NZ
489db.getCollection('Websites').find({urlContainsLangCodeInPath:true, geoLocationCountryCode: "NZ"}).count()
4905
491
492# sort websites that contain /mi(/) in path by geoLocationCountryCode
493# https://www.quackit.com/mongodb/tutorial/mongodb_sort_query_results.cfm
494db.getCollection('Websites').find({urlContainsLangCodeInPath:true}).sort({geoLocationCountryCode: 1})
495
496Actually, I want to sort by count. See https://docs.mongodb.com/manual/reference/operator/aggregation/sortByCount/
497
498
499# PROJECTION:
500db.getCollection('Websites').find({geoLocationCountryCode: {$ne:"nz"}}, {geoLocationCountryCode:1, urlContainsLangCodeInPath: 1})
501
502https://docs.mongodb.com/manual/aggregation/
503EXAMPLE:
504db.orders.aggregate([
505 { $match: { status: "A" } },
506 { $group: { _id: "$cust_id", total: { $sum: "$amount" } } }
507])
508
509X db.Websites.aggregate([{ $match:{urlContainsLangCodeInPath:true}}, $group: {geoLocationCountryCode:1, total: $count}])
510
511
512X db.Websites.aggregate([
513 { $match:{urlContainsLangCodeInPath:true}},
514 {$group: {geoLocationCountryCode:1}}
515])
516
517WORKS (but an "unwind" will get rid of "null"):
518db.Websites.aggregate([
519 { $match:{urlContainsLangCodeInPath:true}},
520 {$group: {_id: "$geoLocationCountryCode", count: {$sum: 1}}},
521 { $sort : { count : -1} }
522])
523
524
525# COUNT OF ALL GEOLOCATION COUNTRIES
526#https://stackoverflow.com/questions/14924495/mongodb-count-num-of-distinct-values-per-field-key
527 # LIST
528 db.Websites.distinct('geoLocationCountryCode');
529
530 # COUNT
531 db.Websites.distinct('geoLocationCountryCode').length;
532
533 # A COUNT WITH QUERY - https://docs.mongodb.com/manual/reference/command/distinct/#dbcmd.distinct
534
535 db.runCommand ( { distinct: "Websites", key: "geoLocationCountryCode", query: { "urlContainsLangCodeInPath": true} } );
536
537 # DISTINCT WITH QUERY WITHOUT COUNT - https://docs.mongodb.com/manual/reference/method/db.collection.distinct/
538 db.Websites.distinct('geoLocationCountryCode', {"urlContainsLangCodeInPath": true});
539
540 #SORTED - https://stackoverflow.com/questions/4759437/get-distinct-values-with-sorted-data
541 db.Websites.distinct('geoLocationCountryCode', {"urlContainsLangCodeInPath": true}).sort();
542
543
544 # count of all sites for which the geolocation is UNKNOWN
545 db.getCollection('Websites').find({geoLocationCountryCode: {$eq:"UNKNOWN"}}).count()
546
547
548# AGGREGATION QUERIES THAT WORK:
549#https://stackoverflow.com/questions/14924495/mongodb-count-num-of-distinct-values-per-field-key
550
551WORKS:
552// count of country codes for all sites
553db.Websites.aggregate([
554
555 { $unwind: "$geoLocationCountryCode" },
556 {
557 $group: {
558 _id: "$geoLocationCountryCode",
559 count: { $sum: 1 }
560 }
561 },
562 { $sort : { count : -1} }
563]);
564
565
566WORKS:
567// count of country codes for sites that have /mi(/) in path
568
569db.Websites.aggregate([
570 {
571 $match: {
572 urlContainsLangCodeInPath: true
573 }
574 },
575 { $unwind: "$geoLocationCountryCode" },
576 {
577 $group: {
578 _id: {$toLower: '$geoLocationCountryCode'},
579 count: { $sum: 1 }
580 }
581 },
582 { $sort : { count : -1} }
583]);
584
585
586WORKS:
587db.Websites.aggregate([
588 {
589 $match: {
590 geoLocationCountryCode: {$ne : "UNKNOWN"}
591 }
592 },
593 { $unwind: "$geoLocationCountryCode" },
594 {
595 $group: {
596 _id: "$geoLocationCountryCode",
597 count: { $sum: 1 }
598 }
599 },
600 { $sort : { count : -1} }
601]);
602
603WORKS:
604db.Websites.aggregate([
605 {
606 $match: {
607 "urlContainsLangCodeInPath": true
608 }
609 },
610 { $unwind: "$geoLocationCountryCode" },
611 {
612 $group: {
613 _id: "$geoLocationCountryCode",
614 count: { $sum: 1 }
615 }
616 },
617 { $sort : { count : -1} }
618]);
619
620
621KEEP ADDITIONAL FIELDS - https://stackoverflow.com/questions/16662405/mongo-group-query-how-to-keep-fields:
622 a. KEEPS ONLY FIRST DOMAIN URL FOR EACH COUNTED COUNTRY CODE:
623
624 db.Websites.aggregate([
625 {
626 $match: {
627 "urlContainsLangCodeInPath": true
628 }
629 },
630 { $unwind: "$geoLocationCountryCode" },
631 {
632 $group: {
633 _id: "$geoLocationCountryCode", count: { $sum: 1 },
634 domain: {$first: '$domain'}
635 }
636 },
637 { $sort : { count : -1} }
638 ]);
639
640 b. KEEP ALL DOMAIN URLS:
641 db.Websites.aggregate([
642 {
643 $match: {
644 "urlContainsLangCodeInPath": true
645 }
646 },
647 { $unwind: "$geoLocationCountryCode" },
648 {
649 $group: {
650 _id: "$geoLocationCountryCode",
651 count: { $sum: 1 },
652 domain: { $addToSet: '$domain' }
653 }
654 },
655 { $sort : { count : -1} }
656 ]);
657
658
659# WANT TO GET THE ABOVE INTO WORLD MAP, use geojson.tools found by Dr Bainbridge
660geojson.tools
661USAGE: https://www.here.xyz/viewer-tool/
662
663
664AIMS:
665* Identify where Maori language is online.
666* How can we identify high quality sites that would be good for a corpus.
667(Related work for other languages to quantifiably answer that)
668
669
670
671
672data-preparation
673docs
674
675
676
677
678/* 1 */
679{
680 "_id" : "US",
681 "count" : 93.0,
682 -95.8,40.33
683}
684
685/* 2 */
686{
687 "_id" : "AU",
688 "count" : 7.0,
689 135.8,-25.33
690}
691
692/* 3 */
693{
694 "_id" : "CN",
695 "count" : 7.0,
696 100.8,
697 32.33
698}
699
700/* 4 */
701{
702 "_id" : "NZ",
703 "count" : 5.0,
704175.8,
705 -40.33
706}
707
708/* 5 */
709{
710 "_id" : "DE",
711 "count" : 5.0,
71210.8,
713 50.33
714}
715
716/* 6 */
717{
718 "_id" : "HK",
719 "count" : 5.0,
720114,
721 22.33
722}
723
724/* 7 */
725{
726 "_id" : "RU",
727 "count" : 4.0,
72838.4,
729 55.5
730}
731
732/* 8 */
733{
734 "_id" : "JP",
735 "count" : 3.0,
736 137.8,
737 36
738}
739
740/* 9 */
741{
742 "_id" : "GB",
743 "count" : 3.0,
744-2,
745 53.33
746}
747
748/* 10 */
749{
750 "_id" : "CA",
751 "count" : 2.0,
752 -105.8,
753 55.33
754}
755
756/* 11 */
757{
758 "_id" : "FR",
759 "count" : 2.0,
760 3,
761 47.33
762}
763
764/* 12 */
765{
766 "_id" : "DK",
767 "count" : 2.0,
768 9.5,
769 55.33
770}
771
772/* 13 British Virgin Islands */
773{
774 "_id" : "VG",
775 "count" : 2.0,
776 -64.8,
777 18.35
778}
779
780/* 14 Ukraine */
781{
782 "_id" : "UA",
783 "count" : 1.0,
784 31.5,
785 48.5
786}
787
788/* 15 */
789{
790 "_id" : "CZ",
791 "count" : 1.0,
792 16.2,
793 49.7
794}
795
796/* 16 Switzerland */
797{
798 "_id" : "CH",
799 "count" : 1.0,
800 8.5,
801 47
802}
803
804/* 17 Zuid-Afrika */
805{
806 "_id" : "ZA",
807 "count" : 1.0,
808 24.2,
809 -30.7
810}
811
812/* 18 */
813{
814 "_id" : "NL",
815 "count" : 1.0,
8165.8,
817 52.33
818}
819
820/* 19 */
821{
822 "_id" : "KR",
823 "count" : 1.0,
824 127.8,
825 36.8
826}
827
828
829/** http://geojson.tools/
830
831
832{
833 "type": "MultiPoint",
834 "coordinates": [
835 [
836 -95.8,
837 40.33
838 ],
839 [
840 135.8,
841 -25.33
842 ],
843 [
844 100.8,
845 32.33
846 ],
847 [
848 175.8,
849 -40.33
850 ],
851 [
852 10.8,
853 50.33
854 ],
855 [
856 10.8,
857 50.33
858 ],
859 [
860 114,
861 22.33
862 ],
863 [
864 38.4,
865 55.5
866 ],
867 [
868 -2,
869 53.33
870 ],
871 [
872 137.8,
873 36
874 ],
875 [
876 -105.8,
877 55.33
878 ],
879 [
880 3,
881 47.33
882 ],
883 [
884 9.5,
885 55.33
886 ],
887 [
888 -64.8,
889 18.35
890 ],
891 [
892 31.5,
893 48.5
894 ],
895 [
896 16.2,
897 49.7
898 ],
899 [
900 8.5,
901 47
902 ],
903 [
904 24.2,
905 -30.7
906 ],
907 [
908 5.8,
909 52.33
910 ],
911 [
912 127.8,
913 36.8
914 ]
915 ]
916}
917
918*/
919
920/* 1 */
921{
922 "_id" : "US",
923 "count" : 93.0,
924 -95.8,40.33
925}
926
927/* 2 */
928{
929 "_id" : "AU",
930 "count" : 7.0,
931 135.8,-25.33
932}
933
934/* 3 */
935{
936 "_id" : "CN",
937 "count" : 7.0,
938 100.8,
939 32.33
940}
941
942/* 4 */
943{
944 "_id" : "NZ",
945 "count" : 5.0,
946175.8,
947 -40.33
948}
949
950/* 5 */
951{
952 "_id" : "DE",
953 "count" : 5.0,
95410.8,
955 50.33
956}
957
958/* 6 */
959{
960 "_id" : "HK",
961 "count" : 5.0,
962114,
963 22.33
964}
965
966/* 7 */
967{
968 "_id" : "RU",
969 "count" : 4.0,
97038.4,
971 55.5
972}
973
974/* 8 */
975{
976 "_id" : "JP",
977 "count" : 3.0,
978 137.8,
979 36
980}
981
982/* 9 */
983{
984 "_id" : "GB",
985 "count" : 3.0,
986-2,
987 53.33
988}
989
990/* 10 */
991{
992 "_id" : "CA",
993 "count" : 2.0,
994 -105.8,
995 55.33
996}
997
998/* 11 */
999{
1000 "_id" : "FR",
1001 "count" : 2.0,
1002 3,
1003 47.33
1004}
1005
1006/* 12 */
1007{
1008 "_id" : "DK",
1009 "count" : 2.0,
1010 9.5,
1011 55.33
1012}
1013
1014/* 13 British Virgin Islands */
1015{
1016 "_id" : "VG",
1017 "count" : 2.0,
1018 -64.8,
1019 18.35
1020}
1021
1022/* 14 Ukraine */
1023{
1024 "_id" : "UA",
1025 "count" : 1.0,
1026 31.5,
1027 48.5
1028}
1029
1030/* 15 */
1031{
1032 "_id" : "CZ",
1033 "count" : 1.0,
1034 16.2,
1035 49.7
1036}
1037
1038/* 16 Switzerland */
1039{
1040 "_id" : "CH",
1041 "count" : 1.0,
1042 8.5,
1043 47
1044}
1045
1046/* 17 Zuid-Afrika */
1047{
1048 "_id" : "ZA",
1049 "count" : 1.0,
1050 24.2,
1051 -30.7
1052}
1053
1054/* 18 */
1055{
1056 "_id" : "NL",
1057 "count" : 1.0,
10585.8,
1059 52.33
1060}
1061
1062/* 19 */
1063{
1064 "_id" : "KR",
1065 "count" : 1.0,
1066 127.8,
1067 36.8
1068}
1069
1070
1071/** http://geojson.tools/
1072
1073
1074{
1075 "type": "MultiPoint",
1076 "coordinates": [
1077 [
1078 -95.8,
1079 40.33
1080 ],
1081 [
1082 135.8,
1083 -25.33
1084 ],
1085 [
1086 100.8,
1087 32.33
1088 ],
1089 [
1090 175.8,
1091 -40.33
1092 ],
1093 [
1094 10.8,
1095 50.33
1096 ],
1097 [
1098 10.8,
1099 50.33
1100 ],
1101 [
1102 114,
1103 22.33
1104 ],
1105 [
1106 38.4,
1107 55.5
1108 ],
1109 [
1110 -2,
1111 53.33
1112 ],
1113 [
1114 137.8,
1115 36
1116 ],
1117 [
1118 -105.8,
1119 55.33
1120 ],
1121 [
1122 3,
1123 47.33
1124 ],
1125 [
1126 9.5,
1127 55.33
1128 ],
1129 [
1130 -64.8,
1131 18.35
1132 ],
1133 [
1134 31.5,
1135 48.5
1136 ],
1137 [
1138 16.2,
1139 49.7
1140 ],
1141 [
1142 8.5,
1143 47
1144 ],
1145 [
1146 24.2,
1147 -30.7
1148 ],
1149 [
1150 5.8,
1151 52.33
1152 ],
1153 [
1154 127.8,
1155 36.8
1156 ]
1157 ]
1158}
1159
1160*/
Note: See TracBrowser for help on using the repository browser.