Changeset 33621 for gs3-extensions/maori-lang-detection/MoreReading/crawling-Nutch.txt

Timestamp:

2019-11-04T20:35:59+13:00 (4 years ago)

Author:

ak19

Message:

Comitting jotted down mongodb related instructions from what Dr Bainbridge did today.

File:

• gs3-extensions/maori-lang-detection/MoreReading/crawling-Nutch.txt (modified) (1 diff)

gs3-extensions/maori-lang-detection/MoreReading/crawling-Nutch.txt

@@ -351 +351 @@
     http://central.maven.org/maven2/org/mongodb/mongo-java-driver/
     https://mongodb.github.io/mongo-java-driver/
+
+
+INSTALLING THE MONGODB SERVER AND MONGO CLIENT ON LINUX
+Need to have sudo and root powers.
+
+https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
+http://www.programmersought.com/article/6500308940/
+
+   52  sudo apt-get install mongodb-clients
+   53  mongo 'mongodb://mongodb.cms.waikato.ac.nz:27017' -u anupama -p
+
+Failed with
+    Error: HostAndPort: host is empty at src/mongo/shell/mongo.js:148
+    exception: connect failed
+
+This is due to a version incompatibility between Client and mongodb Server.
+The solution is to follow instructions at http://www.programmersought.com/article/6500308940/
+and then https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
+as below:
+
+   54  sudo apt-get purge mongodb-clients
+   55  sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
+   56  echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
+   57  sudo apt-get update
+   58  sudo apt-get install mongodb-clients
+   59  mongo 'mongodb://mongodb.cms.waikato.ac.nz:27017' -u anupama -p
+(still doesn't work)
+   60  sudo apt-get install -y mongodb-org
+The above ensures an up to date mongo client but installs the mongodb server too. Maybe this is the only step that is needed to install up-to-date mongo client and mongodb server?
+   72  sudo service mongod status
+
+  103  sudo service mongod start
+"mongod" stands for mongo-daemon. This runs the mongo db server listening for client connections
+  104  sudo service mongod status
+   88  sudo service mongod stop
+
+
+RUNNING AND USING THE MONGO CLIENT SHELL:
+Among the many things you can do with the Mongo client shell, one can use it to find the mongo client version (which is the version of the shell) and the mongo db version.
+
+To run the mongo client shell WITHOUT loading a db:
+
+
+    wharariki:[880]/Scratch/ak19/gs3-extensions/maori-lang-detection>mongo --shell -nodb
+    MongoDB shell version: 2.6.10 <<<<<<<<<-------------------<<<< MONGO CLIENT VERSION
+
+    type "help" for help
+    > help
+        db.help()                    help on db methods
+        db.mycoll.help()             help on collection methods
+        sh.help()                    sharding helpers
+        rs.help()                    replica set helpers
+        help admin                   administrative help
+        help connect                 connecting to a db help
+        help keys                    key shortcuts
+        help misc                    misc things to know
+        help mr                      mapreduce
+
+        show dbs                     show database names
+        show collections             show collections in current database
+        show users                   show users in current database
+        show profile                 show most recent system.profile entries with time >= 1ms
+        show logs                    show the accessible logger names
+        show log [name]              prints out the last segment of log in memory, 'global' is default
+        use <db_name>                set current database
+        db.foo.find()                list objects in collection foo
+        db.foo.find( { a : 1 } )     list objects in foo where a == 1
+        it                           result of the last line evaluated; use to further iterate
+        DBQuery.shellBatchSize = x   set default number of items to display on shell
+        exit                         quit the mongo shell
+
+    > help connect
+
+    Normally one specifies the server on the mongo shell command line.  Run mongo --help to see those options.
+    Additional connections may be opened:
+
+        var x = new Mongo('host[:port]');
+        var mydb = x.getDB('mydb');
+      or
+        var mydb = connect('host[:port]/mydb');
+
+    Note: the REPL prompt only auto-reports getLastError() for the shell command line connection.
+
+    Getting help on connect options: 
+
+    > var x = new Mongo('mongodb.cms.waikato.ac.nz:27017');
+    > var mydb = x.getDB('anupama');
+
+    > mydb.connect.help()
+    DBCollection help
+        db.connect.find().help() - show DBCursor help
+        db.connect.count()
+        db.connect.copyTo(newColl) - duplicates collection by copying all documents to newColl; no indexes are copied.
+        db.connect.convertToCapped(maxBytes) - calls {convertToCapped:'connect', size:maxBytes}} command
+        db.connect.dataSize()
+        db.connect.distinct( key ) - e.g. db.connect.distinct( 'x' )
+        db.connect.drop() drop the collection
+        db.connect.dropIndex(index) - e.g. db.connect.dropIndex( "indexName" ) or db.connect.dropIndex( { "indexKey" : 1 } )
+        db.connect.dropIndexes()
+        db.connect.ensureIndex(keypattern[,options]) - options is an object with these possible fields: name, unique, dropDups
+        db.connect.reIndex()
+        db.connect.find([query],[fields]) - query is an optional query filter. fields is optional set of fields to return.
+                                                  e.g. db.connect.find( {x:77} , {name:1, x:1} )
+        db.connect.find(...).count()
+        db.connect.find(...).limit(n)
+        db.connect.find(...).skip(n)
+        db.connect.find(...).sort(...)
+        db.connect.findOne([query])
+        db.connect.findAndModify( { update : ... , remove : bool [, query: {}, sort: {}, 'new': false] } )
+        db.connect.getDB() get DB object associated with collection
+        db.connect.getPlanCache() get query plan cache associated with collection
+        db.connect.getIndexes()
+        db.connect.group( { key : ..., initial: ..., reduce : ...[, cond: ...] } )
+        db.connect.insert(obj)
+        db.connect.mapReduce( mapFunction , reduceFunction , <optional params> )
+        db.connect.aggregate( [pipeline], <optional params> ) - performs an aggregation on a collection; returns a cursor
+        db.connect.remove(query)
+        db.connect.renameCollection( newName , <dropTarget> ) renames the collection.
+        db.connect.runCommand( name , <options> ) runs a db command with the given name where the first param is the collection name
+        db.connect.save(obj)
+        db.connect.stats()
+        db.connect.storageSize() - includes free space allocated to this collection
+        db.connect.totalIndexSize() - size in bytes of all the indexes
+        db.connect.totalSize() - storage allocated for all data and indexes
+        db.connect.update(query, object[, upsert_bool, multi_bool]) - instead of two flags, you can pass an object with fields: upsert, multi
+        db.connect.validate( <full> ) - SLOW
+        db.connect.getShardVersion() - only for use with sharding
+        db.connect.getShardDistribution() - prints statistics about data distribution in the cluster
+        db.connect.getSplitKeysForChunks( <maxChunkSize> ) - calculates split points over all chunks and returns splitter function
+        db.connect.getWriteConcern() - returns the write concern used for any operations on this collection, inherited from server/db if set
+        db.connect.setWriteConcern( <write concern doc> ) - sets the write concern for writes to the collection
+        db.connect.unsetWriteConcern( <write concern doc> ) - unsets the write concern for writes to the collection
+    > mydb.version()
+    4.0.13 <<<<<<<<<-------------------<<<< MONGODB SERVER VERSION
+
+(Check Mongo server version: https://stackoverflow.com/questions/38160412/how-to-find-the-exact-version-of-installed-mongodb)
+
+Finally we now know the mongodb server version: 4.0.13
+This version didn't work with our mongo client (shell) version of 2.6.10. And that's we had to upgrade the client.
+
+
+
+
+MONGO DB ROBO 3T
+1. Download "Double Pack" from https://robomongo.org/
+2. Untar its contents. Then untar the tarball in that.
+3. Run:
+    wharariki:[110]~/Downloads/robo3t-1.3.1-linux-x86_64-7419c406>./bin/robo3t
+