Changeset 34011 for other-projects/maori-lang-detection/mongodb-data/piechart_data2.txt

Timestamp:

2020-03-10T20:45:18+13:00 (4 years ago)

Author:

ak19

Message:

Piechart data for sites prepared for crawling and the piecharts for these

File:

• other-projects/maori-lang-detection/mongodb-data/piechart_data2.txt (modified) (1 diff)

other-projects/maori-lang-detection/mongodb-data/piechart_data2.txt

@@ -106 +106 @@
 07. 11316 empty status unknown
 
+
+
+============
+
+
+1463 sites prepared for crawling
+1447 sites crawled (16 were autotranslated or otherwise irrelevant)
+1446 crawled sites contained dump.txt files (1 site was missing dump.txt) - 1446 sites in mongodb
+619 sites not finished crawling
+1027 sites where dump.txt contained text:start denoting text content, so 419 sites with no text content
+
+
+
+16 uncrawled irrelevant sites pruned away
+1 failed crawl of site (text dump missing)
+1446 crawled sites in MongoDB
+
+
+Graph title: Breakdown of the 1463 sites prepared for crawling
+* 16 uncrawled irrelevant sites pruned away
+* 1 sites failed to properly crawl (text dump missing)
+* 619 incompletely crawled sites
+* 827 completely crawled sites
+
+
+Graph title: Breakdown of the 1463 sites prepared for crawling
+* 16 uncrawled irrelevant sites pruned away
+* 1 sites failed to properly crawl (text dump missing)
+* 419 crawled sites with no text content
+    - 150 crawled sites with 0-size dump.txt files [crawled sites with empty dump.txt files] See below.
+    - 269 crawled sites where dump.txt had no text content
+* 1027 crawled sites with text content (WebPages collection in MongoDB will have webpage documents for these sites)
+
+
+
+ 
+# All the dump.txt files that are 0 bytes (no content):
+# https://stackoverflow.com/questions/15703664/find-all-zero-byte-files-in-directory-and-subdirectories
+wharariki:[396]/Scratch/ak19/maori-lang-detection/crawled>find . -name "dump.txt" -size 0 | sort | wc
+    150     150    2550
+
+