Changeset 34011 for other-projects

Timestamp:

2020-03-10T20:45:18+13:00 (4 years ago)

Author:

ak19

Message:

Piechart data for sites prepared for crawling and the piecharts for these

Location:

other-projects/maori-lang-detection/mongodb-data

Files:

• pieChart4a_sitesPreparedForCrawling.png (added)
• pieChart4b_sitesPreparedForCrawling.svg (added)
• pieChart4c_screenshotSitesPreparedForCrawling.png (added)
• pieChart5a_sitesPreparedForCrawling.png (added)
• pieChart5b_sitesPreparedForCrawling.svg (added)
• pieChart5c_screenshotSitesPreparedForCrawling.png (added)
• piechart_data.txt (modified) (1 diff)
• piechart_data2.txt (modified) (1 diff)

other-projects/maori-lang-detection/mongodb-data/piechart_data.txt

@@ -487 +487 @@
     metadata _csh_ :        ^@^@^@^@
 
+
+------------
+
+
+Would like to do something like:
+wharariki:[378]/Scratch/ak19/maori-lang-detection/crawled>find . -name UNFINISHED | grep -l text:start */dump.txt | wc
+
+
+Would like to find how many and which of the unfinished websites had a dump.txt with no text content
+AND how many of the completely crawled websites had a dump.txt with no text content.
+
+
+--------------
+
+
+
+wharariki:[393]/Scratch/ak19/maori-lang-detection/crawled>grep -l "text:start" */dump.txt
+
+wharariki:[388]/Scratch/ak19/maori-lang-detection/crawled>less 01461/dump.txt 
+wharariki:[389]/Scratch/ak19/maori-lang-detection/crawled>less 01453/dump.txt 
+wharariki:[390]/Scratch/ak19/maori-lang-detection/crawled>less 01447/dump.txt 
+wharariki:[391]/Scratch/ak19/maori-lang-detection/crawled>less 01446/dump.txt 
+wharariki:[392]/Scratch/ak19/maori-lang-detection/crawled>less 01445/dump.txt 
+ 
+# All the dump.txt files that are 0 bytes (no content):
+# https://stackoverflow.com/questions/15703664/find-all-zero-byte-files-in-directory-and-subdirectories
+wharariki:[396]/Scratch/ak19/maori-lang-detection/crawled>find . -name "dump.txt" -size 0 | sort | wc
+    150     150    2550
+
+
+Examples of empty dump.txt files (listed with: find . -name "dump.txt" -size 0 | sort):
+    wharariki:[400]/Scratch/ak19/maori-lang-detection/crawled>less ../tmp/to_crawl.THE_VERSION_USED/sites/00014/seedURLs.txt 
+    wharariki:[401]/Scratch/ak19/maori-lang-detection/crawled>less ../tmp/to_crawl.THE_VERSION_USED/sites/01461/seedURLs.txt 
+    wharariki:[402]/Scratch/ak19/maori-lang-detection/crawled>less ../tmp/to_crawl.THE_VERSION_USED/sites/01447/seedURLs.txt 
+    wharariki:[403]/Scratch/ak19/maori-lang-detection/crawled>less ../tmp/to_crawl.THE_VERSION_USED/sites/01422/seedURLs.txt 
+    
+
+
+
+=======

other-projects/maori-lang-detection/mongodb-data/piechart_data2.txt

@@ -106 +106 @@
 07. 11316 empty status unknown
 
+
+
+============
+
+
+1463 sites prepared for crawling
+1447 sites crawled (16 were autotranslated or otherwise irrelevant)
+1446 crawled sites contained dump.txt files (1 site was missing dump.txt) - 1446 sites in mongodb
+619 sites not finished crawling
+1027 sites where dump.txt contained text:start denoting text content, so 419 sites with no text content
+
+
+
+16 uncrawled irrelevant sites pruned away
+1 failed crawl of site (text dump missing)
+1446 crawled sites in MongoDB
+
+
+Graph title: Breakdown of the 1463 sites prepared for crawling
+* 16 uncrawled irrelevant sites pruned away
+* 1 sites failed to properly crawl (text dump missing)
+* 619 incompletely crawled sites
+* 827 completely crawled sites
+
+
+Graph title: Breakdown of the 1463 sites prepared for crawling
+* 16 uncrawled irrelevant sites pruned away
+* 1 sites failed to properly crawl (text dump missing)
+* 419 crawled sites with no text content
+    - 150 crawled sites with 0-size dump.txt files [crawled sites with empty dump.txt files] See below.
+    - 269 crawled sites where dump.txt had no text content
+* 1027 crawled sites with text content (WebPages collection in MongoDB will have webpage documents for these sites)
+
+
+
+ 
+# All the dump.txt files that are 0 bytes (no content):
+# https://stackoverflow.com/questions/15703664/find-all-zero-byte-files-in-directory-and-subdirectories
+wharariki:[396]/Scratch/ak19/maori-lang-detection/crawled>find . -name "dump.txt" -size 0 | sort | wc
+    150     150    2550
+
+