Changeset 34011 for other-projects/maori-lang-detection/mongodb-data/piechart_data.txt

Timestamp:

2020-03-10T20:45:18+13:00 (4 years ago)

Author:

ak19

Message:

Piechart data for sites prepared for crawling and the piecharts for these

File:

• other-projects/maori-lang-detection/mongodb-data/piechart_data.txt (modified) (1 diff)

other-projects/maori-lang-detection/mongodb-data/piechart_data.txt

@@ -487 +487 @@
     metadata _csh_ :        ^@^@^@^@
 
+
+------------
+
+
+Would like to do something like:
+wharariki:[378]/Scratch/ak19/maori-lang-detection/crawled>find . -name UNFINISHED | grep -l text:start */dump.txt | wc
+
+
+Would like to find how many and which of the unfinished websites had a dump.txt with no text content
+AND how many of the completely crawled websites had a dump.txt with no text content.
+
+
+--------------
+
+
+
+wharariki:[393]/Scratch/ak19/maori-lang-detection/crawled>grep -l "text:start" */dump.txt
+
+wharariki:[388]/Scratch/ak19/maori-lang-detection/crawled>less 01461/dump.txt 
+wharariki:[389]/Scratch/ak19/maori-lang-detection/crawled>less 01453/dump.txt 
+wharariki:[390]/Scratch/ak19/maori-lang-detection/crawled>less 01447/dump.txt 
+wharariki:[391]/Scratch/ak19/maori-lang-detection/crawled>less 01446/dump.txt 
+wharariki:[392]/Scratch/ak19/maori-lang-detection/crawled>less 01445/dump.txt 
+ 
+# All the dump.txt files that are 0 bytes (no content):
+# https://stackoverflow.com/questions/15703664/find-all-zero-byte-files-in-directory-and-subdirectories
+wharariki:[396]/Scratch/ak19/maori-lang-detection/crawled>find . -name "dump.txt" -size 0 | sort | wc
+    150     150    2550
+
+
+Examples of empty dump.txt files (listed with: find . -name "dump.txt" -size 0 | sort):
+    wharariki:[400]/Scratch/ak19/maori-lang-detection/crawled>less ../tmp/to_crawl.THE_VERSION_USED/sites/00014/seedURLs.txt 
+    wharariki:[401]/Scratch/ak19/maori-lang-detection/crawled>less ../tmp/to_crawl.THE_VERSION_USED/sites/01461/seedURLs.txt 
+    wharariki:[402]/Scratch/ak19/maori-lang-detection/crawled>less ../tmp/to_crawl.THE_VERSION_USED/sites/01447/seedURLs.txt 
+    wharariki:[403]/Scratch/ak19/maori-lang-detection/crawled>less ../tmp/to_crawl.THE_VERSION_USED/sites/01422/seedURLs.txt 
+    
+
+
+
+=======