Changeset 33986 for other-projects

Timestamp:

2020-02-28T22:07:29+13:00 (4 years ago)

Author:

ak19

Message:

Dr Bainbridge investigated the original data set more

File:

• other-projects/maori-lang-detection/mongodb-data/piechart_data.txt (modified) (2 diffs)

other-projects/maori-lang-detection/mongodb-data/piechart_data.txt

@@ +1 @@
+"UPPER BOUND"
+
 blacklisted
 greylisted
@@ -173 +175 @@
 119874
 
+---------------------------
 
+#Number of crawled pages with 0 content in dump.txt because the page was inaccessible when crawling (protocolStatus: NOTFOUND)
+wharariki:[646]/Scratch/ak19/maori-lang-detection/crawled>fgrep -a  'NOTFOUND' 0*/dump.txt | grep protocolStatus | wc
+   3276    9828  419259
+
+#Number of dump.txt files (sites) that had text:start in them vs those that didn't:
+wharariki:[647]/Scratch/ak19/maori-lang-detection/crawled>fgrep -l text:start */dump.txt | wc
+   1027    1027   15405
+wharariki:[648]/Scratch/ak19/maori-lang-detection/crawled>fgrep text:start */dump.txt | wc
+   1027    4108   35945
+
+# number of dump.txt files
+wharariki:[652]/Scratch/ak19/maori-lang-detection/crawled>find . -name "dump.txt" | wc
+   1446    1446   24582
+wharariki:[653]/Scratch/ak19/maori-lang-detection/crawled>
+
+
+Look to see if commoncrawl has a field for how much text there is on the page.
+Else this is a useful feature for them to add.
+
+