source: other-projects/maori-lang-detection/mongodb-data/piechart_data.txt@ 34001

The 12 month period CommonCrawl crawl data that we used:

https://commoncrawl.org/2018/10/september-2018-crawl-archive-now-available/
- contains 2.8 billion web pages and 220 TiB of uncompressed content
- contains 500 million new URLs, not contained in any crawl archive before
https://commoncrawl.org/2018/10/october-2018-crawl-archive-now-available/
- 3.0 billion web pages and 240 TiB of uncompressed content
- 600 million new URLs, not contained in any crawl archive before
https://commoncrawl.org/2018/11/november-2018-crawl-archive-now-available/
- 2.6 billion web pages or 220 TiB of uncompressed content
- 640 million new URLs, not contained in any crawl archive before
https://commoncrawl.org/2018/12/december-2018-crawl-archive-now-available/
- 3.1 billion web pages or 250 TiB of uncompressed content,
- 735 million URLs not contained in any crawl archive before
https://commoncrawl.org/2019/01/january-2019-crawl-archive-now-available/
- 2.85 billion web pages or 240 TiB of uncompressed content
- 850 million URLs not contained in any crawl archive before.
https://commoncrawl.org/2019/03/february-2019-crawl-archive-now-available/
- 2.9 billion web pages or 225 TiB of uncompressed content
- 750 million URLs not contained in any crawl archive before
https://commoncrawl.org/2019/04/march-2019-crawl-archive-now-available/
- 2.55 billion web pages or 210 TiB of uncompressed content
- 660 million URLs not contained in any crawl archive before
https://commoncrawl.org/2019/04/april-2019-crawl-archive-now-available/
- 2.5 billion web pages or 198 TiB of uncompressed content
- 750 million URLs not contained in any crawl archive before
https://commoncrawl.org/2019/05/may-2019-crawl-archive-now-available/
- 2.65 billion web pages or 220 TiB of uncompressed content
- 825 million URLs not contained in any crawl archive before
https://commoncrawl.org/2019/07/june-2019-crawl-archive-now-available/
- 2.6 billion web pages or 220 TiB of uncompressed content
- 880 million URLs not contained in any crawl archive before
https://commoncrawl.org/2019/07/july-2019-crawl-archive-now-available/
- 2.6 billion web pages or 220 TiB of uncompressed content
- 810 million URLs not contained in any crawl archive before
https://commoncrawl.org/2019/08/august-2019-crawl-archive-now-available/
- 2.95 billion web pages or 260 TiB of uncompressed content
- 1.1 billion URLs not contained in any crawl archive before

= 9100 million or 9.1 billion new URLs not contained in any crawl archive before 
+ taking the first crawl month's figure of 2.8 billion - 500 million new URL in first crawl month = 11.4 billion URLs? At least?
---------------------------------------------

"UPPER BOUND"

blacklisted
greylisted
skipped crawling
unfinished (crawling)

Sites crawled and ingested into mongodb:
- domains shortlisted
- not shortlisted


Not included: only areas of interest of sites otherwise too big to exhaustively crawl were crawled. Not the rest. For example, not all of wikipedia but only mi.wikipedia.org. Not all of blogspot, only blogspot blogs indicated by common crawl results for MRI. Not all of docs.google.com, only the specific pages that turned up in common crawl for MRI.


1. ALL DOMAINS FROM CC-CRAWL:

Total counts from CommonCrawl (i.e. unique domain count across discardURLs + greyListed.txt + keepURLs.txt) 

wharariki:[1153]/Scratch/ak19/maori-lang-detection/src>java -cp ".:../conf:../lib/*" org.greenstone.atea.AllDomainCount
Counting all domains and urls in keepURLs.txt + discardURLs.txt + greyListed.txt
    Count of unique domains: 3074
    Count of unique basic domains (stripped of protocol and www): 2791
    Line count: 75559
    Actual unique URL count: 38717
    Unique basic URL count (stripped of protocol and www): 32827
******************************************************

[X 1588 domains from discardURLs + 288 (-1) greylistedURLs + 1462 (+1) keepURLs = 3338 domains]

Line count above is correct and consistent with the following: 23794+4485+47280=75559

But instead of domain/unique domain or URL/basic unique URL counts. The union of:
- domains of the following: 1588+288+1462 = 3338
- unique basic domains of the following (stripped of protocol and www): 1415+277+1362 = 3054
- basic URL count = 10290 + 2751 + 25683 = 38724
- basic unique URL count (stripped of protocol and www) = 9656 + 2727 + 20451 = 32834


wharariki:[1154]/Scratch/ak19/maori-lang-detection/src>java -cp ".:../conf:../lib/*" org.greenstone.atea.AllDomainCount ../tmp/to_crawl.THE_VERSION_USED/discardURLs.txt 
Counting all domains and urls in discardURLs.txt
    Count of unique domains: 1588
    Count of unique basic domains (stripped of protocol and www): 1415
    Line count: 23794
    Actual unique URL count: 10290
    Unique basic URL count (stripped of protocol and www): 9656
******************************************************
wharariki:[1155]/Scratch/ak19/maori-lang-detection/src>java -cp ".:../conf:../lib/*" org.greenstone.atea.AllDomainCount ../tmp/to_crawl.THE_VERSION_USED/greyListed.txt
Counting all domains and urls in greyListed.txt
    Count of unique domains: 288
    Count of unique basic domains (stripped of protocol and www): 277
    Line count: 4485
    Actual unique URL count: 2751
    Unique basic URL count (stripped of protocol and www): 2727
******************************************************
wharariki:[1156]/Scratch/ak19/maori-lang-detection/src>java -cp ".:../conf:../lib/*" org.greenstone.atea.AllDomainCount ../tmp/to_crawl.THE_VERSION_USED/keepURLs.txt 
Counting all domains and urls in keepURLs.txt
    Count of unique domains: 1464
    Count of unique basic domains (stripped of protocol and www): 1362
    Line count: 47280
    Actual unique URL count: 25683
    Unique basic URL count (stripped of protocol and www): 20451
******************************************************


XXXXXXXXXX
wharariki:[1159]/Scratch/ak19/maori-lang-detection/src>java -cp ".:../conf:../lib/*" org.greenstone.atea.AllDomainCount ../tmp/to_crawl.THE_VERSION_USED/seedURLs.txt 
Counting all domains and urls in seedURLs.txt
    Count of unique domains: 1462
    Count of unique basic domains (stripped of protocol and www): 1360
    Line count: 25679
    Actual unique URL count: 25679
    Unique basic URL count (stripped of protocol and www): 20447
******************************************************
XXXXXXXXXX

seedURLs is a subset of keepURLs.


2a. DISCARDED URLS:
URLS that are blacklisted + those pages with too little text content (under an arbitrary min threshold)

> wc -l discardURLs.txt 
23794

b. GREYLISTED URLS:
> wc -l greyListed.txt
4485


c. keepURLs (the URLs we kept for further processing):
wc -l keepURLs.txt 
47280 keepURLs.txt


d. Of the keepURLs, 4 more webpages ultimately irrelevant sites at unprocessed-topsite-matches.txt.

3 not in MRI but of the same domain, one is just a gallery of holiday pictures.

> less unprocessed-topsite-matches.txt 
    The following domain with seedURLs are on a major/top 500 site
    for which no allowed URL pattern regex has been specified.
    Specify one for this domain in the tab-spaced sites-too-big-to-exhaustively-crawl.txt file
        http://familypedia.wikia.com/wiki/Property:Father?limit=500&offset=0
        http://familypedia.wikia.com/wiki/Property:Mother?limit=250&offset=0
        http://familypedia.wikia.com/wiki/Property:Mother?limit=500&offset=0
        https://get.google.com/albumarchive/112997211423463224598/album/AF1QipM73RVcpCT2gpp5XhDUawnfyUDBbuJbeCEbVckl


e. After duplicates further pruned out from what remained of keepURLs - the seedURLs for Nutch:

wc -l seedURLs.txt 
25679 seedURLs.txt

wharariki:[1111]/Scratch/ak19/maori-lang-detection/src>java -cp ".:../conf:../lib/*" org.greenstone.atea.UniqueDomainCount ../tmp/to_crawl.THE_VERSION_USED/seedURLs.txt 
In file ../tmp/to_crawl.THE_VERSION_USED/seedURLs.txt:
    Count of domains: 1462
    Count of unique domains: 1360


But anglican.org was wrongly greylisted and added back in
-> 1463 domains.

3a. Num URLs prepared for crawling:
wharariki:[119]/Scratch/ak19/maori-lang-detection/tmp/to_crawl.THE_VERSION_USED>wc -l seedURLs.txt 
25679 seedURLs.txt

b. Num sites prepared for crawling (https://stackoverflow.com/questions/17648033/counting-number-of-directories-in-a-specific-directory):


wharariki:[147]/Scratch/ak19/maori-lang-detection/tmp/to_crawl.THE_VERSION_USED/sites>echo */ | wc
      1    1463   10241

(2nd number)
OR: sites>find . -mindepth 1 -maxdepth 1 -type d | wc -l
1463


/Scratch/ak19/maori-lang-detection/tmp/to_crawl.THE_VERSION_USED/sites/ also contains subfolders up to 01463


[maori-lang-detection/tmp/to_crawl.THE_VERSION_USED>emacs all-domain-urls.txt
1462+1 (for the greylisted anglican.org) = 1463]

4. Num sites crawled:
wharariki:[155]/Scratch/ak19/maori-lang-detection/crawled>find . -mindepth 1 -maxdepth 1 -type d | wc -l
1447
wharariki:[156]/Scratch/ak19/maori-lang-detection/crawled>echo */ | wc
      1    1447   10129

5. Number of sites not finished crawling (using Nutch at max crawl depth 10):
wharariki:[158]/Scratch/ak19/maori-lang-detection/crawled>find . -name "UNFINISHED" | wc -l
619


6. Number of sites in MongoDB:
1446

Not: 00179, 00485-00495, 00499-00502, 01067* (No dump.txt, but website is repeated in 01408)

* 01067 is listed under sites crawled, but not ingested into mongodb.

In siteID ranges of 00100s, 00400s, 00500s, 01000s, each have less than 100 sites in mongodbs:
99, 88, 97, 99

and 64/64 sites in siteIDs 1400-1463.

=> 1+12+3+1 = 17 sites not crawled of the 1463 = 1446 sites ingested in mongodb.


7. Despite 25679 non-duplicate seedURLs and many more pages crawled from those seeds,
the number of web pages ingested into mongodb are less than about 5 times as much,
because only crawled web pages with non-empty text were ingested into mongodb.

Num pages in MongoDB:
db.getCollection('Webpages').find({}).count()
119874

---------------------------

#Number of crawled pages with 0 content in dump.txt because the page was inaccessible when crawling (protocolStatus: NOTFOUND)
wharariki:[646]/Scratch/ak19/maori-lang-detection/crawled>fgrep -a  'NOTFOUND' 0*/dump.txt | grep protocolStatus | wc
   3276    9828  419259

#Number of dump.txt files (sites) that had text:start in them vs those that didn't:
wharariki:[647]/Scratch/ak19/maori-lang-detection/crawled>fgrep -l text:start */dump.txt | wc
   1027    1027   15405
wharariki:[648]/Scratch/ak19/maori-lang-detection/crawled>fgrep text:start */dump.txt | wc
   1027    4108   35945

# number of dump.txt files
wharariki:[652]/Scratch/ak19/maori-lang-detection/crawled>find . -name "dump.txt" | wc
   1446    1446   24582
wharariki:[653]/Scratch/ak19/maori-lang-detection/crawled>


Look to see if commoncrawl has a field for how much text there is on the page.
Else this is a useful feature for them to add.