Changeset 33414 for gs3-extensions

Timestamp:

2019-08-13T21:57:58+12:00 (5 years ago)

Author:

ak19

Message:

Adding important links

File:

• gs3-extensions/maori-lang-detection/MoreReading/CommonCrawl.txt (modified) (1 diff)

gs3-extensions/maori-lang-detection/MoreReading/CommonCrawl.txt

@@ +1 @@
+There's already python code for getting text:
+
+https://spark-in.me/post/parsing-common-crawl-in-two-simple-commands
+https://gist.github.com/Smerity/afe7430fdb4371015466
+
+https://spark-in.me/post/parsing-common-crawl-in-two-simple-commands
+
+"But it turns out - it is not. This can be attributed to the effort that has been made to make the CC more accessible. The killer feature for me was the presence of their index weighting only ~200Gb, that also features a language detection option, i.e. you do not need to analyze top-level-domains or do any significant data mining."
+
+What does the "language detection option" discussion above mean?
+
+------------
+Skipping CrawlDiagnostics (see below) and robots.txt gz files:
+
+http://commoncrawl.org/2018/08/august-2018-crawl-archive-now-available/
+
+"HTTP 304 notmodified" responses are now stored as WARC revisit records in the "crawldiagnostics" subset along with 404s, redirects and other non-200 responses. For now the revisit records contain a payload digest although there is no payload sent together with HTTP 304 responses. The stupid reason is that the columnar index requires the digest field and we want to make sure that all tools continue to work as expected. The SHA-1 digest of an empty payload (zero bytes) is used for the revisit records.
+
+http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#revisit
+‘revisit’
+General
+
+A ‘revisit’ record describes the revisitation of content already archived, and might include only an abbreviated content body which has to be interpreted relative to a previous record. Most typically, a ‘revisit’ record is used instead of a ‘response’ or ‘resource’ record to indicate that the content visited was either a complete or substantial duplicate of material previously archived.
+...
+
+-------
 
 WET FILES: