Changeset 33470

Timestamp:

2019-09-13T22:53:23+12:00 (5 years ago)

Author:

ak19

Message:

A new script to reduce keepURLs.txt to unique URLs, 1 from each unique domain. Run this script after WETProcessor.java is run on a folder of warc.wet(.gz) files that contain WET records where primary language=MRI. The WETProcessor.java would have produced important keepURLs.txt and the less important discardURLs.txt.

File:

• gs3-extensions/maori-lang-detection/bin/script/unique_mri_domains_from_cc.sh (added)