Changeset 33470

Timestamp:
13.09.2019 22:53:23 (5 weeks ago)
Author:
ak19
Message:

A new script to reduce keepURLs.txt to unique URLs, 1 from each unique domain. Run this script after WETProcessor.java is run on a folder of warc.wet(.gz) files that contain WET records where primary language=MRI. The WETProcessor.java would have produced important keepURLs.txt and the less important discardURLs.txt.

Files:
1 added