Context Navigation

url-blacklist-filter.txt

Timestamp:

2019-10-16T20:00:09+13:00 (5 years ago)

Author:

ak19

Message:

batchcrawl.sh now does what it should have from the start, which is to move the log.out and UNFINISHED files into the output folder instead of leaving them in the input folder, as the input to_crawl folder can and does get replaced all the time, every time I regenerate it after black/white/greylisting more urls. 2. Blacklisted more adult sites, greylisted more product sites and .ru, .pl and .tk domains with whitelisting in the whitelist file. 3. CCWETProcessor now looks out for additional adult sites based on URL and adds them to its blacklist in memory (not the file) and logs the domain for checking and manually adding to the blacklist file.

File:

-              r33568
+              r33569
 zh-min-nan.wiktionary.org
+######
 # unwanted domains
 .video-chat.
 …
 acba.osb-land.com
+# just get rid of any URL containing "livejasmin"
+## livejasmin
+# Actually: do that in the code (CCWETProcessor) with a log message,
+# since we actually need to get rid of any sites in entirety that contain
+# any url with the string "livejasmin"
+# So run the program once, check the log for messages mentioning "additional"
+# adult sites found and add their domains in here.
+anigma-beauty.com
+adultfeet.com
+atopian.org
+bellydancingvideo.net
+bmmodelsagency.com
+brucknergallery.com
+fuckvidz.org
+photobattle.net
+votekat.info
+# Similar to above, the following contained the string "jasmin" in the URL
+teenycuties.com
+a.tiles.mapbox.com
+blazingteens.net
+redtubeporn.info
+osb-land.com
+totallyhotmales.com
+babeevents.com
+talkserver.de
+hehechat.org
+fetish-nights.com
+lesslove.com
+hebertsvideo.com
 # sounds like some pirating site
 ^http://pirateguides.com/
 …
 # not sure about the domain name and/or full url seems like it belongs here
 abcutie.com
+# only had a single seedURL and it quickly redirected to an adult site
+apparactes.gq

Note: See TracChangeset for help on using the changeset viewer.