C / C++网络爬虫

Summary Whalebot is open-source web crawler. It is intended to be simple, fast and memory efficient. It was created as a targeted spider, but you may use it as common. Current release 0.02 Current state. Bold - done, normal - TODO If something broken or you have an idea, please visit https://groups.google.com/forum/#!forum/whalebot Usages It was used for collecting papers on target thematic from http://citeseerx.ist.psu.edu for my master degree work Candidates for logo were collected using whalebot Eating own dogs food (links for url parsing benchmark) Features Simple configuration from command line and text

whalebot
3dparty
google-url
googleurl
build
debug.vsprops
dbm
test
mapping_test.h
webspider
include
prefix.h
statsem_string
CMakeLists.txt
3dparty
google-url
googleurl
third_party
icu
build
using_icu.vsprops