Home
Sign Up
Sign In
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
Sign Up Now
For Existing Member
Sign In
V2EX
›
网址库
用python 做爬虫,抓取网站,在抓取的过程中会碰到重复的网址,随着抓取网址的越来越多,网址库越来越大,如果每次爬到网址都去网址库对比一下 是否重复,这样的结果就是效率越来越低了,有什么办法或者算法 提高过滤重复网址的效率?
Python
•
soho176
•
Apr 11, 2013
• Lastly replied by
h4x3rotab
23
About
·
Help
·
Advertise
·
Blog
·
API
·
FAQ
·
Solana
·
865 Online
Highest 6679
·
Select Language
创意工作者们的社区
World is powered by solitude
VERSION: 3.9.8.5 · 7ms ·
UTC 21:02
·
PVG 05:02
·
LAX 14:02
·
JFK 17:02
♥ Do have faith in what you're doing.
❯