xiaoyu9527
V2EX  ›  问与答

scrapy 下载卡住 如何设置超时?

  •  
  •   xiaoyu9527 · Aug 8, 2017 · 8596 views
    This topic created in 3291 days ago, the information mentioned may be changed or developed.

    正在下载 xxxx 正在保存 xxxx

    就卡住了

    这里写的是一个循环语句。

    1 replies    2017-08-11 14:08:00 +08:00
    860670496
        1
    860670496  
       Aug 11, 2017
    settings 里自带参数,比如:

    DOWNLOAD_TIMEOUT = 300

    如果觉得不能满足要求就写个继承 DownloadTimeoutMiddleware 的类就好

    https://doc.scrapy.org/en/1.4/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.downloadtimeout

    顺带蛋疼的话还可以利用下面这个类获取状态写进度条
    class scrapy.downloadermiddlewares.stats.DownloaderStats
    About   ·   Help   ·   Advertise   ·   Blog   ·   API   ·   FAQ   ·   Solana   ·   1582 Online   Highest 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 22ms · UTC 16:40 · PVG 00:40 · LAX 09:40 · JFK 12:40
    ♥ Do have faith in what you're doing.