python - Scrapy Crawling Speed is so Slow (6 pages / min)! -


i new scrapy , build project using scrapy startproject zhanlang. when start spider scrapy crawl zhanlang -o zhanlang.csv ,it works slowly!only 6pags/min! here code:

def after_login(self, response):     #the site should log in,this function todo after login     yield request(url="https://movie.douban.com/subject/26363254/comments?start=0&limit=20&sort=new_score&status=p",                      meta={'cookiejar': response.meta['cookiejar']},                      callback=self.parse                    )     def parse(self,response):     item = zhanlangitem()     comment in response.xpath('//div[@class="comment-item"]'):         item['name'] = comment.xpath('./div[@class="avatar"]/a/@title').extract_first()         item['text'] = comment.xpath('./div[@class="comment"]/p/text()').extract()         item['vote'] = comment.xpath('.//span[@class="votes"]/text()').extract_first()         yield item     next_page_url = response.xpath('//a[@class="next"]/@href').extract()[0]     next_page_url = "https://movie.douban.com/subject/26363254/comments"+next_page_url     if next_page_url not none:         print next_page_url         yield request(url=next_page_url,                     meta={'cookiejar': response.meta['cookiejar']},                      callback=self.parse                        )

here settings:

download_delay = 0.5 # download delay setting honor 1 of: concurrent_requests_per_domain = 16 #concurrent_requests_per_ip = 16  downloader_middlewares = {       'scrapy.downloadermiddlewares.useragent.useragentmiddleware':none,       'zhanlang.middlewares.randomuseragentmiddleware':400,   }

my middlewares.py are:

from fake_useragent import useragent import requests, random, json  import base64  class randomuseragentmiddleware(object): # random choice useragent     def __init__(self, crawler):         super(randomuseragentmiddleware, self).__init__()         self.ua = useragent()         self.ua_type = crawler.settings.get('random_ua_type', 'random')     @classmethod     def from_crawler(cls, crawler):         return cls(crawler)     def process_request(self, request, spider):         def get_ua():             return getattr(self.ua, self.ua_type)         request.headers.setdefault('user-agent', get_ua())

why crawls slowly? should increase speed?thanks


Comments

Post a Comment

Popular posts from this blog

Is there a better way to structure post methods in Class Based Views -

i'm using class based views in django , never specify url in ajax calls , omit action parameter in form because know these requests go through post method of class based view. notice view - profile page make many requests depending on user updating on page - post method become convoluted , if/else mess example: def post(self, request, *args, **kwargs): # handle user changing available date if request.post.get('availabledate') != none: self.updateavailabledate(request) return jsonresponse({'result':'success'}) if request.post.get('newprojectname') != none: creative_user = creativeuserprofile.objects.get(id = request.user.id) project = project.create_project(creative_user, request.post['newprojectname']) project.save() return jsonresponse({'projectid': project.id }) if request.post.get('projectimage') != none , request.files['file']: proj...
Read more

performance - Why is XCHG reg, reg a 3 micro-op instruction on modern Intel architectures? -

i'm doing micro-optimization on performance critical part of code , came across sequence of instructions (in at&t syntax): add %rax, %rbx mov %rdx, %rax mov %rbx, %rdx i thought had use case xchg allow me shave instruction , write: add %rbx, %rax xchg %rax, %rdx however, dimay found agner fog's instruction tables , xchg 3 micro-op instruction 2 cycle latency on sandy bridge, ivy bridge, broadwell, haswell , skylake. 3 whole micro-ops , 2 cycles of latency! 3 micro-ops throws off 4-1-1-1 cadence , 2 cycle latency makes worse original in best case since last 2 instructions in original might execute in parallel. now... cpu might breaking instruction micro-ops equivalent to: mov %rax, %tmp mov %rdx, %rax mov %tmp, %rdx where tmp anonymous internal register , suppose last 2 micro-ops run in parallel latency 2 cycles. given register renaming occurs on these micro-architectures, though, doesn't make sense me done way. why wouldn't register renamer ...
Read more

c# - Asp.net web api : redirect unauthorized requst to forbidden page -

im trying redirect unauthorized request forbidden page instead i'm getting forbidden page in response body , how can fix ? here's startup class : app.createperowincontext(storecontext.create); app.createperowincontext<applicationusermanager>(applicationusermanager.create); app.createperowincontext<applicationsigninmanager>(applicationsigninmanager.create); app.usecookieauthentication(new cookieauthenticationoptions { authenticationtype = defaultauthenticationtypes.applicationcookie, expiretimespan = timespan.fromdays(30), }); app.useexternalsignincookie(defaultauthenticationtypes.externalcookie); app.usetwofactorsignincookie(defaultauthenticationtypes.twofactorcookie, timespan.fromminutes(5)); app.usetwofactorrememberbrowsercookie(defaultauthenticationtypes.twofactorrememberbrowsercookie); the method im trying reach : [httpget] [authorize(roles = "admin")] public string getcurrentusername() { return userman...
Read more