用Scrapy訂製自己的爬蟲抓取數據

bestlong

Scrapy（http://scrapy.org/）是一个快速的高级别网页抓取框架，可用来抓取网站并从中提取其中结构化数据。它可用于数据挖掘数据监测和自动化测试等等。它是一套基于基于Twisted的纯python实现的爬虫框架

首先安装框架所需环境

1. apt-get install python-twisted python-libxml2
   
2. apt-get install python-pyopenssl python-simplejson

複製代碼

再安装Scrapy

1. wget http://scrapy.org/releases/0.9/Scrapy-0.9.tar.gz
   
2. tar zxf scrapy-X.X.X.tar.gz
   
3. cd scrapy-X.X.X
   
4. python setup.py install

複製代碼

其他系统其他安装方式看http://doc.scrapy.org/intro/install.html

例如面对下面的页面数据：

1. <table width="100%" border="0" align="center" cellpadding="0" cellspacing="0" style="margin-top:15px;" class="datatable">
   
2. <tr>
   
3.         <th width=106><div align="center">名称</div></th>
   
4.         <th width=87><div align="center" class='red'>价格</div></th>
   
5.         <th width=208><div align="center">描述</div></th>
   
6. </tr>
   
7. <tr>
   
8.         <td width=106><div align='center'>产品名称</div></td>
   
9.         <td width=87 class='red'><div align='left'>产品价格</div></td>
   
10.         <td width=208><div align='left'>产品描述</div></td></td>
    
11. </tr>
    
12. <tr>
    
13.         <td width=106><div align='center'>产品名称</div></td>
    
14.         <td width=87 class='red'><div align='left'>产品价格</div></td>
    
15.         <td width=208><div align='left'>产品描述</div></td>
    
16. </tr>
    
17. ....
    
18. </table>

複製代碼

新建一个Scrapy项目

1. python scrapy-ctl.py startproject nuxnu

複製代碼

修改/nuxnu/nuxnu/items.py为：

1. # Define here the models for your scraped items
   
2. #
   
3. # See documentation in:
   
4. # http://doc.scrapy.org/topics/items.html
   
5. 
   
6. from scrapy.item import Item, Field
   
7. 
   
8. class NuxnuItem(Item):
   
9.     # define the fields for your item here like:
   
10.         name = Field()
    
11.         price = Field()
    
12.         desc = Field()

複製代碼

在/nuxnu/nuxnu/spider/目录下新建爬虫程序nuxnu_spider.py

1. from scrapy.spider import BaseSpider
   
2. from scrapy.selector import HtmlXPathSelector
   
3. 
   
4. from nuxnu.items import NuxnuItem
   
5. 
   
6. class NuxnuSpider(BaseSpider):
   
7.         name = "xxx.xxx.xxx"
   
8.         start_urls = [
   
9.                 "http://xxx.xxx.xxx/date.html"
   
10.         ]
    
11. 
    
12.         def parse(self,response):
    
13.                 hxs = HtmlXPathSelector (response)
    
14.                 sites = hxs.select('//table[@class="datatable"]/tr')
    
15.                 items = []
    
16.                 for site in sites:
    
17.                         item = NuxnuItem()
    
18.                         item['name'] = site.select('td[1]/div/text()').extract()
    
19.                         item['price'] = site.select('td[2]/div/text()').extract()
    
20.                         item['desc'] = site.select('td[3]/div/text()').extract()
    
21. 
    
22.                         items.append(item)
    
23.                 return items
    
24. 
    
25. SPIDER = NuxnuSpider()

複製代碼

注：xxx.xxx.xxx为我们要抓取的站点，而上面那个表格中的数据为http://xxx.xxx.xxx/date.html页的内容。

之后再修改/nuxnu/nuxnu/pipelines.py

1. # Define your item pipelines here
   
2. #
   
3. # Don't forget to add your pipeline to the ITEM_PIPELINES setting
   
4. # See: http://doc.scrapy.org/topics/item-pipeline.html
   
5. 
   
6. from os import path
   
7. from scrapy.core import signals
   
8. from scrapy.xlib.pydispatch import dispatcher
   
9. 
   
10. class StorePipeline(object):
    
11.     filename = 'flower.list'
    
12.     def __init__(self):
    
13.         self.f = None
    
14.         dispatcher.connect(self.open, signals.engine_started)
    
15.         dispatcher.connect(self.close, signals.engine_stopped)
    
16. 
    
17.     def process_item(self, domain, item):
    
18.         self.f.write(str(item['name'])+ '\n')
    
19.         return item
    
20. 
    
21.     def open(self):
    
22.         if path.exists(self.filename):
    
23.             self.f = open(self.filename, 'a')
    
24.         else:
    
25.             self.f = open(self.filename, 'w')
    
26. 
    
27.     def close(self):
    
28.         self.f.close() if self.f is not None else None

複製代碼

修改/nuxnu/nuxnu/settings.py

1. # Scrapy settings for nuxnu project
   
2. #
   
3. # For simplicity, this file contains only the most important settings by
   
4. # default. All the other settings are documented here:
   
5. #
   
6. #     http://doc.scrapy.org/topics/settings.html
   
7. #
   
8. # Or you can copy and paste them from where they're defined in Scrapy:
   
9. #
   
10. #     scrapy/conf/default_settings.py
    
11. #
    
12. 
    
13. BOT_NAME = 'nuxnu'
    
14. BOT_VERSION = '1.0'
    
15. 
    
16. SPIDER_MODULES = ['nuxnu.spiders']
    
17. NEWSPIDER_MODULE = 'nuxnu.spiders'
    
18. DEFAULT_ITEM_CLASS = 'nuxnu.items.NuxnuItem'
    
19. USER_AGENT = '%s/%s' % (BOT_NAME, BOT_VERSION)
    
20. ITEM_PIPELINES = ['nuxnu.pipelines.StorePipeline']

複製代碼

剩下来的就是运行了，回到/nuxnu/目录下

1. python scrapy-ctl.py crawl xxx.xxx.xxx

複製代碼

OK了。

具体的 Tutorial 和文档看：
http://doc.scrapy.org/intro/tutorial.html
http://doc.scrapy.org/index.html


參考資料 用Scrapy訂製自己的爬蟲抓取數據