Python爬蟲之爬取我愛我家二手房數(shù)據(jù)
更新時間：2021年05月28日 10:55:28 作者：勤奮的清風(fēng)
我愛我家的數(shù)據(jù)相對來說抓取難度不大,基本無反爬措施.
但若按照規(guī)則構(gòu)造頁面鏈接進(jìn)行抓取,會出現(xiàn)部分頁面無法獲取到數(shù)據(jù)的情況.在網(wǎng)上看了幾個博客,基本上都是較為簡單的獲取數(shù)據(jù),未解決這個問題,在實際應(yīng)用中會出錯,本文有非常詳細(xì)的代碼示例,需要的朋友可以參考下
一、問題說明
首先，運行下述代碼，復(fù)現(xiàn)問題：
# -*-coding:utf-8-*-
import re
import requests
from bs4 import BeautifulSoup
cookie = 'PHPSESSID=aivms4ufg15sbrj0qgboo3c6gj; HMF_CI=4d8ff20092e9832daed8fe5eb0475663812603504e007aca93e6630c00b84dc207; _ga=GA1.2.556271139.1620784679; gr_user_id=4c878c8f-406b-46a0-86ee-a9baf2267477; _dx_uzZo5y=68b673b0aaec1f296c34e36c9e9d378bdb2050ab4638a066872a36f781c888efa97af3b5; smidV2=20210512095758ff7656962db3adf41fa8fdc8ddc02ecb00bac57209becfaa0; yfx_c_g_u_id_10000001=_ck21051209583410015104784406594; __TD_deviceId=41HK9PMCSF7GOT8G; zufang_cookiekey=["%7B%22url%22%3A%22%2Fzufang%2F_%25E9%2595%25BF%25E6%2598%25A5%25E6%25A1%25A5%3Fzn%3D%25E9%2595%25BF%25E6%2598%25A5%25E6%25A1%25A5%22%2C%22x%22%3A%220%22%2C%22y%22%3A%220%22%2C%22name%22%3A%22%E9%95%BF%E6%98%A5%E6%A1%A5%22%2C%22total%22%3A%220%22%7D","%7B%22url%22%3A%22%2Fzufang%2F_%25E8%258B%258F%25E5%25B7%259E%25E8%25A1%2597%3Fzn%3D%25E8%258B%258F%25E5%25B7%259E%25E8%25A1%2597%22%2C%22x%22%3A%220%22%2C%22y%22%3A%220%22%2C%22name%22%3A%22%E8%8B%8F%E5%B7%9E%E8%A1%97%22%2C%22total%22%3A%220%22%7D","%7B%22url%22%3A%22%2Fzufang%2F_%25E8%258B%258F%25E5%25B7%259E%25E6%25A1%25A5%3Fzn%3D%25E8%258B%258F%25E5%25B7%259E%25E6%25A1%25A5%22%2C%22x%22%3A%220%22%2C%22y%22%3A%220%22%2C%22name%22%3A%22%E8%8B%8F%E5%B7%9E%E6%A1%A5%22%2C%22total%22%3A%220%22%7D"]; ershoufang_cookiekey=["%7B%22url%22%3A%22%2Fzufang%2F_%25E9%2595%25BF%25E6%2598%25A5%25E6%25A1%25A5%3Fzn%3D%25E9%2595%25BF%25E6%2598%25A5%25E6%25A1%25A5%22%2C%22x%22%3A%220%22%2C%22y%22%3A%220%22%2C%22name%22%3A%22%E9%95%BF%E6%98%A5%E6%A1%A5%22%2C%22total%22%3A%220%22%7D","%7B%22url%22%3A%22%2Fershoufang%2F_%25E8%258B%258F%25E5%25B7%259E%25E6%25A1%25A5%3Fzn%3D%25E8%258B%258F%25E5%25B7%259E%25E6%25A1%25A5%22%2C%22x%22%3A%220%22%2C%22y%22%3A%220%22%2C%22name%22%3A%22%E8%8B%8F%E5%B7%9E%E6%A1%A5%22%2C%22total%22%3A%220%22%7D"]; zufang_BROWSES=501465046,501446051,90241951,90178388,90056278,90187979,501390110,90164392,90168076,501472221,501434480,501480593,501438374,501456072,90194547,90223523,501476326,90245144; historyCity=["\u5317\u4eac"]; _gid=GA1.2.23153704.1621410645; Hm_lvt_94ed3d23572054a86ed341d64b267ec6=1620784715,1621410646; _Jo0OQK=4958FA78A5CC420C425C480565EB46670E81832D8173C5B3CFE61303A51DE43E320422D6C7A15892C5B8B66971ED1B97A7334F0B591B193EBECAAB0E446D805316B26107A0B847CA53375B268E06EC955BB75B268E06EC955BB9D992FB153179892GJ1Z1OA==; ershoufang_BROWSES=501129552; domain=bj; 8fcfcf2bd7c58141_gr_session_id=61676ce2-ea23-4f77-8165-12edcc9ed902; 8fcfcf2bd7c58141_gr_session_id_61676ce2-ea23-4f77-8165-12edcc9ed902=true; yfx_f_l_v_t_10000001=f_t_1620784714003__r_t_1621471673953__v_t_1621474304616__r_c_2; Hm_lpvt_94ed3d23572054a86ed341d64b267ec6=1621475617'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36',
'Cookie': cookie.encode("utf-8").decode("latin1")
}
def run():
base_url = 'https://bj.5i5j.com/ershoufang/xichengqu/n%d/'
for page in range(1, 11):
url = base_url % page
print(url)
html = requests.get(url, headers=headers).text
soup = BeautifulSoup(html, 'lxml')
try:
for li in soup.find('div', class_='list-con-box').find('ul', class_='pList').find_all('li'):
title = li.find('h3', class_='listTit').get_text() # 名稱
# print(title)
except Exception as e:
print(e)
print(html)
break
if __name__ == '__main__':
run()
運行后會發(fā)現(xiàn)，在抓取https://bj.5i5j.com/ershoufang/xichengqu/n1/(也可能是其他頁碼)時，會報錯：'NoneType' object has no attribute 'find'，觀察輸出的html信息，可以發(fā)現(xiàn)html內(nèi)容為：<HTML><HEAD><script>window.location. rel="external nofollow" ;</script></HEAD><BODY>，但此鏈接在瀏覽器訪問是可以看到數(shù)據(jù)的，但鏈接會被重定向，重定向后的url即為上面這個html的href內(nèi)容。因此，可以合理的推斷，針對部分頁碼鏈接，我愛我家不會直接返回數(shù)據(jù)，但會返回帶有正確鏈接的信息，通過正則表達(dá)式獲取該鏈接即可正確抓取數(shù)據(jù)。
二、解決方法
在下面的完整代碼中，采取的解決方法是：
1.首先判斷當(dāng)前html是否含有數(shù)據(jù)
2.若無數(shù)據(jù)，則通過正則表達(dá)式獲取正確鏈接
3.重新獲取html數(shù)據(jù)
if '<HTML><HEAD><script>window.location.href=' in html: url = re.search(r'.*?href="(.+)" rel="external nofollow" rel="external nofollow" .*?', html).group(1) html = requests.get(url, headers=headers).text
三、完整代碼
# -*-coding:utf-8-*-
import os
import re
import requests
import csv
import time
from bs4 import BeautifulSoup
folder_path = os.path.split(os.path.abspath(__file__))[0] + os.sep # 獲取當(dāng)前文件所在目錄
cookie = 'PHPSESSID=aivms4ufg15sbrj0qgboo3c6gj; HMF_CI=4d8ff20092e9832daed8fe5eb0475663812603504e007aca93e6630c00b84dc207; _ga=GA1.2.556271139.1620784679; gr_user_id=4c878c8f-406b-46a0-86ee-a9baf2267477; _dx_uzZo5y=68b673b0aaec1f296c34e36c9e9d378bdb2050ab4638a066872a36f781c888efa97af3b5; smidV2=20210512095758ff7656962db3adf41fa8fdc8ddc02ecb00bac57209becfaa0; yfx_c_g_u_id_10000001=_ck21051209583410015104784406594; __TD_deviceId=41HK9PMCSF7GOT8G; zufang_cookiekey=["%7B%22url%22%3A%22%2Fzufang%2F_%25E9%2595%25BF%25E6%2598%25A5%25E6%25A1%25A5%3Fzn%3D%25E9%2595%25BF%25E6%2598%25A5%25E6%25A1%25A5%22%2C%22x%22%3A%220%22%2C%22y%22%3A%220%22%2C%22name%22%3A%22%E9%95%BF%E6%98%A5%E6%A1%A5%22%2C%22total%22%3A%220%22%7D","%7B%22url%22%3A%22%2Fzufang%2F_%25E8%258B%258F%25E5%25B7%259E%25E8%25A1%2597%3Fzn%3D%25E8%258B%258F%25E5%25B7%259E%25E8%25A1%2597%22%2C%22x%22%3A%220%22%2C%22y%22%3A%220%22%2C%22name%22%3A%22%E8%8B%8F%E5%B7%9E%E8%A1%97%22%2C%22total%22%3A%220%22%7D","%7B%22url%22%3A%22%2Fzufang%2F_%25E8%258B%258F%25E5%25B7%259E%25E6%25A1%25A5%3Fzn%3D%25E8%258B%258F%25E5%25B7%259E%25E6%25A1%25A5%22%2C%22x%22%3A%220%22%2C%22y%22%3A%220%22%2C%22name%22%3A%22%E8%8B%8F%E5%B7%9E%E6%A1%A5%22%2C%22total%22%3A%220%22%7D"]; ershoufang_cookiekey=["%7B%22url%22%3A%22%2Fzufang%2F_%25E9%2595%25BF%25E6%2598%25A5%25E6%25A1%25A5%3Fzn%3D%25E9%2595%25BF%25E6%2598%25A5%25E6%25A1%25A5%22%2C%22x%22%3A%220%22%2C%22y%22%3A%220%22%2C%22name%22%3A%22%E9%95%BF%E6%98%A5%E6%A1%A5%22%2C%22total%22%3A%220%22%7D","%7B%22url%22%3A%22%2Fershoufang%2F_%25E8%258B%258F%25E5%25B7%259E%25E6%25A1%25A5%3Fzn%3D%25E8%258B%258F%25E5%25B7%259E%25E6%25A1%25A5%22%2C%22x%22%3A%220%22%2C%22y%22%3A%220%22%2C%22name%22%3A%22%E8%8B%8F%E5%B7%9E%E6%A1%A5%22%2C%22total%22%3A%220%22%7D"]; zufang_BROWSES=501465046,501446051,90241951,90178388,90056278,90187979,501390110,90164392,90168076,501472221,501434480,501480593,501438374,501456072,90194547,90223523,501476326,90245144; historyCity=["\u5317\u4eac"]; _gid=GA1.2.23153704.1621410645; Hm_lvt_94ed3d23572054a86ed341d64b267ec6=1620784715,1621410646; _Jo0OQK=4958FA78A5CC420C425C480565EB46670E81832D8173C5B3CFE61303A51DE43E320422D6C7A15892C5B8B66971ED1B97A7334F0B591B193EBECAAB0E446D805316B26107A0B847CA53375B268E06EC955BB75B268E06EC955BB9D992FB153179892GJ1Z1OA==; ershoufang_BROWSES=501129552; domain=bj; 8fcfcf2bd7c58141_gr_session_id=61676ce2-ea23-4f77-8165-12edcc9ed902; 8fcfcf2bd7c58141_gr_session_id_61676ce2-ea23-4f77-8165-12edcc9ed902=true; yfx_f_l_v_t_10000001=f_t_1620784714003__r_t_1621471673953__v_t_1621474304616__r_c_2; Hm_lpvt_94ed3d23572054a86ed341d64b267ec6=1621475617'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36',
'Cookie': cookie.encode("utf-8").decode("latin1")
}
def get_page(url):
"""獲取網(wǎng)頁原始數(shù)據(jù)"""
global headers
html = requests.get(url, headers=headers).text
return html
def extract_info(html):
"""解析網(wǎng)頁數(shù)據(jù)，抽取出房源相關(guān)信息"""
host = 'https://bj.5i5j.com'
soup = BeautifulSoup(html, 'lxml')
data = []
for li in soup.find('div', class_='list-con-box').find('ul', class_='pList').find_all('li'):
try:
title = li.find('h3', class_='listTit').get_text() # 名稱
url = host + li.find('h3', class_='listTit').a['href'] # 鏈接
info_li = li.find('div', class_='listX') # 每個房源核心信息都在這里
p1 = info_li.find_all('p')[0].get_text() # 獲取第一段
info1 = [i.strip() for i in p1.split(' · ')]
# 戶型、面積、朝向、樓層、裝修、建成時間
house_type, area, direction, floor, decoration, build_year = info1
p2 = info_li.find_all('p')[1].get_text() # 獲取第二段
info2 = [i.replace(' ', '') for i in p2.split('·')]
# 小區(qū)、位于幾環(huán)、交通信息
if len(info2) == 2:
residence, ring = info2
transport = '' # 部分房源無交通信息
elif len(info2) == 3:
residence, ring, transport = info2
else:
residence, ring, transport = ['', '', '']
p3 = info_li.find_all('p')[2].get_text() # 獲取第三段
info3 = [i.replace(' ', '') for i in p3.split('·')]
# 關(guān)注人數(shù)、帶看次數(shù)、發(fā)布時間
try:
watch, arrive, release_year = info3
except Exception as e:
print(info2, '獲取帶看、發(fā)布日期信息出錯')
watch, arrive, release_year = ['', '', '']
total_price = li.find('p', class_='redC').get_text().strip() # 房源總價
univalence = li.find('div', class_='jia').find_all('p')[1].get_text().replace('單價', '') # 房源單價
else_info = li.find('div', class_='listTag').get_text()
data.append([title, url, house_type, area, direction, floor, decoration, residence, ring,
transport, total_price, univalence, build_year, release_year, watch, arrive, else_info])
except Exception as e:
print('extract_info: ', e)
return data
def crawl():
esf_url = 'https://bj.5i5j.com/ershoufang/' # 主頁網(wǎng)址
fields = ['城區(qū)', '名稱', '鏈接', '戶型', '面積', '朝向', '樓層', '裝修', '小區(qū)', '環(huán)', '交通情況', '總價', '單價',
'建成時間', '發(fā)布時間', '關(guān)注', '帶看', '其他信息']
f = open(folder_path + 'data' + os.sep + '北京二手房-我愛我家.csv', 'w', newline='', encoding='gb18030')
writer = csv.writer(f, delimiter=',') # 以逗號分割
writer.writerow(fields)
page = 1
regex = re.compile(r'.*?href="(.+)" rel="external nofollow" rel="external nofollow" .*?')
while True:
url = esf_url + 'n%s/' % page # 構(gòu)造頁面鏈接
if page == 1:
url = esf_url
html = get_page(url)
# 部分頁面鏈接無法獲取數(shù)據(jù)，需進(jìn)行判斷，并從返回html內(nèi)容中獲取正確鏈接，重新獲取html
if '<HTML><HEAD><script>window.location.href=' in html:
url = regex.search(html).group(1)
html = requests.get(url, headers=headers).text
print(url)
data = extract_info(html)
if data:
writer.writerows(data)
page += 1
f.close()
if __name__ == '__main__':
crawl() # 啟動爬蟲
四、數(shù)據(jù)展示
截至2021年5月23日，共獲取數(shù)據(jù)62943條，基本上將我愛我家官網(wǎng)上北京地區(qū)的二手房數(shù)據(jù)全部抓取下來了。
到此這篇關(guān)于Python爬蟲之爬取我愛我家二手房數(shù)據(jù)的文章就介紹到這了,更多相關(guān)Python爬取二手房數(shù)據(jù)內(nèi)容請搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家！
相關(guān)文章
對于list，由于她的確非常非常龐雜，在python中應(yīng)用非常廣泛，所以，雖然已經(jīng)介紹完畢了基礎(chǔ)內(nèi)容，這里還要用一講深入一點點，往往越深入越...2014-09-09
文件名中卻都含有xxx有聲下載，使用腳本將其去掉。腳本練習(xí)了os.rename重命名方法，str.partition方法使用， 及正則match，search方法區(qū)別2014-01-01
這篇文章主要介紹了用python制作北上廣深——地鐵線路動態(tài)圖，文中的示例代碼講解詳細(xì)，對我們的工作或?qū)W習(xí)都有一定的價值，感興趣的同學(xué)可以學(xué)習(xí)一下2021-12-12
這篇文章主要介紹了python通過文件頭判斷文件類型，需要的朋友可以參考下2015-10-10
Tensorflow tensor 數(shù)學(xué)運算和邏輯運算方式
這篇文章主要介紹了Tensorflow tensor 數(shù)學(xué)運算和邏輯運算方式，具有很好的參考價值，希望對大家有所幫助。一起跟隨小編過來看看吧2020-06-06
unittest庫unittest庫是python的內(nèi)置庫,用來對程序進(jìn)行測試,下面這篇文章主要給大家介紹了關(guān)于python中單元測試框架unittest基本用法的相關(guān)資料,需要的朋友可以參考下2022-09-09
這篇文章主要介紹了Python with語句和過程抽取思想,本文給大家介紹的非常詳細(xì)，具有一定的參考借鑒價值，需要的朋友可以參考下2019-12-12
Python matplotlib實現(xiàn)條形統(tǒng)計圖
這篇文章主要為大家詳細(xì)介紹了Python matplotlib實現(xiàn)條形統(tǒng)計圖，文中示例代碼介紹的非常詳細(xì)，具有一定的參考價值，感興趣的小伙伴們可以參考一下2022-04-04
這篇文章主要介紹了python多線程互斥鎖與死鎖，多線程間的資源競爭下文我們就以task1()，task2()兩個函數(shù)為例，分別將對全局變量num加一重復(fù)一千萬次循環(huán)，具有一定得參考價值，需要的小伙伴可以參考一下2022-02-02
這篇文章主要介紹了python使用線程封裝的一個簡單定時器類,實例分析了Python線程的使用及定時器類的實現(xiàn)技巧,需要的朋友可以參考下2015-05-05