如何编写 Python 程序爬取新浪军事论坛？-Python教程-PHP中文网

如何编写 Python 程序爬取新浪军事论坛？

php中文网

发布： 2016-06-06 16:22:00

原创

1841人浏览过

回复内容：

context_re = r'(.*?)'

你准备的这个正则表达式啊，truncated！断在了
这里，所以只能爬第一段。

爬取新浪军事论坛需要做三件事：

一、

上CSDN汪海老师的专栏，http://blog.csdn.net/column/details/why-bug.html，学习一个。

二、

按F12看一下前端。

三、

<span class="kn">from</span> <span class="nn">bs4</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span>
<span class="kn">import</span> <span class="nn">requests</span>

<span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">"http://club.mil.news.sina.com.cn/thread-666013-1-1.html?retcode=0"</span><span class="p">)</span> <span class="c">#硬点网址</span>
<span class="n">response</span><span class="o">.</span><span class="n">encoding</span> <span class="o">=</span> <span class="s">'gb18030'</span> <span class="c">#中文编码</span>
<span class="n">soup</span> <span class="o">=</span> <span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">text</span><span class="p">,</span> <span class="s">'html.parser'</span><span class="p">)</span> <span class="c">#构建BeautifulSoup对象</span>


<span class="n">divs</span> <span class="o">=</span> <span class="n">soup</span><span class="p">(</span><span class="s">'div'</span><span class="p">,</span> <span class="s">'mainbox'</span><span class="p">)</span>
 <span class="c">#每个楼层</span>

<span class="k">for</span> <span class="n">div</span> <span class="ow">in</span> <span class="n">divs</span><span class="p">:</span>
    <span class="n">comments</span> <span class="o">=</span> <span class="n">div</span><span class="o">.</span><span class="n">find_all</span><span class="p">(</span><span class="s">'div'</span><span class="p">,</span><span class="s">'cont f14'</span><span class="p">)</span> <span class="c">#每个楼层的正文</span>
        
    <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'Sina_Military_Club.txt'</span><span class="p">,</span><span class="s">'a'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
        <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s">'</span><span class="se">\n</span><span class="s">'</span><span class="o">+</span><span class="nb">str</span><span class="p">(</span><span class="n">comments</span><span class="p">)</span><span class="o">+</span><span class="s">'</span><span class="se">\n</span><span class="s">'</span><span class="p">)</span>

登录后复制

刚好几个小时前就在写一个爬取网站会员（公司）资料的小程序
具体的编程问题就不回答了，跟用什么语言写代码无关，关键是你要分析好这个页面的html代码结构，写出合适的正则表达式来进行匹配，如果想简化的话，可以进行分次匹配（比如先得到

里面的第一个

里面的内容就是原帖的地址，然后再进一步处理）
大数据分析就不会了，还请赐教。

<span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">bs4</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span>

<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">"http://club.mil.news.sina.com.cn/thread-666013-1-1.html"</span><span class="p">)</span>
<span class="n">r</span><span class="o">.</span><span class="n">encoding</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">apparent_encoding</span>
<span class="n">soup</span> <span class="o">=</span> <span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">soup</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">attrs</span><span class="o">=</span><span class="p">{</span><span class="s">"class"</span><span class="p">:</span> <span class="s">"cont f14"</span><span class="p">})</span>
<span class="k">print</span> <span class="n">result</span><span class="o">.</span><span class="n">text</span>

登录后复制

用beautifulSoup吧,正则太多了看着都头疼. 先用了BeautifulSoup爬取数据

<span class="c"># -*- coding:utf-8 -*-</span>

<span class="kn">import</span> <span class="nn">re</span><span class="o">,</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">bs4</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="nb">reload</span><span class="p">(</span><span class="n">sys</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">setdefaultencoding</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">)</span>

<span class="n">url</span> <span class="o">=</span> <span class="s">"http://club.mil.news.sina.com.cn/viewthread.php?tid=666013&extra=page%3D1&page=1"</span>

<span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="n">req</span><span class="o">.</span><span class="n">encoding</span> <span class="o">=</span> <span class="n">req</span><span class="o">.</span><span class="n">apparent_encoding</span>
<span class="n">html</span> <span class="o">=</span> <span class="n">req</span><span class="o">.</span><span class="n">text</span>

<span class="n">soup</span> <span class="o">=</span> <span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">html</span><span class="p">)</span>

<span class="nb">file</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s">'sina_club.txt'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">for</span> <span class="n">tag</span> <span class="ow">in</span> <span class="n">soup</span><span class="o">.</span><span class="n">find_all</span><span class="p">(</span><span class="s">'div'</span><span class="p">,</span> <span class="n">attrs</span> <span class="o">=</span> <span class="p">{</span><span class="s">'class'</span><span class="p">:</span> <span class="s">"cont f14"</span><span class="p">}):</span>
    <span class="n">word</span> <span class="o">=</span> <span class="n">tag</span><span class="o">.</span><span class="n">get_text</span><span class="p">()</span>
    <span class="n">line1</span> <span class="o">=</span> <span class="s">"---------------评论"</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">+</span> <span class="s">"---------------------"</span> <span class="o">+</span> <span class="s">"</span><span class="se">\n</span><span class="s">"</span>
    <span class="n">line2</span> <span class="o">=</span> <span class="n">word</span> <span class="o">+</span> <span class="s">"</span><span class="se">\n</span><span class="s">"</span>
    <span class="n">line</span> <span class="o">=</span> <span class="n">line1</span> <span class="o">+</span> <span class="n">line2</span>
    <span class="n">x</span> <span class="o">+=</span> <span class="mi">1</span>
    <span class="nb">file</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="nb">file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>

登录后复制

哎，扒就扒吧，发了paper能不能告诉我刊号页数让我看一下？我们自己都没做大数据分析…… 建议用一下正则测试工具你需要pyquery，可以使用jquery一样的语法。你值得拥有。
https://pythonhosted.org/pyquery/