一个神奇的蜘蛛?项目，适用于数据采集任务，源码结构简洁明了。
index页面示例：
---
项目地址https://github.com/lixi5338619/magical_spider
使用指南1、配置settings.py，启动flask服务
2、参考demo文件夹中的代码进行测试，主要通过runflow.py运行。
代码语言：javascript代码运行次数：0运行复制```javascript import requestshost = 'http://127.0.0.1:5000'def magical_start(project_name,base_url = 'https://www.php.cn/link/b6f05a7baab2fe0eea07e59bd5b0b317'): # 1、创建浏览器并选择session_id result = requests.post(f'{host}/create',data={'name':project_name,'url':base_url}).json() session_id,process_url = result['session_id'],result['process_url'] return session_id,process_urldef magical_request(session_id,process_url,request_url): # 2、请求浏览器_xhr data = {'session_id':session_id,'process_url':process_url, 'request_url':request_url,'request_type':'get'} result = requests.post(f'{host}/xhr',data=data).json() return result['result']def magical_close(session_id,process_url,process_name): # 4、关闭浏览器 close_data = {'session_id':session_id,'process_url':process_url,'process_name':process_name} requests.post(f'{host}/close',data=close_data).json()
<code>
3、测试代码
<p>GET请求</p><p>代码语言：javascript代码运行次数：0<svg fill="none" height="16" viewbox="0 0 16 16" width="16" xmlns="<a href="https://www.php.cn/link/c9041cfd2a40932691855abd98fd219a">http://www.w3.org/2000/svg"><path</a> d="M6.66666 10.9999L10.6667 7.99992L6.66666 4.99992V10.9999ZM7.99999 1.33325C4.31999 1.33325 1.33333 4.31992 1.33333 7.99992C1.33333 11.6799 4.31999 14.6666 7.99999 14.6666C11.68 14.6666 14.6667 11.6799 14.6667 7.99992C14.6667 4.31992 11.68 1.33325 7.99999 1.33325ZM7.99999 13.3333C5.05999 13.3333 2.66666 10.9399 2.66666 7.99992C2.66666 5.05992 5.05999 2.66659 7.99999 2.66659C10.94 2.66659 13.3333 5.05992 13.3333 7.99992C13.3333 10.9399 10.94 13.3333 7.99999 13.3333Z" fill="currentcolor"></path></svg>运行<svg fill="none" height="16" viewbox="0 0 16 16" width="16" xmlns="<a href="https://www.php.cn/link/c9041cfd2a40932691855abd98fd219a">http://www.w3.org/2000/svg"><path</a> clip-rule="evenodd" d="M4.5 15.5V3.5H14.5V15.5H4.5ZM12.5 5.5H6.5V13.5H12.5V5.5ZM9.5 2.5H3.5V12.5H1.5V0.5H11.5V2.5H9.5Z" fill="currentcolor" fill-rule="evenodd"></path></svg>复制<code>javascript from demo.runflow import magical_start,magical_request,magical_closeproject_name = 'cnipa'base_url = 'https://www.cnipa.gov.cn'session_id,process_url = magical_start(project_name,base_url)print(len(magical_request(session_id, process_url,'https://www.cnipa.gov.cn/col/col57/index.html')))magical_close(session_id,process_url,project_name)</code></pre><p>POST请求</p><p>代码语言：javascript代码运行次数：0<svg fill="none" height="16" viewbox="0 0 16 16" width="16" xmlns="http://www.w3.org/2000/svg"><path d="M6.66666 10.9999L10.6667 7.99992L6.66666 4.99992V10.9999ZM7.99999 1.33325C4.31999 1.33325 1.33333 4.31992 1.33333 7.99992C1.33333 11.6799 4.31999 14.6666 7.99999 14.6666C11.68 14.6666 14.6667 11.6799 14.6667 7.99992C14.6667 4.31992 11.68 1.33325 7.99999 1.33325ZM7.99999 13.3333C5.05999 13.3333 2.66666 10.9399 2.66666 7.99992C2.66666 5.05992 5.05999 2.66659 7.99999 2.66659C10.94 2.66659 13.3333 5.05992 13.3333 7.99992C13.3333 10.9399 10.94 13.3333 7.99999 13.3333Z" fill="currentcolor"></path></svg>运行<svg fill="none" height="16" viewbox="0 0 16 16" width="16" xmlns="http://www.w3.org/2000/svg"><path clip-rule="evenodd" d="M4.5 15.5V3.5H14.5V15.5H4.5ZM12.5 5.5H6.5V13.5H12.5V5.5ZM9.5 2.5H3.5V12.5H1.5V0.5H11.5V2.5H9.5Z" fill="currentcolor" fill-rule="evenodd"></path></svg>复制</code>javascript
from demo.runflow import magical_start,magical_request,magical_closeimport jsonproject_name = 'chinadrugtrials'base_url = '<a href="https://www.php.cn/link/1751a9a215df522dcf207ba51b327643"><a href="https://www.php.cn/link/1751a9a215df522dcf207ba51b327643">http://www.chinadrugtrials.org.cn'session_id,process_url</a></a> = magical_start(project_name,base_url)data = {"id": "","ckm_index": "","sort": "desc","sort2": "","rule": "CTR","secondLevel": "0","currentpage": "2","keywords": "","reg_no": "","indication": "","case_no": "","drugs_name": "","drugs_type": "","appliers": "","communities": "","researchers": "","agencies": "","state": ""}formdata = json.dumps(data)print(magical_request(session_id=session_id, process_url=process_url, request_url='<a href="https://www.php.cn/link/44f95c37a24495521a98b63f0bbf4268"><a href="https://www.php.cn/link/44f95c37a24495521a98b63f0bbf4268">https://www.php.cn/link/44f95c37a24495521a98b63f0bbf4268</a></a>', request_type='post',formdata=formdata ))magical_close(session_id,process_url,project_name)</p><pre class="brush:php;toolbar:false;"><code></p><p>4、index页面可以查看和管理当前运行中的任务，同时可以查看系统的内存和磁盘使用情况。</p><p>5、demo文件夹包含任务流程汇总runflow.py，以及抖音和药监局的案例，提供了单任务和多任务的示例。</p><hr /><p>linux部署1.安装chrome（选择安装位置） yum install <a href="https://www.php.cn/link/6b17d006a2ed6f12f07c7ea60b8002b5">https://www.php.cn/link/6b17d006a2ed6f12f07c7ea60b8002b5</a></p>
<div class="aritcle_card">
<a class="aritcle_card_img" href="/ai/1607">
<img src="https://img.php.cn/upload/ai_manual/000/969/633/68b6dac7dd9e4837.png" alt="集简云">
</a>
<div class="aritcle_card_info">
<a href="/ai/1607">集简云</a>
<p>软件集成平台，快速建立企业自动化与智能化</p>
<div class="">
<img src="/static/images/card_xiazai.png" alt="集简云">
<span>22</span>
</div>
</div>
<a href="/ai/1607" class="aritcle_card_btn">
<span>查看详情</span>
<img src="/static/images/cardxiayige-3.png" alt="集简云">
</a>
</div>
<p>2.检查chrome版本 google-chrome --version</p><p>3.安装对应版本的chromedriver_linux64 例如，我的chrome版本是104.0.5112.79 wget <a href="https://www.php.cn/link/5bb5f8d030f5e6e4b6d137c6990f0772">https://www.php.cn/link/5bb5f8d030f5e6e4b6d137c6990f0772</a></p><p>4.解压chromedriver unzip chromedriver_linux64</p><p>5.授权chromedriver chmod 777 chromedriver</p><p>6.修改项目代码settings.py中的chromedriver路径</p><p>7.安装python依赖并启动flask项目</p><p>Python依赖：flask、sqlite3、selenium、websockets、opencv-python、numpyflask启动方式：python3 server.py8.开启服务器端口访问权限</p><p>9.运行项目测试</code>以上就是magical_spider远程采集方案的详细内容，更多请关注php中文网其它相关文章！
每个人都需要一台速度更快、更稳定的 PC。随着时间的推移，垃圾文件、旧注册表数据和不必要的后台进程会占用资源并降低性能。幸运的是，许多工具可以让 Windows 保持平稳运行。
广告
收藏
收藏
收藏
Copyright 2014-2025 https://www.php.cn/ All Rights Reserved | php.cn | 湘ICP备2023035733号
403
