利用OpenReview API与网页抓取获取会议论文数据教程

pip install openreview

import openreview

# 实例化新的OpenReview API客户端
# 注意：baseurl应指向api2.openreview.net
client = openreview.api.OpenReviewClient(baseurl='https://api2.openreview.net')

# 获取指定会议的所有投稿
# 'venueid'是识别会议的关键，格式通常为 '会议名称.cc/年份/Conference'
submissions = client.get_all_notes(content={'venueid':'NeurIPS.cc/2023/Conference'})

# 从投稿对象中提取论文标题
papers_titles = [s.content['title']['value'] for s in submissions]

# 打印前10个标题进行验证
print("NeurIPS 2023 论文标题（前10个）:")
for title in papers_titles[:10]:
    print(f"- {title}")

NeurIPS 2023 论文标题（前10个）:
- Online PCA in Converging Self-consistent Field Equations
- Don’t blame Dataset Shift! Shortcut Learning due to Gradients and Cross Entropy
- On Slicing Optimality for Mutual Information
- k-Median Clustering via Metric Embedding: Towards Better Initialization with Differential Privacy
- Information Maximization Perspective of Orthogonal Matching Pursuit with Applications to Explainable AI
- STEVE-1: A Generative Model for Text-to-Behavior in Minecraft
- AMAG: Additive, Multiplicative and Adaptive Graph Neural Network For Forecasting Neuron Activity
- Conditional Matrix Flows for Gaussian Graphical Models
- Representational Strengths and Limitations of Transformers
- Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small Scorer

pip install requests lxml

import requests
from lxml.html import fromstring

# 目标网页URL
url = 'https://openaccess.thecvf.com/CVPR2023?day=all'

# 发送HTTP GET请求获取网页内容
response_text = requests.get(url).text

# 使用lxml解析HTML内容
response_html = fromstring(response_text)

# 使用XPath表达式选择论文标题元素
# 这里的XPath '/html/body/dl/dt/a/text()' 是根据目标网页的HTML结构确定的
# 它定位到 <dt> 标签下的 <a> 标签的文本内容
elements = response_html.xpath('//*[@id="content"]/dl/dt/a/text()')

# 打印前10个标题进行验证
print("\nCVPR 2023 论文标题（前10个）:")
for title in elements[:10]:
    print(f"- {title}")

CVPR 2023 论文标题（前10个）:
- GFPose: Learning 3D Human Pose Prior With Gradient Fields
- CXTrack: Improving 3D Point Cloud Tracking With Contextual Information
- Deep Frequency Filtering for Domain Generalization
- Frame Flexible Network
- Unsupervised Cumulative Domain Adaptation for Foggy Scene Optical Flow
- NoisyTwins: Class-Consistent and Diverse Image Generation Through StyleGANs
- DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-Aware Scene Synthesis
- Revisiting Self-Similarity: Structural Embedding for Image Retrieval
- Minimizing the Accumulated Trajectory Error To Improve Dataset Distillation
- Decoupling-and-Aggregating for Image Exposure Correction