Python中解析多行缩进文本元数据：利用正则表达式高效提取键值对

import requests

url = 'https://bioconductor.org/packages/release/bioc/VIEWS'
response = requests.get(url)
data = response.text

# 尝试通过双换行符分割每个元数据块
package_list = data.split('

')

# 错误示例：这种方法无法处理多行值
# package_dict = {
#     package_chunk.split(':')[0]: package_chunk.split(':')[1]
#     for package_chunk in package_list if ':' in package_chunk
# }
# print(package_dict)

import re
import requests

# 目标URL
url = "https://bioconductor.org/packages/release/bioc/VIEWS"

# 发送HTTP请求获取数据
print(f"正在从 {url} 获取数据...")
response = requests.get(url)
data = response.text
print("数据获取成功，开始解析...")

# 编译正则表达式模式
# re.S (DOTALL): 使 '.' 匹配包括换行符在内的所有字符
# re.M (MULTILINE): 使 '^' 和 '$' 匹配每一行的开头和结尾
pat = re.compile(
    r"^([^s][^:]*): (.+?)s*(?=^[^s][^:]*:|Z)", flags=re.S | re.M
)

# 用于存储所有解析出的包元数据的列表
out = []

# 按双换行符分割成独立的元数据块
for chunk in data.split("

"):
    if chunk.strip():  # 确保块非空
        # 使用正则表达式查找当前块中的所有键值对
        # findall 返回一个列表，每个元素是匹配到的键和值的元组
        parsed_items = pat.findall(chunk)
        if parsed_items:
            # 将键值对元组列表转换为字典
            out.append(dict(parsed_items))

# 打印解析结果的前几个字典，以便查看结构
print("
解析结果示例 (前2个包):")
for i, package_info in enumerate(out[:2]):
    print(f"--- Package {i+1} ---")
    for key, value in package_info.items():
        print(f"  {key}: {value.replace('\n', ' ').strip()}") # 替换换行符并去除首尾空白
print(f"
共解析出 {len(out)} 个包的元数据。")

正在从 https://bioconductor.org/packages/release/bioc/VIEWS 获取数据...
数据获取成功，开始解析...

解析结果示例 (前2个包):
--- Package 1 ---
  Package: a4
  Version: 1.44.0
  Depends: a4Base, a4Preproc, a4Classif, a4Core, a4Reporting
  Suggests: MLP, nlcv, ALL, Cairo, Rgraphviz, GOstats
  License: GPL-3
  MD5sum: cc696d3373a9f258d293f2d966da11d5
  NeedsCompilation: no
  Title: Automated Affymetrix Array Analysis Umbrella Package
  Description: Umbrella package is available for the entire Automated Affymetrix Array Analysis suite of package.
  biocViews: Microarray
  Author: Willem Talloen [aut], Tobias Verbeke [aut], Laure Cougnaud [cre]
  Maintainer: Laure Cougnaud <<a class="__cf_email__" data-cfemail="ddb1bca8afb8f3beb2a8bab3bca8b99db2adb8b3bcb3bcb1a4a9b4beaef3b8a8" href="/cdn-cgi/l/email-protection">[email protected]</a>>
  git_url: https://git.bioconductor.org/packages/a4
  git_branch: RELEASE_3_15
  git_last_commit: 5b0fc5a
  git_last_commit_date: 2022-04-26
  Date/Publication: 2022-04-26
  source.ver: src/contrib/a4_1.44.0.tar.gz
  win.binary.ver: bin/windows/contrib/4.2/a4_1.44.0.zip
  mac.binary.ver: bin/macosx/contrib/4.2/a4_1.44.0.tgz
  vignettes: vignettes/a4/inst/doc/a4vignette.pdf
  vignetteTitles: a4vignette
  hasREADME: FALSE
  hasNEWS: TRUE
  hasINSTALL: FALSE
  hasLICENSE: FALSE
  Rfiles: vignettes/a4/inst/doc/a4vignette.R
  dependencyCount: 82
--- Package 2 ---
  Package: a4Base
  Version: 1.44.0
  Depends: a4Preproc, a4Core
  Imports: methods, graphics, grid, Biobase, annaffy, mpm, genefilter, limma, multtest, glmnet, gplots
  Suggests: Cairo, ALL, hgu95av2.db, nlcv
  Enhances: gridSVG, JavaGD
  License: GPL-3
  MD5sum: 094c0a1c87b18ff8f16a3dbe4d06da64
  NeedsCompilation: no
  Title: Automated Affymetrix Array Analysis Base Package
  Description: Base utility functions are available for the Automated Affymetrix Array Analysis set of packages.
  biocViews: Microarray
  Author: Willem Talloen [aut], Tine Casneuf [aut], An De Bondt [aut], Steven Osselaer [aut], Hinrich Goehlmann [aut], Willem Ligtenberg [aut], Tobias Verbeke [aut], Laure Cougnaud [cre]
  Maintainer: Laure Cougnaud <<a class="__cf_email__" data-cfemail="a1cdc0d4d3c48fc2ced4c6cfc0d4c5e1ced1c4cfc0cfc0cdd8d5c8c2d28fc4d4" href="/cdn-cgi/l/email-protection">[email protected]</a>>
  git_url: https://git.bioconductor.org/packages/a4Base
  git_branch: RELEASE_3_15
  git_last_commit: 9ae69e0
  git_last_commit_date: 2022-04-26
  Date/Publication: 2022-04-26
  source.ver: src/contrib/a4Base_1.44.0.tar.gz
  win.binary.ver: bin/windows/contrib/4.2/a4Base_1.44.0.zip
  mac.binary.ver: bin/macosx/contrib/4.2/a4Base_1.44.0.tgz
  hasREADME: FALSE
  hasNEWS: TRUE
  hasINSTALL: FALSE
  hasLICENSE: FALSE
  dependsOnMe: a4
  suggestsMe: epimutacions
  dependencyCount: 73

共解析出 1000 多个包的元数据。