V2EX = way to explore
V2EX 是一个关于分享和探索的地方
Sign Up Now
For Existing Member  Sign In
helloworld000
V2EX  ›  问与答

请问爬虫高手为什么每次 scrapy 用 xpath 的 extract_first() 或者 get() 总是会有这个 class 的前缀出现???

  •  
  •   helloworld000 · Aug 28, 2020 · 1800 views
    This topic created in 2080 days ago, the information mentioned may be changed or developed.

    比如 stackoverflow 的 这个 userpage https://stackoverflow.com/users/542251/liam

    我要得到用户的自我介绍这个 class 里的内容

    response.xpath("//div[@class='grid--cell mt16 s-prose profile-user--bio ']").get()

    最后的结果(如下)

    '<div class="grid--cell mt16 s-prose profile-user--bio ">\r\n<p>Father, Husband, Rock Climber and Developer with 20 years experience in the industry (in that order).</p>\n\n<hr>\n\n<p><strong>Any fool can write code that a computer can understand. Good programmers write code that humans can understand.</strong></p>\n\n<p><a href="https://en.wikiquote.org/wiki/Martin_Fowler" rel="nofollow noreferrer">Martin Fowler (2008)</a></p>\n\n<hr>\n\n<p>Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. <strong>We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil</strong>. Yet we should not pass up our opportunities in that critical 3%.</p>\n\n<p><a href="https://en.wikiquote.org/wiki/Donald_Knuth" rel="nofollow noreferrer">Donald Ervin Knuth (Professor Emeritus at Stanford University, and winner of the 1974 Turing Award)</a></p>\n\n<hr>\n    </div>'
    
    

    怎么前面和后面还是有这个<div class="grid--cell mt16 s-prose profile-user--bio ">?

    难道这个前缀不应该没了吗? 如果我在后面加了个 text()的话就只剩下 ( '\r\n' ) 了...

    scrapy 里的 xpath 不能像 lxml 里的 html 一样直接取 text_content()吗?

    3 replies    2020-08-29 16:38:27 +08:00
    crella
        1
    crella  
       Aug 29, 2020 via Android
    看看有没有.innerHtml 的函数?不懂 python
    Kobayashi
        2
    Kobayashi  
       Aug 29, 2020 via Android
    不能。把标签内所有内容拼接起来一次返回,目前只有 lxml.html 和 bs4 支持,后者其实也是封装前者。

    scrapy 内置解析器是 parsel,也是依赖 lxml 。先取出这个 div HTML 代码,临时引入 lxml.html 包装为 HTML Element,再调用 text_content() 就可以了。
    Guidance3204
        3
    Guidance3204  
       Aug 29, 2020
    xpath 写的不对,大概应该是 //div[@class="classname"]//p//text(), 数据没在 <div> 下面,在 <p> 里面呢
    About   ·   Help   ·   Advertise   ·   Blog   ·   API   ·   FAQ   ·   Solana   ·   2839 Online   Highest 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 54ms · UTC 15:24 · PVG 23:24 · LAX 08:24 · JFK 11:24
    ♥ Do have faith in what you're doing.