python alexa result parsing with lxml.etree

Question

I am using alexa api from aws but I find difficult in parse the result to get what I want

alexa api return an object tree <type 'lxml.etree._ElementTree'>

I use this code to print the tree

from lxml import etree
root = tree.getroot()
print etree.tostring(root)

I get xml below

<aws:UrlInfoResponse xmlns:aws="http://alexa.amazonaws.com/doc/2005-10-05/"><aws:Response xmlns:aws="http://awis.amazonaws.com/doc/2005-07-11"><aws:OperationRequest><aws:RequestId>ccf3f263-ab76-ab63-db99-244666044e85</aws:RequestId></aws:OperationRequest><aws:UrlInfoResult><aws:Alexa>

  <aws:ContentData>
    <aws:DataUrl type="canonical">google.com/</aws:DataUrl>
    <aws:SiteData>
      <aws:Title>Google</aws:Title>
      <aws:Description>Enables users to search the world's information, including webpages, images, and videos. Offers unique features and search technology.</aws:Description>
      <aws:OnlineSince>15-Sep-1997</aws:OnlineSince>
    </aws:SiteData>
    <aws:LinksInCount>3453627</aws:LinksInCount>
  </aws:ContentData>
  <aws:TrafficData>
    <aws:DataUrl type="canonical">google.com/</aws:DataUrl>
    <aws:Rank>1</aws:Rank>
  </aws:TrafficData>
</aws:Alexa></aws:UrlInfoResult><aws:ResponseStatus xmlns:aws="http://alexa.amazonaws.com/doc/2005-10-05/"><aws:StatusCode>Success</aws:StatusCode></aws:ResponseStatus></aws:Response></aws:UrlInfoResponse>

I use root.find('LinksInCount').text to get value of element but it does not work.

I want to know how to get the text 3453627 of aws:LinksInCount

Answer 1

You run into two challenges:

• XML using namespaces
• two namespaces sharing the same namespace prefix

XML document with reused prefix for 2 different namespaces

You see "aws:" prefix, but it is used for two different namespaces:

xmlns:aws="http://alexa.amazonaws.com/doc/2005-10-05/"
xmlns:aws="http://awis.amazonaws.com/doc/2005-07-11"

Using the same namespace prefix in XML is completely legal. The rule is, the later one is valid.

xmlstr = """
<?xml version="1.0"?>
<aws:UrlInfoResponse xmlns:aws="http://alexa.amazonaws.com/doc/2005-10-05/">
  <aws:Response xmlns:aws="http://awis.amazonaws.com/doc/2005-07-11">
    <aws:OperationRequest>
      <aws:RequestId>ccf3f263-ab76-ab63-db99-244666044e85</aws:RequestId>
    </aws:OperationRequest>
    <aws:UrlInfoResult>
      <aws:Alexa>
        <aws:ContentData>
          <aws:DataUrl type="canonical">google.com/</aws:DataUrl>
          <aws:SiteData>
            <aws:Title>Google</aws:Title>
            <aws:Description>Enables users to search the world's information, including webpages, images, and videos. Offers unique features and search technology.</aws:Description>
            <aws:OnlineSince>15-Sep-1997</aws:OnlineSince>
          </aws:SiteData>
          <aws:LinksInCount>3453627</aws:LinksInCount>
        </aws:ContentData>
        <aws:TrafficData>
          <aws:DataUrl type="canonical">google.com/</aws:DataUrl>
          <aws:Rank>1</aws:Rank>
        </aws:TrafficData>
      </aws:Alexa>
    </aws:UrlInfoResult>
    <aws:ResponseStatus xmlns:aws="http://alexa.amazonaws.com/doc/2005-10-05/">
      <aws:StatusCode>Success</aws:StatusCode>
    </aws:ResponseStatus>
  </aws:Response>
</aws:UrlInfoResponse>
"""

Next challenge is, how to search for namespaced elements.

I prefer using xpath, and for it, you can use whatever namespace you like in the xpath expression, but you have to tell the xpath call what you meant by those prefixes. This is done by namespaces dictionary:

from lxml import etree
doc = etree.fromstring(xmlstr.strip())

namespaces = {"aws": "http://awis.amazonaws.com/doc/2005-07-11"}
texts = doc.xpath("//aws:LinksInCount/text()", namespaces=namespaces)
print texts[0]