How to get mobile response of webpage using java or jsoup

问题

I am trying to get response of youtube.com using java with JSoup.

I am able to get the response of youtube using JSoup as follows, it returns the desktop website's response

         String str = "https://www.youtube.com/";
         doc = Jsoup.connect(str)
                    .userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36")
                    .get();

Same way, I am trying to get the response for mobile version to this same site as follows,

            doc = Jsoup.connect("https://"+url2.getHost()+"/search?q="+q)
                        .userAgent("Mozilla/5.0(Linux; U; Android 2.2; en-gb; LG-P500 Build/FRF91) AppleWebKit/533.0 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1")
                        .get();

But this gives only desktop/laptop version response and not the mobile response.

How to get the mobile response from jsoup.

Thanks in advance.

回答1:

I think that maybe your User-Agent isn't quite correct.

I've just tried it with the following and appear to have hit the mobile YouTube site:

String mob = "https://m.youtube.com/";
         mobile = Jsoup.connect(mob)
                    .userAgent("Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1")
                    .get();

Update I've had a look in more detail and I believe that the page content is being modified by some Javascript once the DOM has loaded.

Looking at the HTML returned by the JSoup code above I get this (note the content div is empty):

 <body id="body" class="atom fusion-tn">

  <div id="player"></div>
  <div id="guide-layout-container">
    <div id="guide-container"></div>
    <div id="content-container">
      <div id="content"></div>
    </div>
    <div id="guide-overlay"></div>
    <div id="lightbox"></div>
    <div id="toast"></div>
    <div id="content-overlay"></div>
  </div>
  <div id="_yt_orientation_detect"></div>

  </body>

Comparing to the HTML viewed in Chrome's dev tools I see this:

JSoup is just an HTML parser, not a web browser. In order to do achieve what you require, I think you might need to look at something like this Is there a way to embed a browser in Java?

回答2:

instead of:

.userAgent("Mozilla/5.0(Linux; U; Android 2.2; en-gb; LG-P500 Build/FRF91) AppleWebKit/533.0 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1")

try:

.userAgent("Opera/12.02 (Android 4.1; Linux; Opera Mobi/ADR-1111101157; U; en-US) Presto/2.9.201 Version/12.02")

other options find on:

http://www.useragentstring.com/pages/useragentstring.php

and search for :

Mobile Browsers

来源：https://stackoverflow.com/questions/36259577/how-to-get-mobile-response-of-webpage-using-java-or-jsoup

标签

java

jsoup