How to get direct link of remote video from embedded url within a url in Android using JSoup?

Question

I had asked the question previously about how to retrieve the embedded url for a video file and have successfully done so. Now I have a different issue. The json response for a WUnderground API webcam response gives the following url:

https://www.wunderground.com/webcams/cadot1/902/show.html

Using JSoup and per the answer to my initial issue I was able to get this embedded link:

https://www.wunderground.com/webcams/cadot1/902/video.html?month=11&year=2016&filename=current.mp4

While trying to "stream" the video from that url to a VideoView, I kept getting the error "cannot play video". Upon looking at the source for that link I noticed that the video file that needs to be played is not referenced in html but rather javascript. How can I get the direct link for the video file that needs to be played? Using JSoup or other process?

The source for the url https://www.wunderground.com/webcams/cadot1/902/video.html?month=11&year=2016&filename=current.mp4 shows the following for the needed video file within a <script> bracket:

url: "//icons.wunderground.com/webcamcurrent/c/a/cadot1/902/current.mp4?e=1480377508"

I am using JSoup to get the embedded url for the video from the response url like so:

    private class VideoLink extends AsyncTask<Void, Void, Void> {
    String title;

    @Override
    protected void onPreExecute() {
        super.onPreExecute();
        mProgressDialog.setTitle("JSOUP Test");
        mProgressDialog.setMessage("Loading...");
        mProgressDialog.setIndeterminate(false);
        mProgressDialog.show();
    }

    @Override
    protected Void doInBackground(Void... params) {
        try {

            // for avoiding javax.net.ssl.SSLProtocolException: handshake alert:  unrecognized_name
            System.setProperty("jsse.enableSNIExtension", "false");

            // WARNING: do it only if security isn't important, otherwise you have
            // to follow this advices: http://stackoverflow.com/a/7745706/1363265
            // Create a trust manager that does not validate certificate chains
            TrustManager[] trustAllCerts = new TrustManager[]{new X509TrustManager(){
                public X509Certificate[] getAcceptedIssuers(){return null;}
                public void checkClientTrusted(X509Certificate[] certs, String authType){}
                public void checkServerTrusted(X509Certificate[] certs, String authType){}
            }};

            // Install the all-trusting trust manager
            try {
                SSLContext sc = SSLContext.getInstance("TLS");
                sc.init(null, trustAllCerts, new SecureRandom());
                HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
            } catch (Exception e) {
                ;
            }

            // Connect to the web site
            Document doc = Jsoup.connect(TEST_URL).get();
            Elements elements = doc.getElementsByClass("videoText");
            // Get the html document title
            for (Element link : elements) {
                String linkHref = link.attr("href");
                // linkHref contains something like video.html?month=11&year=2016&filename=current.mp4
                // TODO check if linkHref ends with current.mp4
                title = linkHref;
            }
        } catch (IOException e) {
            e.printStackTrace();
            mProgressDialog.dismiss();
        }
        return null;
    }

    @Override
    protected void onPostExecute(Void result) {
        // Set title into TextView
        resultTxt.setText(title);
        String resVid = TEST_URL;
        Log.d(TAG, "URL: " + resVid);
        Uri resUri = Uri.parse(resVid);
        try {
            // Start the MediaController
            MediaController mediacontroller = new MediaController(
                    MainActivity.this);
            mediacontroller.setAnchorView(resultVidVw);
            // Get the URL from String VideoURL
            Uri video = Uri.parse(resVid);
            resultVidVw.setMediaController(mediacontroller);
            resultVidVw.setVideoURI(video);

        } catch (Exception e) {
            Log.e("Error", e.getMessage());
            e.printStackTrace();
        }

        resultVidVw.requestFocus();
        resultVidVw.setOnPreparedListener(new MediaPlayer.OnPreparedListener() {
            // Close the progress bar and play the video
            public void onPrepared(MediaPlayer mp) {
                mProgressDialog.dismiss();
                resultVidVw.start();
            }
        });
    }
}

Please note that I need to do this on every JSONObject in the response array.

Answer 1

This is how you can GET the file:

(Notice: the Extraction part only works with current html of the site and if that changes, it may not work correctly!)

String url = "https://www.wunderground.com/webcams/cadot1/902/video.html";
int timeout = 100 * 1000;

// Extract video URL
Document doc = Jsoup.connect(url).timeout(timeout).get();
Element script = doc.getElementById("inner-content")
        .getElementsByTag("script").last();
String content = script.data();
int indexOfUrl = content.indexOf("url");
int indexOfComma = content.indexOf(',', indexOfUrl);
String videoUrl = "https:" + content.substring(indexOfUrl + 6, indexOfComma - 1);
System.out.println(videoUrl);

[Output: https://icons.wunderground.com/webcamcurrent/c/a/cadot1/902/current.mp4?e=1481246112]

Now you can get the file by specifying .ignoreContentType(true) in order to avoid org.jsoup.UnsupportedMimeTypeException and .maxBodySize(0) to remove the limit on file size.

// Get video file
byte[] video = Jsoup.connect(videoUrl)
        .ignoreContentType(true).timeout(timeout).maxBodySize(0)
        .execute().bodyAsBytes();

I don't know if you can play it in Android or not but I think you can save it using org.apache.commons.io.FileUtils (I tested it in Java SE but not Android development environment.)

// Save video file
org.apache.commons.io.FileUtils.writeByteArrayToFile(new File("test.mp4"), video);