http

Retrieve HTML content from an infinite scroll page (Facebook)

对着背影说爱祢 提交于 2021-02-10 17:31:49
问题 I would like tor retrieve HTML data from a dynamic web page, like for example a public Facebook page: https://www.facebook.com/bbcnews/ (public content, without login) For example, in this page, we have an infinite scroll, and we have to go at the bottom of the page to load more posts. My current code is here: URL url = new URL("https://www.facebook.com/bbcnews/"); BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream())); BufferedWriter writer = new BufferedWriter

How to get the feed URL(s) from a website?

自作多情 提交于 2021-02-10 16:11:31
问题 As per the official documentation, properly setup websites should indicate the URL of their RSS / Atom feed(s) when asked politely: GET / HTTP/1.1 Host: example.com Accept: application/rss+xml, application/xhtml+xml, text/html When an HTTP server (or server-side script) gets this, it should redirect the HTTP client to the feed. It should do this with an HTTP 302 Found. Something like: HTTP/1.1 302 Found Location: http://example.com/feed I'm trying to get this response, without luck: request(

How to get the feed URL(s) from a website?

这一生的挚爱 提交于 2021-02-10 16:09:19
问题 As per the official documentation, properly setup websites should indicate the URL of their RSS / Atom feed(s) when asked politely: GET / HTTP/1.1 Host: example.com Accept: application/rss+xml, application/xhtml+xml, text/html When an HTTP server (or server-side script) gets this, it should redirect the HTTP client to the feed. It should do this with an HTTP 302 Found. Something like: HTTP/1.1 302 Found Location: http://example.com/feed I'm trying to get this response, without luck: request(

HTTP GET using cURL is giving SSL error

妖精的绣舞 提交于 2021-02-10 16:02:33
问题 I have written a REST server which listens on port 8000. I am trying to invoke the api from the same machine using the cURL command - curl -H "accept: application/json" https://localhost:8000/status -v I am getting the following error * About to connect() to localhost port 8000 (#0) * Trying ::1... * Connected to localhost (::1) port 8000 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * NSS error -12263 (SSL_ERROR_RX_RECORD

HTTP GET using cURL is giving SSL error

可紊 提交于 2021-02-10 15:59:41
问题 I have written a REST server which listens on port 8000. I am trying to invoke the api from the same machine using the cURL command - curl -H "accept: application/json" https://localhost:8000/status -v I am getting the following error * About to connect() to localhost port 8000 (#0) * Trying ::1... * Connected to localhost (::1) port 8000 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * NSS error -12263 (SSL_ERROR_RX_RECORD

Apache responses with http/1.0 even if request is http/1.1

落花浮王杯 提交于 2021-02-10 14:23:03
问题 I configured Apache Http server version 2.4 as proxy for backend server (Glassfish App Server actually). MPM is worker . Configuration is mostly default. Configuration for proxy ProxyPass /context/ http://backend.com:8080/context/ keepalive=On ttl=25 timeout=300 max=50 ProxyPassReverse /context/ http://backend.com:8080/context/ I open page in browser and it sends GET http://example.com/context HTTP/1.1 to Apache. Apache returns response HTTP/1.0 200 OK with Connection: close header. But

Selenium Python - Get Network response body

泪湿孤枕 提交于 2021-02-10 14:17:09
问题 I use Selenium to react to the reception of data following a GET request from a website. The API called by the website is not public, so if I use the URL of the request to retrieve the data, I get {"message":"Unauthenticated."} . All I've managed to do so far is to retrieve the header of the response. I found here that using driver.execute_cdp_cmd('Network.getResponseBody', {...}) might be a solution to my problem. Here is a sample of my code: import json from selenium import webdriver from

Selenium Python - Get Network response body

走远了吗. 提交于 2021-02-10 14:15:44
问题 I use Selenium to react to the reception of data following a GET request from a website. The API called by the website is not public, so if I use the URL of the request to retrieve the data, I get {"message":"Unauthenticated."} . All I've managed to do so far is to retrieve the header of the response. I found here that using driver.execute_cdp_cmd('Network.getResponseBody', {...}) might be a solution to my problem. Here is a sample of my code: import json from selenium import webdriver from

How to pass Java date as path variable in a rest get call

微笑、不失礼 提交于 2021-02-10 12:32:31
问题 There is an existing api which is having the below code : @GetMapping(value = "/getAmount") public long getAmount(){ Date d = new Date(); long amountFetchedFromDb = callToDatabase(d); return amountFetchedFromDb; } Now I need to change the functionality as below: @GetMapping(value = "/getAmount") public long getAmount(){ Date d = new Date(); <CALL TO A NEW REST API PASSING IT THE DATE d AND THE NEW API WILL MAKE THE DB CALL AND RETURN THE AMOUNT> return amount; } Now, I have created a new rest

TRESTRequest: Is it possible to use custom media types in a POST request?

蹲街弑〆低调 提交于 2021-02-10 12:25:35
问题 We have an API that expects our own vendor specific content type for example application/vnd.xxxx.custom.custom-data+json but looking through the source code of REST.Client it seems to always default to one of the ContentTypes in REST.Types for example when assigning ctNone in my body request it will default to ctAPPLICATION_X_WWW_FORM_URLENCODED . I've tried assigning the content type directly to the TRESTClient.ContentType property but that gets overwritten by the TRESTRequest.ContentType