How to use linux curl to login and retrieve data once logged?

问题

I have a device Stiebel Eltron heat pump home and I would like to use Linux shell curl (not php) curl to login (POST) and retrieve (GET) data once logged

Here is how my curl login POST call looks like (I used Firebug + persist option to copy/paste the below):

curl --data-urlencode 'userName=tutu&password=xx' \
'https://thesite/api/login?noCacheDummyValue=1459356436185' -X POST \
 -H 'Host: thesite' \
 -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0' \
 -H 'Accept: application/json, text/plain, */*' \
 -H 'Accept-Language: en-US,en;q=0.7,fr-FR;q=0.3' \
 -H 'Accept-Encoding: gzip, deflate, br' \
 -H 'X-Requested-With: XMLHttpRequest' \
 -H 'Content-Type: application/json;charset=utf-8' \
 -H 'Referer: https://thesite/mobile/app/app.html' \
 -H 'Content-Length: 44' \
 -H 'Cookie: JSESSIONID=qhs02mfeip2p1n5n4t2rj1huu'

But this gives me nothing in the output:

 sh myproj.sh 
curl: (52) Empty reply from server

Q1: What should I collect from this first `POST`, and how to do it?

I tried to get the cookie with --cookie-jar cookie.txt but nothing showed up in my current directory. However I sent cookies with the header... shouldn't I get them back?

After that I would like to continue the session I (think I) opened so to collect data from a GET that is (same method as above):

curl 'https://thesite/api/data/1036493/heatEnergy/today?noCacheDummyValue=1459356438440' \
     -H 'Host: thesite' \
     -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0' \
     -H 'Accept: application/json, text/plain, */*' \
     -H 'Accept-Language: en-US,en;q=0.7,fr-FR;q=0.3' \
     -H 'Accept-Encoding: gzip, deflate, br' \
     -H 'X-Requested-With: XMLHttpRequest' \
     -H 'Referer: https://thesite/mobile/dashboard/dash.html' \
     -H 'Cookie: JSESSIONID=qhs02mfeip2p1n5n4t2rj1huu'

Q2: How do I keep the session opened to perform the `GET`above?

I tried --next option idea from here to call the GET after the POST; without success, --next option is not recognized on my Ubuntu Linux.

If it can help understanding, the page I should get is full of javascript (angularjs I think).

回答1:

From your POST call it is obvious that you are trying to do an ajax XMLHttpRequest. Likely this is expecting json encoded data. However you are providing url-encoded data.

Probably you should try using something along

... -X POST -H "Content-Type: application/json" -d '{"userName":"tutu","password":"xx"}' 'https://thesite/api/login?noCacheDummyValue=1459356436185'

To make the login work, you need to match the way the server is maintaining the session. If session info is referred to via a cookie yo may just extract the cookie from the POST call (as you said you tried already, but on an error related response)

If the server does not use a straight way of referencing the session you might not succeed in getting your GET call working. If you are lucky maybe you then just need to parse a string from the page (fragment) or json data returned.

So monitor what a working handshake (e.g using a browser) is exchanging over the wire. And try identifing the important pieces of data. Then, put those together to form your GET call.

回答2:

Thanks to rpy, I was able to do what I wanted:

Collect the session cookie:

curl -a -X POST -d '{"userName":"tutu","password":"xx"}'                  \
 --trace-ascii debugdump.txt --cookie-jar cookies.txt                     \
-H 'Content-Type: application/json;charset=utf-8'                         \
-H 'Referer: https://thesite/mobile/app/app.html'                         \
   'https://thesite/api/login?noCacheDummyValue=1459356436185' -o my.post

todo: check result is "ok" in my.post
todo: get content of cookies.txt into ${COOKIE}

Use it to retrieve the raw data I wanted:

curl -a \
-H 'Referer: https://thesite/mobile/dashboard/dash.html'   \
-H 'Cookie:JSESSIONID='${COOKIE}                           \
   'https://thesite/api/data/1036493/outTemp/lastMonth?noCacheDummyValue=1459972646687'   \
   -o lm_outTemp.get \
-L 'https://thesite/api/data/1036493/heatEnergy/lastMonth?noCacheDummyValue=1459972646694' \
   -o lm_heatEnergy.get

And I already started working on how to process the data here: Firefox - Collecting the data used by LAB.min.js

来源：https://stackoverflow.com/questions/36315280/how-to-use-linux-curl-to-login-and-retrieve-data-once-logged

标签

angularjs