Shell: Connecting to a website and accessing a field

Question

I want to write a script that takes an argument which is text, opens a connection to a specific website and input the arg into text field using the field\'s ID. Is this poss

Answer 1

... "opens a connection to a specific website and input the arg into text field using the field's ID" ...

You mean you want to fill & send a HTML <form> ... </form>, right?

I would use curl (http://curl.haxx.se/). With curl you can automate HTTP POST requests very easily, suppose you have website with following form (excerpt from: http://curl.haxx.se/docs/httpscripting.html):

<form method="POST" action="junk.cgi">
      <input type=text name="birthyear">
      <input type=submit name=press value=" OK ">
</form>

this command would fill & send the form (let's pretend that form is available on http://www.example.com/when.cgi):

curl --data "birthyear=1905&press=%20OK%20" http://www.example.com/when.cgi

Answer 2

think you need to get to grips with forms first before attempting this.

Meaning you could reproduce thml file locally which includes all the values of the form, the forms action could be the end url steps 3 and 4, also look into auto submit java scripts

The final hyperlink well once the form is submitted - if the last step by parsing outcome of the post and then using curl or wget or something that would act as the click

E2A the problem with a bash script is my concept above of creating the form is bull crap since to then execute a java script command line browser or links/lynx/wget/curl etc will be a challenge.

tne first question is does the form support both get and post - if form action can only be post then you will not be able to send form fields as variables i.e.

http://destinatio-form-url.com/acceptform.cgi?user=something&address=something_else

This above exampe is how you coud generate your form values if get is supported if however you need to post then a form needs to be gnerated with form action set to post to get to that url and it goes like I said you need to create the form.

If assuming you can send it via above format then the thing to watch for is where the response in the clickable link is if its another click away - you can see the problem if however it is returned on the same page submitted to then - it be pretty easy to parse the html by grepping for something specific and grepping/awking until you get exact url which you fire off,

take a look at my answer here

bash script to login to webpage

This is how you would go about in java authenticating grabbing a cookie then progressing as a logged in user, the thing there is all you need to post your form

All I am saying is it is possible in bash but for url processing maybe done in a better language which gives you all the libraries to do this and makes it elegant rather than calling all sorts of system commands

The example given is in Java but could be in any language, perl,php,python and so forth and they should all have libraries for this task, for Perl look up LWP html in google and lots of specific libraries like LWP HTML Parser and so forth you could use

Anyhow all the best

I think links supports java scripts if it helps..

it is like lynx but has a lot more addons

    apt-cache search links|grep browser
amule-gnome-support - ed2k links handling support for GNOME web browsers
elinks - advanced text-mode WWW browser
elinks-data - advanced text-mode WWW browser - data files
elinks-doc - advanced text-mode WWW browser - documentation
elinks-lite - advanced text-mode WWW browser - lightweight version
libhtmlunit-core-js-java - GUI-Less browser for Java programs - JavaScript engine
libhtmlunit-java - GUI-Less browser for Java programs
libjenkins-htmlunit-java - Jenkins branch of HtmlUnit browser testing for web apps
libphp-snoopy - Snoopy is a PHP class that simulates a web browser
links - Web browser running in text mode
links2 - Web browser running in both graphics and text mode
man2html - browse man pages in your web browser
surf - simple web browser

Answer 3

If you know exactly which field you need to fill, then this can be done using lynx. Assume you get the string S with your script as an input argument. Then you create a command script, which will guide lynx through its behaviour.

For example, suppose S=foo, and your field is the second field in the web page. After that, there are two more fields, and then the submit button. After that you wait for the page to load and click the hyperlink (after that you exit). The web page is www.something.com.

The command script would be in a file bar.txt:

key <tab> //get to first field
key <tab> //get to second field
key f     //input f
key o     //input o
key o     //input o
key <tab> //get to third field
key <tab> //get to fourth field
key <tab> //get to sumbit button
key ^J    //click submit and wait for load
key <tab> //get to hyperlink
key ^J    //click hyperlink and wait for load
key Q     //exit
key y     //confirm exit

The main command would then be lynx www.something.com -accept_all_cookies -cmd_script=bar.txt

Now all you need to do is dynamically create the input string.

#!/bin/bash
script=bar.txt
input=$1
webpage=www.something.com
len=${#input}
echo 'key <tab>' > $script
echo 'key <tab>' >> $script
for i in `echo $input|fold -w1` 
do
    echo 'key '$i >> $script
done
echo 'key <tab>' >> $script
echo 'key <tab>' >> $script
echo 'key <tab>' >> $script
echo 'key ^J' >> $script
echo 'key <tab>' >> $script
echo 'key ^J' >> $script
echo 'key Q' >> $script
echo 'key y' >> $script

lnyx $webpage -accept_all_cookies -cmd_script=bar.txt

Now all you need to do is save the script, modify it to be executable and call it ./script your_string

Answer 4

To get you started, here is my script to order today's lunch from our local canteen:

URL="https://lunch.com/lunch/cgi-bin/order.cgi"

O="order=Order"
A="amount_%d=%%d&amount_foil_container_%d=%%d"

function order_lunch() {
  if [[ -n "$@" ]]; then
    curl -u "$USER":"$PASSWORD" \
         -d $(printf $(printf "$O&$A&$A&$A&$A" 0 0 1 1 2 2 3 3) \
                     "${@:2:8}") \
         "$URL";
  else
    echo "Nothing to order.";
  fi;
}

Where input is a string in the following format

2012-08-23 1 0 0 0 0 0 0 0

where each field denotes a different dish, i.e. a 1 in the first position after the date is "1 pasta"

Good luck.