sooo close. Need help with geturl and data functions

arcanedreams · Post by **arcanedreams** » Thu Sep 06, 2007 9:40 pm

Here is what I have so far. Dont worry about the username and password..I just made those up for testing purposes.

bbind pub - !joined joined

proc joined {nick host handle chan text} {
    

set query "http://forums.shooshtime.com/login.php?vb_login_username=zooshbot&vb_login_password=zoosh&s=&do=login"
set http [::http::geturl $query]
set html [::http::data $http] 


regexp {<strong>Welcome, (.*?) </strong><br />} $html - uname
putserv "PRIVMSG $chan :$uname"

That link will log you into the site.

From there, I can't get it to find any info on the page. I'm guessing it has something to do with the way it handles the link...perhaps it is taking the info from the actual login.php page

How do I get around this?

rosc2112 · Post by **rosc2112** » Fri Sep 07, 2007 12:18 am

Best bet is to debug what you are getting in your data var, by looking at it with putcmdlog lines after getting the webpage, like:

Code: Select all

set query "http://somewhere.com/$varwhatever"
set page [::http::geturl $query]
set html [::http::data $page] 

set count 0
foreach line [split $html \n] {
       incr count
       putcmdlog "line$count '$line'"
}

It's also helpful to [catch] the geturl errors so you can see them, for example (from one of the movie grabber scripts I did):

Code: Select all

	set movieurl "http://www.filmspot.com"
	catch {set page [::http::geturl $movieurl -timeout $::movietimeout]} error
	if {[string match -nocase "*couldn't open socket*" $error]} {
		puthelp "PRIVMSG $chan :Error: couldn't connect to filmspot.com..Try again later."
		return
	}
	if { [::http::status $page] == "timeout" } {
		puthelp "PRIVMSG $chan :Error: Connection to filmspot.com timed out..Try again later."
		return
	}
	set html [::http::data $page]
	::http::cleanup $page
        # Now we have data to work with and test the contents, etc..
	if {[regexp $regexm $html match moviedata]} {
             # the $regexm var is set earlier in the script depending on 
             # which commandline was given to the script

Thats a pretty standard template for any url-grabber script.

arcanedreams · Post by **arcanedreams** » Fri Sep 07, 2007 2:15 am

Not sure how to view the putcmdlog, so I just used putlog instead.

only error that showed up was expected. It was an error stating that the $uname variable couldn't be found.

I had it set to spam the channel with all the variables in the entire script.

It will go something like this:

$query: http://forums.shooshtime.com/login.php? ... =&do=login

$http: ::http::5

Then it will stop there and never output the $html variable.

I have tried simply changing the url to just the standard http://forums.shooshitme, and the $html variable will show up like this:

$html: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

So it looks to me like when I use the url that I need to have the bot login to the site, it has trouble at the ::http::data part

If I had to guess I would say the http::data part isn't returning because after logging in, you are redirected to index.php, where the call in the url was to login.php

I also returned the value for ::http::size, and it comes back as 0
For the ::http::code it returns :HTTP/1.1 302 Found

Any idea why, or how to fix it?

Or perhaps an alternative way to get the bot to login to the site and grab info while its logged in?

rosc2112 · Post by **rosc2112** » Sat Sep 08, 2007 12:58 pm

There are newlines in the output, so you won't see them, which is why I suggested using foreach [split $html \n] to output each line.

arcanedreams · Post by **arcanedreams** » Sat Sep 08, 2007 1:59 pm

well I got it to work. As I suspected, when you log in, it redirects you. So I had to grab the meta data for the url to the re-direction.

Full working code:

Code: Select all

package require http


#############################
#######Profile Grabber#######
#############################
bind pub - !profile profile

proc profile {nick host handle chan user} {
    

set url "http://forums.shooshtime.com/login.php?vb_login_username=zooshbot&vb_login_password=zoosh&s=&do=login&url=/member.php?username=$user"
#putserv "PRIVMSG $chan :$url"

set http [::http::geturl $url]
#putserv "PRIVMSG $chan :$http"

#################
##get meta info##
#################
upvar #0 $http state
        array set meta [set ${http}(meta)]
        if {![info exist meta(Location)]} {
           return $http
        }
        set redirect $meta(Location)
        unset meta

#putserv "PRIVMSG $chan :Redirecting to $redirect"


###################
##Set and cleanup##
###################
set newhttp [::http::geturl $redirect]
set size [::http::size $newhttp]
set code [::http::code $newhttp]
set data [::http::data $newhttp]

::http::cleanup $newhttp


##############
##Debug info##
##############
#putserv "PRIVMSG $chan :Size of html returned, $size"
#putserv "PRIVMSG $chan :Html code returned, $code"
#putserv "PRIVMSG $chan :Html data, $data"

#regexp {To remove (.*?)you must} $data - notloggedin
#regexp {has not registered and therefore (.*?) to view} $data - noprofile
#putserv "PRIVMSG $chan :$notloggedin"
#putserv "PRIVMSG $chan :$noprofile"


###############
##Search Data##
###############
regexp {Total Posts: <strong>(.*?)</strong>} $data - tposts
regexp {Join Date: <strong>(.*?)</strong>} $data - joined
regexp {Referrals: <strong>(.*?)</strong>} $data - referrals
regexp {<td>
						<strong>Birthday</strong>:<br />
						(.*?)
					</td>} $data - bday
regexp {User Notes: <strong>(.*?)</strong>} $data - unotes



##########
##Output##
##########
putserv "PRIVMSG $chan :User Status for $user"
putserv "PRIVMSG $chan :Joined on $joined"
putserv "PRIVMSG $chan :$tposts Total Posts"
putserv "PRIVMSG $chan :$referrals Total Referrals"
putserv "PRIVMSG $chan :$unotes Total User Notes"
putserv "PRIVMSG $chan :Was Born On $bday"
}