HTTP/TLS Package

[w00f]

Hey guys,
i got this little problem within my wget tcl procedure when calling http::get to reach one ssl server. Works fine if i remove the ssl from the URL (-https +http).

here's is a bit of the script

Code: Select all

package require http
package require tls
http::register https 443 [list ::tls::socket -require 0 -request 1]

proc s:wget { url } {

       catch {set token [http::geturl $url -binary 1 -timeout 10000]} error
.....
       if {[http::ncode $token] != 200} {
                s:debug "[http::ncode $token] HTTP error"
                return 0
       }
}

It breaks in the http::ncode check ([http::ncode $token] = "", no code returned).

.errorInfo
can't read "state(sock)": no such element in array
while executing
"fileevent $state(sock) writable {}"


Thanks in advance.

[speechles]

Checking for a numeric code when one isn't available happens. Your method isn't a reliable way to check for the full range of errors which can occur during http transactions and parsing the response. This is the way I would handle it.....

Code: Select all

package require http
package require tls
http::register https 443 [list ::tls::socket -require 0 -request 1]

proc s:wget { url } {
	http::config -useragent "Mozilla/EggdropWget"
	catch {set token [http::geturl $url -binary 1 -timeout 10000]} error
	# error condition 1, invalid socket or other general error
	if {![string match -nocase "::http::*" $error]} {
		s:debug "Error: [string totitle [string map {"\n" " | "} $error]] \( $url \)"
		return 0
	}
	# error condition 2, http error
	if {![string equal -nocase [::http::status $token] "ok"]} {
		s:debug "Http error: [string totitle [::http::status $token]] \( $url \)"
		http::cleanup $token
		return 0
	}
	# recursive redirect support, 300's
	# this doesn't check for redirect to self nor does it check
	# for nesting too deep, if you want these I can supply them
	# but for using trusted urls, this shouldn't be necessary.
	if {[string match "*[http::ncode $token]*" "303|302|301" ]} {
		upvar #0 $token state
		foreach {name value} $state(meta) {
			if {[regexp -nocase ^location$ $name]} {
				if {![string match "http*" $value]} {
					# fix our locations if needed
					if {![string match "/" [string index $value 0]]} {
						set value "[join [lrange [split $url "/"] 0 2] "/"]/$value"
					} else {
						set value "[join [lrange [split $url "/"] 0 2] "/"]$value"
					}
				}
				s:wget $value
				# only the last iteration from our recursion is required
				# we save time by using return on prior recurses
				return
			}
		}
	}
	# waaay down here, we finally check the ncode for 400 or 500 codes
	if {[string match 4* [http::ncode $token]] || [string match 5* [http::ncode $token]]} {
		s:debug "Http resource is not evailable: [http::ncode $token] \( $url \)"
		return 0
	}
	# uncomment the three lines below to return the http::data
	# check for error by string equaling 0, if it isn't 0 it contains
	# the data returned from the http transaction
	# ---
	# set data [http::data $token]
	# http::cleanup $token
	# return $data
}

[w00f]

Thanks speechles, i've added the http status if (all the others are there already)

the status returned is 'eof', which is weird

Code: Select all

[11:10p]  jackass    • s:wget "https://xpto.com/index.php?action=dl&id=133"
[11:10p]  -modtcl    • [DEBUG] URL: https://xpto.com/index.php?action=dl&id=6437"
[11:10p]  -modtcl    • [DEBUG] Weirdo status returned, eof
[11:10p]  -modtcl    • 0
[11:10p]  jackass    • set errorInfo
[11:10p]  -modtcl    • can't read "state(sock)": no such element in array
[11:10p]  -modtcl    • while executing
[11:10p]  -modtcl    • "fileevent $state(sock) writable {}"

it worked just fine few days ago, dunno what changed since.
Something on the server maybe \:

[nml375]

w00f:
The EOF state simply means that the connection was dropped while the script was still retrieving the http headers (prematurely closed). Most likely the ssl handshake went wrong.

[w00f]

:/ That's what i though, just tried /usr/bin/wget with the SSL link and it worked like a charm.

So, it's most likely a SSL package problem ?

Using the non-ssl link isn't really a problem, i was just "curious"

Thanks :)

[nml375]

Well, usually it's a matter of setting up a proper CA. I rare cases there might be some issues finding a common set of ciphers between the client and the server.
I'm not familiar with the tsl implementation, but in general when debugging OpenSSL issues, using the openssl s_client commandline client can be very helpful as it can use user-set CA's, verbose output, cerificate chain list, etc.

[w00f]

Well, i've updated the SSL certs (using openssl) then rehashed the certs, tested with curl/wget and it worked, everything seems to work but TCL =/

here's the output from curl

Code: Select all

root@tehplanet:~# curl -I --capath /etc/ssl/certs https://xpto.com
HTTP/1.1 200 OK
Date: Tue, 02 Mar 2010 22:59:58 GMT
Server: Apache
X-Powered-By: PHP/5.2.11
Set-Cookie: PHPSESSID=29o1qivkuab8taakmfhhlg9ig5; path=/
Expires: Wed, 03 Mar 2010 04:59:58 GMT
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
Pragma:
Last-Modified: Tue, 02 Mar 2010 22:59:58 GMT
Connection: close
Content-Type: text/html; charset=UTF-8
root@tehplanet:~#

guess i'll have to exec wget if i want to use the SSL link.

[nml375]

Are you using the very same CA's with your tcl-script? I see that your xpto.com site uses a self-signed certificate. Unless this is listed as a trusted cert for your tls/tcl script, the connection would most likely be dropped.

Also, reading the docs for the tls tcl library, there are several additional parameters for tls connections, including -cadir and -cafile (specifying the root CA), and -command which may be used as a callback to track the status/failure of the tls/ssl handshake.

Further, depending on the version of tcl, you might have to patch the http-package. Supposedly, it's been submitted to the tcl developers to be included since v8.2.1

[speechles]

[11:10p] jackass • s:wget "https://xpto.com/index.php?action=dl&id=133"
[11:10p] -modtcl • [DEBUG] URL: https://xpto.com/index.php?action=dl&id=6437"

It's redirecting you. You've changed my messages to make it harder for me to tell, but this is obviously not the the url you chose. It's the one your being re-directed to.

Code: Select all

PHPSESSID=29o1qivkuab8taakmfhhlg9ig5

It wants cookies too. This is why it's redirecting you. You aren't supplying the cookies during the redirect though. So all you'll get is an eof. Try the code below for a better outcome ...

Note: What follows can work as "the textbook example" for anyone wishing to do the same or anything remotely similar. This covers every possible aspect and scenario that could arise from the http transaction (with the exception of security certificate authentications which nml375 discussed). This also includes redirect-to-self and nesting-too-deep when handling redirect traversals, even handles malicious url's safely. I've seen some poor web scripts that could benefit greatly just by reusing this code. Feel free to reuse it yourself for any non-malicious intents or purposes.

Code: Select all

package require http
package require tls
http::register https 443 [list ::tls::socket -require 0 -request 1]

# recursive wget with cookies and referer
proc s:wget { url {refer ""} {cookies ""} { re 0 } } {
   http::config -useragent "Mozilla/EggdropWget"
   # if we have cookies, let's use em ;)
   if {![string length $cookies]} {
      catch {set token [http::geturl $url -binary 1 -timeout 10000]} error
   } else {
      catch {set http [::http::geturl $url -binary 1 -headers [list "Referer" "$refer" "Cookie" "[string trim [join $cookies {;}] {;}]" ] -timeout 10000]} error
   }
   # error condition 1, invalid socket or other general error
   if {![string match -nocase "::http::*" $error]} {
      s:debug "Error: [string totitle [string map {"\n" " | "} $error]] \( $url \)"
      return 0
   }
   # error condition 2, http error
   if {![string equal -nocase [::http::status $token] "ok"]} {
      s:debug "Http error: [string totitle [::http::status $token]] \( $url \)"
      http::cleanup $token
      return 0
   }
   upvar #0 $token state
   # iterate through the meta array to grab cookies
   foreach {name value} $state(meta) {
      # do we have cookies?                                                           
         if {[regexp -nocase ^Set-Cookie$ $name]} {
         # yes, add them to cookie list                                                        
         lappend ourCookies [lindex [split $value {;}] 0]
      }
   }
   # if no cookies this iteration remember cookies from last
   if {![info exists ourCookies] && [string length $cookies]} {
      set ourCookies $cookies
   }
   # recursive redirect support, 300's
   # the full gambit of browser support, hopefully ... ;)
   if {[string match "*[http::ncode $token]*" "303|302|301" ]} {
      foreach {name value} $state(meta) {
         if {[regexp -nocase ^location$ $name]} {
            if {![string match "http*" $value]} {
               # fix our locations if needed
               if {![string match "/" [string index $value 0]]} {
                  set value "[join [lrange [split $url "/"] 0 2] "/"]/$value"
               } else {
                  set value "[join [lrange [split $url "/"] 0 2] "/"]$value"
               }
            }
            # catch redirect to self's. There is one rule:
            # A url can redirect to itself a few times to attempt to
            # gain proper cookies, or referers. This is hard-coded at 2.
            # We catch the 3rd time and poison our recursion with it.
            # This will stop the madness ;)
            if {[string match [string map {" " "%20"} $value] $url]} {
               if {![info exists poison]} {
                  set poison 1 
               } else {
                  incr poison
                  if {$poison > 2} {
                    s:debug "HTTP Error: Redirect error self to self \(3rd instance poisoned\) \( $url \)""
                    return
                  }
               }
            }
            # poison any nested recursion over 10 traversals deep. no legitimate
            # site needs to do this. EVER!
            if {[incr re] > 10} {
              s:debug "HTTP Error: Redirect error (>10 too deep) \( $url \)"
              return 
            }
            # recursive redirect by passing cookies and referer
            # this is what makes it now work! :)
            s:wget [string map {" " "%20"} $value] $url $ourCookies $re
            # only the last iteration from our recursion is required
            # we save time by using return on prior recurses. this does
            # not poison the recursion because we have invoked ourself
            # before we poison the unneeded iteration.
            return
         }
      }
   }
   # waaay down here, we finally check the ncode for 400 or 500 codes
   if {[string match 4* [http::ncode $token]] || [string match 5* [http::ncode $token]]} {
      s:debug "Http resource is not evailable: [http::ncode $token] \( $url \)"
      return 0
   }
   # uncomment the three lines below to return the http::data
   # check for error by string equaling 0, if it isn't 0 it contains
   # the data returned from the http transaction. If it's an empty
   # reply "", it means the site is either malicious or causing you
   # to have endless redirects these are poison and no html is
   # returned.
   # ---
   # set data [http::data $token]
   # http::cleanup $token
   # return $data
}

[Elfriede]

Code: Select all

set data [http::data $token] 

May i ask one question to that code.
What would be the easiest way to check $data, if theres eg html inside ?

[speechles]

Code: Select all

set data [http::data $token] 

May i ask one question to that code.
What would be the easiest way to check $data, if theres eg html inside ?

The easiest way is by knowing how html is constructed. There is a <head*> as well as a <body*> to most html pages. If these are present there is usually most always, a <title*> as well.

The http::geturl used above in my example meant for the original poster, w00f, includes "-binary 1". This omits the encoding as everything is bit-oriented. HTML obtained using this method probably won't appear correct for any languages that aren't composed of "ascii" characters (utf-8, for example, will lose its sequencing and render things incorrectly) as the HTML loses the encoding associated with the transaction when done in binary.

[w00f]

Yes nml375, i've already tried -cadir and -cafile options, same error.
The certificate it's not self-signed,

Code: Select all

(curl verbose output)

* successfully set certificate verify locations:
*   CAfile: none
* CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*        subject: /O=www.xpto.com/OU=Domain Control Validated/CN=www.xpto.com
*        start date: 2009-11-11 05:55:55 GMT
*        expire date: 2011-11-11 05:55:55 GMT
*        subjectAltName: xpto.com matched
*        issuer: /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07                                                                  
* SSL certificate verify ok.

I'm running Tcl 8.5.


Thanks for the code speechles, but the output is the same (status: eof)
ah! that "redirect" was a typo in the script, my bad.

Anyway, thanks guys =)

[nml375]

Ah well, was reported self-signed when I attempted to connect at that time.. nevertheless, try these settings:

Code: Select all

http::register https 443 [list ::tls::socket -require 0 -request 1 -command ::tls::callback]
set ::tls::debug 3
set ::tls::logcmd putlog

That should dump quite a lot of data into the logs, hopefully revealing what is going on.

Edit: Sorry 'bout that, seems I was investigating the certificate of xpto.org, not xpto.com

Edit again: Should be -command, not -callback

[w00f]

here's the log/handshake

Code: Select all

[01:40a]  -modtcl    • TLS/sock18: handshake/start: before/connect initialization
[01:40a]  -modtcl    • TLS/sock18: connect/loop: before/connect initialization
[01:40a]  -modtcl    • TLS/sock18: connect/loop: SSLv2/v3 write client hello A
[01:40a]  -modtcl    • TLS/sock18: connect/exit: SSLv2/v3 read server hello A
[01:40a]  -modtcl    • TLS/sock18: connect/loop: SSLv3 read server hello A
[01:40a]  -modtcl    • TLS/sock18: connect/exit: SSLv3 read server certificate A
[01:40a]  -modtcl    • TLS/sock18: connect/exit: SSLv3 read server certificate A
[01:40a]  -modtcl    • TLS/sock18: verify/3: Bad Cert: self signed certificate in certificate chain (rc = 0)
[01:40a]  -modtcl    • TLS/sock18: verify/3: /L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class 2 Policy Validation Authority/CN=http://www.valicert.com//emailAddress=info@valicert.com
[01:40a]  -modtcl    • TLS/sock18: verify/2: /C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
[01:40a]  -modtcl    • TLS/sock18: verify/1: /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287
[01:40a]  -modtcl    • TLS/sock18: verify/0: /O=www.xpto.com/OU=Domain Control Validated/CN=www.xpto.com
[01:40a]  -modtcl    • TLS/sock18: connect/loop: SSLv3 read server certificate A
[01:40a]  -modtcl    • TLS/sock18: connect/loop: SSLv3 read server key exchange A
[01:40a]  -modtcl    • TLS/sock18: connect/loop: SSLv3 read server done A
[01:40a]  -modtcl    • TLS/sock18: connect/loop: SSLv3 write client key exchange A
[01:40a]  -modtcl    • TLS/sock18: connect/loop: SSLv3 write change cipher spec A
[01:40a]  -modtcl    • TLS/sock18: connect/loop: SSLv3 write finished A
[01:40a]  -modtcl    • TLS/sock18: connect/loop: SSLv3 flush data
[01:40a]  -modtcl    • TLS/sock18: connect/exit: SSLv3 read finished A
[01:40a]  -modtcl    • TLS/sock18: connect/loop: SSLv3 read finished A
[01:40a]  -modtcl    • TLS/sock18: handshake/done: SSL negotiation finished successfully
[01:40a]  -modtcl    • TLS/sock18: connect/exit: SSL negotiation finished successfully

Then it hangs till times out

Code: Select all

[01:41a]  -modtcl    • TLS/sock18: alert/write: close notify
[01:41a]  -modtcl    • [DEBUG] Request timeout (15s).

timeout only happens with the tls callback.
what does the "verify/3" log means? It's not a self-signed cert lol

[nml375]

That was some interresting results.
The message regarding a self-signed cert means there was one certificate in the chain (not necessary yours) that was self-signed. Root CA certificates are generally self-signed, since there is no "higher" cert to sign it. Instead, we tell our clients (web browsers, etc) that we explicitly trust this cert, and any cert it has signed. An intermediate cert should however never be self-signed.

Setting the ::tls::debug variable to anything higher than 0, wil tell ::tl::callback to approve any cert, regardless of it's validity. The fact that your connection now doesn't close prematurely would suggest there might be some issue with the site-cert or your CA. However, the fact that it now freezes/times out suggests that the web server is not doing too well...

If you keep the ::tls::callback, but set ::tls::debug to 0, do you get the same behaviour?