UNOFFICIAL incith-google 2.1x (Nov30,2o12)

[speechles]

<speechles> !e egg
<sp33chy> 10446 Auctions | 88+12 Brown XLD1 Coturnix Quail Hatching, $6.99($15.00), 0 Bids, 39m @ http://cgi.ebay.com/_W0QQItemZ180283735618 | 6 MOTTLED BLACK TURKEY HATCHING EGGS, $15.50($12.00), 6 Bids, 29m @ http://cgi.ebay.com/_W0QQItemZ370081106690 | 6 "RARE" CHOCOLATE TURKEY HATCHING EGGS, $12.50($12.00), 3 Bids, 19m @ http://cgi.ebay.com/_W0QQItemZ370081104521
<speechles> !e .co.uk egg
<sp33chy> 6,222 matches | MODERN HANDMADE NATURAL WOOL RUG duck eg, £49.99(+£7.99), Buy It Now, 37m @ http://cgi.ebay.co.uk/_W0QQItemZ370081688160 | MODERN,WOOL,BROWN & DUCK EGG BLUE DAMASK, £58.99(+£11.95), 1 Bid , 44m @ http://cgi.ebay.co.uk/_W0QQItemZ370079770996 | A++ 2 pcs RARE Dinosaur BONE + EGG SHEL, £5.19(+£9.29), 1 Bid , 55m @ http://cgi.ebay.co.uk/_W0QQItemZ150287227642

Above is a top/bottom example of both templates together in action. At any time you may be directed to one or the other. Full multi-lingual support is of course 100% operational as well.

<speechles> !e .co.uk egg
<sp33chy> redirected (302): http://search.ebay.co.uk/egg_W0QQpqryZegg -> http://shop.ebay.co.uk/items/_W0QQ_nkwZ ... omZQQ_mdoZ :: cookie: ebay=%5Ecompare-items%3D%7Edomain%3D.ebay.co.uk%7E%5E; Domain=.ebay.co.uk; Path=/ns1=BAQAAARwLwlp8AAaAANgAQ0qcfohjNjN8NTE1XjEyMjAyMzM5OTIyMDBeMF5eMF4zMjAxMjAwYThiMDVeM14yMV41MF4yXjNeM14yXjFeMF4xXjBeMF4wi5OZz9LVANqJjDWOkfH+Lgyp7ak*; Domain=.ebay.co.uk; Expires=Tue, 01-Sep-2009 01:53:
<sp33chy> url (200): http://shop.ebay.co.uk/items/_W0QQ_nkwZ ... omZQQ_mdoZ charset: iso8859-1 encode_string: iso8859-1 :: cookie: ebay=%5Ecompare-items%3D%7Edomain%3D.ebay.co.uk%7E%5E; Domain=.ebay.co.uk; Path=/ns1=BAQAAARwLwlp8AAaAANgAQ0qcfohjNjN8NTE1XjEyMjAyMzM5OTIyMDBeMF5eMF4zMjAxMjAwYThiMDVeM14yMV41MF4yXjNeM14yXjFeMF4xXjBeMF4wi5OZz9LVANqJjDWOkfH+Lgyp7ak*; Domain=.ebay.co.uk; Expires=Tue, 01-Sep-2009 01:53:12 GMT; Pa

The new ebay template is now included and 100% functional. There may slight problems with some items I've not run across yet. Adding this new template added new abilities to this script that were required. First off, you have an entirely new debug line as shown above. This includes cookie handling (which ebay requires for some servers) and now this script can intelligently redirect and supports full cookie handling. The redirects are recursive of sorts without endless looping, so can handle an infinite number which the debug output will diligently report to your debugnick (for example, ebay italy requires 3-5 redirects with full cookie support before any html will be revealed [see quote below]). This will be needed when adding full support for iGN later, they require cookies as well and more than a single redirect. This can now easily be accommodated for with the added abilities.

The only broken functions will be both related to gamespot. With their recent addition of ajax programming it is too difficult to attempt to scrape their search results any longer. Therefore, you will find both !game and !review always return the same static message stating no results found. This is not a bug nor a misfeature. Simply put, this fox has been outsmarted.

Without further delay, you can get the new script HERE or the 1st page of this thread. There may some inconsistencies with ebay at the moment because of it's new template. You may also notice that some sites now work that refused to before. Credit this entirely to the redirect and cookie support.

<speechles> !e .it subato
<sp33chy> redirected (302): http://search.ebay.it/subato_W0QQpqryZsubato -> http://shop.ebay.it/cqtranslate/_W0QQ_n ... omZQQ_mdoZ :: cookie: ebay=%5Ecompare-items%3D%7Edomain%3D.ebay.it%7E%5E; Domain=.ebay.it; Path=/ns1=BAQAAARwLwlp8AAaAANgARkqchSdjNjZ8NTE1XjEyMjAyMzU2ODczMzVeMF5eMF4zYTAxMjEwODhhMDdeM14wXjUwXjJeNV41XjReM14yXjFeMF4xXjBeMF4wCqSCuKOL2aiEAbYKksGbJk2CmS0*; Domain=.ebay.it; Expires=Tue, 01-Sep-2009 02
<sp33chy> redirected (302): http://shop.ebay.it/cqtranslate/_W0QQ_n ... omZQQ_mdoZ -> http://shop.ebay.it/items/__subato_W0QQ_nkwZsubato :: cookie: ebay=%5Ecompare-items%3D%7Edomain%3D.ebay.it%7E%5Ecv%3D15555%5E; Domain=.ebay.it; Path=/ns1=BAQAAARwLwlp8AAaAANgARUqchShjNjV8NTYzXjEyMjAyMzU2ODgxMTleXjFeM3wyfDUwfDR8NXwxMXw3fDQzXjFeMl40XjJeMTJeMTJeMl4xXjFeMF4xXjDKhmQHwjiMXaAKVCRD2rGGj2R25A**; Domain=.ebay.it; Expires=
<sp33chy> redirected (302): http://shop.ebay.it/items/__subato_W0QQ_nkwZsubato -> http://shop.ebay.it:80/items/___W0QQ_nkwZsubato :: cookie: ebay=%5Ecompare-items%3D%7Edomain%3D.ebay.it%7E%5Ecv%3D15555%5E; Domain=.ebay.it; Path=/ns1=BAQAAARwLwlp8AAaAANgARUqchShjNjV8NTYzXjEyMjAyMzU2ODg0ODVeXjFeM3wyfDUwfDR8NXwxMXw3fDQzXjFeMl40XjJeMTJeMTJeMl4xXjFeMF4xXjDh+sljWM2yyj5len0DsLaIjYEVCw**; Domain=.ebay.it; Expires=Tue, 01-Sep-2009 02:21:28
<sp33chy> redirected (302): http://shop.ebay.it:80/items/___W0QQ_nkwZsubato -> http://shop.ebay.it/items/subito_W0QQ_n ... uscZsubato :: cookie: ds2=alss/0.subato48bca328^; Domain=.ebay.it; Path=/ebay=%5Ecompare-items%3D%7Edomain%3D.ebay.it%7E%5Ecv%3D15555%5E; Domain=.ebay.it; Path=/ns1=BAQAAARwLwlp8AAaAANgARUqchShjNjV8NTYzXjEyMjAyMzU2ODg1NjNeXjFeM3wyfDUwfDR8NXwxMXw3fDQzXjFeMl40XjJeMTJeMTJeMl4xXjFeMF4xXjAjyY
<sp33chy> url (200): http://shop.ebay.it/items/subito_W0QQ_n ... uscZsubato charset: iso8859-1 encode_string: iso8859-1 :: cookie: ds2=alss/0.subato48bca328^; Domain=.ebay.it; Path=/ebay=%5Ecompare-items%3D%7Edomain%3D.ebay.it%7E%5Ecv%3D15555%5E; Domain=.ebay.it; Path=/ns1=BAQAAARwLwlp8AAaAANgARUqchShjNjV8NTYzXjEyMjAyMzU2ODg1NjNeXjFeM3wyfDUwfDR8NXwxMXw3fDQzXjFeMl40XjJeMTJeMTJeMl4xXjFeMF4xXjAjyYvToUyldchz4VWOU8

Jumping through hoops is not a problem. It just takes a little bit longer to get there.. heh

Some people might complain about google not working. For those I suggest you take a look at the debug output. If you see url (403): displayed along with google's sorry page and showing an empty cookie field. You will need to visit HERE and do what it suggests. Eventually I will add a method to inject google cookies into this script. So that you can use a browser on the same machine to generate the cookie and pass the captcha requirements. Then this script will read that same cookie into itself and use it to pass authentication and avoid this sorry page. Until then the 403 error if displayed means google is blocking your bot and this script from working.

Other than that, Enjoy, and most importantly.. have a fun

[madwoota]

The only broken functions will be both related to gamespot. With their recent addition of ajax programming it is too difficult to attempt to scrape their search results any longer. Therefore, you will find both !game and !review always return the same static message stating no results found. This is not a bug nor a misfeature. Simply put, this fox has been outsmarted.

If you're interested, it's not terribly difficult to get that ajax data. That IS kinda the whole point of it... If you're happy to accept suggestions from someone as lame as me of course?
Fyi, it's about 3 lines of code and the result is so easy to parse - even I could do it!

[grindking]

Some people might complain about google not working. For those I suggest you take a look at the debug output. If you see url (403): displayed along with google's sorry page and showing an empty cookie field. You will need to visit HERE and do what it suggests. Eventually I will add a method to inject google cookies into this script. So that you can use a browser on the same machine to generate the cookie and pass the captcha requirements. Then this script will read that same cookie into itself and use it to pass authentication and avoid this sorry page. Until then the 403 error if displayed means google is blocking your bot and this script from working.

I think this may be happening to me but need a bit more info on the debug settings. i have the variable debug/debugnicks setup accordingly, but are the debug outputs usually printed on the console if you're chatting with the bot? I had an old ig-debug.txt in the root directory of my installation but it never is recreated or nothing is printed to it if i touch the file and give it write access.

basically google stopped working awhile ago, thought it was my linux machine somehow because some other friends of mine on efnet (em0) have it working perfectly. i decided to go ahead and install a windrop on my machine even though i have no idea if this is a good test, but the script DOES in fact work on the windrop install. I recompiled a newer 1.6.19 egg on my linux machine that it wasn't working on and the script still doesn't work.

so even without the debug printing for me, that's pretty strange unless it's acting exactly how you describe

what's the best way to test this other than the steps you've already mentioned, or a better debug output? i'm sure i'm doing something wrong in that regard atleast.

UPDATE: actually i have been getting the debug reports properly from the windrop but not my old eggdrop (v1.6.18/tcl8.4/http2.5) on my linux machine.

i could safely assume something is going on with my linux machine i think atleast? hehe, ill wait to hear back i just saw i was receiving debug from the windrop and my settings are the same on the linux side yet i don't. maybe another script conflicting?

[speechles]

The only broken functions will be both related to gamespot. With their recent addition of ajax programming it is too difficult to attempt to scrape their search results any longer. Therefore, you will find both !game and !review always return the same static message stating no results found. This is not a bug nor a misfeature. Simply put, this fox has been outsmarted.

If you're interested, it's not terribly difficult to get that ajax data. That IS kinda the whole point of it... If you're happy to accept suggestions from someone as lame as me of course?
Fyi, it's about 3 lines of code and the result is so easy to parse - even I could do it!

I've figured it out actually. What ya gotta do is keep track of 4 things. The referring page must be gamespot, the cookie must be a valid session (these are easy) and finally, the coup-de-grace, the X headers "X-Requested-With XMLHttpRequest" and his cousin "X-Request JSON". If all these header fields are given in 3rd page attempt, one being based upon "http://www.gamespot.com/pages/search/se ... &sort=rank" rather than the original gamespot url, I now see the xml which the javascript would insert into the html. The old template of course can't scrape these raw results but the new one surely will. Expect it shortly and gamespot will work once again. Hurray ;D

Code: Select all

      if {$switch == 12} {
        # grab the ajax data
        set http [::http::geturl "http://www.gamespot.com/pages/search/search_ajax.php?q=${input}&type=game&offset=0&tags_only=false&sort=rank" -query "" -headers "X-Requested-With XMLHttpRequest X-Request JSON Referer $query Cookie [list [string trimright $cookies "; "]]" -timeout [expr 1000 * 10]]
        # strip those escapes and get that xml
        set html [string map {"\" ""} [::http::data $http]] 
        upvar #0 $http state
        set incithcharset [string map -nocase {"UTF-" "utf-" "iso-" "iso" "windows-" "cp" "shift_jis" "shiftjis"} $state(charset)]
      }

Basically, you just add that between the #REDIRECT part in the fetch_html right before the ::http::cleanup is called but after the last closing brace } and you can clearly see the xml waiting to be scraped in ig-debug.txt .. nice

Oh yeah, and the cookie building method originally used was flawed somewhat, the new script will correct that along with any odd behavior caused as a result..

<speechles> !game mario kart
<sp33chy> 17 games | Game: Mario Kart Wii (WII) (Review Score: 8.5 | Release Date: Apr 27, 2008) - The venerable kart racer heads to the Wii and adds online competition to the mix. @ http://www.gamespot.com/wii/driving/mar ... index.html | Game: Mario Kart DS (DS) (Review Score: 9.2 | Release Date: Nov 14, 2005) - Now you can play Mario Kart against opponents all over the world with Mario
<sp33chy> Kart DS. Mario Kart DS includes new tracks, as well as retro tracks selected from all the previous Mario Kart versions. @ http://www.gamespot.com/ds/driving/mari ... index.html | Game: Mario Kart: Double Dash!! (GC) (Review Score: 7.9 | Release Date: Nov 17, 2003) - Despite the fact that it's still essentially the same game as Super Mario Kart, there is no denying the innate
<sp33chy> quality of the production of Double Dash. @ http://www.gamespot.com/gamecube/drivin ... index.html

Just a teaser for now. Gamespot !game is fully functional, next up is review. Should have something for the public tomorrow.

[speechles]

I think this may be happening to me but need a bit more info on the debug settings. i have the variable debug/debugnicks setup accordingly, but are the debug outputs usually printed on the console if you're chatting with the bot? I had an old ig-debug.txt in the root directory of my installation but it never is recreated or nothing is printed to it if i touch the file and give it write access.

basically google stopped working awhile ago, thought it was my linux machine somehow because some other friends of mine on efnet (em0) have it working perfectly. i decided to go ahead and install a windrop on my machine even though i have no idea if this is a good test, but the script DOES in fact work on the windrop install. I recompiled a newer 1.6.19 egg on my linux machine that it wasn't working on and the script still doesn't work.

so even without the debug printing for me, that's pretty strange unless it's acting exactly how you describe

what's the best way to test this other than the steps you've already mentioned, or a better debug output? i'm sure i'm doing something wrong in that regard atleast.

UPDATE: actually i have been getting the debug reports properly from the windrop but not my old eggdrop (v1.6.18/tcl8.4/http2.5) on my linux machine.

i could safely assume something is going on with my linux machine i think atleast? hehe, ill wait to hear back i just saw i was receiving debug from the windrop and my settings are the same on the linux side yet i don't. maybe another script conflicting?

<bot> redirected (302): http://www.google.com/search?q=stuff&sa ... all&num=10 -> http://sorry.google.com/sorry/?continue ... 26num%3D10 :: cookie:
<bot> url (403): http://sorry.google.com/sorry/?continue ... 26num%3D10 charset: utf-8 encode_string: iso8859-1 :: cookie:

The debug output does not appear on the console or partyline. It will appear in a seperate query/pm window as if the bot is messaging you. This is what a 403 sorry redirect from google looks like. You must complete the captcha using a capable browser on the IP the bot is on. It will then generate a cookie entry. Presently the script attempts bare requests to google before attempting to offer cookies. But if I knew what the filename of the cookie was and the path to said cookie. Instead of doing the initial bare request to google, I could instead use the cookie generated by the capable browser (checking expiration and if expired simply ask to generate the cookie again with a capable browser) and pass that to google immitating it and avoiding the sorry page. This will eventually make it into the script as the quote above is from the bot of a friend of mine. Since I have the ability to test with him this feature it should be easy to accomplish and avoid that sorry once and for all.

Also, the script has a hidden debug feature discussed before. If you look in your eggdrop's root directory there should be a file called ig-debug.txt. This file will contain the latest html the bot has retrieved. So if you issue a !google anything and download the file from your bot immediately afterwards. You can rename it to test.html and load that into a browser. If it is indeed the google sorry page contained in it, that will solve the riddle of the non functional eggdrop

[grindking]

Also, the script has a hidden debug feature discussed before. If you look in your eggdrop's root directory there should be a file called ig-debug.txt. This file will contain the latest html the bot has retrieved. So if you issue a !google anything and download the file from your bot immediately afterwards. You can rename it to test.html and load that into a browser. If it is indeed the google sorry page contained in it, that will solve the riddle of the non functional eggdrop

yah in the last part of my previous update i said i was able to get the debug output working properply (on the windrop ONLY) and as of now my linux eggdrop won't even message me the debug output on attempts made.

for some reason my ig-debug.txt continues to stay at 0 bytes even on attempts as well, so i must have some other issue going on with my machine at this point. i'm looking into all that for now and thanks for the help.

[speechles]

Also, the script has a hidden debug feature discussed before. If you look in your eggdrop's root directory there should be a file called ig-debug.txt. This file will contain the latest html the bot has retrieved. So if you issue a !google anything and download the file from your bot immediately afterwards. You can rename it to test.html and load that into a browser. If it is indeed the google sorry page contained in it, that will solve the riddle of the non functional eggdrop

yah in the last part of my previous update i said i was able to get the debug output working properply (on the windrop ONLY) and as of now my linux eggdrop won't even message me the debug output on attempts made.

for some reason my ig-debug.txt continues to stay at 0 bytes even on attempts as well, so i must have some other issue going on with my machine at this point. i'm looking into all that for now and thanks for the help.

If you have .set enabled, you can try logging onto the bots partyline before issuing the command. Right after it's done, type .set errorInfo on the partyline and it might shed more like on the exact problem causing the interpreter to fail. If you consistantly get a 0byte ig-debug.txt the problem is likely occuring during http gets. The script only checks for a timeout or a simple socket error, more advanced errors aren't interpreted and the page body is attempted to be stored into ig-debug.txt. The debug line (if it was shown to you) would reveal the exact ncode error occuring if there were one. Since it doesn't even get that far i'm guessing the parser itself isn't getting variables it needs and is causing tcl errors because it expects some html to parse rather than an empty file. This halts the script execution and returns. Checking the contents of .set errorInfo would confirm this.

[grindking]

If you have .set enabled, you can try logging onto the bots partyline before issuing the command. Right after it's done, type .set errorInfo on the partyline and it might shed more like on the exact problem causing the interpreter to fail. If you consistantly get a 0byte ig-debug.txt the problem is likely occuring during http gets. The script only checks for a timeout or a simple socket error, more advanced errors aren't interpreted and the page body is attempted to be stored into ig-debug.txt. The debug line (if it was shown to you) would reveal the exact ncode error occuring if there were one. Since it doesn't even get that far i'm guessing the parser itself isn't getting variables it needs and is causing tcl errors because it expects some html to parse rather than an empty file. This halts the script execution and returns. Checking the contents of .set errorInfo would confirm this.

took me a bit to comment out the unbinding of tcl/set in the cfg hehe..

anyway here's some output from errorInfo:

Unknown option -urlencoding, must be: -accept, -proxyfilter, -proxyhost, -proxyport, -useragent
while executing
"::http::config -useragent $ua -urlencoding "utf-8""
(procedure "fetch_html" line 254)
invoked from within
"fetch_html $input 1"
(procedure "google" line 21)
invoked from within
"google $search"
(procedure "send_output" line 56)
invoked from within
"send_output "$input" "$chan" "$nick" "$uhand""
(procedure "incith::google::public_message" line 16)
invoked from within
"incith::google::public_message $_pubm1 $_pubm2 $_pubm3 $_pubm4 $_pubm5"

[speechles]

If you have .set enabled, you can try logging onto the bots partyline before issuing the command. Right after it's done, type .set errorInfo on the partyline and it might shed more like on the exact problem causing the interpreter to fail. If you consistantly get a 0byte ig-debug.txt the problem is likely occuring during http gets. The script only checks for a timeout or a simple socket error, more advanced errors aren't interpreted and the page body is attempted to be stored into ig-debug.txt. The debug line (if it was shown to you) would reveal the exact ncode error occuring if there were one. Since it doesn't even get that far i'm guessing the parser itself isn't getting variables it needs and is causing tcl errors because it expects some html to parse rather than an empty file. This halts the script execution and returns. Checking the contents of .set errorInfo would confirm this.

took me a bit to comment out the unbinding of tcl/set in the cfg hehe..

anyway here's some output from errorInfo:

Unknown option -urlencoding, must be: -accept, -proxyfilter, -proxyhost, -proxyport, -useragent
while executing
"::http::config -useragent $ua -urlencoding "utf-8""
(procedure "fetch_html" line 254)
invoked from within
"fetch_html $input 1"
(procedure "google" line 21)
invoked from within
"google $search"
(procedure "send_output" line 56)
invoked from within
"send_output "$input" "$chan" "$nick" "$uhand""
(procedure "incith::google::public_message" line 16)
invoked from within
"incith::google::public_message $_pubm1 $_pubm2 $_pubm3 $_pubm4 $_pubm5"

I knew it would be something amazingly simple. That means you truly arent using http 2.5 even tho you think you are. What you need to do is go to your /lib/tcl8.*/ folder and copy http.tcl version 2.5 to the root of that folder. It's fine if you have them in folders as well, but the dominant copy of http is the one in the root. To enforce this, place 2.5 there and it should solve this problem for you. Make sure you DO NOT source http within your eggdrop.conf this will cause a conflict and possibly crash your bot.

[grindking]

I knew it would be something amazingly simple. That means you truly arent using http 2.5 even tho you think you are. What you need to do is go to your /lib/tcl8.*/ folder and copy http.tcl version 2.5 to the root of that folder. It's fine if you have them in folders as well, but the dominant copy of http is the one in the root. To enforce this, place 2.5 there and it should solve this problem for you. Make sure you DO NOT source http within your eggdrop.conf this will cause a conflict and possibly crash your bot.

well i've located some older versions of http.tcl and cleaned up my libs a bit so that now i only have tcl8.4 libs there, with only http 2.5 tcl.

i'm still getting the same Unknown option -urlencoding, must be: -accept, -proxyfilter, -proxyhost, -proxyport, -useragent for some unknown reason and maybe just cuz it's getting late or something. i guess i'm going to have to look more into it tomorrow cuz it sure doesn't make sense.

i mean it says $Id: http.tcl,v 2.5.001 2004/09/08 13:36:10 perpleXa Exp $

as my version in every http.tcl file on my linux machine


thanks for all the help btw, this is just an unstable debian install with normal packages and stuff, so maybe it's related to that. i was going to install tcl8.5 for s* and giggles but figured it might introduce other points of failure.

[speechles]

I knew it would be something amazingly simple. That means you truly arent using http 2.5 even tho you think you are. What you need to do is go to your /lib/tcl8.*/ folder and copy http.tcl version 2.5 to the root of that folder. It's fine if you have them in folders as well, but the dominant copy of http is the one in the root. To enforce this, place 2.5 there and it should solve this problem for you. Make sure you DO NOT source http within your eggdrop.conf this will cause a conflict and possibly crash your bot.

well i've located some older versions of http.tcl and cleaned up my libs a bit so that now i only have tcl8.4 libs there, with only http 2.5 tcl.

i'm still getting the same Unknown option -urlencoding, must be: -accept, -proxyfilter, -proxyhost, -proxyport, -useragent for some unknown reason and maybe just cuz it's getting late or something. i guess i'm going to have to look more into it tomorrow cuz it sure doesn't make sense.

i mean it says $Id: http.tcl,v 2.5.001 2004/09/08 13:36:10 perpleXa Exp $

as my version in every http.tcl file on my linux machine


thanks for all the help btw, this is just an unstable debian install with normal packages and stuff, so maybe it's related to that. i was going to install tcl8.5 for s* and giggles but figured it might introduce other points of failure.

There is another way to fix it. You can safely remove the -urlencoding "utf-8" and the -urlencoding "iso8859-1" from the ::http::config portions of the script. It should still behave exactly as it does with or without them (for the most part). The only reason for adding them is to ensure encodings are always 100% correctly interpreted.

Code: Select all

set http [::http::config -useragent $ua -urlencoding "utf-8"]
set http [::http::config -useragent $ua]

For example, in wikipedia the above can be done. You can do the same for every instance of -urlencoding.

[grindking]

There is another way to fix it. You can safely remove the -urlencoding "utf-8" and the -urlencoding "iso8859-1" from the ::http::config portions of the script. It should still behave exactly as it does with or without them (for the most part). The only reason for adding them is to ensure encodings are always 100% correctly interpreted.

Code: Select all

set http [::http::config -useragent $ua -urlencoding "utf-8"]
set http [::http::config -useragent $ua]

For example, in wikipedia the above can be done. You can do the same for every instance of -urlencoding.

i just did that really quickly to see and that fixed it. anything i should do maybe to use it without removing that, or is that some bizarre instance because of my installation?

[madwoota]

I've figured it out actually. What ya gotta do is keep track of 4 things.

No you dont.

The referring page must be gamespot

No it doesn't.

the cookie must be a valid session

Not required.

and finally, the coup-de-grace, the X headers "X-Requested-With XMLHttpRequest" and his cousin "X-Request JSON".

Win!

Save yourself some time and just send the json req straight at search_ajax.php with the X- headers and set http::config to -accept "application/json".

[testebr]

Save yourself some time and just send the json req straight at search_ajax.php with the X- headers and set http::config to -accept "application/json".

Really, very very simple like my test with php curl:

Code: Select all

<?

$curl = curl_init();

$url = 'http://www.gamespot.com/pages/search/search_ajax.php?q=grid&type=game&offset=0&tags_only=false&sort=rank';

curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_HTTPHEADER, array("Accept: application/json","X-Requested-With: XMLHttpRequest"));
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);

$result = curl_exec($curl);

echo $result;

?>

No cookies, no session, no referrer, nothing more...

And here the live test with code above: http://t7ds.dreamhosters.com/gamespot.php

[speechles]

<speechles> !review mgs
<sp33chy> Metal Gear Solid 4: Guns of the Patriots for PS3 (us10 them9.3 you9.5) « Konami Kojima Productions Modern Action Adventure Release: Jun 12, 2008 » « ESRB: Mature » « Rank: 10 of 51,409 (up by 1) PS3 Rank: 2 of 572 Highest Rank:1 in 2008 » « Number of Players: 1 Player, 16 Online » « Metal Gear Solid 4 is an awe-inspiring synthesis of dramatic storytelling and entertaining gameplay. » @
<sp33chy> http://www.gamespot.com/ps3/adventure/m ... index.html ...... Video: Franchise Player: Metal Gear Solid 4: Guns of the Patriots @ http://www.gamespot.com/ps3/adventure/m ... e-patriots

<speechles> !r halo @ x360
<sp33chy> Halo 3 for Xbox 360 (us9.5 them9.4 you8.9) « Microsoft Game Studios Bungie Software Sci-Fi First-Person... Release: Sep 25, 2007 » « ESRB: Mature » « Rank: 31 of 51,409 (up by 3) X360 Rank: 11 of 858 » « Number of Players: 1-4, 16 Online | Offline Modes: Competitive, Cooperative, Team Oriented | Online Modes: Competitive, Cooperative, Team Oriented » « Halo 3 builds upon the concepts of Halo 2 in ways
<sp33chy> that you'd expect, but there are also new modes and options that send the series in exciting new directions. » @ http://www.gamespot.com/xbox360/action/halo3/index.html ...... Video: Video Q&A: Shane Kim on more Gears @ http://www.gamespot.com/xbox360/action/ ... more-gears

Review is now quite elaborate. All sorts of goodies to read now, even the top video link referenced in the article. There are bound to be some anamolies as I haven't had time to fully check every single game against the parser. I took a simplistic approach to the parsing because of how gamespot uses weird spacing, tabs, and other clutter to make it difficult. This is why anamolies might pop up every now and then.

You can find the new script HERE or as always the first post of this thread. Remember, most important, have a fun