UNOFFICIAL incith-google 2.1x (Nov30,2o12)

neofutur · Post by **neofutur** » Wed Oct 14, 2009 5:45 pm

ajc13 wrote:Looking for some assistance, my apologies if this is the wrong spot.

When I attempt to invoke '!google' I receive the following:
Tcl error [incith::google::public_message]: can't read "state(body)": no such variable

same here, same message :
Tcl error [incith::google::public_message]: can't read "state(body)": no such variable

it seems google change their homepage recently to obfuscate results . . .

wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" "http://www.google.com/search?btnI=&q=Deprecated Function"

recently became very different than :

wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" "http://www.google.fr/search?btnI=&q=Deprecated Function"

the google.com page seems now very obfuscated

speechles · Post by **speechles** » Wed Oct 14, 2009 6:38 pm

ajc13 wrote:Looking for some assistance, my apologies if this is the wrong spot.

When I attempt to invoke '!google' I receive the following:
Tcl error [incith::google::public_message]: can't read "state(body)": no such variable

Suggestions/redirections?

incith-google 1.9.9t (Sep25,2oo9)

running eggdrop v1.6.19+ctcpfix
OS: Linux 2.6.28-15-server
Tcl library: /usr/share/tcltk/tcl8.5
Tcl version: 8.5.6 (header version 8.5.6)
Tcl is threaded.

[23:05] Incith:Google compression test successful, found Trf package! Gzip enabled.
[23:05] - UNOFFICIAL incith:google-1.9.9t loaded.

Appears it found and enabled gzip. You can try disabling support for this.

Code: Select all

# Change this to 0 to disable gzip completely
variable use_gzip 1

If the error still persists, then I'll need more information from you. Such as right after this occurs, and the file ig-debug.txt is created in your eggdrop's root. What does your copy of it contain? Is it empty? Btw, ig-debug.txt contains the gathered html after a successful get/strip. Gzip inflation occurs before this file is written/created. Gzip inflation will only occur if the sending website has indicated the data is gzip encoded as well. Hopefully merely turning off gzip support solves it, although for me it works well. The only difference is I'm using the zlib package to support it. Haven't hide time to test using Trf nor had any users using Trf complain. So this may be the first problem concerning those using that package. The method (headerless unzip) using Trf to support gzip was borrowed from scottey's rss synd script and as such is expected to work the same in either script.

Also, will be a new version shortly to fix a few issues I've found and corrected in ebay (shipping/bid fix), google (result totals work again), and youtube (HD fixes). There are still bound to be tiny little inconsistencies here or there and since the scope of this script is so large I focus more time keeping the larger things that work right doing so, than the minor few which aren't. Stay tuned ;P

ajc13 · Post by **ajc13** » Wed Oct 14, 2009 6:55 pm

Thanks for the response.

speechles wrote: Appears it found and enabled gzip. You can try disabling support for this.

It initially complained that Trf was not found.

This is an Ubuntu host, so I installed the libtrf-tcl package (2.1.2~20071113-2).

Setting the disable and retesting.

On retest, I get the bot blocking hanging (high cpu%), ig-debug.txt contains... an ugly line from google...

speechles · Post by **speechles** » Wed Oct 14, 2009 7:00 pm

neofutur wrote:wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" "http://www.google.fr/search?btnI=&q=Deprecated Function"

the google.com page seems now very obfuscated

This is most likely because the IP used is still one in use by the script/bot. If you've allowed gzip and the script can find support for it in either Trf/zlib packages or finds the commands it needs already available it will affix a header attribute to each query it makes. This attribute tells the website to send the reply back gzipped (compressed). What your seeing with wget is probably their reply sent to you compressed as well since it's made using the same IP as the script which just made a gzip request earlier. Make sense?

speechles · Post by **speechles** » Wed Oct 14, 2009 7:04 pm

ajc13 wrote:On retest, I get the bot blocking hanging (high cpu%), ig-debug.txt contains... an ugly line from google...

Ugly line? Does it look like compressed data?
btw, It's always going to be just a single line. The script strips all newlines, the .txt is made after the html is cleaned up and right before it's sent back to the main procedure to do further processing.

And.. does just !google not work. Have you tried any other triggers? Do any of these suffer similarly? It's easier for me to spot the source of the problem since I'm not experiencing it if you could do a little detective work as well. Have you changed "debugnick" within the config to your nickname. If so, does the bot message you the query it just made? What was the query string? With these answers I should be able to correct the problem.

neofutur · Post by **neofutur** » Wed Oct 14, 2009 7:14 pm

speechles wrote:
neofutur wrote:wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" "http://www.google.fr/search?btnI=&q=Deprecated Function"

the google.com page seems now very obfuscated
This is most likely because the IP used is still one in use by the script/bot. If you've allowed gzip and the script can find support for it in either Trf/zlib packages or finds the commands it needs already available it will affix a header attribute to each query it makes. This attribute tells the website to send the reply back gzipped (compressed). What your seeing with wget is probably their reply sent to you compressed as well since it's made using the same IP as the script which just made a gzip request earlier. Make sense?

no i made it from another Ip adress, and the was not gzipped

neofutur · Post by **neofutur** » Wed Oct 14, 2009 7:21 pm

speechles wrote:
ajc13 wrote: Tcl error [incith::google::public_message]: can't read "state(body)": no such variable
Appears it found and enabled gzip. You can try disabling support for this.
Code: Select all
# Change this to 0 to disable gzip completely
variable use_gzip 1

this workaround worked for me !
the script is working with variable use_gzip 0

and I get the same error again if i go back to use_gzip 1

thanks for your answers !

to help you i also tried the debugnick
the debug works when I have use_gzip 0 but i receive nothing whent having use_gzip 1

same for ig-debug.txt, the file is written when I have use_gzip 0 but nothing is written in ig-debug.txt when I have use_gzip 1

ajc13 · Post by **ajc13** » Wed Oct 14, 2009 7:29 pm

speechles wrote: Ugly line? Does it look like compressed data?
btw, It's always going to be just a single line. The script strips all newlines, the .txt is made after the html is cleaned up and right before it's sent back to the main procedure to do further processing.

And.. does just !google not work. Have you tried any other triggers? Do any of these suffer similarly? It's easier for me to spot the source of the problem since I'm not experiencing it if you could do a little detective work as well. Have you changed "debugnick" within the config to your nickname.

Am trying mate, appreciate the patience.

Yes, the single line of dense html, javascript.

After setting the debug tried .google utah and bot retrieved a maps line and received a msg from the bot with the query.

I do '.google google' and it does not post the query in private and the bot then falls offline - it does manage to create your ig-debug.txt though.

It appears to get stuck processing that result?

That result posted:
http://pastebin.com/m2392959a

tscolin · Post by **tscolin** » Thu Oct 15, 2009 2:53 pm

i get this error when using google fight

[14:52] Tcl error [incith::google::public_message]: can't read "matches1": no such variable

?fight doesnt work

speechles · Post by **speechles** » Fri Oct 16, 2009 6:51 pm

ajc13 wrote:i get this error when using google fight

[14:52] Tcl error [incith::google::public_message]: can't read "matches1": no such variable

?fight doesnt work

This is related to !google's "total_results" not showing (ie. 523,423 results). Whether or not you display them this is used internally by the script with google searches to determine how many results you want versus how many there are. This is faster than the normally used method of keep recursing the page matching/removing until you can't anymore or you've reached the number of results you need. If there are 2 results and you've set it to show 4, using this method would cause a noticeable delay when returning only 1, 2 or 3 results compared to 4, 5, 6 . This is because the last (few) loop(s) of the regexp/regsub combination (parsing) will not match. But the time checking that this will not match will be a time penalty. The alternative faster way is to scrape out the results ahead of time, and when the amount we want is more than the results scraped then we save the time penalty. We know when to stop perfectly. If the script cannot scrape results, it will accept this time penalty and parse the old imperfect way, but of course !fight requires these totals and assumes it will get them and throws that error. The new detection also includes all prior detection schemes as fall-backs where the newest scheme each site uses takes precedence, and if these fail, it attempts each template before and so-on. So compatibility between regional servers use of new changes is kept at the level it previously was. As no previous template was removed, simply new ones added. There are probably a few which could be pruned, but all this would do is give the "no search results" found message faster.

ajc13 wrote:Am trying mate, appreciate the patience.

Yes, the single line of dense html, javascript.

After setting the debug tried .google utah and bot retrieved a maps line and received a msg from the bot with the query.

I do '.google google' and it does not post the query in private and the bot then falls offline - it does manage to create your ig-debug.txt though.

It appears to get stuck processing that result?

That result posted:
http://pastebin.com/m2392959a

Correct you are, for some reason it continuously matches and the "short_answers" is the problem here. Along with keeping the results short which is basically using any "one-box" answers, or several other sepecial templates as the only result that shows. The problem with this, is it also skipped those "inter-google" links, where google injects links to its own services for some things which is sometimes annoying. To skip this needless "feature" of google, I used a crude but fast method of detecting when to do so. To do this right the matching portion needs to be removed before recursion occurs and the same portion matches again and again infinitely. This is what is happening, so to avoid it this "hidden benefit" of short_answers will cease to exist. Once I properly find a method to skip these types of links which doesn't cause infinite loops a new option will be created to cover these types, rather than tying the two together. So for now to avoid this problem, set "short_answers" to 0. This will cause more results that you might want to appear with calculations, weather, etc.. but this weekend possibly a new version which has been tested somewhat thoroughly (/whois sp33chy on efnet if your curious where it's tested) will be released to fix everything *fingers crossed*

Jonathan1683 · Post by **Jonathan1683** » Sun Oct 18, 2009 1:21 pm

is there a way to fix this?
Tcl error [mess_join]: couldn't open "data/h_message.dat": no such file or directory
Tcl error [incith::google::public_message]: invalid command name "zlib"

I checked with the admin and he told me it was running 8.6 TCL can I just disable zlib ?

/home/admin# locate tcl86
./usr/local/lib/libtcl86.so
./usr/local/lib/libtcl86.so.1
./usr/local/lib/libtcl86.a

speechles · Post by **speechles** » Sun Oct 18, 2009 4:25 pm

Jonathan1683 wrote:is there a way to fix this?
Tcl error [mess_join]: couldn't open "data/h_message.dat": no such file or directory
Tcl error [incith::google::public_message]: invalid command name "zlib"

I checked with the admin and he told me it was running 8.6 TCL can I just disable zlib ?

/home/admin# locate tcl86
./usr/local/lib/libtcl86.so
./usr/local/lib/libtcl86.so.1
./usr/local/lib/libtcl86.a

Both those problems aren't inherited from this script. Only the bottom error is. This is related to tcl8.6 which has two flavors (where zlib is concerned) depending on the beta version they are running. Some include zlib, some do not. Their flavor obviously does not. That error will not show normally useless your using a deprecated version of the script (prior to version s) or attempt self made changes (mentioned hacks). Update to the most recent version found on the 1st link of this thread. Then find the option below:

Code: Select all

    variable use_gzip 1

Change it to 0. Now it won't attempt to use gzip compression for anything, solved. Make sure to please READ the config section completely and set ALL you options as you see fit. Leaving them default is fine to get the script up and operating fast. After this initial "testing-phase" it is expected you will then MODIFY the config section completely creating a theme and feel your users will prefer. If you need help designing a look/feel that is right make sure to mention so. It's probably already possible to do using existing configuration options which is why I suggest you not ignore these powerful settings and make use of them.

Check this thread in an hour or so, version "u" will be available which corrects all the problems mentioned above and then some..

Jonathan1683 · Post by **Jonathan1683** » Sun Oct 18, 2009 9:55 pm

Thank you, I am not sure about what tcl he is using and I don't think he knows much about it either so I really don't know what to do in this regard. I changed that option to 0 and now it seems to be working.

Thank you

speechles · Post by **speechles** » Wed Oct 21, 2009 5:33 pm

This update was going to occur over the weekend, but google threw me a curve ball. They changed templates at the last second. It would've been foolish to release it as intended because of this as it would've broken immediately.

Code: Select all

# skip self results? google sometimes uses links to it's own
# services within normal google searches. This will skip _any_ google
# links from appearing within google search results. For some
# queries this will omit obvious replies which is normal.
# ------
variable skip_self 1

In a nutshell this is the new option discussed above. This was married to the short_answers option before, now it's divorced and has a small meager apartment on the bad side of town.. But.. There is something you need to understand about this option, besides it being newly single. It will skip any, and by any I mean literally _ANY_ google result that appears. Type !g google, and normally you would expect to see google reference itself as your results... But instead, you merely get the total results found, yet no results shown. And if you've disabled total results, you get nothing at all. As if the bot didn't do anything. This is normal behavior, not a bug. Moving on....

!fight works now. Total results now appear with most results (a few are broken, sure). Also tinkered with the way wiki(pedia/media) tries matching subtag #anchors . This part is still a work-in-progress (read this as it's slow, not as slow as before, but it's still a dog). Also went ahead and fixed some minor inconsistencies, such as youtube giving you total results even if you've disabled them and a few others like this.

Incith:Google v1.9.9u ... Have a fun

Note: The problem with short_answers and looping has been resolved. This was caused by a mistake forgetting to unset variables before continue'ing the while, which carries over those variables into the loop. The parsing in this loop uses unset variables to mean no matches left. With these still set, this then repeats again, and forever, or until your bot pings out. But even then it still continues to eat the cpu, up until the process is terminated. This will never happen again. All variables are now safely reset before this happens with skip_self.

transacid · Post by **transacid** » Sun Oct 25, 2009 7:36 am

there seems to be an error in the latest version, or i'm doing something wrong

Code: Select all

Tcl error [incith::google::public_message]: list must have an even number of elements