Durby - 0.3.0. New project page on code.google.com

[radix]

on line 1131 change

Code: Select all

proc CharsetToEncoding {charset} {

to

Code: Select all

proc http::CharsetToEncoding {charset} {

if you have specified http.tcl in eggdrop.conf remove it unless it is required by something else.

[lee8oi]

A few things:

Thanks radix! good tip. we do have testing bots that used various versions of http and some are definitly specified in the eggdrop.conf.

Speechles has mentioned having a new version of webby available that fixes the encoding issues and improves on a couple features.

I tried some experimenting when I was working on my version of the encoding fixes. During the process speechles fixed it himself. So now its just a matter of merging with the new version of webby.

Volunteers are welcome for this project if anyone wants to help rebase, or give me some time to get my current project ready for release and I'll revisit this and update things.

Thanks again guys for your reports and input!

[lee8oi]

This update rebases Durby on the latest webby found in speechles script archive. This version should have all the encoding issues sorted out. Durby now adds a new feature: Nick ignore. You can now set the script to ignore certain nicks in channel. Latest release is available on github.

[FightingNavyman]

got some errors when I try to do to rehash my bot so durby 0.2.8 to load.

[1:09:34pm] * Quits: @R2D2 (R2D2@P2PNET-3CA29397.zomgbbq.com) (Client exited)

[1:09:34pm] <R2D2> [13:09:36] durby: Cannot find zlib or trf package! Gzipped url queries will not be used. Enjoy the slow lane!
————————————————————
DCC session closed

[lee8oi]

You'll get the same message (not an error) in webby as well. this means you don't have zlib or trf support correctly installed. Tcl 8.6b has the necessary packages to satisfy the zlib requirements. Install that and recompile your bot and you should see this message change to 'enjoy the fast lane' instead. but that message only applies to using the --gz option so you can safely ignore it if you don't use --gz.

[lee8oi]

Durby has moved to a new project page on google code at https://code.google.com/p/durby/

[x0x]

Would it be possible to also detect URL's without http? And I don't think the URL shortners work when detecting urls. I only see the title displayed.

[x0x]

Found a bug as well

http://screenrant.com/24-season-9-fox

<title> Fox to Bring Back ’24′ for Season 9</title>

IRC: ~ Fox to Bring Back 242 for Season 9

[speechles]

Found a bug as well

http://screenrant.com/24-season-9-fox

<title> Fox to Bring Back ’24′ for Season 9</title>

IRC: ~ Fox to Bring Back 242 for Season 9

Patching your eggdrop for utf-8 support solves this without requiring any changes. But for those with unpatched eggdrops/windrops this is indeed annoying. It isn't durby's fault. This is actually, inherited from "webby" unfortunately.. but you've discovered a legacy problem that has always annoyed me. It's an issue with transcoding entities (&#8217;24&#8242; turns into ’24′) correctly into their proper encodings within a string already encoded. I've taken some time to rewrite the problem procedure, and hopefully this works for both unpatched and patched bots, regardless..
**crosses fingers**

Find the procedure named:
webbydescdecode

Replace it entirely with the procedure found below:

Code: Select all

proc webbydescdecode {text char} {
   # code below is neccessary to prevent numerous html markups
   # from appearing in the output (ie, ", ᘧ, etc)
   # stolen (borrowed is a better term) from perplexa's urban
   # dictionary script..
   if {![string match *&* $text]} {return $text}
   if {[string match "*;*" $char]} {set char [string trim $char {;}] }
   set escapes {
		  \xa0 ¡ \xa1 ¢ \xa2 £ \xa3 ¤ \xa4
		¥ \xa5 ¦ \xa6 § \xa7 ¨ \xa8 © \xa9
		ª \xaa « \xab ¬ \xac ­ \xad ® \xae
		¯ \xaf ° \xb0 ± \xb1 ² \xb2 ³ \xb3
		´ \xb4 µ \xb5 ¶ \xb6 · \xb7 ¸ \xb8
		¹ \xb9 º \xba » \xbb ¼ \xbc ½ \xbd
		¾ \xbe ¿ \xbf À \xc0 Á \xc1 Â \xc2
		Ã \xc3 Ä \xc4 Å \xc5 Æ \xc6 Ç \xc7
		È \xc8 É \xc9 Ê \xca Ë \xcb Ì \xcc
		Í \xcd Î \xce Ï \xcf Ð \xd0 Ñ \xd1
		Ò \xd2 Ó \xd3 Ô \xd4 Õ \xd5 Ö \xd6
		× \xd7 Ø \xd8 Ù \xd9 Ú \xda Û \xdb
		Ü \xdc Ý \xdd Þ \xde ß \xdf à \xe0
		á \xe1 â \xe2 ã \xe3 ä \xe4 å \xe5
		æ \xe6 ç \xe7 è \xe8 é \xe9 ê \xea
		ë \xeb ì \xec í \xed î \xee ï \xef
		ð \xf0 ñ \xf1 ò \xf2 ó \xf3 ô \xf4
		õ \xf5 ö \xf6 ÷ \xf7 ø \xf8 ù \xf9
		ú \xfa û \xfb ü \xfc ý \xfd þ \xfe
		ÿ \xff ƒ \u192 Α \u391 Β \u392 Γ \u393 Δ \u394
		Ε \u395 Ζ \u396 Η \u397 Θ \u398 Ι \u399
		Κ \u39A Λ \u39B Μ \u39C Ν \u39D Ξ \u39E
		Ο \u39F Π \u3A0 Ρ \u3A1 Σ \u3A3 Τ \u3A4
		Υ \u3A5 Φ \u3A6 Χ \u3A7 Ψ \u3A8 Ω \u3A9
		α \u3B1 β \u3B2 γ \u3B3 δ \u3B4 ε \u3B5
		ζ \u3B6 η \u3B7 θ \u3B8 ι \u3B9 κ \u3BA
		λ \u3BB μ \u3BC ν \u3BD ξ \u3BE ο \u3BF
		π \u3C0 ρ \u3C1 ς \u3C2 σ \u3C3 τ \u3C4
		υ \u3C5 φ \u3C6 χ \u3C7 ψ \u3C8 ω \u3C9
		ϑ \u3D1 ϒ \u3D2 ϖ \u3D6 • \u2022
		… \u2026 ′ \u2032 ″ \u2033 ‾ \u203E
		⁄ \u2044 ℘ \u2118 ℑ \u2111 ℜ \u211C
		™ \u2122 ℵ \u2135 ← \u2190 ↑ \u2191
		→ \u2192 ↓ \u2193 ↔ \u2194 ↵ \u21B5
		⇐ \u21D0 ⇑ \u21D1 ⇒ \u21D2 ⇓ \u21D3 ⇔ \u21D4
		∀ \u2200 ∂ \u2202 ∃ \u2203 ∅ \u2205
		∇ \u2207 ∈ \u2208 ∉ \u2209 ∋ \u220B ∏ \u220F
		∑ \u2211 − \u2212 ∗ \u2217 √ \u221A
		∝ \u221D ∞ \u221E ∠ \u2220 ∧ \u2227 ∨ \u2228
		∩ \u2229 ∪ \u222A ∫ \u222B ∴ \u2234 ∼ \u223C
		≅ \u2245 ≈ \u2248 ≠ \u2260 ≡ \u2261 ≤ \u2264
		≥ \u2265 ⊂ \u2282 ⊃ \u2283 ⊄ \u2284 ⊆ \u2286
		⊇ \u2287 ⊕ \u2295 ⊗ \u2297 ⊥ \u22A5
		⋅ \u22C5 ⌈ \u2308 ⌉ \u2309 ⌊ \u230A
		⌋ \u230B 〈 \u2329 〉 \u232A ◊ \u25CA
		♠ \u2660 ♣ \u2663 ♥ \u2665 ♦ \u2666
		" \x22 & \x26 < \x3C > \x3E O&Elig; \u152 œ \u153
		Š \u160 š \u161 Ÿ \u178 ˆ \u2C6
		˜ \u2DC   \u2002   \u2003   \u2009
		‌ \u200C ‍ \u200D ‎ \u200E ‏ \u200F – \u2013
		— \u2014 ‘ \u2018 ’ \u2019 ‚ \u201A
		“ \u201C ” \u201D „ \u201E † \u2020
		‡ \u2021 ‰ \u2030 ‹ \u2039 › \u203A
		€ \u20AC &apos; \u0027 ‎ "" ‏ "" ‬ "" ‭ ""
		‮ ""
   };
  set text [string map [list "\]" "\\\]" "\[" "\\\[" "\$" "\\\$" "\" "\\\"] [string map $escapes $text]]
  regsub -all -- {&#([[:digit:]]{1,5});} $text {[encoding convertto $char [format %c [string trimleft "\1" "0"]]]} text
  regsub -all -- {&#x([[:xdigit:]]{1,4});} $text {[encoding converto $char [format %c [scan "\1" %x]]]} text
  return [subst "$text"]
}

It works now for me in my minimal testing. This fix is also included in the newest download of webby, incith-google, and birdy/twitter (silent updates ftw!). Let me know if you still experience rendering issues or if this breaks things entirely now...
**crosses fingers again**

[x0x]

Nice! Seems to work! No issues so far haha.

[x0x]

Another bug, it was there already before your fix. Only happens to certain URLs.

<ME> http://gizmodo.com/watch-nasas-emergenc ... -501627379
<BOT> durby: Invalid command name "zlib" ( http://gizmodo.com/watch-nasas-emergenc ... -501627379 )
<BOT> ~ 0

When I rehash my bot, I do see this error:

durby: Cannot find zlib or trf package! Gzipped url queries will not be used. Enjoy the slow lane!
durby 0.3.0 has been loaded.

Might have something to do with it.

-edit-

Same problem after installing tcl-trf. Script recognizes it too:

durby: Found trf package. Fast lane activated!
durby 0.3.0 has been loaded.

<ME> http://gizmodo.com
<BOT> durby: Invalid command name "zlib" ( http://gizmodo.com )
<BOT> ~ 0

[speechles]

The problem is, the http package version you are using unfortunately. It is a mistake in tcl8.5, where it is assumed zlib is present. Even if you have installed trf correctly, the problem is your version of http package is zlib dependent. It cannot work without it. The only alternative is, to compile against tcl8.6b2 or higher. This tcl version natively includes zlib. The other alternative is to compile tcl locally, removing http 2.7 from the lib folder and replacing it with http 2.5.

<speechles> !webby http://gizmodo.com
<sp33chy> Gizmodo - The Gadget Guide ( http://is.gd/zMhtyP )( 200; text/html; utf-8; 68504 bytes (gzip); 222576 bytes )
<sp33chy> The Gadget Guide

As you can see, to view gizmodo, you MUST use gzip. Some sites, you must use TLS/https.

Webby: Found zlib package. Fast lane activated!
Webby: https supported: tls package found.

For your bot to fully support the future, you want to get both zlib and tls supported.

Update: Appears this occurs infrequently, they will serve you a gzip request, sometimes. Sometimes requesting gzip, will return plain-text response. Sometimes plain-text will return gzip. Not every single time, but... at peak times? Possibly? Let's just assume it is. Let's also assume the future is now, but... sometimes it isn't.

## The request should be sent back gzip, it isn't..
<speechles> !webby http://gizmodo.com --gz
<sp33chy> Gizmodo - The Gadget Guide ( http://tinyurl.com/suee2 )( 200; text/html; utf-8; 221863 bytes )
<sp33chy> The Gadget Guide

## The request should be sent back plain-text, it isn't...
<speechles> !webby http://gizmodo.com
<sp33chy> Gizmodo - The Gadget Guide ( http://is.gd/zMhtyP )( 200; text/html; utf-8; 68820 bytes (gzip); 222580 bytes )
<sp33chy> The Gadget Guide

And here's what it looks like within the request... full headers..

<speechles> !webby http://gizmodo.com --header --xheader --html
<sp33chy> Gizmodo - The Gadget Guide ( http://is.gd/zMhtyP )( 200; text/html; utf-8; 68558 bytes (gzip); 222584 bytes )
<sp33chy> ntCoent-Length=221869; Via=1.1 varnish; P3P=CP="IDC DSP COR CURa ADMa OUR IND PHY ONL COM STA"; Date=Sun, 12 May 2013 01:53:56 GMT; Content-Type=text/html; charset=utf-8; Content-Length=45713; Content-Encoding=gzip; Connection=close; Cache-Control=public, max-age=100; Age=59; Accept-Ranges=bytes
<sp33chy> x-cdn-view=mantle-root; X-Timer=S1368323636.443116426,VS0,VE0; X-Served-By=cache-s29-SJC2; X-Cache-Hits=38; X-Cache=HIT
<sp33chy> Metas: viewport=width=device-width,initial-scale=1.0;
<sp33chy> Metas: kinja:meta=%7B%22hackerspaceCssMd5%22%3A%227a40dde694f259a7a27031f5a3e7a718%22%2C%22spaceCssMd5%22%3A%225705ea850434beb072d9aef9746d0a5d%22%2C%22frontCssMd5%22%3A%22e4388a3f525f1b3674e732196d7681a6%22%2C%22templatesEnUsJsMd5%22%3A%2243c22c0836fbb5caf2b5157bacefdae5%22%2C%22io9CssMd5%22%3A%222018a4600ca16abc1ab2eb739d42a1c4%22%2C%22tinymceCssMd5%22%3A%22d646c23ed4acc4aa1cf446a2d8aef8fa%22%2C%22natgeo80sCssMd5%2
<sp33chy> Metas: kinja:mode=live; kinja:page-type=frontpage; ROBOTS=INDEX, FOLLOW; og:title=Gizmodo - The Gadget Guide; og:type=blog; og:image=http://gawker.com/assets/images/logos/t ... 00x200.png; twitter:card=summary; twitter:site=@gizmodo; og:description=The Gadget Guide; description=The Gadget Guide; og:locale=en_US; og:site_name=Gizmodo; fb:app_id=44615671688;
<sp33chy> Metas: google-site-verification=FUkM9gDOR_WvsjOMuGnUUhhYv5zvRaQHCMmNeRHvvhQ

Here is the problem: X-Served-By=cache-s29-SJC2

The cache is at fault depending on which you are served from. The same issue happens with some wikipedia/mediawiki articles. The caches are to blame when they cache they wrong type.

[x0x]

The problem is that I need to stick with TCL 8.5 for now (long story).

Is there another way to fix this?

[hille]

Great script!
Got one question. I have a urlshortener on my bot. I use .uf to trigger it.
I don't want durby to write the title of the url when I use the urlshortener.
Is it possible to add the .uf trigger in the ignorelist?

[speechles]

Is it possible to add the .uf trigger in the ignorelist?

Yes. Let's make it an ignore list though first.

Find these lines:

Code: Select all

		if {[string match -nocase "!webby*" $word] || [string match -nocase "!durby*" $word]} {
			return 0
		} elseif {[string match -nocase "*://*" $word] || [string match -nocase "www.*" $word]} {

Replace them with these lines, found below:

Code: Select all

		set ignores [list "!webby" "!durby" ".uf"] ; foreach ignore $ignores { if {[string match -nocase $ignore* $word]} { return 0 } }
		if {[string match -nocase "*://*" $word] || [string match -nocase "www.*" $word]} {

Now to add them, add them to "ignores" list.