RSS news by demond

[demond]

since the other thread that people have been posting in about this wasn't actually relevant from the beginning, I'm opening this dedicated to the subject new thread

for those who don't know what the fuss is all about: I made a RSS news script for my own purposes (it still doesn't have an official name, I haven't released it officially yet) and I posted the beta version on several occasions here on these forums, after seeing that people tend to have problems with other RSS scripts; my version differs from all other similar scripts in the mechanism of parsing - it uses the object-oriented and XML facilities of TclLib and TclXML, also the RSS package from BDK, posted on Tcl'ers Wiki; and as it (unfortunately) turns out, most people are not used to installing third-party Tcl packages and therefore have been having problems with my script's requirements

I'm a big proponent of software component re-use, and I think we should encourage usage of good (and de facto standard) component packages like TclLib and TclXML

I'm also inclined to release the script officially, but that would depend on how many people actually manage to successfully install the required packages

[greenbear]

Installing aditional packages will probably be a big pita for everyone running bots from their cheap $5 a month shell accounts...

[demond]

well you certainly have a point here, although if you have at least 5mb quota you should be able to install TclLib and TclXML in your homedir:

Code: Select all

[demond@whitepine demond]$ du -h -d0 lib/tcllib1.7/ lib/tclxml3.0/
3.1M    lib/tcllib1.7/
156K    lib/tclxml3.0/

[demond]

I might rewrite this thing so that it would no longer depend on external packages, but it will continue to be rather minimalistic RSS solution, I'm not interested in adding bells & whistles

[sKy]

I think that`s an good idea.

Well, RSS. I know what RSS means but i think its inconnu as well. Please don`t get me wrong, but if you want some feedback you then should try to explain in some easy sentence what RSS is and what the purpose of your script is.

Installing aditional packages will probably be a big pita for everyone running bots from their cheap $5 a month shell accounts...

I think he is right :/.

[tonyrayo]

I believe demond has been brief because there has been no offical release... or maybe even because no one should try to use the script at this stage that doesn't know what an rss parser is and how it would work based on looking at source... or maybe it's just because demond refuses to wear domed shape hats!!! PORQUE!!!!

[demond]

an official release that doesn't require external packages is coming, stay tuned

[demond]

ok folks, here it is, feedback is much appreciated:

Code: Select all

#######################################################################
#                                                                     #
# rssnews.tcl - RSS news announcer for eggdrop by demond@demond.net   #
#                                                                     #
#               this will announce the updated news from RSS feed(s), #
#               periodically polling the feed(s); supports multiple   #
#               channels, one feed per channel; you only need to set  #
#               the feeds array, see below                            #
#                                                                     #
#        Usage: !rss <news#|*> (read news# or * for headlines list)   #
#                                                                     #
#######################################################################

package require Tcl 8.3
package require eggdrop 1.6
package require http 2.0

namespace eval rssnews {

# set your feed(s) sources here: channel, poll frequency in mins, feed URL
#
#set feeds(#chan1) {17 http://www.osnews.com/files/recent.rdf}
#set feeds(#chan2) {11 http://news.google.com/news?ned=us&topic=h&output=rss}

variable version "rssnews-1.0"

variable timeout 20 ;# seconds

bind pub  - !rss [namespace current]::news
bind time - *    [namespace current]::timer

putlog "$version by demond loaded"

proc timer {min hour day month year} {
	variable feeds
	if {[info exists feeds]} {
	if {$min} {set min [string trimleft $min 0]}
	foreach {chan feed} [array get feeds] {
		if {$min && $min % [lindex $feed 0] == 0} {
			fetch [lindex $feed 1] $chan
		}
	}}
}

proc fetch {url chan} {
	variable timeout
	variable version; variable token
	set to [expr {$timeout * 1000}]
	set cmd [namespace current]::callback
	::http::config -useragent "$version by demond"
	if {[catch {set t [::http::geturl $url -command $cmd -timeout $to]} err]} {
		putlog "$version: ERROR($chan): $err"
	} {
		set token($t) $chan
	}
}

proc callback {t} {
	variable version; variable token
	set chan $token($t)
	switch -exact [::http::status $t] {
	"timeout" {
		putlog "$version: ERROR($chan): timeout"
	}
	"error" {
		putlog "$version: ERROR($chan): [::http::error $t]"
	}
	"ok" {
		if {[::http::ncode $t] != 200} {
			putlog "$version: ERROR($chan): [::http::code $t]"
		} {
			process [::http::data $t] $chan
		}
	}
	default {
		putlog "$version: ERROR($chan): got EOF from socket"
	}}
	::http::cleanup $t
}

proc process {data chan} {
	variable news; variable hash
	set idx 1; set news($chan) {}
	regsub -all {(?i)<items.*?>.*?</items>} $data {} data
	foreach {foo item} [regexp -all -inline {(?i)<item.*?>(.*?)</item>} $data] {
		regexp {(?i)<title>(.*?)</title>}             $item -> title
		regexp {(?i)<link>(.*?)</link}                $item -> link
		regexp {(?i)<description>(.*?)</description>} $item -> descr
		strip title descr
		if {[info exists hash($chan)]} {
		if {[lsearch -exact $hash($chan) [md5 $title]] == -1} {
			if {[botonchan $chan]} {
				puthelp "privmsg $chan :($idx) $title"
			}
		}}
		lappend news($chan) [list $title $link $descr]
		lappend hashes [md5 $title]
		incr idx
	}
	set hash($chan) $hashes
}

proc strip {args} {
	variable html
	foreach a $args {
		upvar $a x
		set amp {& &}
		set x [string map $amp $x]
		set x [string map $html $x]
		regsub -all {<[^<]+?>} $x {} x
	}
}

proc news {nick uhost hand chan text} {
	variable news; variable feeds
	set num [lindex [split $text] 0]
	if {$num == ""} {
		puthelp "notice $nick :Usage: $::lastbind <news#|*>"
		return
	}
	if {$num != "*" && ![string is integer $num]} {
		puthelp "notice $nick :argument must be number or *"
		return
	}
	if {![info exists news($chan)]} {
		puthelp "notice $nick :no news for this channel"
		return
	}
	if {$num == "*"} {
		set idx 1
		puthelp "notice $nick :News source: [lindex $feeds($chan) 1]"
		foreach item $news($chan) {
			puthelp "notice $nick :($idx) [lindex $item 0]"
			incr idx
		}
		return 1
	} {
	if {$num < 1 || $num > [llength $news($chan)]} {
		puthelp "notice $nick :no such news index, try $::lastbind *"
	} {
		set idx [expr {$num-1}]
		puthelp "notice $nick :......title($num): [lindex [lindex $news($chan) $idx] 0]"
		puthelp "notice $nick :description($num): [lindex [lindex $news($chan) $idx] 2]"
		puthelp "notice $nick :.......link($num): [lindex [lindex $news($chan) $idx] 1]"
		return 1
	}}
}

variable html {
	"     \x22  &apos;     \x27  &      \x26  <       \x3C
	>       \x3E        \x20  ¡    \xA1  ¤   \xA4
	¢     \xA2  £    \xA3  ¥      \xA5  ¦   \xA6
	§     \xA7  ¨      \xA8  ©     \xA9  ª     \xAA
	«    \xAB  ¬      \xAC  ­      \xAD  ®      \xAE
	¯     \xAF  °      \xB0  ±   \xB1  ²     \xB2
	³     \xB3  ´    \xB4  µ    \xB5  ¶     \xB6
	·   \xB7  ¸    \xB8  ¹     \xB9  º     \xBA
	»    \xBB  ¼   \xBC  ½   \xBD  ¾   \xBE
	¿   \xBF  ×    \xD7  ÷   \xF7  À   \xC0
	Á   \xC1  Â    \xC2  Ã   \xC3  Ä     \xC4
	Å    \xC5  Æ    \xC6  Ç   \xC7  È   \xC8
	É   \xC9  Ê    \xCA  Ë     \xCB  Ì   \xCC
	Í   \xCD  Î    \xCE  Ï     \xCF  Ð      \xD0
	Ñ   \xD1  Ò   \xD2  Ó   \xD3  Ô    \xD4
	Õ   \xD5  Ö     \xD6  Ø   \xD8  Ù   \xD9
	Ú   \xDA  Û    \xDB  Ü     \xDC  Ý   \xDD
	Þ    \xDE  ß    \xDF  à   \xE0  á   \xE1
	â    \xE2  ã   \xE3  ä     \xE4  å    \xE5
	æ    \xE6  ç   \xE7  è   \xE8  é   \xE9
	ê    \xEA  ë     \xEB  ì   \xEC  í   \xED
	î    \xEE  ï     \xEF  ð      \xF0  ñ   \xF1
	ò   \xF2  ó   \xF3  ô    \xF4  õ   \xF5
	ö     \xF6  ø   \xF8  ù   \xF9  ú   \xFA
	û    \xFB  ü     \xFC  ý   \xFD  þ    \xFE
	ÿ     \xFF  '      \x27  <p> \x20 <br> \x20 \n \x20
}

}

[demond]

' on the last line actually needs to be & #39; (without space between & and #)

[r00tw00t]

Wow, after two hours of trying to get your 'beta' version working, I found this. One single file does it all. kudos!

Now if I wanned the bot to say the headline + link in the channel, how do I do? Like: <eggdrop> [nameofnewssource]: [headline] - [link]

And everytime a new headline comes up it will say it in the #. (somekind of pseudo-live news feeder.)

Thanks!

EDIT: huh. The script already does that. I was refreshing the RSS until a new headline was up, and checked if the bot had said that. The poll frequency is 3 min but it took 15-20 minutes for the bot to say the headline. Laggy?

[demond]

you set the poll frequency in the feeds array to 3mins but it took 15mins for an update headline(s) to show up? that means the RSS feed got updated 12mins (or more) after the script started polling

[r00tw00t]

http://www.fragbite.com/xml/headlines_rss.php i use this rss but the script doesnt seem to handle it?

[demond]

you don't have <description> tags in that feed

I'll fix that in 1.1 by providing default values for <title>, <link> and <description> tags

[demond]

okay 1.1 is out with that fixed and improved handling of HTML encoding

find rssnews.tcl here, I'll also submit it to Tcl script archive

[sarius]

Sweet!

It works for most the the news websites so far. But is it possible to include CDATA support too?

It can't seem to read from this site - http://soccernet.espn.go.com/rss/news

Really appreciate the work done. Cheers!