This is the new home of the egghelp.org community forum.
All data has been migrated (including user logins/passwords) to a new phpBB version.


For more information, see this announcement post. Click the X in the top right-corner of this box to dismiss this message.

Issues with TCL encoding on Eggdrop

Help for those learning Tcl or writing their own scripts.
Post Reply
c
chronoz
Voice
Posts: 2
Joined: Thu May 19, 2011 4:02 pm

Issues with TCL encoding on Eggdrop

Post by chronoz »

I have installed Eggdrop on a new Debian server, but it keeps having issues with processing special characters.

Eggdrop is running utf-8. I have even manually enforced TCL encoding to utf-8 in the script. And I have tried recompiling Eggdrop with instructions from http://eggwiki.org/Utf-8.

Code: Select all

	22:00 <@me> !tr fr I have prepared lots of cookies for the entire family.
	22:00 <@bot> J'ai préparé beaucoup de biscuits pour toute la famille.
	22:00 <@me> !tr ar The special characters are processed.
	22:00 <@bot> êêÃE ÃEùçÃDìé çÃDãíñÃA çÃDîçõé.
(Also see a previous Question asked, that did not get solved: http://stackoverflow.com/questions/6008 ... on-eggdrop)

Code: Select all

	namespace eval gTranslator {

	# Factor this out into a helper
	proc getJson url {
	  set tok [http::geturl $url]
	  set res [json::json2dict [http::data $tok]]
	  http::cleanup $tok
	  return $res
	}
	# How to decode _decimal_ entities; WARNING: high magic factor within!
	proc decodeEntities str {
	  set str [string map {\[ {\[} \] {\]} \$ {\$} \\ \\\\} $str]
	  subst [regsub -all {&#(\d+);} $str {[format %c \1]}]
	}

	bind pub - !tr gTranslator::translate
	proc translate { nick uhost handle chan text } {
	  package require http
	  package require json
	  set lngto [string tolower [lindex [split $text] 0]]
	  set text [http::formatQuery q [join [lrange [split $text] 1 end]]]
	  set dturl "http://ajax.googleapis.com/ajax/services/language/detect?v=1.0&q=$text"

	  set lng [dict get [getJson $dturl] responseData language]

	  if { $lng == $lngto } {
		putserv "PRIVMSG $chan :\002Error\002 translating $lng to $lngto."
		return 0
	  }
	  set trurl "http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&langpair=$lng%7c$lngto&$text"
	  putlog $trurl

	  set res [getJson $trurl]

	  putlog $res
	  #putserv "PRIVMSG $chan :Language detected: $lng"

	  set translated [decodeEntities [dict get $res responseData translatedText]]

	  putserv "PRIVMSG $chan :[encoding convertto utf-8 $translated]"
	}
	}
User avatar
speechles
Revered One
Posts: 1398
Joined: Sat Aug 26, 2006 10:19 pm
Location: emerald triangle, california (coastal redwoods)

Post by speechles »

<speechl3s> !tr @fr I am a stranger --swapo
<sp33chy> Google: (auto->fr) Romanian to French translation
<sp33chy> Je suis un étranger

<speechl3s> !tr @fr I am a stranger
<sp33chy> Google: (auto->fr) Romanian to French translation
<sp33chy> Je suis un étranger

<speechl3s> !tr @zh I am a stranger
<sp33chy> Google: (auto->zh) Malay to Chinese translation
<sp33chy> 我是一个蠕虫

<speechl3s> !tr en@zh I am a stranger
<sp33chy> Google: (en->zh) English to Chinese translation
<sp33chy> 我是个陌生人

<speechl3s> !tr @ar I am a stranger
<sp33chy> Google: (auto->ar) English to Arabic translation
<sp33chy> أنا غريب

<speechl3s> !tr @fr I have prepared lots of cookies for the entire family.
<sp33chy> Google: (auto->fr) English to French translation
<sp33chy> J'ai préparé beaucoup de biscuits pour toute la famille.

<speechl3s> !tr @fr I have prepared lots of cookies for the entire family. --swapo
<sp33chy> Google: (auto->fr) English to French translation
<sp33chy> J'ai préparé beaucoup de biscuits pour toute la famille.

<speechl3s> !tr @ar The special characters are processed.
<sp33chy> Google: (auto->ar) English to Arabic translation
<sp33chy> تتم معالجة الأحرف الخاصة.

<speechl3s> !tr تتم معالجة الأحرف الخاصة
<sp33chy> Google: (auto->en) Arabic to English translation
<sp33chy> Special characters are processed
Let me (the google magician) explain a few things. Notice my translate script works. Correctly, and at the same time I can break it, and make it work like yours does.

So let me explain what you are doing wrong....

You've patched your bot. Hopefully correctly.. Let's move on..

Code: Select all

set dturl "http://ajax.googleapis.com/ajax/services/language/detect?v=1.0&q=$text" 
First off, the url has no google encodings within it. Why do you not use &oe= and &ie= to define your input and output encodings to google? Also, your format query has "q" and $text within it. Why do you then add q=$text??! O_o;;?!
Change that to look like it does below...

Code: Select all

set dturl "http://ajax.googleapis.com/ajax/services/language/detect?v=1.0&ie=utf-8&oe=utf-8$text"
Then...

Code: Select all

set text [http::formatQuery q [join [lrange [split $text] 1 end]]]
You http::formatQuery before you've ever configured your url encoding to utf-8.. WHY?
Change it to look like below...

Code: Select all

http::config -urlencoding {utf-8}
set text [http::formatQuery q [join [lrange [split $text] 1 end]]]
Finally, and this is the one likely causing your issue...

Code: Select all

putserv "PRIVMSG $chan :[encoding convertto utf-8 $translated]" 
Why are you converting encoding at all? Your bots system encoding is already utf-8 if you've properly patched your eggdrop. Doing that again will cause double conversion and mangle your characters..
Change this as well to look like below:

Code: Select all

putserv "PRIVMSG $chan :$translated"


Finally, change this line below

Code: Select all

package require http
to this one..

Code: Select all

package require http 2.5
url-encoding was added as an option in http version 2.5. ;)

Doing all that I say, making these changes. You will notice your translate script now works. No thanks required. I've found out how to make perfect utf-8 several years ago ;)
c
chronoz
Voice
Posts: 2
Joined: Thu May 19, 2011 4:02 pm

Post by chronoz »

Thanks! Removing the [encoding convertto utf-8 $translated] indeed solved the issue. I added it before recompiling eggdrop for utf-8 to see if it would solve the issue.

Recompiling eggdrop for utf-8 actually solved the issue, while adding [encoding convertto utf-8 $translated] broke it again.

Appreciate the thorough response!
User avatar
speechles
Revered One
Posts: 1398
Joined: Sat Aug 26, 2006 10:19 pm
Location: emerald triangle, california (coastal redwoods)

Post by speechles »

Also, seems your script needs food badly, it is about to die. Remember, don't shoot food. Red elf needs food badly. Red elf is dead. When does this happen you ask? Well... deprecation has already begun in the form of rate limits. On December 1st, 2011 it's put to rest entirely.

See here -> http://code.google.com/apis/language/tr ... rview.html

It's dead jim.


But keep in mind, that there is another way...
"It is important to stress that the Translate API is not the Google Translate web site, nor the Google Translate Web element."

So Incith:Google's translate function will continue to function as it always has...
/me smiles gleefully and feels a bit sad for chronoz at the same time..
Post Reply