Problem when using character set latin1 instead of utf-8

Talk about Multi-Language tricks, problems, errors etc.

Moderator: alexandrleonenko

hermanfransen
Junior
Junior
Posts: 11
Joined: Jan 19th, '09, 13:47
Location: Amsterdam - The Netherlands

Problem when using character set latin1 instead of utf-8

Postby hermanfransen » Jan 19th, '09, 14:40

I am a newbie here and very impressed by this software. I use the payed version 5.0.2.

I have a problem when using latin1 character set.

At the beginning of the document I put:
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
At the beginning of the php code I put:
header("Content-Type: text/html; charset=iso-8859-1"); (or charset=latin1)
After database connection I put:
$db_conn->query('SET NAMES latin1 COLLATE latin1_general_ci');
And the datagrid settings are:
$dg_encoding = "latin1";
$dg_collation = "latin1_general_ci";
In the MySQL database I recreated the table to make sure this is ok:
CHARSET=latin1 COLLATE=latin1_general_ci

When I add a new record in datagrid with special characters like é, è, ü, €, after updating they all give a question mark "?".

When I change to utf-8 no problems occur:

header("Content-Type: text/html; charset=utf-8");
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" >
$db_conn->query('SET NAMES utf8 COLLATE utf8_unicode_ci');
$dg_encoding = "utf8";
$dg_collation = "utf8_unicode_ci";

Because my database is already in latin1, it's a lot of work to translate this into utf-8. Hopefully I am doing something wrong instead of a bug.

Any help would be very appreciated.

Greetings,
Herman Fransen
From Amsterdam

zewa
Site Moderator
Site Moderator
Posts: 15
Joined: Jan 15th, '09, 06:45
Location: Graz - Styria - Austria
Contact:

Re: Problem when using character set latin1 instead of utf-8

Postby zewa » Jan 21st, '09, 19:19

Hi Hermann

Can you tell me what way the values have been inserted into the table?
Are there too those ? for the special characters?

also please try querying directly an INSERT statement with
$db_conn->query(''), with some special chars and look if its correctly done this way.

Greetings
Zewa

zewa
Site Moderator
Site Moderator
Posts: 15
Joined: Jan 15th, '09, 06:45
Location: Graz - Styria - Austria
Contact:

Re: Problem when using character set latin1 instead of utf-8

Postby zewa » Jan 22nd, '09, 07:17

please also check this thread
http://www.apphp.com/forum/viewtopic.php?f=11&t=13

Greetings
Zewa

hermanfransen
Junior
Junior
Posts: 11
Joined: Jan 19th, '09, 13:47
Location: Amsterdam - The Netherlands

Re: Problem when using character set latin1 instead of utf-8

Postby hermanfransen » Jan 23rd, '09, 15:02

Hi Zewa,

When I use $db_conn->query('') the special characters are inserted correctly (tested é, €). After the insert they are showed correctly in view mode. When I go to edit mode they are correctly showed too. But after I do an update, they are changed to "?". In phpMyAdmin I can see with "select hex(field)" that the character is actual changed.

Thanks a lot for your help in this problem.

Greetings,
Herman

p.s. I read the thread about the place of the statement $dg_encoding = ...

zewa
Site Moderator
Site Moderator
Posts: 15
Joined: Jan 15th, '09, 06:45
Location: Graz - Styria - Austria
Contact:

Re: Problem when using character set latin1 instead of utf-8

Postby zewa » Jan 23rd, '09, 18:38

Could you please try following test script ...

http://zewa.ze.funpic.de/datagrid/testDB.rar

its just a simple testDB with only one table and some german umlauts in it.

The test1.php is a simple datagrid setup.
Place it in the root datagrid folder for proper working.

If this works for you, than it must be some special settings of your file not working.


Another solution is simply to leave your DB as latin1 as it is, and keeping UTF-8 as the
character set, as utf-8 should be able to represent every char.

EDIT:
Another thing could be some Apache or MY.ini settings.
My my.ini has by default utf8 as collation server and character set.

Im using XAMPP 1.6.6.a right out of the box for testing the Datagrid related stuff and for development.

Greetings
Zewa

hermanfransen
Junior
Junior
Posts: 11
Joined: Jan 19th, '09, 13:47
Location: Amsterdam - The Netherlands

Re: Problem when using character set latin1 instead of utf-8

Postby hermanfransen » Jan 27th, '09, 12:03

Hi Zewa,

Your script was a problem because in my account I can't add an extra database. Also in the root of datagrid the script didn't work (strange, I don't understand this). So I added the table in my existing database and put the script in another place (not root datagrid). And then the outcome was that the special characters (é è € etc.) where not shown correctly.

I decided to stop trying to solve this problem. I converted everything to utf-8, it was a lot of work but finaly everything works now between my datagrid and my existing website.

Thanks a lot for your help!!

Kind regards,
Herman Fransen

zewa
Site Moderator
Site Moderator
Posts: 15
Joined: Jan 15th, '09, 06:45
Location: Graz - Styria - Austria
Contact:

Re: Problem when using character set latin1 instead of utf-8

Postby zewa » Jan 27th, '09, 19:07

mmhm sad ... but as stated it would have been enough only to set every collation to utf-8 no mather
the database is latin1.

Greetings
Zewa

hugomp
Master
Master
Posts: 217
Joined: Feb 20th, '09, 12:44

Re: Problem when using character set latin1 instead of utf-8

Postby hugomp » Mar 12th, '09, 07:38

Hello Zewa

Same language problem here...

My solution:
Comment in datagrid.class.php file:
//$this->encoding = "utf8";
//$this->collation = "utf8_unicode_ci";

around line 650....

Then change:
...................
foreach($_POST as $fldName => $fldValue){
// update all fields, excepting uploading fields
if(!strpos($fldName, "file_act") && !strpos($fldName, "_operation_randomize_code")){
$fldName = substr($fldName, 3, strlen($fldName));
$fldValue = $this->IsDatePrepare($fldName,$fldValue);
$fldValue = stripslashes($fldValue); // added not to save backslashes in Database
if(!$this->IsReadonly($fldName) && !$this->IsValidator($fldName)){
if (intval($ind) >= 1) $sql .= ", ";
if($this->IsText($fldName)){
..................................
protected function GetFieldValueByType($field_value, $ind, $row, $field_name="", $m_field_req_type=""){
// Un-quote string quoted by mysql_real_escape_string()
if(get_magic_quotes_gpc()) {
if(ini_get('magic_quotes_sybase')) {
$field_value = str_replace("''", "'", $field_value);
} else {
//$field_value = str_replace("''", "'", $field_value); // removed
$field_value = str_replace('"', "&quot;", $field_value); // double quotation mark, added
$field_value = str_replace("'", "&#039;", $field_value); // single quotation mark, added

$field_value = stripslashes($field_value);
}
}
.................................................
case "textarea":
$field_value=stripslashes($field_value); // added
$field_value = str_replace('"', "&quot;", $field_value); // double quotation mark
$field_value = str_replace("'", "&#039;", $field_value); // single quotation mark

..........................................................
case "textbox":
$field_value=stripslashes($field_value); // added
$field_value = str_replace('"', "&quot;", $field_value); // double quotation mark
$field_value = str_replace("'", "&#039;", $field_value); // single quotation mark

................................................................

Works fine with versions 4.2.8, 5.0.2 and 5.0.3
Best regards


Return to “Multi-Language implementation”