[mythtv-commits] Ticket #5946: Fix (unimportant) data corruption in database character set conversion

MythTV mythtv at cvs.mythtv.org
Sat Nov 29 18:01:39 UTC 2008

#5946: Fix (unimportant) data corruption in database character set conversion
 Reporter:  sphery <mtdean at thirdcontact.com>  |       Owner:  ijr    
     Type:  patch                             |      Status:  new    
 Priority:  minor                             |   Milestone:  unknown
Component:  mythtv                            |     Version:  head   
 Severity:  medium                            |     Mlocked:  0      
 The conversion from CHAR(latin1)->BINARY->CHAR(utf8) performed for some
 fields in database upgrades 1216 and 1217 results in null-padded data in
 the CHAR field.  MySQL expects CHAR field data to be space-padded, so it
 never strips the null characters from the data when returning query
 results or when manipulating data.  The null characters then break
 comparisons (meaning we get new "uncorrupted" values placed in the DB in
 addition to the corrupted values) and can be displayed, for example, in

 The attached patch changes the conversion of all multi-character CHAR
 fields to use CHAR(latin1)->VARBINARY->CHAR(utf8) to prevent the null-
 padding corruption from occurring.  Specifically, the affected fields are
 people.name, programgenres.genre, programrating.{system,rating},
 recordedrating.{system.rating} .

 There are still columns of type CHAR(1) for which I didn't change the
 intermediate BINARY data type, and it's possible that in some databases
 values for fields which are not defined as "NOT NULL" may be converted to
 a single null character when they are meant to be blank (which would be
 represented internally as a single space).  The additional CHAR(1) fields
 include cardinput.sharable,
 dtv_multiplex.{inversion,polarity,bandwidth,transmission_mode}, and
 programgenres.relevance ; however, in my database, none include empty
 values, so none are corrupted.

 Though this will change the database upgrade history, it will simply
 prevent corruption of data for users who have not yet upgraded their
 databases.  I will be submitting a patch that fixes the corrupt data for
 those who have already converted to trunk.

 Users should not attempt to fix the data as it's really unimportant data
 and most likely there exist uncorrupted data for any current listings data
 and the upcoming patch will fix it properly.  Fixing it properly is not
 trivial because of key uniqueness constraints which will be violated due
 to the existence of duplicates whose only difference is the null-padding
 corruption.  The new, uncorrupted values will have been inserted by
 mythfilldatabase/EIT, so are likely to exist if MythTV
 (mythbackend/mythfilldatabase) has ever been run since the database

Ticket URL: <http://svn.mythtv.org/trac/ticket/5946>
MythTV <http://www.mythtv.org/>

More information about the mythtv-commits mailing list