1
2
3
4
5
6
7
8
9
10
11
12

SELECT * FROM TABLE
WHERE NOT COLUMN ~ ( '^('||
 $$[\09\0A\0D\x20-\x7E]|$$||               -- ASCII
 $$[\xC2-\xDF][\x80-\xBF]|$$||             -- non-overlong 2-byte
  $$\xE0[\xA0-\xBF][\x80-\xBF]|$$||        -- excluding overlongs
 $$[\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}|$$||  -- straight 3-byte
  $$\xED[\x80-\x9F][\x80-\xBF]|$$||        -- excluding surrogates
  $$\xF0[\x90-\xBF][\x80-\xBF]{2}|$$||     -- planes 1-3
 $$[\xF1-\xF3][\x80-\xBF]{3}|$$||          -- planes 4-15
  $$\xF4[\x80-\x8F][\x80-\xBF]{2}$$||      -- plane 16
 ')*$' )
;

Or, if you have iconv on your system, as most UNIX variants do, you can pg_dumpall a database and run the following command on it:

if (@iconv($input, 'UTF-8', 'UTF-8') == $input) echo "Good UTF-8!"; else echo "Nope."

Then take a plain text pg_dump of the database in UTF-8. Let's call it "db.orig.dmp". Strip all problem characters:

iconv -f UTF-8 -t UTF-8 -c <db .orig.dmp >db.stripped.sql
</db>

Compare both files:

diff -u db.orig.dmp db.stripped.sql

That output will show all lines containing a problem character. Now the tedious part: Use "db.orig.dmp" to find out which tables they belong to, locate the records in the database by primary key and fix them. Alternatively, you can use iconv's auto-repair if you know which bytes give you a problem.

For example, if you know that the trouble stems only from 0x80 bytes that should be Euro symbols, you could:

iconv -f UTF-8 -t UTF-8 --byte-subst="&lt;0x%x>" < db.orig.sql | sed -e 's/&lt;0x80>/EUR/g' >db.fixed.sql

The resulting "db.fixed.sql" could then be loaded into the new database.

Yet, for all its merits and addictive features, PostgreSQL is a beast to upgrade between major version releases.

If you were going from 8.2.x to the latest stable at the time of this writing, 8.3.x, then the move from the 8.2 series to 8.3 would require you to dump/restore your entire database. Not a simple chore if you have a 100 GB worth of critical data, but you have no choice as the innards of PG typically change between "major upgrades". For this, replication technologies such as Slony are a useful option as you can install the new database separately, replicate the data over, and then make a switch when the two databases are mirrored — thereby avoiding the downtime of the hassle (here's an informative PGCON presentation).

Fortunately, minor version upgrades are much simpler.

No dump/restore is needed. If you were to go from, say 8.2.3 to 8.2.9, all you would have to do is to upgrade the RPMs. I'm assuming Linux here but the same would work for other platforms.

1. Check what version you have currently installed:

   rpm -qa | grep postgres | tee /root/rpm_pgsql

2. Stop your postgres server

   /sbin/service postgresql stop

3. We will now simply execute the RPM upgrade command, which of course assumes that you have the latest RPMs downloaded from the PG website. Notice below that they're all lumped into the same command to prevent any (unlikely) possibility of dependency issues, so make sure all of this appears on one line — the backslash is included here for formatting (so it won't break on the command line).

   rpm -Uvh 
     postgresql-8.2.9-1PGDG.rhel4.i386.rpm 
     postgresql-contrib-8.2.9-1PGDG.rhel4.i386.rpm 
     postgresql-devel-8.2.9-1PGDG.rhel4.i386.rpm 
     postgresql-libs-8.2.9-1PGDG.rhel4.i386.rpm 
     postgresql-server-8.2.9-1PGDG.rhel4.i386.rpm

4. Hopefully that went without a hitch. Now restart the postgresql server:

   service postgresql restart

That's it. That should do it. It is highly unlikely that you'll see errors. And if you're running an older 8.2.x, it's highly recommended you upgrade to 8.2.9 anyway due to performance increases and some bug fixes.

The recent upgrades of Cpanel have been a pain because they've broken mysqlhotcopy. Not kosher. Fortunately fixing this is pretty easy, by simply downgrading the DBD module of Perl, which mysqlhotcopy relies on.

Usually the error is something like this:

To solve this, simply downgrade the DBD to version 4.001. Create a script anywhere on your root folder as, say, dbd-downgrade.sh:

1
2
3
4
5
6
7
8
9

#!/bin/sh
cd /usr/src
wget -O dbd.tar.gz  "http://search.cpan.org/CPAN/authors/id/C/CA/CAPTTOFU/DBD-mysql-4.001.tar.gz"
gzip -cd dbd.tar.gz | tar xf -
rm -rf dbd.tar.gz
cd DBD-mysql-4.001
perl Makefile.PL
make
make install

Then of course chmod the file to executable (644) and run it:

Then try your mysqlhotcopy command again. It should work. The only thing to consider is that if your Cpanel is setup to upgrade automatically, it may rebuild DBD. You may need to downgrade the DBD again, until this thing is fixed!

If after this update, you see the error for RELOAD privileges, like this…

DBD::mysql::db do failed: Access denied; you need the RELOAD privilege for this operation at /usr/bin/mysqlhotcopy line 472.

…then simply login to mysql as "root" user through the root SSH terminal, and execute the following command:

mysql>  grant all privileges on *.* to root@localhost;
mysql>  flush privileges;

Done. That should do it. Let me know if it doesn't.

A more long term fix

MySQL bug list has a couple of suggested fixes. If you are comfortable with Perl, you can edit the code in /usr/bin/mysqlhotcopy and have it solved in a better way.

Latest Cpanel updates are breaking Movable type installations, wreaking havoc with 500 Server Error messages and causing core dumps all over the place. Here's the kludgish workaround, that, well, works.

A user on MT forums suggested that downgrading to DBD::Mysql 2.9007 should do the trick. It does. Six Apart seems to be aware of it.

Here's a little script for Cpanel/WHM users to execute on their SSH shells. Save it as, say, "dbidowngrade.sh" in your root folder, CHMOD it to 755, and execute it with "./dbidowngrade.sh" at the command prompt.

Important Note:

Yes, some people will have problems with Cpanel updating their DBD::Mysql with the /upcp script, in which case you should probably set up a cron job to execute the above script on a regular basis. I have it up as hourly and it takes just a few seconds. I have commented out the lines in green because you don't need to download the file from CPAN more than once.

Please remember, this is merely a temporary solution. Movable Type should of course have an update soon, and Cpanel should shortly start providing the 3.0001 version in the near future.

Update: July 8, 2005

The latest update of DBD::Mysql, 3.0001_3, works for me. So here's the new code:

1
2
3
4
5
6
7
8
9
10
11

#!/bin/sh
cd /usr/src
 
wget -O dbd.tar.gz "http://search.cpan.org/CPAN/authors/id/C/CA/CAPTTOFU/DBD-mysql-3.0001_3.tar.gz"
gzip -cd dbd.tar.gz | tar xf -
rm -rf dbd.tar.gz
 
cd DBD-mysql-3.0001_3
perl Makefile.PL
make
make install

Update: July 6, 2005

Looks like the DBD::Mysql author has released a 3.0001_2 developer update. I haven't been able to install it, it gives me a bunch of errors but that's par for the course from 'developer' versions of software. YMMV. I'll stick with the 2.9007 for now.

Original method: Downgrading surely works

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

#!/bin/sh
#---------------------------------------------
# Script to downgrade to DBD::mysql 2.9007
#---------------------------------------------
cd /usr/src
wget -O dbd.tar.gz "http://www.cpan.org/modules/by-module/DBD/DBD-mysql-2.9007.tar.gz"
gzip -cd dbd.tar.gz | tar xf -
rm -rf dbd.tar.gz
cd DBD-mysql-2.9007
perl Makefile.PL
make
make install
 
# Clean up
cd /usr/src
rm -rf DBD-mysql-2.9007