1
2
3
4
5
6
7
8
9
10
11
12

SELECT * FROM TABLE
WHERE NOT COLUMN ~ ( '^('||
 $$[\09\0A\0D\x20-\x7E]|$$||               -- ASCII
 $$[\xC2-\xDF][\x80-\xBF]|$$||             -- non-overlong 2-byte
  $$\xE0[\xA0-\xBF][\x80-\xBF]|$$||        -- excluding overlongs
 $$[\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}|$$||  -- straight 3-byte
  $$\xED[\x80-\x9F][\x80-\xBF]|$$||        -- excluding surrogates
  $$\xF0[\x90-\xBF][\x80-\xBF]{2}|$$||     -- planes 1-3
 $$[\xF1-\xF3][\x80-\xBF]{3}|$$||          -- planes 4-15
  $$\xF4[\x80-\x8F][\x80-\xBF]{2}$$||      -- plane 16
 ')*$' )
;

Or, if you have iconv on your system, as most UNIX variants do, you can pg_dumpall a database and run the following command on it:

if (@iconv($input, 'UTF-8', 'UTF-8') == $input) echo "Good UTF-8!"; else echo "Nope."

Then take a plain text pg_dump of the database in UTF-8. Let's call it "db.orig.dmp". Strip all problem characters:

iconv -f UTF-8 -t UTF-8 -c <db .orig.dmp >db.stripped.sql
</db>

Compare both files:

diff -u db.orig.dmp db.stripped.sql

That output will show all lines containing a problem character. Now the tedious part: Use "db.orig.dmp" to find out which tables they belong to, locate the records in the database by primary key and fix them. Alternatively, you can use iconv's auto-repair if you know which bytes give you a problem.

For example, if you know that the trouble stems only from 0x80 bytes that should be Euro symbols, you could:

iconv -f UTF-8 -t UTF-8 --byte-subst="&lt;0x%x>" < db.orig.sql | sed -e 's/&lt;0x80>/EUR/g' >db.fixed.sql

The resulting "db.fixed.sql" could then be loaded into the new database.

Yet, for all its merits and addictive features, PostgreSQL is a beast to upgrade between major version releases.

If you were going from 8.2.x to the latest stable at the time of this writing, 8.3.x, then the move from the 8.2 series to 8.3 would require you to dump/restore your entire database. Not a simple chore if you have a 100 GB worth of critical data, but you have no choice as the innards of PG typically change between "major upgrades". For this, replication technologies such as Slony are a useful option as you can install the new database separately, replicate the data over, and then make a switch when the two databases are mirrored — thereby avoiding the downtime of the hassle (here's an informative PGCON presentation).

Fortunately, minor version upgrades are much simpler.

No dump/restore is needed. If you were to go from, say 8.2.3 to 8.2.9, all you would have to do is to upgrade the RPMs. I'm assuming Linux here but the same would work for other platforms.

1. Check what version you have currently installed:

   rpm -qa | grep postgres | tee /root/rpm_pgsql

2. Stop your postgres server

   /sbin/service postgresql stop

3. We will now simply execute the RPM upgrade command, which of course assumes that you have the latest RPMs downloaded from the PG website. Notice below that they're all lumped into the same command to prevent any (unlikely) possibility of dependency issues, so make sure all of this appears on one line — the backslash is included here for formatting (so it won't break on the command line).

   rpm -Uvh 
     postgresql-8.2.9-1PGDG.rhel4.i386.rpm 
     postgresql-contrib-8.2.9-1PGDG.rhel4.i386.rpm 
     postgresql-devel-8.2.9-1PGDG.rhel4.i386.rpm 
     postgresql-libs-8.2.9-1PGDG.rhel4.i386.rpm 
     postgresql-server-8.2.9-1PGDG.rhel4.i386.rpm

4. Hopefully that went without a hitch. Now restart the postgresql server:

   service postgresql restart

That's it. That should do it. It is highly unlikely that you'll see errors. And if you're running an older 8.2.x, it's highly recommended you upgrade to 8.2.9 anyway due to performance increases and some bug fixes.