1
2
3
4
5
6
7
8
9
10
11
12

SELECT * FROM TABLE
WHERE NOT COLUMN ~ ( '^('||
 $$[\09\0A\0D\x20-\x7E]|$$||               -- ASCII
 $$[\xC2-\xDF][\x80-\xBF]|$$||             -- non-overlong 2-byte
  $$\xE0[\xA0-\xBF][\x80-\xBF]|$$||        -- excluding overlongs
 $$[\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}|$$||  -- straight 3-byte
  $$\xED[\x80-\x9F][\x80-\xBF]|$$||        -- excluding surrogates
  $$\xF0[\x90-\xBF][\x80-\xBF]{2}|$$||     -- planes 1-3
 $$[\xF1-\xF3][\x80-\xBF]{3}|$$||          -- planes 4-15
  $$\xF4[\x80-\x8F][\x80-\xBF]{2}$$||      -- plane 16
 ')*$' )
;

Or, if you have iconv on your system, as most UNIX variants do, you can pg_dumpall a database and run the following command on it:

if (@iconv($input, 'UTF-8', 'UTF-8') == $input) echo "Good UTF-8!"; else echo "Nope."

Then take a plain text pg_dump of the database in UTF-8. Let's call it "db.orig.dmp". Strip all problem characters:

iconv -f UTF-8 -t UTF-8 -c <db .orig.dmp >db.stripped.sql
</db>

Compare both files:

diff -u db.orig.dmp db.stripped.sql

That output will show all lines containing a problem character. Now the tedious part: Use "db.orig.dmp" to find out which tables they belong to, locate the records in the database by primary key and fix them. Alternatively, you can use iconv's auto-repair if you know which bytes give you a problem.

For example, if you know that the trouble stems only from 0x80 bytes that should be Euro symbols, you could:

iconv -f UTF-8 -t UTF-8 --byte-subst="&lt;0x%x>" < db.orig.sql | sed -e 's/&lt;0x80>/EUR/g' >db.fixed.sql

The resulting "db.fixed.sql" could then be loaded into the new database.

The ALTER TABLE ...ALTER COLUMN...TYPE... command is useful only if you want to alter the data somehow, or change the data type. Otherwise, it'll be an aeon before this finishes even inside a transaction on a database of any meaningful size.

Until now, I was not familiar with any sensible mechanism to simply change the size in PG. But yesterday, Tom Lane himself suggested something ubercool in the list.

Let's assume for the sake of simplicity that your table is called "TABLE1" and your column is "COL1". You can find the size of your "COL1" column by issuing the following query on the system tables:

SELECT atttypmod FROM pg_attribute
WHERE attrelid = 'TABLE1'::regclass
AND attname = 'COL1';
 
atttypmod
-----------
24
(1 ROW)

This means that the size is 20 (4 is added for legacy reasons, we're told). You can now conveniently change this to a varchar(35) size by issuing this command:

UPDATE pg_attribute SET atttypmod = 35+4
WHERE attrelid = 'TABLE1'::regclass
AND attname = 'COL1';
 
UPDATE 1

Note that I manually added the 4 to the desired size of 35..again, for some legacy reasons inside PG. Done. That's it. Should we check?

d TABLE1
 
TABLE "public.TABLE1"
COLUMN  |  TYPE                 | Modifiers
--------+-----------------------+-----------
COL1    | CHARACTER VARYING(35) |

Such a simple yet effective trick. Of course it'd be nicer if this is somehow included in a more proper way in the database, but this does the job.

Yet, for all its merits and addictive features, PostgreSQL is a beast to upgrade between major version releases.

If you were going from 8.2.x to the latest stable at the time of this writing, 8.3.x, then the move from the 8.2 series to 8.3 would require you to dump/restore your entire database. Not a simple chore if you have a 100 GB worth of critical data, but you have no choice as the innards of PG typically change between "major upgrades". For this, replication technologies such as Slony are a useful option as you can install the new database separately, replicate the data over, and then make a switch when the two databases are mirrored — thereby avoiding the downtime of the hassle (here's an informative PGCON presentation).

Fortunately, minor version upgrades are much simpler.

No dump/restore is needed. If you were to go from, say 8.2.3 to 8.2.9, all you would have to do is to upgrade the RPMs. I'm assuming Linux here but the same would work for other platforms.

1. Check what version you have currently installed:

   rpm -qa | grep postgres | tee /root/rpm_pgsql

2. Stop your postgres server

   /sbin/service postgresql stop

3. We will now simply execute the RPM upgrade command, which of course assumes that you have the latest RPMs downloaded from the PG website. Notice below that they're all lumped into the same command to prevent any (unlikely) possibility of dependency issues, so make sure all of this appears on one line — the backslash is included here for formatting (so it won't break on the command line).

   rpm -Uvh 
     postgresql-8.2.9-1PGDG.rhel4.i386.rpm 
     postgresql-contrib-8.2.9-1PGDG.rhel4.i386.rpm 
     postgresql-devel-8.2.9-1PGDG.rhel4.i386.rpm 
     postgresql-libs-8.2.9-1PGDG.rhel4.i386.rpm 
     postgresql-server-8.2.9-1PGDG.rhel4.i386.rpm

4. Hopefully that went without a hitch. Now restart the postgresql server:

   service postgresql restart

That's it. That should do it. It is highly unlikely that you'll see errors. And if you're running an older 8.2.x, it's highly recommended you upgrade to 8.2.9 anyway due to performance increases and some bug fixes.