Archive
Tag "postgresql"

Finding non-UTF8 values in PostgreSQL

23 Jul, 2009
Databases

This reg­exp worked for me.

1
2
3
4
5
6
7
8
9
10
11
12

SELECT * FROM TABLE
WHERE NOT COLUMN ~ ( '^('||
$$[\09\0A\0D\x20-\x7E]|$$||               -- ASCII
$$[\xC2-\xDF][\x80-\xBF]|$$||             -- non-overlong 2-byte
$$\xE0[\xA0-\xBF][\x80-\xBF]|$$||        -- excluding overlongs
$$[\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}|$$||  -- straight 3-byte
$$\xED[\x80-\x9F][\x80-\xBF]|$$||        -- excluding surrogates
$$\xF0[\x90-\xBF][\x80-\xBF]{2}|$$||     -- planes 1-3
$$[\xF1-\xF3][\x80-\xBF]{3}|$$||          -- planes 4-15
$$\xF4[\x80-\x8F][\x80-\xBF]{2}$$||      -- plane 16
')*$' )
;

Or, if you have iconv on your sys­tem, as most UNIX vari­ants do, you can pg_dumpall a data­base and run the fol­low­ing com­mand on it:

if (@iconv($input, 'UTF-8', 'UTF-8') == $input) echo "Good UTF-8!"; else echo "Nope."

Then take a plain text pg_dump of the data­base in UTF-8. Let's call it "db.orig.dmp". Strip all prob­lem characters:

iconv -f UTF-8 -t UTF-8 -c <db .orig.dmp >db.stripped.sql
</db>

Com­pare both files:

diff -u db.orig.dmp db.stripped.sql

That out­put will show all lines con­tain­ing a prob­lem char­ac­ter. Now the tedious part: Use "db.orig.dmp" to find out which tables they belong to, locate the records in the data­base by pri­mary key and fix them. Alter­na­tively, you can use iconv's auto-repair if you know which bytes give you a problem.

For exam­ple, if you know that the trou­ble stems only from 0×80 bytes that should be Euro sym­bols, you could:

iconv -f UTF-8 -t UTF-8 --byte-subst="&lt;0x%x>" < db.orig.sql | sed -e 's/&lt;0x80>/EUR/g' >db.fixed.sql

The result­ing "db.fixed.sql" could then be loaded into the new database.

Read More

Upgrade PostgreSQL within the same "minor version" (8.2.3 -> 8.2.9)

28 Apr, 2008
Databases

Post­greSQL is a full-fledged enterprise-grade data­base brought into the pub­lic domain. Many home­made web­mas­ters now have to deal with what is a very viable — if not an out­right bet­ter — alter­na­tive to the rag­ing pop­u­lar­ity of a sig­nif­i­cantly sim­pler MySQL.

Yet, for all its mer­its and addic­tive fea­tures, Post­greSQL is a beast to upgrade between major ver­sion releases.

If you were going from 8.2.x to the lat­est sta­ble at the time of this writ­ing, 8.3.x, then the move from the 8.2 series to 8.3 would require you to dump/restore your entire data­base. Not a sim­ple chore if you have a 100 GB worth of crit­i­cal data, but you have no choice as the innards of PG typ­i­cally change between "major upgrades". For this, repli­ca­tion tech­nolo­gies such as Slony are a use­ful option as you can install the new data­base sep­a­rately, repli­cate the data over, and then make a switch when the two data­bases are mir­rored — thereby avoid­ing the down­time of the has­sle (here's an infor­ma­tive PGCON pre­sen­ta­tion).

For­tu­nately, minor ver­sion upgrades are much simpler.

No dump/restore is needed. If you were to go from, say 8.2.3 to 8.2.9, all you would have to do is to upgrade the RPMs. I'm assum­ing Linux here but the same would work for other platforms.

  1. Check what ver­sion you have cur­rently installed: 
    rpm -qa | grep postgres | tee /root/rpm_pgsql
  2. Stop your post­gres server 
    /sbin/service postgresql stop
  3. We will now sim­ply exe­cute the RPM upgrade com­mand, which of course assumes that you have the lat­est RPMs down­loaded from the PG web­site. Notice below that they're all lumped into the same com­mand to pre­vent any (unlikely) pos­si­bil­ity of depen­dency issues, so make sure all of this appears on one line — the back­slash is included here for for­mat­ting (so it won't break on the com­mand line). 
    rpm -Uvh
postgresql-8.2.9-1PGDG.rhel4.i386.rpm
postgresql-contrib-8.2.9-1PGDG.rhel4.i386.rpm
postgresql-devel-8.2.9-1PGDG.rhel4.i386.rpm
postgresql-libs-8.2.9-1PGDG.rhel4.i386.rpm
postgresql-server-8.2.9-1PGDG.rhel4.i386.rpm
  4. Hope­fully that went with­out a hitch. Now restart the post­gresql server: 
    service postgresql restart

That's it. That should do it. It is highly unlikely that you'll see errors. And if you're run­ning an older 8.2.x, it's highly rec­om­mended you upgrade to 8.2.9 any­way due to per­for­mance increases and some bug fixes.

Read More

TOPICS

© 2010 —Sniptools. All Rights Reserved.