There will always be some compatibility issues between different hardware. Look at the compatibility between different computers of the past. The Atari STFM and the STE and the Amiga A500 and A1200. Both upgrading systems had some compatibility issues with software originally written for the older system. This is always due to games coders taking full advantage of system hardware right down to the basic core level, coding directly to the memory registers of each chip in the system, so that when a new improved system is released the memory registers will be different and may cause issues and compatibility problems.

As for the PS3, yes there are some compatibility issues, but as Demon said, Sony did the wise thing and adding the compatibility into the firmware of the PS3 so as problems with specific games are found they can be fixed and released in the next firmware update. Sony have already been doing this and already released a firmware update that fixed the compatibility with many PS2 and PS1 games.

There is an online database of PS3 compatibility here It is in Japanese so might be hard to navigate easily but it does contain information about PS3 compatibility with each of the complete 7,841 games in the Sony PS1 and PS2 database!

UPDATE: I also found the US compatibility status page now here which gives very detailed information about the compatibility of each game you search for, saying exactly what the issues currently are.

BTW, did you know that the PS3 actually contains the PS2 CPU and GPU processors to aid in better compatibility of older PS2 games? Much as the PS2 contains the original PS1's CPU for better compatibility with PS1 games. This combined with the firmware configuration settings for PS2 compatibility, each time games are found to contain some compatibility issues they can be fixed by Sony.

And most of the compatibility issues do not mean a game will not run at all. A lot of the issues are things like missing audio in menu screens or background music in game. Such things are annoying by the games are playable. There are not that many games that will not run at all.

This is in stark contrast to the Xbox 360 where backwards compatibility with the older Xbox games requires direct compatibility code for each game to run, and so only the games that have code written to run on the 360 will work, which is not many out of the complete Xbox back catalogue.