Here I'll collect some minor issues and their solutions, which are often surprising...
Drupal comes with site logos in .png format. If I create my own site logos in .png format and try to blend the background, then they look fine in FF, but the background is different in IE. Why does this happen?
The solution is to use .jpg instead.
Sometimes, FF (don't know about IE) draws only the top half of a picture (especially the user pictures). This can be fixed by enabling PHP output_buffering 4096.
At some point, image uploading stopped working. This was shortly after updating Drupal, and I found out that I had forgotton to assign the new files to the drupal group.