I don't think it's a really big issue, but I agree with you wholeheartedly.

The color of the post should only change when it hits a negative score. A post downvoted by only one person is a reflection of that person rather than the post. Also, since they start at 1 point anyhow, it would be a very elegant system.