I actually live off the passive income (I've been an affiliate full-time for almost 4 years). While it has been quite a ride and I can't really share my most profitable sites/tools (without my wife - who is also my business partner - getting me back for it), I will say that it can be a good thing. That said, it is also an extremely competitive business. It is very easy to generate $50/month off a little blog, but generating $10,000/month takes a lot more out of you and carries far more risk.

If you've got something that is growing or doing well already, see if you can grow it - when there is little competition in whatever segment it is an opportunity not to be missed. They are rare.