Startup has a very specific definition. It refers to a company that aims to solve a problem and/or create something new in a particular industry. This means that it is a subset of the small business category; the corner deli that just opened up last week is a small business, but not a startup. On the other hand, a new tech firm that is trying to figure out an innovative alternative to email is both a small business and a startup.

I'd say the biggest thing that distinguishes startups from other small businesses is the risk factor. When your goal is to solve challenging problems in your field, then by definition your chances of success are low. After all, the problems are categorized as "challenging for a reason: nobody has been able to find a simple and elegant solution to them yet. Whereas a typical small business operates on relatively safe territory - they are most likely providing identical products and services as other players in their industry. So their risk is a lot lower.

I've generally worked in "startup" environments, and I actually do mentally translate that word into "small business." I consider it to have more realistic connotations. YMMV of course.

A corner deli is a small business but is clearly not a startup. Why lump in this kind of small business with a different kind of small business?

Granted the term is probably overused, not in this case though. Picplum is building a scalable web business and has taken on funding, seems pretty much a startup to me.