1. You're overqualified
It's not just a cliche--you really can be overqualified for a position. It's especially true in a tight economy. A candidate that is more qualified would require a higher salary and benefits package for a competitive offer and for long-term retention. If someone else interviewed who fit the qualifications but didn't overly exceed them, it might be in the company's best interest to hire him and save the cash.
2. You don't know the right people
You may be great on paper, and you may interview really well--but if another candidate is the employer's tennis instructor's daughter, you might be out of luck. This isn't a reason you can't really avoid. Your best bet is to make sure you follow up with a genuinely appreciative phone call or note. Leaving a positive impression will keep you in that employer's mind if other opportunities arise.
3. You hit it off, just not professionally
Having a good rapport with your interviewer is great--however, if you bonded over your love of tequila shooters, you may have made a friend and not an employer. In a less extreme example, you may be very easy to interview but still not right for the position. While being friendly and personable are two very important traits, they won't guarantee you the job.
4. You came with conditions
You may be a good fit for the job, but if you come with strings attached, you may not get hired. If you can't see yourself sticking to the position long-term, or if prior commitments mean you'll have to work odd hours, it could take you out of the running. If possible, come in condition-free or at least willing to compromise. However, if you have a restriction that is non-negotiable, it's only fair to both of you to bring it up in the interview--there's no sense in wasting time if the situation won't work out.
5. An unpredictable reason
Often, the reason is one you may never know. You may get an unrelated job simply for having a shared interest with the interviewer, or perhaps because you have a skill the employer hopes to learn from you. It may be as simple as two or more candidates being equally qualified, and you lost the coin toss.
It may also be a simple reason like an off-the-cuff comment you probably shouldn't have made, or a more blatant reason like answering your cell phone during an interview (never a good move, no matter how friendly you are with the interviewer!). Be honest with yourself about the interview process--if you can think of a slip up, learn from your mistake and keep it in mind for your next interview.
The bottom line:
Whatever the reason, do your best to learn from it and apply it to your next interview. Don't be afraid to politely follow up and inquire about why you weren't right for the position--just make sure you don't come off as sulking. Be professional, and thank them for helping you to understand what you did wrong, or where you can improve. After all, if you made it to the interview stage once, you're likely to do it again.