The Waistcoat is not a versatile piece of any man’s wardrobe. It's a niche item, and can really only work well in specifi contexts. It can complement formal to business casual outfits - working better in formal than anything else. It is considerably more difficult to achieve a coherent ensemble than with most other pieces of apparel, and is rarely worn well. Seamless integration is significantly harder than you would think. It is definitely not a beginner’s item.
Fit
A waistcoat, especially when worn without a jacket, needs fit really, really damn well. Worn sans jacket, a slim silhouette is essential - otherwise you will look boxy as hell. Furthermore, the waistcoat must cover the trousers' waistband. Should a patch of belt or shirt be exposed, the entire outfit will crumble. And everyone will laugh. And no-one will like you. (Note: I may or may not be exaggerating slightly) In all seriousness though, make it cover your waistband. Avoid looking like this
In all likelihood, you won't be wearing a waistcoat THAT often, so it makes sense to just have one tailor-made. It'll look better - and bespoke waistcoats generally aren't that expensive.
The Three Piece Suit
A waistcoat worn as part of a three piece suit lends an air of seriousness and sophistication. Roger Sterling in Mad Men is a good example:
Be sure to match the waistcoat with the suit colour and fabric exactly. A small amount of discrepancy causes disharmony. If you’re going to wear a waistcoat a different colour to the suit, make it contrast by a significant amount. The character Lane Pryce in Mad Men does it beautifully:
This is the easiest, and best look to achieve, in my opinion.
The Odd Waistcoat
As a general rule, waistcoats worn sans jacket tend to look like you lost your jacket, but there are a few things you can do to pull it off.
Texture is your best friend. Look for tweed especially
Pattern is pretty much essential. Look for waistcoats with two or more colours
Fit has to be spot on. As in, perfect. Don't look like a box
Contrast with your pants and shirt.
The Casual Waistcoat
I'm going to be perfectly honest. Waistcoats aren't really a casual item. However, by following the same general rules, as before, you can dress it down somewhat. At best, you can wear it with chinos, wingtips, a knit tie and an unstructured blazer. At best.
This gentlemen does it particularly well. He has the contrasting waistcoat, with texture, in an outfit that it supported by his other items. You can take a look at his blog here
Belts v Suspenders
Suspenders are the recommended go-to item for a waistcoat outfit. Belts add more bulk to an area where a slim fit is paramount, and have a tendency to slide down the hip, risking a reveal of your belt. A suspender avoids this entirely, while keeping the trousers uniformly held up. As Alan Flusser eloquently put it, "Belts and vests should choose different dance partners"
Quick Tip
Trousers hang differently when held up by suspenders as opposed to belts. If you intend to wear vests regularly, designate one pair of trousers as your "vest trousers" and have them tailored while wearing suspenders - to ensure that the trousers hang correctly you wear them.
Button Etiquette
Most waistcoats require the bottom button to be left unfastened. The origin of this practise is somewhat dubious, but can be perhaps traced back to King Edward VII, whose expanding waistline required he leave the bottom button undone. It has also been suggested that the custom originated to prevent the waistcoat riding up when on horseback. Whatever the origin, it has become a convention that perseveres even today. It is not only a matter of etiquette, but also one of practicality. When fastened, the bottom of the waistcoat pulls sideways, which causes bulging and causes it to become misshapen. When left undone, this stress is avoided entirely. With most modern waistcoats, the bottom button should never be fastened, as it throws off the tailoring of the suit, effectively destroying your silhouette.
Tie v Bowtie
This is a situation where a bowtie can actually work better than a tie. A tie can make the "V" crowded, while a Bowtie can work particularly well in helping the area "breathe". Either is acceptable here.
Parting Notes
If the waistcoat has a silk pseudo lining on the back, it indicates that it is meant for wear as part of a three piece suit, and is not meant for solo wear
Avoid a black waistcoat in any situation that would not require a black suit. It’s simply too formal
Double breasted are too "costumy" in anything other than black tie. Even then, be careful
Don't ever wear one without a tie. The "V" shape made looks really empty. Furthermore, you shouldn't really wear a waistcoat in any situation where you wouldn't wear a tie.
In conclusion, it is definitely possible to wear a waistcoat in modern society. It's just pretty easy to fuck it up. Good luck.
EDIT: I changed the opening paragraph. The waistcoat is not a versatile piece. It really is a niche item. Sorry for the confusion