First up, plastic. You may call it polymer or rotomolding, but it's plastic. For 90%+ this is the material of choice and if you don't have a requirement that pushes you toward a different material you have already found your home. Plastic is the cheapest material covered in this article, yet still one of the most durable.
Plastic isn't perfect by a long shot. Plastic boats can be flexy in rough water. They tend to be heavier than their hard-shell cousins and have been known to deform under weight and/or heat. The finish isn't as nice as other materials since regular plastic just doesn't shine.
Fiberglass suffers no such issues. Fiberglass looks great, weighs less than plastic, and resists flexing and deforming. With 'glass the paddling is generally more direct and feels very solid but think twice about that beach landing or navigating a rock garden. Fiberglass can crack and scratch in a way that plastic just doesn't, and a glass boat may cost double or more what a comparable plastic boat costs.
That leaves the exotics and hybrids. Let's consider Kevlar. Kevlar is exactly like fiberglass except that Kevlar's properties allow for a thinner hull that is just as strong as 'glass. That means Kevlar boats are lighter, but weight is a funny bowling partner. A fiberglass Current Designs will be as light or lighter than a Kevlar Wilderness. Current builds a boat just above the minimum strength standard (thinner materials) while Wilderness builds a boat just under maximum strength standards (thicker material). The only real rule is that a comparable boat from the same manufaturer is 5-10 LBS lighter in Kevlar than fiberglass for a $500 - $1000 premium when new. That's $100 per pound when most of us could just skip a meal or two for the same weight saving results. Short of lifting a kayak onto a roof rack, what difference does a few pounds make? If racing is your passion you can always find a race-specific boat generally built a bit fragile, light, and without frills. For touring, recreation and fitness wight isn't going to change all that much in performance.
There are some variations on plastic worth noting. Companies like P&H and Prijon use techniques to build plastic boats that are sturdier and stiffer than the competition. Both companies products are worth a look but the technology comes at a small premium. Even the standard roto plastic boats have gotten quite good anymore, and are less prone to the deformation issues of boats from fifteen years ago.
You may also consider the new high-density polymers that look and act a lot like fiberglass but are in fact a thin walled, high quality plastic. Boats like Eddyline look great, are extremely light, resist scratching and have crazy durability. While these boats aren't quite as stiff and unflexing as fiberglass they are better than standard plastic. Note that the pricing is closer to fiberglass in most cases.
The bottom line is that plastic is the first line of defense and the natural choice if you aren't sure. High quality, high density plastics are great if you can get a boat in your budget. Fiberglass makes for a great paddling experience but only if you are prepared to be more careful. Kevlar is nice, but fiberglass is very nearly as good.