Fortunately, there is a basic method that will allow you to begin to critically assess fly selection. That method is based (in no particular order) on four characteristics: Size, shape, color and behavior.
Size. Refers simply to the actual physical dimensions of the organism you're tying or buying a fly to imitate. It could be five-millimeter-long midge or a foot-long squid. In a majority of situations, flies that are too big (say a size 14 when the natural is an 18), or too small (18 when it should be a 14), aren't accepted as readily by the fish (and often will not be accepted at all). And remember: Size differentials on smaller organisms are much more obvious than on larger organisms.
Shape. Refers to the overall look of the organism being imitated. Does it have obvious upright wings? Perhaps it has a long, cylindrical body? How does it hold its legs? These are the kinds of questions you need to ask yourself, and once you've answered them, select a fly that has the most obvious shapes designed into it. Just remember this: Fish don't see as well as humans, so perfect, photorealistic shape is not necessary for a successful fly. Indeed, many of the best flies are quite impressionistic -- just giving the general outline that's needed.
Color. Can be extremely important at times, and not so important at others. I've seen fish that wanted midges that were one particular shade of green. Identical flies (but with different greens or other colors) presented identically simply did not work nearly as well as a fly with the right green. On the other hand, there are times (when simply "searching" for fish, for example), that a big, all-black fly can work wonders. It's not the color in and of itself, but rather the strong silhouette provided by the blackness.
Behavior. Means what the organism is doing. Is it swimming? Resting? Trying to pull itself free from the water? A fly that is designed to mimic that motion is already one step toward getting a fish to take. An imitation of a leech or baitfish will be more successful if it has parts on it that undulate and wave in the water like fins or tails. Conversely, an imitation of a dead mayfly spinner should float along looking, well, dead.
These four characteristics are a starting point to understanding fly selection. It's also important to understand the relationship of those four characteristics to each other and a few general guidelines as to which characteristics are most important in which circumstances. Before proceeding, it's important to know for whom you're buying or tying a fly. If you want a fly to hang on your wall as part of a collection, then you're buying or tying for yourself. However, if you're selecting a fly to fish with, you're buying for the FISH. Selecting a fly simply because it's "a nice-looking tie" is a poor criteria when it comes to impressing the finny denizens of the deep. Rather, selecting a fly that exhibits the necessary elements of the natural in ways that can be perceived by the fish is what you're after.
Among the four characteristics, there is one that will be the "triggering device" that signals a fish to investigate the fly. Referred to as the "Primary Trigger," this characteristic needs to be present in the fly in order for it to be truly successful. The three other characteristics ("Secondary Triggers") would assume a supporting role for the Primary Trigger, making the whole of the imitation complete in the fish's eyes.
A fly that exhibits the Primary Trigger will initially get looked at by the fish. Then, the Secondary Triggers become important as the fish scrutinizes the fly more closely (this may only take a second). If the secondary triggers are not present, or are past the tolerance point of that particular fish, then the fish will not take the fly.
In addition, the Primary Trigger can be enhanced ("supercharged," if you will) in order to make the fly appear more real than the real thing. These "Super Triggers" can be based in design, materials and presentation, and getting to know what they are can make you a more successful angler.
The physical structure and life cycle stages of a food organism can have a marked effect on how the four characteristics are interrelated. For example, there may be a time during the life-cycle of a mayfly when shape is the Primary Trigger to the fish. At other times color may be Primary Trigger. Although there's really no hard and fast rule for determining which is more important initially (outside of reading what others have discovered), you can work this way: If the organism being imitated is exceptionally large or small, then size could be the Primary Trigger. If the organism has a very distinct shape (obvious upright wings, for example), then shape may be the place to start. If the color of an organism is bright, or reflects significant light, then color could be your Primary characteristic. If the organism moves (or doesn't move) in a distinct manner, then behavior could be the trigger you want to imitate first off.
Of course, there are organisms that exhibit all of the above, and sometimes pure trial-and-error is needed to discern the Primary Trigger. I've found that when it comes to mayflies, shape is often the number one characteristic (see the photos). In insects like damselfly nymphs, which are strongly-undulating swimmers, behavior is definitely the ticket. When it comes to minute creatures like midges, size can be very important, but then so can surface impression (really a subset of shape). I've rarely seen color as the Primary Trigger in most organisms, but there have certainly been times when that was true.
When it comes to purely "attractor" flies, behavior is a good Primary place to start (lots of motion). Size and color can also be quite important, as a fly that's easy for the fish to see may have a better chance of getting taken.