A/C fan does not blow air, Controls work OK

I noticed that my A/C fan works when it wants to.  I hit the AUTO button and
the indicators behave normally. For instance, I turn it to 70 and it adjusts
normally, you can hear the compressor work, but the air does not blow into
the cabin.  At random times, the air would kick in and work as normal.  
However, other times it may take a while to start blowing air.  It doesn't
matter what temperature I adjust it to, it does the same thing.  Even if I do
not hit the AUTO button and just turn the fan knob up so that I get regular
outside air to blow into the cabin, it won't start when I demand it.  Maybe
after 5 or even 20 minutes I'll get the fan to start.  Do you have any idea
what it could be?  Any help would be enormously appreciated! :-)