grinding from the rear end

Over the past few weeks I have noticed that there is a low, grinding, growling sound
coming somewhere in the rear of the car. Some things I know:

A) it's not the breaks
B) only happens when the car is in gear
C) only happens when the clutch is disengaged (not depressed)
D) is mainly noticeable while decelerating, but not hard deceleration
E) is very faint or unnoticeable while accelerating
F) seems to get ever so slightly worse every time I drive it

I'm about to order a 17mm hex driver and change the fluid in the transaxle, but my
mechanic says he thinks it's a CV if anything. Any additional thoughts?