Interior trunk access panel won't unlock!

I recently purchased this vehicle from a family friend, while I was cleaning I
noticed it had a trunk access panel behind the fold down arm rest/cup
holder in the back seat.

I tried my key (both ways), but it doesn't go in all the way nor does it
actually unlock it. The visible metal piece of the latch wiggles, but that's all I
get. My key is NOT a valet key, it works to unlock/lock the glove box as well
as the trunk (it also starts the engine, and works on all doors.)

The previous owner didn't even know the car had this feature, so he wasn't
much help either. The image shows the location of the access panel and
the lock on it. I don't necessarily NEED access to the trunk from the interior,
but it would be nice!