sold car without title in hand, now buyer wants refund

I put my '99 Mazda Protege up for sale with a sign in the window showing my
phone number and the asking price of $400. I got a buyer interested almost
immediately. I was very honest with him that this is an old car with over 325k
miles, but it runs very well. The one problem it had is that the battery died,
which I did by sitting with the car off but the headlights on (not used to having
manual lights). This was no problem to the buyer, who met up with me that
night to see if his buddy could jump start the car. It took and the car was in
satisfactory order for the sale to continue. The next day, the buyer paid me the
$400 in cash which I took into my apartment and placed in my safe, where I
believed my title was.

Unfortunately, this is where things go very wrong. I could not find my title
anywhere, probably because I am in the midst of moving for the second time in
two months. Anyway, I came back outside and told the buyer that I couldn't
find the title but I would request a duplicate from the DOT that evening after
work. He agreed to this. I contacted him later to let him know I would get the
title to him next week as I couldn't get it immediately on the spot. He said okay
to that.

I texted him later and asked if he wanted me to go out and take the plates off
the car because we are both moving out of the same apartment complex right
now. He said he could take care of that, and I thought that was fine because
we were definitely going to see each other again to do the title transfer and I'd
just take the plates and my custom bracket back then. Then he tells me they're
using the car to move him to his new place and they want to keep the plates
on so they don't get pulled over for running around with no plates on the car,
which made sense because he's not on the title yet, but I'm now realizing was
a huge risk on my part because my name is still associated with this car that
he may have messed up.

Tonight, after all this earlier today, he calls me THREE times in quick
succession well after 11pm on a Friday night. I thought about ignoring his calls
in all honesty but when the third one came in I answered. He immediately
started in about how the car was suddenly having problems and that he thinks
I screwed him over. He wants his money back. I tell him that most of the
money went straight toward bills related to my move and that I don't have any
way to remedy the situation at 11:30pm on a Friday night. I asked if we could
talk in the morning, which he wouldn't agree to and kept badgering me, saying
I was being dishonest, that there was something wrong with the car beside the
battery. I had no knowledge of any other issues, beside me killing the battery
on a car I won't need in my next living arrangement, which is why I'm selling it
in the first place. He kept talking about helping me scrap the car, which I did
not agree to. He told me he's going to return the battery he bought to
Autozone and tried using that against me too, even though he literally knew
and agreed to purchasing and installing a battery. I finally got him to agree to
speaking tomorrow morning when something could possibly be negotiated.

My question is can he really demand money from me? We've agreed to the
sale, in front of three witnesses (2 of his friends and 1 of mine), shook hands,
exchanged money which has been deposited in my bank account and used,
and we agreed to the title exchange happening next week because of a simple
human error of misplacement. I communicated everything I knew to him clearly
and, I mean, it's a 1999 car with 325k miles that I sold for $400 in cash. We
didn't even exchange last names (because we didn't get to the title transfer
phase yet). He didn't exactly go to a dealership for a big spender. All that to
say, it's a junky car that he immediately used to move himself into a new
apartment on one of the first few cold days of the year with a new battery that
I'm pretty sure he didn't charge properly. How do I know that he didn't cause a
new problem with the vehicle? I don't want to be a jerk to this kid, but I've
never sold a car like this before myself and frankly don't want to be screwed
over or out of money. Since there was written agreement (via text messages)
about the title transfer happening next week AND he already paid for the car,
he shouldn't be able to come after me or something, right? I mean, the car is
his. I want to go back to the DOT tomorrow morning when they open and see
if I can expedite the title duplicate for an extra fee and get it on the spot before
I have to talk to this kid again. Maybe having the title in hand will get him to
calm down about all this? Maybe?

What should I do to remedy this situation? Does he actually have a leg to
stand on with demanding his money back? Do I have the right to say no since
we agreed to the sale as is with the condition of obtaining the title by next
week?

Please help. Thank you.