inside door leather panel keeps coming of

It should have blind plastic fasteners that secure it to the door, pop the panel off and see if any are missing or broken. You can buy replacements at most auto parts stores, they'll have a wide variety of different types. Just pop them back in. The panel is likely held on by more than just the plastic fasteners, there should also be some screws and/or bolts located around the armrest or along the bottom of the panel. If any part of the panel itself, especially the area that holds the plastic fasteners, is broken, you can fix it with an epoxy glue.

Worst case you would need a new door panel.

I believe you’re talking about the small leather panel just above the armrest. If that is indeed the case, it falls off because an inferior adhesive was used at the factory to secure the panel. You can fix this by scrapping all the old glue residue off, then masking the door panel with tape and paper then applying new glue to the back of the leather as well as the corresponding surface of the door, lastly carefully reapply the panel with and push the edges in the seam with a credit card or something. Plenty of videos on YouTube tackle this issue. I replaced my panel with some contrasting red fabric I got from Walmart so creativity is possible