If the entire panel was loose, it wasn't installed correctly. The arm rest/door pull hooks are enough to retain most of the panel with the bottom hooks doing the rest.
Pop out the insert around the door release lever. Slide the panel over the lever and place it against the door, higher than you...