I can't be of much help because I've never used it. But the emulator's website says zip files should work:

Currently, Vita3K supports .pkg, NoNpDrm, FAGDec, or manually decrypted games (Vitamin dumps are not supported and Maidump is unstable). The games should be in a .zip or .vpk format if you want to install them from the emulator, or if you prefer to copy them yourself, you can drag and drop the game folder in your pref_path/ux0/app folder (not applicable for NoNpDrm dumps/.pkg files).

https://vita3k.org/quickstart.html

Edit: also where did you yoink the rom from? And what are the contents of the zip file?