Unfortunately, that’s not possible, the Dialog object is not available in custom code for list views. But you can open the dialog from a panel, as you mention.
The idea is to have a button (added by JS in custom code) which open a Sharepoint site in a dialog. Currently, the site has to be opened in a new browser tab which is not very handy for the user.
Do I understand it right, you have now a configuration when a list items are opened in a new tab? By default, they are opened in the same tab. Do you have any custom code that implements this behavior?
You can also try to turn on “Launch forms in dialog” in List Settings -> Advanced Settings, when the site is in Classic UI mode.