Use the controller icon before starting the game to re-map a button (I used one of the back triggers). After you select the button you want to use, assign it to the Del key on the num pad. Quick fix, and works like a charm.

Steam Deck doesn't always have perfect controller mapping out of the gate... but is super-flexible once you learn how to configure custom button mappings!

For anyone else finding this in the future: yes, Steam Input can let you remap any key you want to any button, so you can assign the Del key to anything you need at any point.

However, Reventure has full controller support and doesn't need the Del key at all (but unfortunately doesn't list the buttons anywhere).

To delete a save with a controller, select it in the menu and hold down at the same time either L1+R1 or L3+R3 (ie LB+RB or LS+RS if you're used to Xbox prompts).