I wanted vim-like navigation for my terminal, so I added:
set editing-mode vi
set keymap vi
To my .inputrc file based on this.
From man readline (my emphasis):
The set of legal keymap names is emacs, emacs-standard, emacs-meta, emacs-ctlx, vi, vi-move, vi-command, and vi-insert. vi is equivalent to vi-command
Thus, in your inputrc you can specify different keybinds for the different modes, so, for example, in command mode you could yank the last argument from the previous command with Altp, but this keybind would have no effect in insert mode (in this simple setup anyway):
set editing-mode vi
set keymap vi-command
# these are for vi-command mode
"\e[A": history-search-backward
"\e[B": history-search-forward
"\ep": yank-last-arg
set keymap vi-insert
# these are for vi-insert mode
"\e[A": history-search-backward
"\e[B": history-search-forward
Control-l: clear-screen