My code did not use a element with a src tag, but instead used multiple children to set a video in multiple formats.
To properly destroy and unload this video, I had to use a combination of multiple answers on this page, which resulted in:
var videoElement = $('#my-video')
videoElement[0].pause() // Pause video
videoElement.empty() // Remove all children
videoElement.load() // Load the now sourceless video
delete videoElement // The call mentioned in other answers
videoElement.remove() // Removing the video element altogether