You could add the path to the gem in the autoload paths for the app.
So, in config/application.rb, within the class Application < Rails::Application ... end block add:
config.autoload_paths += %W(#{config.root}/vendor/gems/my_gem/lib)
config.autoload_paths += Dir["#{config.root}/vendor/gems/my_gem/lib/**/"]
Then, when running your development server, all files in there should be reloaded.