I installed Jenkins on my vserver. When I had a look at htop Jenkins was running with 30 thread, each was allowed to allocate 247MB memory and up to 1181MB virtual memory.>
http://winstone.sourceforge.net/#commandLine is the official command-line reference - but as I mention in the comment --handlerCountMax (or --handlerCountStartup, for that matter) did not seem to work for me. Try it yourself (here's how to run Jenkins in stand-alone mode).
You may want to try to reduce the number of executors as well as disabling plugins you do not need and see what happens.
Please keep in mind, however, that if you plan to continue using Jenkins seriously, you should plan for more resources, not less: as the number of your jobs grows, so will the resource utilization.