According to this link, there is an option to set MaxRamSize manually to restrict the JVM to not use memory beyond this. But I have not seen any documentation of the same. I
Why I'm looking for this kind of an option to run the application inside a container (Like Docker) and prevent the application from being killed by the OOM Killer.
This is exactly the option you are looking for. Moreover, it was properly integrated with cgroups (~docker) as part of JDK-8170888.
Note that this flag does not prevent JVM from allocating more memory than its value, but rather gives a hint "I know my physical RAM limit is X, please allocate within X", so JVM will share this value between Java heap and native memory. But if e.g. your application has native memory leak or classloader leak, then this limit will be reached anyway.
JVM is complaining on MaxRAM=1073741824B because it doesn't expect B in the end, it should be declared as -XX:MaxRAM=1073741824.
See the description of the flag: Real memory size (in bytes) used to set maximum heap size.