I\'m using the following code to work with Git in a Java application. I have a valid key (use it all the time), and this specific code has work for me before with the same
You read a file named .pem and de-base64 all of it and treat the result as PKCS8-unencrypted, apparently successfully. This means the file was NOT PEM-format. PEM format at minimum MUST have the dash-BEGIN and dash-END lines to be valid, which if not removed cause de-base64 to either fail or be wrong. (Some PEM formats also have 822-style headers which must be handled.)
You appear to be using BouncyCastle, but in my versions there is no PKCS8Generator constructor that takes only RSAPrivateKey. The closest thing that works is JcaPKCS8Generator (RSAPrivateKey implements PrivateKey, OutputEncryptor=null) (i.e. a different but related class, and two arguments not one).
PemWriter is buffered, and you didn't flush it before looking at the underlying StringWriter. As a result writer.toString().getBytes() is an empty/zero-length array, which JSch rightly considers invalid.
With #2 and #3 fixed and using my input, and calling JSch directly instead of via JGit, it works for me.