I can use \\p{Punct} to match all punctuations(including underscore).
\\p{Punct}
And I wanted to exclude all apostrophes strictly inside a word.
You could group all punctuations, you are interested in, manually and exclude the apostrophe. Then combine that group with your rule for finding the right apostrophes (that are not within a word) by an OR.