Is there any way to completely ignore line break and tab characters etc. in RegEx? For instance, the line break and tab characters could be found anywhere and in any order
If you just want that regex to match that input, all you need to do is specify Singleline mode:
Regex.Matches(input, @"\[CustomToken).*?(/\])", RegexOptions.Singleline);
The dot metacharacter normally matches any character except linefeed (\n). Singleline mode, also known as "dot-matches-all" or "DOTALL" mode, allows it to match linefeeds as well.