For anyone in the future, I solved this by adding the following css :
*:not(input) {
user-select: none;
}
The above basically disable text selection on all elements, which during my testing disallowed zooming on webpage. FYI: I haven't dived deep into details, just stating its effects.