I have a simple pandas pyinstaller exe which is over 40MB.
My exe example:
import collections import csv import selenium import pandas print(\'hi\')
try setting up your environment with a virtualenv, and install in there only the required libraries
some details on working with virtual env are here: https://virtualenv.pypa.io/en/stable/