PyInstaller: a module is not included into --onefile, but works fine with --onedir

Question

I'm using PyInstaller to bundle my application into one .exe file. The problem is that it works fine with --onedir option, but can't find a module when built with --onefile.

Both --onedir and --onefile say during the building process:

<...>
INFO: Analyzing hidden import 'sklearn.utils.sparsetools._graph_validation'
<...>

Running the instance created with --onedir works fine, but the instance produced by --onefile dies:

<...>
  File "_min_spanning_tree.pyx", line 8, in init sklearn.utils.mst._min_spanning
_tree (sklearn\utils\sparsetools\_min_spanning_tree.c:4754)
ImportError: No module named _graph_validation

Here are my .spec files

onedir.spec

# -*- mode: python -*-
a = Analysis(['../../brainactivity.py'],
             hiddenimports=['greenlet', 'sklearn.utils.sparsetools._graph_validation', 'sklearn.utils.sparsetools._graph_tools', 'scipy.special._ufuncs_cxx', 'sklearn.utils.lgamma', 'sklearn.utils.weight_vector'],
             hookspath=None,
             runtime_hooks=None)
pyz = PYZ(a.pure)
exe = EXE(pyz,
          a.scripts,
          exclude_binaries=True,
          name='brainactivity.exe',
          debug=False,
          strip=None,
          upx=True,
          console=True,)
coll = COLLECT(exe,
               a.binaries,
               [('./data/201305182224-DF-facial-3-420.csv', '../../data/201305182224-DF-facial-3-420.csv', 'DATA')],
               [('./model/brain_20k_colored_properly.obj', '../../model/brain_20k_colored_properly.obj', 'DATA')],
               [('brain_fragment_shader.glsl', '../../brain_fragment_shader.glsl', 'DATA')],
               [('brain_vertex_shader.glsl', '../../brain_vertex_shader.glsl', 'DATA')],
               a.zipfiles,
               a.datas,
               strip=None,
               upx=True,
               name='brainactivity')

onefile.spec

# -*- mode: python -*-
a = Analysis(['../../brainactivity.py'],
             hiddenimports=['greenlet', 'sklearn.utils.sparsetools._graph_validation', 'sklearn.utils.sparsetools._graph_tools', 'scipy.special._ufuncs_cxx', 'sklearn.utils.lgamma', 'sklearn.utils.weight_vector'],
             hookspath='.',
             runtime_hooks=None)
pyz = PYZ(a.pure)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          [('./data/201305182224-DF-facial-3-420.csv', '../../data/201305182224-DF-facial-3-420.csv', 'DATA')],
          [('./model/brain_20k_colored_properly.obj', '../../model/brain_20k_colored_properly.obj', 'DATA')],
          [('brain_fragment_shader.glsl', '../../brain_fragment_shader.glsl', 'DATA')],
          [('brain_vertex_shader.glsl', '../../brain_vertex_shader.glsl', 'DATA')],
          a.zipfiles,
          a.datas,
          name='brainactivity.exe',
          debug=False,
          strip=None,
          upx=True,
          console=True )

Answer 1

I had the same error. The solution is to create a hook for sklearn. Generally u need to create a hook file like this

hiddenimports = ['sklearn.utils.sparsetools._graph_validation'] 

and save this in in a file with name hook-modulename.py in the same folder. But this will import only _graph_validation. This might lead to error for another module. Best to import all the submodules in a package by

from hookutils import collect_submodules
hiddenimports = collect_submodules('sklearn') 

and save it to a hook file in the same folder. For me i had to create 2 hook file. one for sklearn and one for scipy.

from hookutils import collect_submodules
hiddenimports = collect_submodules('scipy') 

after saving them i used below command to run

pyinstaller --additional-hooks-dir=. myfile.py

for better understanding follow http://pythonhosted.org/PyInstaller/hooks.html#understanding-pyinstaller-hooks

Answer 2

just import following packages in the script which is going to convert into exe file

 import xgboost
 import sklearn.ensemble
 import sklearn.tree
 import pickle
 import pandas as pd
 import sklearn.neighbors.typedefs
 import sklearn.neighbors.quad_tree
 import sklearn.tree._utils
 import cython
 import sklearn
 import sklearn.utils._cython_blas
 import numpy as np
 import joblib
 from sklearn.preprocessing import StandardScaler

this help me in solving this issue.