Parsing with libclang; unable to parse certain tokens (Python in Windows)

Question

I have some code (taken and adapted from here and here), which uses libclang to parse C++ sourcefiles in Python (Widnows) and get all of its declaration

Answer 1

By default, libclang doesn't add the compiler system include path.

Always make sure you've checked the diagnostics - like compiler error messages, they tend to indicate how to resolve any issues. In this case, it would have been a reasonably obvious there was an include issue:

, spelling "'iostream' file not found">

If you make sure libclang adds those paths, it should start working.

This question includes an approach to solving this problem. This seems to be a recurring theme on Stackoverflow, so I wrote ccsyspath to help find those paths on OSX, Linux and Windows. Simplifying your code slightly:

import clang.cindex
clang.cindex.Config.set_library_file('C:/Program Files (x86)/LLVM/bin/libclang.dll')
import ccsyspath

index = clang.cindex.Index.create()

args    = '-x c++ --std=c++11'.split()
syspath = ccsyspath.system_include_paths('clang++')
incargs = [ b'-I' + inc for inc in syspath ]
args    = args + incargs

trans_unit = index.parse('test.cpp', args=args)
for node in trans_unit.cursor.walk_preorder():
    if node.location.file is None:
        continue
    if node.location.file.name != 'test.cpp':
        continue
    if node.kind.is_declaration():
        print(node.kind, node.location)

Where my args end up being:

['-x',
 'c++',
 '--std=c++11',
 '-IC:\\Program Files (x86)\\LLVM\\bin\\..\\lib\\clang\\3.8.0\\include',
 '-IC:\\Program Files (x86)\\Microsoft Visual Studio 12.0\\VC\\include',
 '-IC:\\Program Files (x86)\\Windows Kits\\8.1\\include\\shared',
 '-IC:\\Program Files (x86)\\Windows Kits\\8.1\\include\\um',
 '-IC:\\Program Files (x86)\\Windows Kits\\8.1\\include\\winrt']

and the output is:

(CursorKind.USING_DIRECTIVE, )
(CursorKind.FUNCTION_DECL, )
(CursorKind.PARM_DECL, )
(CursorKind.PARM_DECL, )
(CursorKind.VAR_DECL, )
(CursorKind.FUNCTION_DECL, )
(CursorKind.PARM_DECL, )
(CursorKind.VAR_DECL, )
(CursorKind.VAR_DECL, )