Tracing Python expression evaluation step by step

Question

I\'m trying to write Python expression evaluation visualizer, that will show how Python expressions are evaluated step by step (for education purposes). Philip Guo\'s Python

Answer 1

Expression stepping is implemented in Thonny IDE.

It uses AST instrumentation, where each (sub)expression e is transformed into after(before(<location info>), e). Functions before and after are dummy functions for causing extra call-events in Python's tracing system. These extra calls notify when (sub)expression evaluation is about to start or has just ended. (Similar dummy functions are added to detect start and end of each statement.)

AST instrumentation and interpretation of these new events is done in thonny.backend.FancyTracer.

Python's AST nodes contain start position of corresponding text ranges, but they are sometimes incorrect. End positions are completely missing. thonny.ast_utils.mark_text_ranges tries to take care of this (but the solution is incomplete at the moment).

It would be nice if somebody extracted relevant functionality from Thonny to a more general package. Maybe even two packages -- one for computing location info for Python AST and other for detailed tracing of Python code. I'd be willing to help with this, if someone took the lead.

Answer 2

Why not use the dis module?

Since CPython compiles Python to bytecode and runs that, looking at the bytecode gives you the best idea of what actually happens.

In [1]: import dis

In [2]: dis.dis('sorted([4, 2, 3, 1] + [5, 6])[1] == 2')
  1           0 LOAD_NAME                0 (sorted)
              3 LOAD_CONST               0 (4)
              6 LOAD_CONST               1 (2)
              9 LOAD_CONST               2 (3)
             12 LOAD_CONST               3 (1)
             15 BUILD_LIST               4
             18 LOAD_CONST               4 (5)
             21 LOAD_CONST               5 (6)
             24 BUILD_LIST               2
             27 BINARY_ADD
             28 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             31 LOAD_CONST               3 (1)
             34 BINARY_SUBSCR
             35 LOAD_CONST               1 (2)
             38 COMPARE_OP               2 (==)
             41 RETURN_VALUE

Edit: An alternative method could be to show the steps one-by-one in IPython:

In [1]: [4, 2, 3, 1]
Out[1]: [4, 2, 3, 1]

In [2]: [4, 2, 3, 1] + [5, 6]
Out[2]: [4, 2, 3, 1, 5, 6]

In [3]: sorted([4, 2, 3, 1, 5, 6])
Out[3]: [1, 2, 3, 4, 5, 6]

In [4]: [1, 2, 3, 4, 5, 6][1]
Out[4]: 2

In [5]: 2 == 2
Out[5]: True

Answer 3

The addition of the two lists is certainly not the first node to be evaluated in that code; I believe there are in fact nine earlier node evaluations - sorted, 4, 2, 3, 1, [4,2,3,1], 5, 6, [5,6]. Not only would you have to determine what order evaluations are performed in, you'd also have to decide which of those evaluations are worth showing.

I think a better approach to your problem would be to modify the AST nodes so that they emit their before/after state as a side-effect of being executed. You wouldn't care about their order, you'd just execute the entire expression once. And there is already a package called macropy that has a tracing feature that does exactly this. Its output isn't quite what you're asking for, but it could probably be modified to be a closer match.