Python: KeyError: 0 - function that creates a list from subsets of data frame accessed via a loop

问题

I have a loop that creates subsets of a larger data frame, which are then used as input for a function for analysis. This function returns a list. I used prints to see where it stops. So for the first run in the loop I can see the subset, the list output of the function, but when it starts again, second run, I see the second subset but than at the function I get the following error:

<ipython-input-11-8f6203e297e3> in ssd(x, y)
      8 
      9     for i in range(x.shape[0]):
---> 10         spread_cumdiff += (x[i] - y[i]) **2
     11 
     12     return spread_cumdiff

Note the above part is the last part "before" the python-error below. In fact, above has 2 more similar blocks, ie function a) which contains function b) which contains the block above.

~/anaconda3/envs/thesis/lib/python3.5/site-packages/pandas/core/series.py in __getitem__(self, key)
    621         key = com._apply_if_callable(key, self)
    622         try:
--> 623             result = self.index.get_value(self, key)
    624 
    625             if not is_scalar(result):

~/anaconda3/envs/thesis/lib/python3.5/site-packages/pandas/core/indexes/base.py in get_value(self, series, key)
   2558         try:
   2559             return self._engine.get_value(s, k,
-> 2560                                           tz=getattr(series.dtype, 'tz', None))
   2561         except KeyError as e1:
   2562             if len(self) > 0 and self.inferred_type in ['integer', 'boolean']:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 0

I have it something along these lines:

df_test = pd.DataFrame(np.random.randint(low=0, high=10, size=(5, 5)),
                    columns=['a', 'b', 'c', 'd', 'e'], 
                    index = ['20100101', '20100102', '20100103', '20100104', '20100105']

dfs = []
N = 3
for x in np.arange(len(df_test)+1)[N:]:
    df1 = df_test.iloc[np.arange(x - N, x)]

    test_list = myfunc(df1) # it takes in df1, makes some computation and returns a
                            # list of 2-element tuples, i.e. [('a', 'b'), ('d', 'e')]

edit: please see below the function:

def ssd(x, y):

    spread_cumdiff = 0

    for i in range(x.shape[0]):
        #print("x_i", x[i])
        #print("y_i", y[i])
        spread_cumdiff += (x[i] - y[i]) **2

    return spread_cumdiff

I tried to use the print function but it doesn't even come that far for the second run of the loop.

def pairs_match(df, p):

    df_norm = df.assign(**df.drop('datetime', 1).pipe(lambda d: d.div(d.shift().bfill()).cumprod()))
    df_norm = df_norm.replace([np.inf, -np.inf], np.nan)

    df_norm.fillna(method = 'ffill', inplace = True)
    df_norm.fillna(method = 'bfill', inplace = True)

    ticker = df_norm.columns.values.tolist()
    ticker.pop(0)
    ticker_list = pd.DataFrame({'ticker': ticker})

    # to be implemented: if length of list list <2, then skip the entire run! 

    all_pairs = list(itertools.permutations(ticker_list.ticker, 2))
    squared = []
    presel_pairs = []

    for i in all_pairs:
        squared.append(ssd(df_norm[i[0]].head(n = train_win), df_norm[i[1]].head(n = train_win)))   # ssd(x,y) function from above

    tbl_dist = pd.DataFrame({'Pair' : all_pairs, 'SSD' : squared})

    ssd_perctl = p
    ssd_thresh = stats.scoreatpercentile(tbl_dist['SSD'], ssd_perctl)

    presel_pairs = tbl_dist[tbl_dist['SSD'] <= ssd_thresh]
    presel_pairs_list = presel_pairs['Pair']
    presel_pairs_list = presel_pairs_list.reset_index(drop = True)

    return presel_pairs_list      

def pairs_match(df, p) returns a list, which is then used in another function.

回答1:

try printing x[i] and y[i] seperaterly so you know which of the 2 causes the keyerror. Also please post the function because without it we have no clue what's going on.

回答2:

As thought, the problem really was at the cumsum-update function. I have rewritten the function as follows:

def ssd(x, y):

    spread_diff_sq = np.subtract(x, y) **2
    spread_diff_sq_cum = spread_diff_sq.cumsum()
    spread_cumdiff = spread_diff_sq_cum.iloc[-1]

    return spread_cumdiff

this doesn't go to the root of the problem, nevertheless it's avoiding the loop.

来源：https://stackoverflow.com/questions/49909524/python-keyerror-0-function-that-creates-a-list-from-subsets-of-data-frame-ac