问题
I have a example excel file data2.xlsx from here, which has a Sheet1 as follows:
Preprocess:
The columns 2018, 2019, 2020, num are object type, which I need to convert to float:
cols = ['2018', '2019', '2020', 'num']
df[cols].replace('--', np.nan, regex=True).astype(float)
Also I need to extract city names from bj, sh, gz, sz from 2019-bj-price-quantity, 2019-sh-price-quantity, 2019-gz-price-quantity, 2019-sz-price-quantity
pattern = '|'.join(['2019-', '-price-quantity'])
df['city'] = df['city'].str.replace(pattern, '')
Finally I need to extract price and quantity of nums for each city and reshape a new dataframe like this:
How could I do that in pandas? Thanks.
Update:
df = pd.read_excel('./data2.xlsx', sheet_name = 'Sheet1', header = None)
df.groupby(df.iloc[:, 0].isna().cumsum()).transform('first')
Out:
0 1 2 3 4
0 2019-bj-price-quantity 2018.0 2019.0 2020.0 num
1 2019-bj-price-quantity 2018.0 2019.0 2020.0 num
2 2019-bj-price-quantity 2018.0 2019.0 2020.0 num
3 2019-bj-price-quantity 2018.0 2019.0 2020.0 num
4 2019-sh-price-quantity 2018.0 2019.0 2020.0 num
5 2019-sh-price-quantity 2018.0 2019.0 2020.0 num
6 2019-sh-price-quantity 2018.0 2019.0 2020.0 num
7 2019-sh-price-quantity 2018.0 2019.0 2020.0 num
8 2019-sh-price-quantity 2018.0 2019.0 2020.0 num
9 NaN NaN NaN NaN NaN
10 2019-gz-price-quantity 2018.0 2019.0 2020.0 num
11 2019-gz-price-quantity 2018.0 2019.0 2020.0 num
12 2019-gz-price-quantity 2018.0 2019.0 2020.0 num
13 2019-gz-price-quantity 2018.0 2019.0 2020.0 num
14 2019-gz-price-quantity 2018.0 2019.0 2020.0 num
15 NaN NaN NaN NaN NaN
16 2019-sz-price-quantity 2018.0 2019.0 2020.0 num
17 2019-sz-price-quantity 2018.0 2019.0 2020.0 num
18 2019-sz-price-quantity 2018.0 2019.0 2020.0 num
19 2019-sz-price-quantity 2018.0 2019.0 2020.0 num
20 2019-sz-price-quantity 2018.0 2019.0 2020.0 num
Reference related: Read dataframe split by nan rows and reshape them into multiple dataframes in Python
回答1:
*note I use column indices when the column name is not certain
You can split tables with
df['city'] = df.groupby(df.iloc[:, 0].isna().cumsum()).transform(first)
df.dropna(subset=df.columns[0], inplace=True)
df = df.loc[df[df.colmns[0]] != df.city]
Now df will have an additional column city with the table title, while the title and empty rows have been discarded. You can access any part of that city column with .str.split.str.get
df.city = df.city.str.split('-').str.get(1)
Finally you want to keep just the num column, which is the easiest step
df = df.iloc[:, [0, 4, 5]]
df = df.pivot(index='city', columns=df.columns[0], values=df.columns[1])
回答2:
My code based on jezrael's great answer, welcome to share better solution or improve it:
# add header=None for default columns names
df = pd.read_excel('./data2.xlsx', sheet_name = 'Sheet1', header=None)
# convert columns by second row
df.columns = df.iloc[1].rename(None)
# create new column `city` by forward filling non missing values by second column
df.insert(0, 'city', df.iloc[:, 0].mask(df.iloc[:, 1].notna()).ffill())
pattern = '|'.join(['2019-', '-price-quantity'])
df['city'] = df['city'].str.replace(pattern, '')
df['year'] = df['year'].str.replace(pattern, '')
# convert floats to integers
df.columns = [int(x) if isinstance(x, float) else x for x in df.columns]
df = df[df.year.isin(['price', 'quantity'])]
df = df[['city', 'year', 'num']]
df['num'] = df['num'].replace('--', np.nan, regex=True).astype(float)
df = df.set_index(['city', 'year']).unstack().reset_index()
df.columns = df.columns.droplevel(0)
df.rename({'year': 'city'}, axis=1, inplace=True)
print(df)
Out:
year price quantity
0 bj 21.0 10.0
1 gz 6.0 15.0
2 sh 12.0 NaN
3 sz 13.0 NaN
来源：https://stackoverflow.com/questions/63250668/read-dataframe-split-by-nan-rows-and-extract-specific-columns-in-python