I have a dataframe in pandas where each column has different value range. For example:
df:
A B C
1000 10 0.5
765 5 0.35
800 7 0.09
Any idea how I can normalize the columns of this dataframe where each value is between 0 and 1?
My desired output is:
A B C
1 1 1
0.765 0.5 0.7
0.8 0.7 0.18(which is 0.09/0.5)
You can use the package sklearn and its associated preprocessing utilities to normalize the data.
from sklearn import preprocessing
x = df.values #returns a numpy array
min_max_scaler = preprocessing.MinMaxScaler()
x_scaled = min_max_scaler.fit_transform(x)
df = pandas.DataFrame(x_scaled)
For more information look at the scikit-learn documentation on preprocessing data: scaling features to a range.
one easy way by using Pandas: (here I want to use mean normalization)
normalized_df=(df-df.mean())/df.std()
to use min-max normalization:
normalized_df=(df-df.min())/(df.max()-df.min())
Based on this post: https://stats.stackexchange.com/questions/70801/how-to-normalize-data-to-0-1-range
You can do the following:
def normalize(df):
result = df.copy()
for feature_name in df.columns:
max_value = df[feature_name].max()
min_value = df[feature_name].min()
result[feature_name] = (df[feature_name] - min_value) / (max_value - min_value)
return result
You don't need to stay worrying about whether your values are negative or positive. And the values should be nicely spread out between 0 and 1.
If you like using the sklearn package, you can keep the column and index names by using pandas loc like so:
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
scaled_values = scaler.fit_transform(df)
df.loc[:,:] = scaled_values
Your problem is actually a simple transform acting on the columns:
def f(s):
return s/s.max()
frame.apply(f, axis=0)
Or even more terse:
frame.apply(lambda x: x/x.max(), axis=0)
Simple is Beautiful:
df["A"] = df["A"] / df["A"].max()
df["B"] = df["B"] / df["B"].max()
df["C"] = df["C"] / df["C"].max()
You can create a list of columns that you want to normalize
column_names_to_normalize = ['A', 'E', 'G', 'sadasdsd', 'lol']
x = df[column_names_to_normalize].values
x_scaled = min_max_scaler.fit_transform(x)
df_temp = pd.DataFrame(x_scaled, columns=column_names_to_normalize, index = df.index)
df[column_names_to_normalize] = df_temp
Your Pandas Dataframe is now normalized only at the columns you want
However, if you want the opposite, select a list of columns that you DON'T want to normalize, you can simply create a list of all columns and remove that non desired ones
column_names_to_not_normalize = ['B', 'J', 'K']
column_names_to_normalize = [x for x in list(df) if x not in column_names_to_not_normalize ]
I think that a better way to do that in pandas is just
df = df/df.max().astype(np.float64)
Edit If in your data frame negative numbers are present you should use instead
df = df/df.loc[df.abs().idxmax()].astype(np.float64)
The solution given by Sandman and Praveen is very well. The only problem with that if you have categorical variables in other columns of your data frame this method will need some adjustments.
My solution to this type of issue is following:
from sklearn import preprocesing
x = pd.concat([df.Numerical1, df.Numerical2,df.Numerical3])
min_max_scaler = preprocessing.MinMaxScaler()
x_scaled = min_max_scaler.fit_transform(x)
x_new = pd.DataFrame(x_scaled)
df = pd.concat([df.Categoricals,x_new])
You might want to have some of columns being normalized and the others be unchanged like some of regression tasks which data labels or categorical columns are unchanged So I suggest you this pythonic way (It's a combination of @shg and @Cina answers ):
features_to_normalize = ['A', 'B', 'C']
# could be ['A','B']
df[features_to_normalize] = df[features_to_normalize].apply(lambda x:(x-x.min()) / (x.max()-x.min()))
def normalize(x):
try:
x = x/np.linalg.norm(x,ord=1)
return x
except :
raise
data = pd.DataFrame.apply(data,normalize)
From the document of pandas,DataFrame structure can apply an operation (function) to itself .
DataFrame.apply(func, axis=0, broadcast=False, raw=False, reduce=None, args=(), **kwds)
Applies function along input axis of DataFrame. Objects passed to functions are Series objects having index either the DataFrame’s index (axis=0) or the columns (axis=1). Return type depends on whether passed function aggregates, or the reduce argument if the DataFrame is empty.
You can apply a custom function to operate the DataFrame .
The following function calculates the Z score:
def standardization(dataset):
""" Standardization of numeric fields, where all values will have mean of zero
and standard deviation of one. (z-score)
Args:
dataset: A `Pandas.Dataframe`
"""
dtypes = list(zip(dataset.dtypes.index, map(str, dataset.dtypes)))
# Normalize numeric columns.
for column, dtype in dtypes:
if dtype == 'float32':
dataset[column] -= dataset[column].mean()
dataset[column] /= dataset[column].std()
return dataset
Note that sklearn uses biased estimator for standard deviation. Consider following normalize example:
import pandas as pd
df = pd.DataFrame({
'A':[1,2,3],
'B':[100,300,500],
'C':list('abc')
})
print(df)
A B C
0 1 100 a
1 2 300 b
2 3 500 c
When normalizing we simply subtract the mean and divide by standard deviation.
df.iloc[:,0:-1] = df.iloc[:,0:-1].apply(lambda x: (x-x.mean())/ x.std(), axis=0)
print(df)
A B C
0 -1.0 -1.0 a
1 0.0 0.0 b
2 1.0 1.0 c
If you do the same thing with sklearn you will get DIFFERENT output!
import pandas as pd
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
df = pd.DataFrame({
'A':[1,2,3],
'B':[100,300,500],
'C':list('abc')
})
df.iloc[:,0:-1] = scaler.fit_transform(df.iloc[:,0:-1].to_numpy())
print(df)
A B C
0 -1.224745 -1.224745 a
1 0.000000 0.000000 b
2 1.224745 1.224745 c
The results are different. However, as per the official documentation of sklearn.preprocessing.scale using biased estimator is UNLIKELY to affect the performance of machine learning algorithms and we can safely use them.
You can do this in one line
DF_test = DF_test.sub(DF_test.mean(axis=0), axis=1)/DF_test.mean(axis=0)
it takes mean for each of the column and then subtracts it(mean) from every row(mean of particular column subtracts from its row only) and divide by mean only. Finally, we what we get is the normalized data set.
This is how you do it column-wise using list comprehension:
[df[col].update((df[col] - df[col].min()) / (df[col].max() - df[col].min())) for col in df.columns]
It is only simple mathematics. The answer should as simple as below.
normed_df = (df - df.min()) / (df.max() - df.min())
来源：https://stackoverflow.com/questions/26414913/normalize-columns-of-pandas-data-frame