Filter out rows based on list of strings in Pandas

Question

I have a large time series data frame (called df), and the first 5 records look like this:

df

         stn     years_of_data  total_minutes         


        

Answer 1

Had a similar question, found this old thread, I think there are other ways to get the same result. My issue with @EdChum's solution for my particular application is that I don't have a list that will be matched exactly. If you have the same issue, .isin isn't meant for that application.

Instead, you can also try a few options, including a numpy.where:

  removelist = ['ayside','rrowview']
  df['flagCol'] = numpy.where(df.stn.str.contains('|'.join(remove_list)),1,0)

Note that this solution doesn't actually remove the matching rows, just flags them. You can copy/slice/drop as you like.

This solution would be useful in the case that you don't know, for example, if the station names are capitalized or not and don't want to go through standardizing text beforehand. numpy.where is usually pretty fast as well, probably not much different from .isin.