Python:如何删除一个特定列为空/NaN的行?
Python:如何删除一个特定列为空/NaN的行?
我有一个CSV文件。我读取了它:
import pandas as pd data = pd.read_csv('my_data.csv', sep=',') data.head()
它的输出结果如下:
id city department sms category 01 khi revenue NaN 0 02 lhr revenue good 1 03 lhr revenue NaN 0
我想要删除所有空值/NaN的sms
列的行。有什么高效的方法可以做到这一点吗?
admin 更改状态以发布 2023年5月25日
你可以使用方法dropna
来实现:
data.dropna(axis=0, subset=('sms', ))
详细了解参数,请查阅文档。
当然,有很多种实现方式,而且性能也有一些微小的差异。除非极致性能很重要,否则我最喜欢用dropna()
,因为它最表达清晰。
import pandas as pd import numpy as np i = 10000000 # generate dataframe with a few columns df = pd.DataFrame(dict( a_number=np.random.randint(0,1e6,size=i), with_nans=np.random.choice([np.nan, 'good', 'bad', 'ok'], size=i), letter=np.random.choice(list('abcdefghijklmnop'), size=i)) ) # using notebook %%timeit a = df.dropna(subset=['with_nans']) #1.29 s ± 112 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) # using notebook %%timeit b = df[~df.with_nans.isnull()] #890 ms ± 59.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) # using notebook %%timeit c = df.query('with_nans == with_nans') #1.71 s ± 100 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
使用具有参数子集的dropna
指定要检查NaN的列:
data = data.dropna(subset=['sms']) print (data) id city department sms category 1 2 lhr revenue good 1
使用boolean indexing
和notnull
的另一种解决方案:
data = data[data['sms'].notnull()] print (data) id city department sms category 1 2 lhr revenue good 1
使用query
的替代方案:
print (data.query("sms == sms")) id city department sms category 1 2 lhr revenue good 1
定时
#[300000 rows x 5 columns] data = pd.concat([data]*100000).reset_index(drop=True) In [123]: %timeit (data.dropna(subset=['sms'])) 100 loops, best of 3: 19.5 ms per loop In [124]: %timeit (data[data['sms'].notnull()]) 100 loops, best of 3: 13.8 ms per loop In [125]: %timeit (data.query("sms == sms")) 10 loops, best of 3: 23.6 ms per loop