在将大型Pandas数据框保存为hdf时出现OverflowError。

我有一个非常大的Pandas数据框（约15GB，8300万行），我希望将其保存为一个h5（或feather）文件。其中一列包含长的ID数字字符串，应该是字符串/对象类型。但是，即使我确保pandas将所有列解析为对象类型：

df = pd.read_csv('data.csv', dtype=object)

print(df.dtypes) # 检查一下

df.to_hdf('df.h5', 'df')

> client_id object

event_id object

account_id object

session_id object

event_timestamp object

# 等等...

我得到以下错误：

File "foo.py", line 14, in

df.to_hdf('df.h5', 'df')

File "/shared_directory/projects/env/lib/python3.6/site-packages/pandas/core/generic.py", line 1996, in to_hdf

return pytables.to_hdf(path_or_buf, key, self, **kwargs)

File "/shared_directory/projects/env/lib/python3.6/site-packages/pandas/io/pytables.py", line 279, in to_hdf

f(store)

File "/shared_directory/projects/env/lib/python3.6/site-packages/pandas/io/pytables.py", line 273, in

f = lambda store: store.put(key, value, **kwargs)

File "/shared_directory/projects/env/lib/python3.6/site-packages/pandas/io/pytables.py", line 890, in put

self._write_to_group(key, value, append=append, **kwargs)

File "/shared_directory/projects/env/lib/python3.6/site-packages/pandas/io/pytables.py", line 1367, in _write_to_group

s.write(obj=value, append=append, complib=complib, **kwargs)

File "/shared_directory/projects/env/lib/python3.6/site-packages/pandas/io/pytables.py", line 2963, in write

self.write_array('block%d_values' % i, blk.values, items=blk_items)

File "/shared_directory/projects/env/lib/python3.6/site-packages/pandas/io/pytables.py", line 2730, in write_array

vlarr.append(value)

File "/shared_directory/projects/env/lib/python3.6/site-packages/tables/vlarray.py", line 547, in append

self._append(nparr, nobjects)

File "tables/hdf5extension.pyx", line 2032, in tables.hdf5extension.VLArray._append

OverflowError: value too large to convert to int

显然它仍然试图将其转换为整数，并失败了。

当运行df.to_feather()时，我遇到了类似的问题：

df.to_feather('df.feather')

File "/shared_directory/projects/env/lib/python3.6/site-packages/pandas/core/frame.py", line 1892, in to_feather

to_feather(self, fname)

File "/shared_directory/projects/env/lib/python3.6/site-packages/pandas/io/feather_format.py", line 83, in to_feather

feather.write_dataframe(df, path)

File "/shared_directory/projects/env/lib/python3.6/site-packages/pyarrow/feather.py", line 182, in write_feather

writer.write(df)

File "/shared_directory/projects/env/lib/python3.6/site-packages/pyarrow/feather.py", line 93, in write

table = Table.from_pandas(df, preserve_index=False)

File "pyarrow/table.pxi", line 1174, in pyarrow.lib.Table.from_pandas

File "/shared_directory/projects/env/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 501, in dataframe_to_arrays

convert_fields))

File "/usr/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator

yield fs.pop().result()

File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result

return self.__get_result()

File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result

raise self._exception

File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run

result = self.fn(*self.args, **self.kwargs)

File "/shared_directory/projects/env/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 487, in convert_column

raise e

File "/shared_directory/projects/env/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 481, in convert_column

result = pa.array(col, type=type_, from_pandas=True, safe=safe)

File "pyarrow/array.pxi", line 191, in pyarrow.lib.array

File "pyarrow/array.pxi", line 78, in pyarrow.lib._ndarray_to_array

File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status

pyarrow.lib.ArrowInvalid: ('Could not convert 1542852887489 with type str: tried to convert to double', 'Conversion failed for column session_id with type object')

所以：

1. 是否任何看起来像数字的内容都会被强制转换为数字存储？
2. NaN的存在是否会影响这里发生的情况？
3. 是否有其他的存储解决方案？什么是最好的选择？