vaex对象dtype dtype('O')没有原生的HDF5等价物。

Question

我使用vaex.from_csv()将csv转换成hdf5。

import vaex
vaex.from_csv("/Users/xxxx/development/vaex/dataAN/testdata1.csv", convert=True)

获取

IPython/core/interactiveshell.py:3331: DtypeWarning: Columns (53,55) have mixed types.Specify dtype option on import or set low_memory=False.
  exec(code_obj, self.user_global_ns, self.user_ns)
ERROR:MainThread:root:error creating dataset for 'EXT_SUB_ACC', with type dtype('O') 
Traceback (most recent call last):
  File "/Users/xxxx/development/vaex/install/vaex/packages/vaex-core/vaex/hdf5/export.py", line 201, in export_hdf5
    array = h5column_output.require_dataset('data', shape=shape, dtype=dtype.newbyteorder(byteorder))
  File "/Users/xxxx/.conda/envs/vaex/lib/python3.7/site-packages/h5py/_hl/group.py", line 191, in require_dataset
    return self.create_dataset(name, *(shape, dtype), **kwds)
  File "/Users/xxxx/.conda/envs/vaex/lib/python3.7/site-packages/h5py/_hl/group.py", line 136, in create_dataset
    dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
  File "/Users/xxxx/.conda/envs/vaex/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 119, in make_new_dset
    tid = h5t.py_create(dtype, logical=1)
  File "h5py/h5t.pyx", line 1634, in h5py.h5t.py_create
  File "h5py/h5t.pyx", line 1656, in h5py.h5t.py_create
  File "h5py/h5t.pyx", line 1711, in h5py.h5t.py_create
TypeError: Object dtype dtype('O') has no native HDF5 equivalent

这些错误是什么意思？

Answer 1

这里给出一个提示。

IPython/core/interactiveshell.py:3331: DtypeWarning: Columns (53,55) have mixed types.Specify dtype option on import or set low_memory=False.

Pandas找不到通用类型，因此使用dtype=object，而dtype不能保存到hdf5中（没有原生机器表示）。如果你强制使用dtype，把这个参数传给from_csv，vaex会把它传给pandas。