使用
PyMongo尝试
db.myCollection.insert(df.to_dict())
出现错误
InvalidDocument: documents must have only string keys, the key was Timestamp('2013-11-23 13:31:00', tz=None)
db.myCollection.insert(df.to_json())
出现错误
TypeError: 'str' object does not support item assignment
db.myCollection.insert({id: df.to_json()})
出现错误
InvalidDocument: documents must have only string a keys, key was <built-in function id>df
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 150 entries, 2013-11-23 13:31:26 to 2013-11-23 13:24:07
Data columns (total 3 columns):
amount 150 non-null values
price 150 non-null values
tid 150 non-null values
dtypes: float64(2), int64(1)
我怀疑是否存在既最快又简单的方法。如果您不担心数据转换，您可以这样做
>>> import json
>>> df = pd.DataFrame.from_dict({'A': {1: datetime.datetime.now()}})
>>> df
A
1 2013-11-23 21:14:34.118531
>>> records = json.loads(df.T.to_json()).values()
>>> db.myCollection.insert(records)
但是如果您尝试加载数据回来，您会得到：
>>> df = read_mongo(db, 'myCollection')
>>> df
A
0 1385241274118531000
>>> df.dtypes
A int64
dtype: object
因此您必须将“A”列转换回
datetimeintfloatstrDataFrame>>> df['A'] = pd.to_datetime(df['A'])
>>> df
A
0 2013-11-23 21:14:34.118531
odo可以使用
做到这一点odo(df, db.myCollection)
我认为这个问题有很酷的想法。就我而言，我花了更多时间来处理大型数据帧的移动。在这种情况下，pandas 往往允许您选择 chunksize（例如 pandas.DataFrame.to_sql 中的示例）。所以我想我可以通过添加我在这个方向上使用的功能来做出贡献。
def write_df_to_mongoDB( my_df,\
database_name = 'mydatabasename' ,\
collection_name = 'mycollectionname',
server = 'localhost',\
mongodb_port = 27017,\
chunk_size = 100):
#"""
#This function take a list and create a collection in MongoDB (you should
#provide the database name, collection, port to connect to the remoete database,
#server of the remote database, local port to tunnel to the other machine)
#
#---------------------------------------------------------------------------
#Parameters / Input
# my_list: the list to send to MongoDB
# database_name: database name
#
# collection_name: collection name (to create)
# server: the server of where the MongoDB database is hosted
# Example: server = 'XXX.XXX.XX.XX'
# this_machine_port: local machine port.
# For example: this_machine_port = '27017'
# remote_port: the port where the database is operating
# For example: remote_port = '27017'
# chunk_size: The number of items of the list that will be send at the
# some time to the database. Default is 100.
#
#Output
# When finished will print "Done"
#----------------------------------------------------------------------------
#FUTURE modifications.
#1. Write to SQL
#2. Write to csv
#----------------------------------------------------------------------------
#30/11/2017: Rafael Valero-Fernandez. Documentation
#"""
#To connect
# import os
# import pandas as pd
# import pymongo
# from pymongo import MongoClient
client = MongoClient('localhost',int(mongodb_port))
db = client[database_name]
collection = db[collection_name]
# To write
collection.delete_many({}) # Destroy the collection
#aux_df=aux_df.drop_duplicates(subset=None, keep='last') # To avoid repetitions
my_list = my_df.to_dict('records')
l = len(my_list)
ran = range(l)
steps=ran[chunk_size::chunk_size]
steps.extend([l])
# Inser chunks of the dataframe
i = 0
for j in steps:
print j
collection.insert_many(my_list[i:j]) # fill de collection
i = j
print('Done')
return
如果您的数据框缺少数据（即无，nan）并且您不希望文档中出现空键值：
db.insert_many(df.to_dict("records")).to_dict("records")from pandas.core.common import _maybe_box_datetimelike
my_list = [dict((k, _maybe_box_datetimelike(v)) for k, v in zip(df.columns, row) if v != None and v == v) for row in df.values]
db.insert_many(my_list)
其中
if v != None and v == vNonenan.insert_manynull我使用以下部分将数据帧插入到数据库中的集合中。
df.reset_index(inplace=True)
data_dict = df.to_dict("records")
myCollection.insert_many(data_dict)
这个怎么样：
db.myCollection.insert({id: df.to_json()})
id 将是该 df 的唯一字符串
只需制作字符串键即可！
import json
dfData = json.dumps(df.to_dict('records'))
savaData = {'_id': 'a8e42ed79f9dae1cefe8781760231ec0', 'df': dfData}
res = client.insert_one(savaData)
##### load dfData
data = client.find_one({'_id': 'a8e42ed79f9dae1cefe8781760231ec0'}).get('df')
dfData = json.loads(data)
df = pd.DataFrame.from_dict(dfData)
如果您想一次发送多份：
db.myCollection.insert_many(df.apply(lambda x: x.to_dict(), axis=1).to_list())
如果您想确保不会引发 InvalidDocument 错误，那么类似以下内容是一个好主意。这是因为 mongo 无法识别 np.int64、np.float64 等类型
from pymongo import MongoClient
client = MongoClient()
db = client.test
col = db.col
def createDocsFromDF(df, collection = None, insertToDB=False):
docs = []
fields = [col for col in df.columns]
for i in range(len(df)):
doc = {col:df[col][i] for col in df.columns if col != 'index'}
for key, val in doc.items():
# we have to do this, because mongo does not recognize these np. types
if type(val) == np.int64:
doc[key] = int(val)
if type(val) == np.float64:
doc[key] = float(val)
if type(val) == np.bool_:
doc[key] = bool(val)
docs.append(doc)
if insertToDB and collection:
db.collection.insert_many(docs)
return docs
对于更新插入，这有效。
for r in df2.to_dict(orient="records"):
db['utest-pd'].update_one({'a':r['a']},{'$set':r})
一次处理一条记录，但似乎
upsert_many使用 PyMongoArrow，这是 MongoDB 专门为此目的构建的 Python 库。它构建在 PyMongo 之上。
PyMongoArrow 允许您轻松快速地将数据从 MongoDB 移动到许多其他数据格式，例如 Pandas DataFrame、NumPy Array 或 Apache Arrow Table。您还可以使用 PyMongoArrow 直接将这些其他数据格式的数据写回 MongoDB。
这一切只需几行代码即可实现。查看文档：https://mongo-arrow.readthedocs.io/en/latest/quickstart.html