我正在尝试将formdata提供给scrapy.FormRequest对象。 formdata是以下结构的字典:
{
"param1": [
{
"paramA": "valueA",
"paramB": "valueB"
}
]
}
通过等效于以下代码,在scrapy shell中运行:
from scrapy import FormRequest
url = 'www.example.com'
method_post = 'POST'
formdata = <the above dict>
fr = FormRequest(url=url, method=method_post, formdata=formdata)
fetch(fr)
并作为回应我收到以下错误:
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/Users/chhk/.local/share/virtualenvs/project/lib/python3.6/site-packages/scrapy/http/request/form.py", line 31, in __init__
querystr = _urlencode(items, self.encoding)
File "/Users/chhk/.local/share/virtualenvs/project/lib/python3.6/site-packages/scrapy/http/request/form.py", line 66, in _urlencode
for k, vs in seq
File "/Users/chhk/.local/share/virtualenvs/project/lib/python3.6/site-packages/scrapy/http/request/form.py", line 67, in <listcomp>
for v in (vs if is_listlike(vs) else [vs])]
File "/Users/chhk/.local/share/virtualenvs/project/lib/python3.6/site-packages/scrapy/utils/python.py", line 119, in to_bytes
'object, got %s' % type(text).__name__)
TypeError: to_bytes must receive a unicode, str or bytes object, got dict
我尝试了各种各样的解决方案,包括整个字符串,各种转义字符以及字典的变化以使其更加令人愉快,但删除此错误的解决方案都不适用于请求(我得到400响应)。
我知道formdata和我正在做的其他事情是正确的,因为我已经成功地在curl中复制了它(formdata是通过-d formdata.txt
提供的)。
有没有办法解决FormRequest无法处理复杂的字典结构?或者我错过了什么?
而不是formdata
你可以尝试使用body
参数。例:
FormRequest(url=url, method=method_post, body=json.dumps(formdata))