如何在mitmproxy容器中捕获服务器的响应并将其以json格式粘贴到文件中？

Question

我需要捕获并填写每个响应的请求文件，该请求正在通过我的mitmproxy容器。

Dockerfile

FROM mitmproxy/mitmproxy:latest

RUN mkdir url_catching
WORKDIR /home/$USER/url_catching
COPY ./url.py .



EXPOSE 8080:8080

ENTRYPOINT ["/usr/bin/mitmdump","-s","./url.py"]

Docker运行

sudo docker run --rm -it -p 8080:8080 mitmdump_url:latest

我的pyhton脚本（很抱歉，我是python的新手）

from mitmproxy import http

def response(flow):
    url_request: str = str(flow.request.pretty_url)
    url_request = url_request.replace("/", "_")
    with open(url_request, "ab") as ofile:
        ofile.write(flow.request.pretty_url.encode())
        ofile.write(flow.request.content)
        ofile.write(flow.response.content)

• 结果是，我为每个请求获取了单独的文件，并且其名称等于请求的url，但是只有url是人类可读的格式，其他所有内容均已编码或丢失。因此，我需要过去以json格式通过代理传递到文件的响应。

Answer 1

[request/responde具有headers +empty line+ body/content

我展示了两种将headers转换为string / bytes的方法。

作为普通文本行

for key, value in flow.response.headers.items():
    ofile.write('{}: {}\n'.format(key, value).encode())

结果

Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 14 Jan 2020 11:51:49 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 181
Connection: keep-alive

作为JSON。我将其转换为dict()，因为标头无法直接转换为JSON

d = dict(flow.request.headers.items())
d = json.dumps(d, indents=2)
ofile.write(d.encode() + b'\n')

结果

{
  "Host": "httpbin.org",
  "User-Agent": "python-requests/2.22.0",
  "Accept-Encoding": "gzip, deflate",
  "Accept": "*/*",
  "Connection": "keep-alive"
}

我也跳过带有'/ static /'的网址

from mitmproxy import http
import json

def response(flow):
    url_request: str = str(flow.request.pretty_url)

    if '/static/' not in url_request:
        url_request = url_request.replace("/", "_")
        with open(url_request + '.txt', "ab") as ofile:

            ofile.write(b'--- url ---\n')
            ofile.write(flow.request.pretty_url.encode() + b'\n')

            ofile.write(b'--- request ---\n')

            ofile.write(b'--- headers ---\n')
            #for key, value in flow.request.headers.items():
            #    ofile.write('{}: {}\n'.format(key, value).encode())
            d = dict(flow.request.headers.items())
            d = json.dumps(d, indents=2)
            ofile.write(d.encode() + b'\n')


            ofile.write(b'--- content ---\n')
            ofile.write(flow.request.content + b'\n')

            ofile.write(b'--- response ---\n')

            ofile.write(b'--- headers ---\n')
            for key, value in flow.response.headers.items():
                ofile.write('{}: {}\n'.format(key, value).encode())

            ofile.write(b'--- content ---\n')
            ofile.write(flow.response.content + b'\n')

要将所有内容都作为一个JSON，您必须首先创建包含所有元素（标题，正文等）的字典，然后使用json.dumps(all_elements)

---

测试代码

import requests

proxy = {
    'http': 'http://localhost:8080',
    'https': 'http://localhost:8080',
}

urls = [
    'https://httpbin.org/get',
    'https://httpbin.org/gzip',
    'https://httpbin.org/brotli',
    'https://httpbin.org/deflate',
    'https://httpbin.org/encoding/utf8',
]

for url in urls:
    print(url)
    r = requests.get(url, proxies=proxy, verify=False)
    print(r.text)

包含结果的文件之一

--- url ---
https://httpbin.org/get
--- request ---
--- headers ---
{
  "Host": "httpbin.org",
  "User-Agent": "python-requests/2.22.0",
  "Accept-Encoding": "gzip, deflate",
  "Accept": "*/*",
  "Connection": "keep-alive"
}
--- content ---

--- response ---
--- headers ---
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 14 Jan 2020 12:06:04 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 181
Connection: keep-alive
--- content ---
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.22.0"
  }, 
  "origin": "83.23.66.224, 83.23.66.224", 
  "url": "https://httpbin.org/get"
}