Minikube apiserver 频繁停止

Question

我在 Microsoft Windows 10 Pro 上使用 minikube v1.11.0。我的minikube经常停止，查找minikube状态

Minikube 状态：

类型：控制平面
主持人：跑步
kubelet：运行
api 服务器：已停止
kubeconfig：已配置

minikube 日志：

Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

Minikube 详细信息：

Minikube 版本：1.11.0
分配内存：4096
处理器：4个虚拟处理器

“minikube ssh dmesg”命令结果：

D:\IMP\DevOps Implementation\Python>minikube ssh dmesg
[    0.000000] Linux version 4.19.107 (jenkins@jenkins) (gcc version 7.4.0 (Buildroot 2019.02.10)) #1 SMP Thu May 28 15:07:17 PDT 2020
[    0.000000] Command line: BOOT_IMAGE=/boot/bzImage root=/dev/sr0 loglevel=3 console=ttyS0 noembed nomodeset norestore waitusb=10 random.trust_cpu=on hw_rng_model=virtio systemd.legacy_systemd_cgroup_controller=yes initrd=/boot/initrd
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[3]:  832, xstate_sizes[3]:   64
[    0.000000] x86/fpu: xstate_offset[4]:  896, xstate_sizes[4]:   64
[    0.000000] x86/fpu: Enabled xstate features 0x1f, context size is 960 bytes, using 'compacted' format.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000f7feffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000f7ff0000-0x00000000f7ffefff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000f7fff000-0x00000000f7ffffff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000101ffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.3 present.
[    0.000000] DMI: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
[    0.000000] Hypervisor detected: Microsoft Hyper-V
[    0.000000] Hyper-V: features 0x2e7f, hints 0x20c2c
[    0.000000] Hyper-V Host Build:18362-10.0-0-0.836
[    0.000000] Hyper-V: LAPIC Timer Frequency: 0x30d40
[    0.000000] tsc: Marking TSC unstable due to running on Hyper-V
[    0.000000] Hyper-V: Using hypercall for remote TLB flush
[    0.000000] tsc: Detected 1800.006 MHz processor
[    0.000686] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000687] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000690] last_pfn = 0x102000 max_arch_pfn = 0x400000000
[    0.000705] MTRR default type: uncachable
[    0.000706] MTRR fixed ranges enabled:
[    0.000707]   00000-9FFFF write-back
[    0.000707]   A0000-DFFFF uncachable
[    0.000707]   E0000-FFFFF write-back
[    0.000708] MTRR variable ranges enabled:
[    0.000709]   0 base 0000000000 mask 7F00000000 write-back
[    0.000709]   1 base 0100000000 mask 7000000000 write-back
[    0.000709]   2 disabled
[    0.000710]   3 disabled
[    0.000710]   4 disabled
[    0.000710]   5 disabled
[    0.000710]   6 disabled
[    0.000711]   7 disabled
[    0.000718] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT
[    0.000727] last_pfn = 0xf7ff0 max_arch_pfn = 0x400000000
[    0.006501] found SMP MP-table at [mem 0x000ff780-0x000ff78f]
[    0.006515] Scanning 1 areas for low memory corruption
[    0.006583] Using GB pages for direct mapping
[    0.006586] BRK [0x8da02000, 0x8da02fff] PGTABLE
[    0.006587] BRK [0x8da03000, 0x8da03fff] PGTABLE
[    0.006588] BRK [0x8da04000, 0x8da04fff] PGTABLE
[    0.006599] BRK [0x8da05000, 0x8da05fff] PGTABLE
[    0.006600] BRK [0x8da06000, 0x8da06fff] PGTABLE
[    0.006625] BRK [0x8da07000, 0x8da07fff] PGTABLE
[    0.006632] BRK [0x8da08000, 0x8da08fff] PGTABLE
[    0.006669] RAMDISK: [mem 0x75db4000-0x7fffffff]
[    0.006709] ACPI: Early table checksum verification disabled

[    3.211586] Freeing unused kernel image memory: 1428K
[    3.217804] Write protecting the kernel read-only data: 20480k
[    3.218711] Freeing unused kernel image memory: 2004K
[    3.218811] Freeing unused kernel image memory: 648K
[    3.218813] Run /init as init process
[    3.420796] psmouse serio1: trackpoint: failed to get extended button data, assuming 3 buttons
[    3.606951] tar (1220) used greatest stack depth: 14064 bytes left
[    3.658466] systemd[1]: systemd 240 running in system mode. (-PAM -AUDIT -SELINUX -IMA -APPARMOR -SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=hybrid)
[    3.658536] systemd[1]: Detected virtualization microsoft.
[    3.658540] systemd[1]: Detected architecture x86-64.
[    3.667515] systemd[1]: Set hostname to <minikube>.
[    3.667543] systemd[1]: Initializing machine ID from random generator.
[    3.667835] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument
[    3.678215] systemd-fstab-generator[1225]: Ignoring "noauto" for root device
[    3.679587] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
[    3.679589] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[    3.684942] systemd[1]: /usr/lib/systemd/system/vmtoolsd.service:7: PIDFile= references path below legacy directory /var/run/, updating /var/run/vmtoolsd.pid \xe2\x86\x92 /run/vmtoolsd.pid; please update the unit file accordingly.
[    3.688167] systemd[1]: /usr/lib/systemd/system/rpc-statd.service:13: PIDFile= references path below legacy directory /var/run/, updating /var/run/rpc.statd.pid \xe2\x86\x92 /run/rpc.statd.pid; please update the unit file accordingly.
[    4.077349] systemd-journald[1482]: Received request to flush runtime journal from PID 1
[    4.082397] journalctl (1875) used greatest stack depth: 14032 bytes left
[    4.328094] hv_vmbus: Vmbus version:5.0
[    4.346743] hv_vmbus: registering driver hid_hyperv
[    4.347880] input: Microsoft Vmbus HID-compliant Mouse as /devices/0006:045E:0621.0001/input/input4
[    4.347953] hid-generic 0006:045E:0621.0001: input: <UNKNOWN> HID v0.01 Mouse [Microsoft Vmbus HID-compliant Mouse] on
[    4.347953] hid-generic 0006:045E:0621.0001: input: <UNKNOWN> HID v0.01 Mouse [Microsoft Vmbus HID-compliant Mouse] on
[    4.348317] hv_vmbus: registering driver hv_storvsc
[    4.349279] hv_vmbus: registering driver hyperv_keyboard
[    4.350678] hv_utils: Registering HyperV Utility Driver
[   52.468905] hv_balloon: Max. dynamic memory size: 4000 MB
[   96.728628] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   96.729009] Bridge firewalling registered
[   96.734466] audit: type=1325 audit(1594295640.425:2): table=nat family=2 entries=0
[   96.734568] audit: type=1300 audit(1594295640.425:2): arch=c000003e syscall=313 success=yes exit=0 a0=5 a1=41a8e6 a2=0 a3=5 items=0 ppid=1078 pid=2441 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=kernel key=(null)
[   96.734671] audit: type=1327 audit(1594295640.425:2): proctitle=2F7362696E2F6D6F6470726F6265002D71002D2D0069707461626C655F6E6174
[   96.753305] audit: type=1325 audit(1594295640.444:3): table=nat family=2 entries=5
[   96.753308] audit: type=1300 audit(1594295640.444:3): arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=1a1ea60 items=0 ppid=2350 pid=2470 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-legacy-multi" subj=kernel key=(null)
[   96.753309] audit: type=1327 audit(1594295640.444:3): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4E00444F434B4552
[  355.252138] audit: type=1325 audit(1594295898.980:72): table=filter family=2 entries=30
[  355.252142] audit: type=1300 audit(1594295898.980:72): arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=207d050 items=0 ppid=2350 pid=6087 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-legacy-multi" subj=kernel key=(null)
[  355.252144] audit: type=1327 audit(1594295898.980:72): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D740066696C746572002D4100444F434B45520000002D6900646F636B657230002D6F00646F636B657230002D7000746370002D64003137322E31372E302E34002D2D64706F727400343433002D6A00414343455054
[  569.369364] docker0: port 13(veth97c2039) entered disabled state
[  569.369455] audit: type=1700 audit(1594296113.095:87): dev=veth97c2039 prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
[  569.377656] audit: type=1300 audit(1594296113.095:87): arch=c000003e syscall=44 success=yes exit=32 a0=e a1=c002858760 a2=20 a3=0 items=0 ppid=1 pid=2350 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="dockerd" exe="/usr/bin/dockerd" subj=kernel key=(null)
[  569.377659] audit: type=1327 audit(1594296113.095:87): proctitle=2F7573722F62696E2F646F636B657264002D48007463703A2F2F302E302E302E303A32333736002D4800756E69783A2F2F2F7661722F72756E2F646F636B65722E736F636B002D2D64656661756C742D756C696D69743D6E6F66696C653D313034383537363A31303438353736002D2D746C73766572696679002D2D746C7363
[  575.768707] audit: type=1325 audit(1594296119.496:88): table=nat family=2 entries=27
[  575.768710] audit: type=1300 audit(1594296119.496:88): arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=10a24b0 items=0 ppid=2350 pid=8742 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-legacy-multi" subj=kernel key=(null)
[  575.768712] audit: type=1327 audit(1594296119.496:88): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4400444F434B4552002D7000746370002D6400302F30002D2D64706F727400343433002D6A00444E4154002D2D746F2D64657374696E6174696F6E003137322E31372E302E343A3434330000002D6900646F636B657230
[  575.770459] audit: type=1325 audit(1594296119.498:89): table=filter family=2 entries=32
[  575.770461] audit: type=1300 audit(1594296119.498:89): arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=17ad4d0 items=0 ppid=2350 pid=8744 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-legacy-multi" subj=kernel key=(null)
[  575.770463] audit: type=1327 audit(1594296119.498:89): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D740066696C746572002D4400444F434B45520000002D6900646F636B657230002D6F00646F636B657230002D7000746370002D64003137322E31372E302E34002D2D64706F727400343433002D6A00414343455054
[  575.772344] audit: type=1325 audit(1594296119.500:90): table=nat family=2 entries=26
[  575.772356] audit: type=1300 audit(1594296119.500:90): arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=814150 items=0 ppid=2350 pid=8746 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-legacy-multi" subj=kernel key=(null)
[  575.772358] audit: type=1327 audit(1594296119.500:90): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4400504F5354524F5554494E47002D7000746370002D73003137322E31372E302E34002D64003137322E31372E302E34002D2D64706F727400343433002D6A004D415351554552414445
[  575.775023] audit: type=1325 audit(1594296119.503:91): table=nat family=2 entries=25
[  576.333512] veth3126200: renamed from eth0
[  576.342533] docker0: port 1(vethb62d4b0) entered disabled state
[  576.518935] docker0: port 2(vetha808974) entered disabled state
[  576.521396] vethe98d518: renamed from eth0
[  576.708284] docker0: port 3(vethc4912f9) entered disabled state
[  576.709295] vethbd779ad: renamed from eth0
[  579.921427] docker0: port 1(vethb62d4b0) entered disabled state
[  579.926503] device vethb62d4b0 left promiscuous mode
[  579.926508] docker0: port 1(vethb62d4b0) entered disabled state
[  580.745101] docker0: port 2(vetha808974) entered disabled state
[  580.748501] device vetha808974 left promiscuous mode
[  580.748514] docker0: port 2(vetha808974) entered disabled state
[  582.042311] docker0: port 3(vethc4912f9) entered disabled state
[  582.047229] device vethc4912f9 left promiscuous mode
[  582.047236] docker0: port 3(vethc4912f9) entered disabled state
[  582.047260] kauditd_printk_skb: 14 callbacks suppressed
[  582.047261] audit: type=1700 audit(1594296125.769:96): dev=vethc4912f9 prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
[  582.050812] audit: type=1300 audit(1594296125.769:96): arch=c000003e syscall=44 success=yes exit=32 a0=e a1=c001dcf7e0 a2=20 a3=0 items=0 ppid=1 pid=2350 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="dockerd" exe="/usr/bin/dockerd" subj=kernel key=(null)
[  582.050918] audit: type=1327 audit(1594296125.769:96): proctitle=2F7573722F62696E2F646F636B657264002D48007463703A2F2F302E302E302E303A32333736002D4800756E69783A2F2F2F7661722F72756E2F646F636B65722E736F636B002D2D64656661756C742D756C696D69743D6E6F66696C653D313034383537363A31303438353736002D2D746C73766572696679002D2D746C7363

Answer 1

根据您提供的信息，此问题最常见的原因可能是缺乏资源，这会导致 apiserver 被驱逐。有几件事需要注意：

1. 增加分配的内存和 CPU 量，以确保足以满足您的使用情况。

2. 检查您是否有足够的存储空间

   /dev/sda1

   。达到阈值可能会导致某些进程被驱逐，包括 apiserver。

3. 如果这还不够，请尝试通过执行以下命令来获取更多日志：

   minikube ssh 'docker logs $(docker ps -a -f name=k8s_kube-api --format={{.ID}})'

4. 最终您可以使用

   minikube stop

   和

   minikube start

   重新启动 minikube（如果您想从头开始，可以在中间添加

   minikube delete

   ）。如果问题再次出现，请尝试使用

   journalctl

   找出其原因。

请告诉我是否有帮助。

Answer 2

我还面临 APISERVER: STOPPED 主要是在系统重新启动后。因此，我使用 minikube stop 停止了 minikube，并在 Windows 笔记本电脑上使用 minikube start --vm-driver=hyperv 再次启动。这帮助我让我的 APISERVER 进入运行状态。

Answer 3

我遇到了类似的问题，通过 ssh 登录 minikube 后解决了。

步骤：

minikube ssh
$ sudo su
$ systemctl daemon-reload
$ systemctl start docker
$ exit
$ exit

输出：

Answer 4

以管理员模式打开Powershell：

第1步：通过“minikube stop”命令停止minikube

第2步：通过“minikube start”启动minikube

第3步：您将观察到'apiserver'处于运行状态。