想要在Jupyter Notebook中运行Spark(scala)内核。正在获取OSError:[WinError 193]%1不是有效的Win32应用程序

问题描述 投票:5回答:1
Traceback (most recent call last):
  File "c:\users\rdx\anaconda3\lib\runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\rdx\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\RDX\Anaconda3\Scripts\ipython.exe\__main__.py", line 9, in <module>
  File "c:\users\rdx\anaconda3\lib\site-packages\IPython\__init__.py", line 119, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "c:\users\rdx\anaconda3\lib\site-packages\traitlets\config\application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-112>", line 2, in initialize
  File "c:\users\rdx\anaconda3\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "c:\users\rdx\anaconda3\lib\site-packages\IPython\terminal\ipapp.py", line 296, in initialize
    super(TerminalIPythonApp, self).initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "c:\users\rdx\anaconda3\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "c:\users\rdx\anaconda3\lib\site-packages\IPython\core\application.py", line 398, in initialize
    self.parse_command_line(argv)
  File "c:\users\rdx\anaconda3\lib\site-packages\IPython\terminal\ipapp.py", line 291, in parse_command_line
    return super(TerminalIPythonApp, self).parse_command_line(argv)
  File "<decorator-gen-4>", line 2, in parse_command_line
  File "c:\users\rdx\anaconda3\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "c:\users\rdx\anaconda3\lib\site-packages\traitlets\config\application.py", line 514, in parse_command_line
    return self.initialize_subcommand(subc, subargv)
  File "c:\users\rdx\anaconda3\lib\site-packages\IPython\core\application.py", line 208, in initialize_subcommand
    return super(BaseIPythonApplication, self).initialize_subcommand(subc, argv)
  File "<decorator-gen-3>", line 2, in initialize_subcommand
  File "c:\users\rdx\anaconda3\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "c:\users\rdx\anaconda3\lib\site-packages\traitlets\config\application.py", line 452, in initialize_subcommand
    self.subapp.initialize(argv)
  File "<decorator-gen-116>", line 2, in initialize
  File "c:\users\rdx\anaconda3\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "c:\users\rdx\anaconda3\lib\site-packages\jupyter_console\app.py", line 138, in initialize
    self.init_shell()
  File "c:\users\rdx\anaconda3\lib\site-packages\jupyter_console\app.py", line 105, in init_shell
    JupyterConsoleApp.initialize(self)
  File "c:\users\rdx\anaconda3\lib\site-packages\jupyter_client\consoleapp.py", line 334, in initialize
    self.init_kernel_manager()
  File "c:\users\rdx\anaconda3\lib\site-packages\jupyter_client\consoleapp.py", line 288, in init_kernel_manager
    self.kernel_manager.start_kernel(**kwargs)
  File "c:\users\rdx\anaconda3\lib\site-packages\jupyter_client\manager.py", line 243, in start_kernel
    **kw)
  File "c:\users\rdx\anaconda3\lib\site-packages\jupyter_client\manager.py", line 189, in _launch_kernel
    return launch_kernel(kernel_cmd, **kw)
  File "c:\users\rdx\anaconda3\lib\site-packages\jupyter_client\launcher.py", line 123, in launch_kernel
    proc = Popen(cmd, **kwargs)
  File "c:\users\rdx\anaconda3\lib\subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "c:\users\rdx\anaconda3\lib\subprocess.py", line 1224, in _execute_child
    startupinfo)
OSError: [WinError 193] %1 is not a valid Win32 application

我遵循了此链接中给出的过程:https://github.com/ibm-et/spark-kernel/wiki/Guide-to-Integrating-the-Spark-Kernel-with-Jupyter

我想在Windows 10上运行它。

我安装了Anaconda和Spark。并且还为这两个环境路径设置了路径。这是我的kernel.json文件:

{
    "display_name": "Spark 1.5.1 (Scala 2.10.4)",
    "language_info": { "name": "scala" },
    "argv": [
        "C:/Users/RDX/spark-kernel-master/dist/spark-kernel/bin/spark-kernel",
        "--profile",
        "{connection_file}"
    ],
    "codemirror_mode": "scala",
    "env": {
        "SPARK_OPTS": "--master=local[2] --driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info",
        "MAX_INTERPRETER_THREADS": "16",
        "CAPTURE_STANDARD_OUT": "true",
        "CAPTURE_STANDARD_ERR": "true",
        "SEND_EMPTY_OUTPUT": "false",
        "SPARK_HOME": "X:\\Softwares\\BIG_Data_files\\spark-2.0.1-bin-hadoop2.7",
        "PYTHONPATH": "X:\\Softwares\\BIG_Data_files\\spark-2.0.1-bin-hadoop2.7/python:X:\\Softwares\\BIG_Data_files\\spark-2.0.1-bin-hadoop2.7/python/lib/py4j-0.10.3-src.zip"
     }
}

在内核列表中可用。但是从笔记本中选择Spark内核时显示错误

ipython kernelspec list 

可用的内核:

pyspark C:\ Users \ RDX.ipython \ kernels \ pysparkpython3 c:\ users \ rdx \ anaconda3 \ lib \ site-packages \ ipykernel \ resourcesspark C:\ ProgramData \ jupyter \ kernels \ spark

apache-spark kernel jupyter-notebook jupyter spark-notebook
1个回答
0
投票

对不起,您解决了这个问题吗?我也有这个问题

© www.soinside.com 2019 - 2024. All rights reserved.