PySpark SparkSession fails to initialize in marimo - NoneType error on getOrCreate()

[tvanderv92]

Hey folks,

I'm trying to get PySpark working in Marimo inside a Docker container (replaced Jupyter with Marimo in https://github.com/tvanderv92/docker-spark-iceberg).

Running into a frustrating issue where SparkSession.builder.getOrCreate() throws:

AttributeError: 'NoneType' object has no attribute 'startswith'
# at /opt/spark/python/pyspark/sql/session.py:472

Works

subprocess.run(['python', '-c', 'from pyspark.sql import SparkSession; spark = SparkSession.builder.master("local[*]").getOrCreate()'])

Fails

from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[*]").getOrCreate()

Tried setting SPARK_HOME, various master URLs, removing SPARK_REMOTE - no luck. Has anyone gotten PySpark working in Marimo (local spark inside a container)?

[Light2Dark]

Does this work in non-container envs? That codepath, does it point to anything?

[tvanderv92]

@Light2Dark, resolved it by including spark connect with an environment variable and exposing it's port, and a minor change in the entrypoint shell script. seems to be working now.