Spark Command: /opt/java/openjdk/bin/java -cp /opt/spark/conf/:/opt/spark/jars/slf4j-api-2.0.17.jar:/opt/spark/jars/* -Dspark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog -Dspark.connect.grpc.binding.address=0.0.0.0 -Dspark.hadoop.fs.s3a.path.style.access=true -Dspark.connect.grpc.binding.port=15002 -Dspark.hadoop.fs.s3a.secret.key=*********(redacted) -Dspark.hadoop.fs.s3a.access.key=*********(redacted) -Dspark.kubernetes.container.image=data-platform-spark:4.1.1 -Dspark.sql.catalog.iceberg.type=hadoop -Dspark.sql.catalog.iceberg.warehouse=s3a://your-bucket/data -Dspark.kubernetes.namespace=spark -Dspark.plugins=org.apache.spark.plugins.connect.SparkConnectPlugin -Dspark.hadoop.fs.s3a.endpoint=http://s3-api.norskdata.duckdns.org -Dspark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem -Dspark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions -Xmx1g -XX:+IgnoreUnrecognizedVMOptions --add-modules=jdk.incubator.vector --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dio.netty.tryReflectionSetAccessible=true -Dio.netty.allocator.type=pooled -Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE --enable-native-access=ALL-UNNAMED org.apache.spark.deploy.master.Master --host spark-connect-pool-master-0 --port 7077 --webui-port 8080
========================================
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
26/07/07 00:40:50 INFO Master: Started daemon with process name: 20@spark-connect-pool-master-0
26/07/07 00:40:50 INFO SignalUtils: Registering signal handler for TERM
26/07/07 00:40:50 INFO SignalUtils: Registering signal handler for HUP
26/07/07 00:40:50 INFO SignalUtils: Registering signal handler for INT
26/07/07 00:40:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
26/07/07 00:40:50 INFO SecurityManager: Changing view acls to: spark
26/07/07 00:40:50 INFO SecurityManager: Changing modify acls to: spark
26/07/07 00:40:50 INFO SecurityManager: Changing view acls groups to: spark
26/07/07 00:40:50 INFO SecurityManager: Changing modify acls groups to: spark
26/07/07 00:40:50 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: spark groups with view permissions: EMPTY; users with modify permissions: spark; groups with modify permissions: EMPTY; RPC SSL disabled
26/07/07 00:40:50 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
26/07/07 00:40:50 INFO Master: Starting Spark master at spark://spark-connect-pool-master-0:7077
26/07/07 00:40:50 INFO Master: Running Spark version 4.1.1
26/07/07 00:40:50 INFO JettyUtils: Start Jetty 0.0.0.0:8080 for MasterUI
26/07/07 00:40:50 INFO Utils: Successfully started service 'MasterUI' on port 8080.
26/07/07 00:40:50 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://spark-connect-pool-master-0:8080
26/07/07 00:40:50 INFO Utils: Successfully started service on port 6066.
26/07/07 00:40:50 INFO StandaloneRestServer: Started REST server for submitting applications on spark-connect-pool-master-0 with port 6066
26/07/07 00:40:50 INFO Master: I have been elected leader! New state: ALIVE
26/07/07 00:40:50 INFO Master: Registering worker 10.42.0.71:39241 with 32 cores, 124.7 GiB RAM