spark-shuffle-site.xml 配置项

在 NodeManager 启动 spark shuffle，可以配置 spark-shuffle-site.xml，设置 spark-shuffle 的参数。

spark-shuffle-site.xml 配置项

  
    spark.shuffle.push.server.mergedShuffleFileManagerImpl
    org.apache.spark.network.shuffle.RemoteBlockPushResolver
    启动 push based shuffle 必须配置。默认为 org.apache.spark.network.shuffle.
NoOpMergedShuffleFileManager 
  
  
  	spark.io.mode
    NIO
    默认 NIO
  
  
  	spark.io.preferDirectBufs
    true
    默认 true
  
  
  	spark.io.connectionTimeout
    120s
    默认为 ${spark.network.timeout}
  
  
  	spark.network.timeout
    120s
    connection idle timeout， 默认为120s, 超过事件的由 IdleStateHandler 进行关闭 
  
  
  	spark.io.connectionCreationTimeout
    120s
    默认 ${spark.io.connectionTimeout}
  
  
  	spark.io.backLog
    -1
    默认 -1, 代表用系统默认的
  
  
  	spark.io.numConnectionsPerPeer
    1
    默认 1
  
  
  	spark.io.serverThreads
    0
     默认0 代表 2x#cores
  
  
  	spark.io.clientThreads
    0
    默认0 代表 2x#cores
  
  
  	spark.io.receiveBuffer
    -1
    默认 -1, 使用系统默认的
  
  
  	spark.io.sendBuffer
    -1
    默认 -1, 使用系统默认的
  
  
  	spark.sasl.timeout
    30s
    默认 30s
  
  
  	spark.io.maxRetries
    3
    默认 3
  
  
  	spark.io.retryWait
    5s
    默认5s
  
  
  	spark.io.lazyFD
    true
    默认true. Whether to initialize FileDescriptor lazily or not. If true, file descriptors are created only when data is going to be transferred. This can reduce the number of open files.
  
  
  	spark.io.enableVerboseMetrics
    false
    默认false.Whether to track Netty memory detailed metrics. If true, the detailed metrics of Netty PoolByteBufAllocator will be gotten, otherwise only general memory usage will be tracked.
  
  
  	spark.io.enableTcpKeepAlive
    false
    默认 false