Spark

单机安装Spark的单机安装是很简单的解压安装包[root@masterspark]#tar-zxvfspark-2.0.0-bin-hadoop2.6.tgz[root@masterspark]#ll总用量0drwxr-xr-x.125005001937月202016spark-2.0.0-bin-hadoop2.6[root@masterspark]#pwd/usr/local/src/spark配置环境变量[root@masterspark]#vi~/.bash_profile#加入环境变量#SparkexportSPARK_HOME=/usr/local/src/spark/spark-2.0.0-bin-hadoop2.6exportPATH=$PATH:$SPARK_HOME/bin:[root@masterspark]#source~/.bash_profile安装测试Spark的shell命令[root@masterspark]#spark-shellUsingSpark'sdefaultlog4jprofile:org/apache/spark/log4j-defaults.propertiesSettingdefaultloglevelto"WARN".Toadjustlogginglevelusesc.setLogLevel(newLevel).20/01/0215:29:53WARNNativeCodeLoader:Unabletoloadnative-hadooplibraryforyourplatform...usingbuiltin-javaclasseswhereapplicable20/01/0215:29:55WARNUtils:Service'SparkUI'couldnotbindonport4040.Attemptingport4041.20/01/0215:29:55WARNSparkContext:UseanexistingSparkContext,someconfigurationmaynottakeeffect.SparkcontextWebUIavailableathttp://192.168.176.61:4041Sparkcontextavailableas'sc'(master=local[*],appid=local-1577950195083).Sparksessionavailableas'spark'.Welcometo______/__/__________//___\\/_\/_`/__/'_//___/.__/\_,_/_//_/\_\version2.0.0/_/UsingScalaversion2.11.8(JavaHotSpot(TM)64-BitServerVM,Java1.8.0_221)Typeinexpressionstohavethemevaluated.Type:helpformoreinformation.scala>启动后的控制台界面和web界面，这样就算安装成功Spark的集群安装在单机模式的安装上，再进行相应的配置修改slaves进入Spakr的conf/目录下，复制一个slaves模板#复制slaves模板[root@masterconf]#cpslaves.templateslaves#打开slaves[root@masterconf]#vislaves#加入节点机器的映射名称，映射在配置Hadoop集群的时候已经配置masterslave1slave2修改spark-env.shexportJAVA_HOME=/usr/lib/jvm/jdk8u191-b12exportSCALA_HOME=/home/modules/spark-2.3.0/examples/src/main/scalaexportHADOOP_HOME=/home/modules/hadoop-2.8.3exportHADOOP_CONF_DIR=/home/modules/hadoop-2.8.3/etc/hadoopexportSPARK_HOME=/home/modules/spark-2.3.0exportSPARK_DIST_CLASSPATH=$(/home/modules/hadoop-2.8.3/bin/hadoopclasspath)exportLD_LIBRARY_PATH=/home/modules/hadoop-2.8.3/lib/nativeexportYARN_CONF_DIR=/home/modules/hadoop-2.8.3/etc/hadoopexportSPARK_MASTER_IP=node1分发配置到其他机器上分发Sparkscp-rspark/root@slave1:/usr/local/src/scp-rspark/root@slave2:/usr/local/src/分发环境变量scp~/.bash_profileroot@slave1:~/.bash_profilescp~/.bash_profileroot@slave1:~/.bash_profile启动Spark集群进入Spark的sbin/目录下执行启动命令[root@mastersbin]#start-all.shorg.apache.spark.deploy.master.Masterrunningasprocess14018.Stopitfirst.master:org.apache.spark.deploy.worker.Workerrunningasprocess14365.Stopitfirst.slave1:org.apache.spark.deploy.worker.Workerrunningasprocess1952.Stopitfirst.slave2:org.apache.spark.deploy.worker.Workerrunningasprocess2616.Stopitfirst.启动测试个机器出现以下就是搭建成功masterslave1slave2NodeManagerNodeManagerNodeManagerJpsJpsJpsDataNodeDataNodeDataNodeWorkerWorkerWorkerNameNodeMasterSecondaryNameNodeResourceManagerSparkSubmitSparkSubmitSparkSubmit进入Master:8080查看集群模式网页