Spark基础-scala学习（一、入门）

Scala解析器的使用

1. REPL：Read（取值）-> Evaluation（求值）-> Print（打印）->Loop（循环）。scala解析器也被称为REPL，会快速编译scala代码为字节码，然后交给JVM执行
2. val result = 1 设置变量不可变
3. var result = 2 可变的变量
4. val name: String = null 声明变量类型
5. val name: Any = "leo"
6. val name1,name2:String = null 声明多个变量
7. val num1,num2=100

数据类型与操作符

1. 基本数据类型：Byte、Char、Short、Int、Long、Float、Double、Boolean
2. scala没有基本数据类型与包装类型的概念，统一都是类
3. 使用以上类型，直接就恶意调用大量的函数，例如，1.toString(),1.to(10)
4. 在scala中，操作符比如+-*/%&|^>><<等其实是数据类型的函数，比如1+1可以写作1.+(1);例如1.to(10) 又可以写作1 to 10
5. scala中没提供++、--操作符，只能用+=和-=。比如counter=1,counter++是错误的，必须写作counter+=1

函数调用与apply()函数

1. 函数调用，不需要传递参数，允许调用函数时省略括号，例如："Hello World".distinct
2. apply函数
   • Scala中使用类名()的形式其实就是类名.apply()的缩写，来创建类对象，而不是new 类名()的方式
   • 例如"Hello World"(6)因为StringOps类中有def apply(n: Int):Char的函数定义，所以"Hello World"(6),实际上是"Hello World".apply(6)的缩写

条件控制与循环

1. if(age > 19) 1 else 8 存在返回值
2. if(age > 18) "adult" else 0 当返回值类型不同，会取公共父类型Any

输入输出

1. val name = readLine("Welcome to House")
2. val age = readInt()
3. for(i <-1 to n) print(i)
4. for(i <-1 until n) println(i) 表示不达到上限
5. for(c <- "Hello World") print(c) 对字符串遍历，类似于java增强for循环

for(i <- 1 to 9;j <- 1 to 9){
    if(j==9){
        println(i*j)
    }else{
        print(i*j+" ")
    }
}

函数入门

1. 定义函数，age为返回值

def sayHello(name:String,age: Int) = {
    if(age > 19) {
        printf("hi %s,you are a big boy\n",name)
        age
    }else{
        printf("hi ,%s,you are a children\n",name)
        age
    }

}

def sum(n:Int) = {
    var result = 0
    for( i<- 1 to n){
        result += i
    }
    result
}

def fab(n: Int): Int = {
    if(n <= 0) 1
    else fab(n-1)+fab(n-2)
}

函数默认参数

1. 默认参数

def sayHello(firstName:String,middleName:String = "",lastName:String="") = firstName + " " +middleName+" "+lastName

1. 带名参数调用，可以不按顺序

sayHello(firstName="hahha",middleName="xxx",lastName="ggg")

变长参数

def sum(nums: Int*)={
    var result = 0
    for(num <- nums){
        result += num
    }
    result
}

sum(1 to 5: _*) //值为15，表示取出1到5的整数相加

def sum2(nums:Int*):Int = {
    if(nums.length == 0) 0
    else nums.head + sum2(nums.tail: _*)
}

lazy值和异常

//定义过程，不会有返回值
def sayHello(name:String):Unit = "Hello,"+name

lazy val lines = fromFile("/home/1.text").mkString
//使用的时候才会执行上面这句
print(lines)

异常

try{
    throw new lllegAlrgumentException("x should not be negative")
}catch{
    case_:lllegAlrgumentException => print("sorry,error")
}finally{
  print("release io ")
}

数组操作之Array、ArrayBuffer以及遍历数组

1. val a = new ArrayInt
2. a(0) = 1给元素赋值
3. val a = Array("hello","world")
4. import scala.collection.mutable.ArrayBuffer
5. var b = ArrayBufferInt
6. b += (2,3,4,5)
7. 结果b.type = ArrayBuffer(1,2,3,4,5)
8. b++=ArrayBuffer(6,7,8,9,10)
9. 结果b.type = ArrayBuffer(1,2,3,4,5,6,7,8,9,10)
10. b.trimEnd(5)
11. b的结果去掉后5位

scala> b.insert(1,2,2,3)

scala> b
res20: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(1, 2, 2, 3, 2, 3, 5, 5, 6, 9, 8)

scala> b.insert(1,1,2,9)

scala> b
res22: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(1, 1, 2, 9, 2, 2, 3, 2, 3, 5, 5, 6, 9, 8)

scala> b.insert(1,7,8)

scala> b
res24: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(1, 7, 8, 1, 2, 9, 2, 2, 3, 2, 3, 5, 5, 6, 9, 8)

scala> b.remove(1)
res25: Int = 7

scala> b
res26: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(1, 8, 1, 2, 9, 2, 2, 3, 2, 3, 5, 5, 6, 9, 8)

scala> b.remove(1,3)

scala> b
res29: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(1, 9, 2, 2, 3, 2, 3, 5, 5, 6, 9, 8)

//互相转换
scala> b.toArray
res30: Array[Int] = Array(1, 9, 2, 2, 3, 2, 3, 5, 5, 6, 9, 8)

scala> b.toBuffer
res31: scala.collection.mutable.Buffer[Int] = ArrayBuffer(1, 9, 2, 2, 3, 2, 3, 5, 5, 6, 9, 8)

scala> for(i <- 0 until b.length) print(b(i))
192232355698
//跳跃遍历
scala> for(i <- 0 until (b.length,2)) print(b(i))
123359
//从尾部遍历
scala> for(i <- (0 until b.length).reverse) print(b(i))
896553232291
//增强for循环
scala> for(i <- b) print(i)
192232355698

scala> val b = Array(9,8,2,2,3,4)
b: Array[Int] = Array(9, 8, 2, 2, 3, 4)

scala> scala.util.Sorting.quickSort(b)

scala> b
res3: Array[Int] = Array(2, 2, 3, 4, 8, 9)

scala> b.mkString
res5: String = 223489

scala> b.mkString(",")
res6: String = 2,2,3,4,8,9

scala> b.toString
res7: String = [I@6d4502ca

yield

scala> val a = Array(1,2,3,4,5)
a: Array[Int] = Array(1, 2, 3, 4, 5)

scala> val a2 = for(ele <- a) yield ele * ele
a2: Array[Int] = Array(1, 4, 9, 16, 25)

scala> val a3 = for(ele <- a if ele % 2==0) yield ele * ele
a3: Array[Int] = Array(4, 16)

scala> a.filter(_%2==0).map(_*2)
res8: Array[Int] = Array(4, 8)

scala> a.filter{_%2==0} map{_*2}
res9: Array[Int] = Array(4, 8)

算法案例：移除第一个负数后的所有负数

scala> import scala.collection.mutable.ArrayBuffer
import scala.collection.mutable.ArrayBuffer

scala> val a = ArrayBuffer[Int]()
a: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer()

scala> a += (1,2,3,4,5,-1,-3,-5,-7)
res10: a.type = ArrayBuffer(1, 2, 3, 4, 5, -1, -3, -5, -7)

scala> :paste
// Entering paste mode (ctrl-D to finish)

var foundFirstNegative = false
var arrayLength = a.length
var index = 0
while(index < arrayLength){
if(a(index)>=0){
index+=1
}else{
if(!foundFirstNegative){foundFirstNegative = true;index+=1}
else{a.remove(index);arrayLength-=1}
}
}

// Exiting paste mode, now interpreting.

foundFirstNegative: Boolean = true
arrayLength: Int = 6
index: Int = 6

scala> a
res12: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(1, 2, 3, 4, 5, -1)

第二种，性能更高，数组更少移动

var foundFirstNegative = false
val keepIndexes = for(i <- 0 until a.length if !foundFirstNegative || a(i)>=0
) yield{
if(a(i)<0) foundFirstNegative = true
i
}
for (i <- 0 until keepIndexes.length){a(i) = a(keepIndexes(i))}
a.trimEnd(a.length - keepIndexes.length)

scala> a
res15: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(1, 2, 3, 4, 5, -1)

创建Map

//不可变Map
scala> val ages = Map("Leo"->30,"den"->20,"jack"->23)
ages: scala.collection.immutable.Map[String,Int] = Map(Leo -> 30, den -> 20, jack -> 23)

scala> ages("Leo")
res0: Int = 30

scala> ages("Leo") = 20
<console>:13: error: value update is not a member of scala.collection.immutable.Map[String,Int]
       ages("Leo") = 20
       ^
//可变Map
scala> val ages = scala.collection.mutable.M
Map   MapBuilder   MapLike   MapProxy   MultiMap   MutableList

scala> val ages = scala.collection.mutable.Map("Leo"->30,"den"->20,"jack"->23)
ages: scala.collection.mutable.Map[String,Int] = Map(jack -> 23, den -> 20, Leo -> 30)

scala> ages("Leo")=24

scala> ages("Leo")
res3: Int = 24
//第三种创建方式
scala> val ages = Map(("Leo",30),("Jen",31),("jack",33))
ages: scala.collection.immutable.Map[String,Int] = Map(Leo -> 30, Jen -> 31, jack -> 33)

//第四种创建方式
scala> val ages = new scala.collection.mutable.HashMap[String,Int]
ages: scala.collection.mutable.HashMap[String,Int] = Map()

scala> ages.put("jack",24)
res5: Option[Int] = None

scala> ages
res6: scala.collection.mutable.HashMap[String,Int] = Map(jack -> 24)
//必须判断是否存在，否则不存在会报错
scala> val age = if(ages.contains("jack")) ages("jack") else 0
age: Int = 24
//或者取默认值
scala> val leoAge = ages.getOrElse("Leo",0)
leoAge: Int = 0
//一次性添加多个key-value对
scala> ages+=("Mike"->35,"Tom"->50)
res9: ages.type = Map(jack -> 24, Mike -> 35, Tom -> 50, Leo -> 30)
//移除元素
scala> ages-="Mike"
res10: ages.type = Map(jack -> 24, Tom -> 50, Leo -> 30)

//Map本身不可变，下列操作通过生成新的Map实现
scala> val aggs = Map("Leo"->24,"jike"->34)
aggs: scala.collection.immutable.Map[String,Int] = Map(Leo -> 24, jike -> 34)

scala> val ages2 = aggs + ("Mike"->20,"Tom"->70)
ages2: scala.collection.immutable.Map[String,Int] = Map(Leo -> 24, jike -> 34, Mike -> 20, Tom -> 70)

scala> val ages3 = aggs-"Leo"
ages3: scala.collection.immutable.Map[String,Int] = Map(jike -> 34)

scala> aggs
res12: scala.collection.immutable.Map[String,Int] = Map(Leo -> 24, jike -> 34)

//Map遍历
scala> for((key,value) <- ages) println(key +" "+value)
jack 24
Tom 50
Leo 30

scala> for(key <- ages.keySet)println(key)
jack
Tom
Leo

scala> for(value <- ages.values) println(value)
24
50
30
//反转key和value
scala> for((key,value) <- ages) yield(value,key)
res16: scala.collection.mutable.HashMap[Int,String] = Map(50 -> Tom, 30 -> Leo, 24 -> jack)

Map排序

//自动对Map的key排序
scala> val ages = scala.collection.immutable.SortedMap("leo"->30,"alice"->14,"jen"->25)
ages: scala.collection.immutable.SortedMap[String,Int] = Map(alice -> 14, jen -> 25, leo -> 30)

//保证插入顺序和读取顺序相同
scala> val aggs = new scala.collection.mutable.LinkedHashMap[String,Int]
aggs: scala.collection.mutable.LinkedHashMap[String,Int] = Map()

scala> aggs("leo")=30
scala> aggs("jike")=40
scala> aggs("alice")=15

scala> aggs
res22: scala.collection.mutable.LinkedHashMap[String,Int] = Map(leo -> 30, jike -> 40, alice -> 15)

Map的元素类型-Tuple

//简单tuple
scala> val t = ("leo",30)
t: (String, Int) = (leo,30)

scala> t._1
res23: String = leo

scala> t._2
res24: Int = 30

//zip操作，将两数组拼接成元组
scala> val names = Array("leo","jack","mike")
names: Array[String] = Array(leo, jack, mike)

scala> val ages = Array(30,24,26)
ages: Array[Int] = Array(30, 24, 26)

scala> val nameAges = names.zip(ages)
nameAges: Array[(String, Int)] = Array((leo,30), (jack,24), (mike,26))

scala> for((name,age)<-nameAges) println(name +":" + age)
leo:30
jack:24
mike:26