#### 问题4: 上传自己修改后的jar 包 + deploy

- 方案1:

1.  将自己的repo 上传到 github. 
2.  修改 `ec2 directory` 里面的 `spark_ec2.py` 里面的 git-repo. 
 
- 方案2:
1. 在本地   `build` 一个jar
2. 将 jar 包通过 `ss` 的方式 上传到 `ec2-master`
3. 通过 `RSYNC` 的方式 将 jar 分发到各个`slave`。 

![](http://wx3.sinaimg.cn/mw690/006BEqu9gy1fgioh9oqiaj30oi0ibdir.jpg)

## 2017-06-03/04

### Benchmark Suite for Apache Spark

 Usecase 1. It enables `quantitative comparison` for Apache Spark system optimizations such as caching policy and memory management optimization, scheduling policy optimization. Researchers and developers can use Spark-Bench to comprehensively evaluate and compare the performance of their optimization and the vanilla Apache Spark.




### 实验过程 (Bench 未完成)

#### 问题1: 安装 maven

Instead to skip tests if you don't want them to be executed.

#### 问题2: Spark-Bench Configurations.
- git clone = download
- build
- 根据 Chen 的再修改 (/spark-bench/conf/env.sh)





#### 问题3: Connect to Amazon EC2 Remotely Using SSH

- Download the .pem file.

- In Amazon Dashboard choose "Instances" from the left side bar, and then select the instance you would like to connect to.

- Click on "Actions", then select "Connect"

- Click on "Connect with a Standalone SSH Client"

- Open up a Terminal window

- Create a directory:

- Move the downloaded .pem file to the .ssh directory we just created:

- Change the permissions of the .pem file so only the root user can read it:

- Create a config file:

- Enter the following text into that config file:

Host *amazonaws.com
IdentityFile ~/.ssh/ec2private.pem
User ec2-user

- Save that file.
- Use the ssh command with your public DNS hostname to connect to your instance.
e.g.:

#### Java JDK 1.8替代 1.7.

1. Installe Java JDK.
Default 1.7 jdk installed on Amazon Linux. Upgrade Java to 1.8

2. Set the JAVA_HOME environment variable.

Some applications such as Apache Maven and Apache Ant require you to set the JAVA_HOME environment variable. On Amazon Linux/CentOS, OpenJDK installs into either /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-2.b17.7.amzn1.x86_64.

JAVA_HOME should point to the directory containing a bin/java executable. Edit user bashrc file and append the following line to export JAVA_HOME

#### 更简洁方案 Java 8 application on EC2

check Java current version

install Java 1.8

change the Java version

Setup JAVA_HOME Variable

Setup JRE_HOME Variable

Setup PATH Variable

#### What is .sh file in conf folder

An SH file is a script programmed for bash, a type of Unix shell (Bourne-Again SHell). It contains instructions written in the Bash language and can be executed by typing text commands within the shell's command-line interface.

#### What is pom.xml 

A Project Object Model (POM) provides all the configuration for a single project. General configuration covers the project's name, its owner and its dependencies on other projects. One can also configure individual phases of the build process, which are implemented as plugins. 

Larger projects should be divided into several modules, or sub-projects, each with its own POM.

![](https://upload.wikimedia.org/wikipedia/commons/thumb/c/cf/Maven_CoC.svg/354px-Maven_CoC.svg.png)

#### What is Maven/mvn

Maven is a build automation tool used primarily for Java projects. 

Maven addresses two aspects of building software: first, it describes how software is built, and second, it describes its dependencies. 

## 2017-06-05

### Spark in Intellij IDE 

#### Bug-1

应该倒入 sql lib

#### Bug-2

问题原因： scala version （还不支持 scala 2.12）

### Spark Web UI
Every SparkContext launches its own instance of Web UI which is available at http://[driver]:4040 by default.

>> All the information that is displayed in web UI is available thanks to JobProgressListener and other SparkListeners. One could say that web UI is a web layer to Spark listeners.

## 2017-06-06 (Bench build 成功) 

### Maven 编译成 1.8 

https://maven.apache.org/plugins/maven-compiler-plugin/examples/set-compiler-source-and-target.html

Setting the -source and -target of the Java Compiler
Sometimes when you may need to compile a certain project to a different version than what you are currently using. The javac can accept such command using -source and -target. The Compiler Plugin can also be configured to provide these options during compilation.

For example, if you want to use the Java 8 language features (-source 1.8) and also want the compiled classes to be compatible with JVM 1.8 (-target 1.8), you can either add the two following properties, which are the default property names for the plugin parameters:

### 更新 yum install Scala 2.11.8 

Install Scala

Spark is based on Scala, so we need to install it early in the process

We’ll also need to tell Ubuntu where we installed Scala. To do this we update a setup file called ‘bashrc’

For editing, we’ll use an application called ‘vi’. This application is a very paired down file editor.

Open the bashrc file
`vi .bashrc`
Use the arrow keys to go to the bottom of the file
Press the {Insert} key
Type:

Press {Esc}
Type :wq and Enter (to save and close)
Ask Ubuntu to read the new bashrc file

Verify that the the new Scala version is recognized. The command below should return something like Scala code runner version 2.11.8 …

### Setup a Spark 2.0 Cluster + R in AWS
https://edgarsdatalab.com/2016/08/25/setup-a-spark-2-0-cluster-r-on-aws/

### Getting Spark, Python, and Jupyter Notebook running on Amazon EC2

https://medium.com/@josemarcialportilla/getting-spark-python-and-jupyter-notebook-running-on-amazon-ec2-dec599e1c297

### yum vs apt-get 


![Success](http://wx2.sinaimg.cn/mw1024/006BEqu9gy1fgbxvod7i8j30y60v4qkc.jpg)

# 六月中旬实验记录

## 2017-06-11 

### Configure the Environment

Add to the following lines to your ~/.bash_profile file.

> Create TensorFlowOnSpark (TFoS) AMI on EC2

> https://github.com/yahoo/TensorFlowOnSpark/wiki/Create_AMI



###  Run a MapReduce job

Run a test MapReduce job using one of the examples included within the Hadoop distribution, e.g.

The output is like:

### Hadoop InputFormat/OutputFormat

Install and compile Hadoop InputFormat/OutputFormat for TFRecords

### Downloading the Script - spark-ec2 folder
([教程](https://sparkour.urizone.net/recipes/spark-ec2/))

If you are running Spark 2.x and need to download the script, visit the `AMPLab spark-ec2` repository. In the Branch: dropdown menu, select `branch-2.0`, then open the Clone or download dropdown menu as shown in the image below. Hovering the mouse over the Download ZIP button will show you the download URL.

### export HADOOP_HOME

### Unsupported major.minor version 52.0

because the jar was compiled in jdk 1.8, but you are trying to run it using a jdk 1.7 environment.


### aclocal - install automake

Where does this come from and how do you fix it?

It's part of Automake . 

On CentOs

### 如何看 log - spark.eventLog.enabled 
1. 为了看log, 你需要先运行spark/sbin/start-history-server.sh 启动history server, 然后再访问端口18080来获得job的运行状态信息

2. 记得在/tmp 下建立文件夹 spark-events

3. 同时在spark/conf/spark-defaults.conf中 加入spark.eventLog.enabled

或者是 ： 

Setting spark-defualts.conf as

原因是：

[Viewing After the Fact](http://spark.apache.org/docs/2.1.0/monitoring.html)

This creates a web interface at http://<server-url>:18080 by default, listing incomplete and completed applications and attempts.

> https://stackoverflow.com/questions/31233830/apache-spark-setting-spark-eventlog-enabled-and-spark-eventlog-dir-at-submit-or

![](http://wx2.sinaimg.cn/mw690/006BEqu9gy1fgioh9pjq0j30po08hdhn.jpg)


For Spark Job setup:

Then check the `Spark History Server`

You can also put on old fashioned Java logging

### Configuring Logging
Spark uses `log4j` for logging. You can configure it by adding a `log4j.properties` file in the conf directory. One way to start is to copy the existing `log4j.properties.template` located there.

## 2017-06-14

拼接代码 - 做实验

### Install bash-completion on Amazon Linux

Go into /src first.

## 2017-06-15 目的是 - Warm up Sparkcontext 

### 网上的warm-up范例

很类似的范例

来自 - Drizzle: Low Latency Execution for Apache Spark

In Drizzle, we introduce group scheduling, where multiple batches (or a group) of computation are scheduled at once. This helps decouple the granularity of task execution from scheduling and amortize the costs of task serialization and launch.

#### warm-up in DrizzleRunningSum.scala

https://github.com/amplab/drizzle-spark/blob/master/examples/src/main/scala/org/apache/spark/examples/DrizzleRunningSum.scala

// Let all the executors join
Thread.sleep(10000)
// Warm up the JVM and copy the JAR out to all the machines etc.
val execIds = sc.parallelize(0 until sc.getExecutorMemoryStatus.size,
  sc.getExecutorMemoryStatus.size).foreach { x =>
  Thread.sleep(1)
} ``` #### SparkKMeans.scala

val data = Array(1, 2, 3, 4, 5)
val distData = sc.parallelize(data)

// Basic map example in scala
scala> val x = sc.parallelize(List("spark", "rdd", "example",  "sample", "example"), 3)
scala> val y = x.map(x => (x, 1))
scala> y.collect
res0: Array[(String, Int)] = Array((spark,1), (rdd,1), (example,1), (sample,1), (example,1))

// rdd y can be re writen with shorter syntax in scala as 
scala> val y = x.map((_, 1))
scala> y.collect
res0: Array[(String, Int)] = Array((spark,1), (rdd,1), (example,1), (sample,1), (example,1))

// Another example of making tuple with string and it's length
scala> val y = x.map(x => (x, x.length))
scala> y.collect
res0: Array[(String, Int)] = Array((spark,5), (rdd,3), (example,7), (sample,6), (example,7))

val startTimeMillis = System.currentTimeMillis()

/* your code goes here */

val endTimeMillis = System.currentTimeMillis()
val durationSeconds = (endTimeMillis - startTimeMillis) / 1000

Method -2 

 // returns the current value of the system timer, in nanoseconds
    System.out.print("time in nanoseconds = ");
    System.out.println(System.nanoTime());

 // returns the current value of the system timer, in milliseconds
    System.out.print("time in milliseconds = ");
    System.out.println(System.currentTimeMillis());
      
      

- press Esc (to leave editing or other mode)
- hit ctrl + v (visual block mode)
- use the up/down arrow keys to select lines you want (it won't highlight everything - it's OK!)
- Shift + i (capital I)
- insert the text you want, i.e. '% '
- press Esc.

new ForkJoinTaskSupport(environment: ForkJoinPool = ForkJoinTasks.defaultForkJoinPool)

scala> import scala.collection.parallel._
import scala.collection.parallel._

scala> val pc = mutable.ParArray(1, 2, 3)
pc: scala.collection.parallel.mutable.ParArray[Int] = ParArray(1, 2, 3)

scala> pc.tasksupport = new ForkJoinTaskSupport(new scala.concurrent.forkjoin.ForkJoinPool(2))
pc.tasksupport: scala.collection.parallel.TaskSupport = scala.collection.parallel.ForkJoinTaskSupport@4a5d484a

scala> pc map { _ + 1 }
res0: scala.collection.parallel.mutable.ParArray[Int] = ParArray(2, 3, 4)

//I am using scala parallel collections.

val largeList = list.par.map(x => largeComputation(x)).toList

largerList.tasksupport = new ForkJoinTaskSupport(
  new scala.concurrent.forkjoin.ForkJoinPool(x)
)
largerList.map(x => largeComputation(x)).toList

scala> pc.tasksupport = new ThreadPoolTaskSupport()
pc.tasksupport: scala.collection.parallel.TaskSupport = scala.collection.parallel.ThreadPoolTaskSupport@1d914a39

scala> pc map { _ + 1 }
res1: scala.collection.parallel.mutable.ParArray[Int] = ParArray(2, 3, 4)

2>&1

echo test > afile.txt

echo test 1> afile.txt

echo test 2> afile.txt

echo test 1>&2 # or echo test >&2

echo test 2>&1

bin/hadoop fs <args>

sc.setLocalProperty("spark.scheduler.pool", "myPool")

// these two jobs (one per action) will run in the myPool pool
rdd.count
rdd.collect

sc.setLocalProperty("spark.scheduler.pool", null)

// this job will run in the default pool
rdd.count

import org.apache.spark.scheduler._


def setJobGroup(groupId: String, description: String, interruptOnCancel: Boolean = false) {
    setLocalProperty(SparkContext.SPARK_JOB_DESCRIPTION, description)
    setLocalProperty(SparkContext.SPARK_JOB_GROUP_ID, groupId)
    // Note: Specifying interruptOnCancel in setJobGroup (rather than cancelJobGroup) avoids
    // changing several public APIs and allows Spark cancellations outside of the cancelJobGroup
    // APIs to also take advantage of this property (e.g., internal job failures or canceling from
    // JobProgressTab UI) on a per-job basis.
    setLocalProperty(SparkContext.SPARK_JOB_INTERRUPT_ON_CANCEL, interruptOnCancel.toString)
  }

  /** Clear the current thread's job group ID and its description. */
  def clearJobGroup() {
    setLocalProperty(SparkContext.SPARK_JOB_DESCRIPTION, null)
    setLocalProperty(SparkContext.SPARK_JOB_GROUP_ID, null)
    setLocalProperty(SparkContext.SPARK_JOB_INTERRUPT_ON_CANCEL, null)
  }

INFO FairSchedulableBuilder: Created pool [poolName], schedulingMode: FIFO, minShare: 0, weight: 1

log4j.logger.org.apache.spark.scheduler.FairSchedulableBuilder=INFO

import org.jblas.DoubleMatrix

Read more about maven at : https://maven.apache.org/
3. Clone repository from github and install locally.

### What is jblas 

 `jblas` is a fast linear algebra library for Java. jblas is based on BLAS and LAPACK, the de-facto industry standard for matrix computations, and uses state-of-the-art implementations like ATLAS for all its computational routines, making jBLAS very fast.

### What is an artifact in Maven/pom.xml

- Project Object Model (POM) 

- An artifact is a file, usually a JAR, that gets deployed to a Maven repository.

- A Maven build produces `one or more` artifacts, such as a compiled JAR and a "sources" JAR.

- Each artifact has a group ID (usually a reversed domain name, like com.example.foo), an artifact ID (just a name), and a version string. The three together uniquely identify the artifact.

- A project's dependencies are specified as artifacts.

In Maven terminology, the `artifact` is the resulting output of the maven build, generally a `jar` or war or other executable file. Artifacts in maven are identified by a coordinate system of `groupId`, `artifactId`, and `version`. Maven uses the groupId, artifactId, and version to identify dependencies (usually other jar files) needed to build and run your code.

## 2017-06-20

### Spark -trace analysis

https://github.com/kayousterhout/trace-analysis

#### Configuring Spark log

1. Spark configuration parameter spark.eventLog.enabled to true
2. the event log is written to a series of files in the folder /tmp/spark-events/ on master machine.
3. logs are stored in a file named EVENT_LOG_1 within the application's folder. 

#### Analyzing

1.  --waterfall-only flag tells the script to just generate the visualization, and skip more complex performance analysis. 
2.  To see all available options, use the flag --help.

For each job in the `EVENT_LOG_1` file, the Python script will output a gnuplot file that, when plotted, will generate a waterfall depicting how time was spent by each of the tasks in the job. The plot files are named `[INPUT_FILENAME]_[JOB_ID]_waterfall.gp`. To plot the waterfall for job 0, for example:

#### Keep in mind:
- ` only` the total amount of time spent in each part of task execution and the start and end time of the task is accurate.
-  Spark does not currently include instrumentation to measure the time spent reading input data from disk or writing job output to disk (the ''Output write wait'' shown in the waterfall is time to write shuffle output to disk, which Spark does have instrumentation for)

#### FAQ 

##### (如何画多个图) tasks seem to be overlapping others.

This typically happens when you try to plot multiple gnuplot files with one command, e.g., with a command like: