This is an installation guide for Apache Hadoop and Apache Spark on a multi node setup. We will use Cloudera v5 as it has the packages for debian. You are free to use Hortonworks ,MapR (or) straight up zips from Hadoop, Spark.

Lets clear the terminology of a node. A node means individual servers, Virtual Machines (VMs) (or) Micro services.

In a multinode setup, we will deploy a topology of

  • One Hadoop namenode
  • Many Hadoop datanode
  • One Spark Master
  • Many Spark Workers

The definition of Hadoop namenode, datanode, Spark can be read from the links here.

Apache Spark Hadoop

In all the nodes

We need java, hence do the following in all the nodes.

Install java

sudo apt-get install openjdk-7-jdk

Install cloudera repo package

wget https://archive.cloudera.com/cdh5/one-click-install/trusty/amd64/cdh5-repository_1.0_all.deb
sudo dpkg -i cdh5-repository_1.0_all.deb
sudo apt-get -y update

For those of you folks from Ceph background, The Hadoop namenode is like Ceph MDS (Metadata server) which just stores pointer to data/additional info about the data.

There is a research project we will look at to see how Ceph can be natively used with Apache Spark. Why do you want do that, well Ceph is native (ugly Java can be gotten rid of)

First node

Lets install Hadoop namenode in the first node.

sudo apt-get -y install hadoop-yarn-resourcemanager hadoop-hdfs-namenode hadoop-yarn-nodemanager hadoop-yarn-proxyserver hadoop-client

Make a directory to store hadoop filesystem and change permission of that directory to hdfs:hdfs

sudo mkdir -p /hadoopstorage/name
sudo chown -R hdfs:hdfs /hadoopstorage/

Edit /etc/hadoop/conf/core-site.xml with the $FIRST_NODE_IP_ADDR to ip address of first node

<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance withthe License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions and limitations under the License.
-->
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://$FIRST_NODE_IP_ADDR:8097</value>
</property>
</configuration>

Edit /etc/hadoop/conf/hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
 <name>dfs.namenode.name.dir</name>
 <value>/hadoopstorage/name/</value>
</property>
<property>
	<name>dfs.permissions.enabled</name>
	<value>false</value>
</property>
</configuration>

Make symbolic links

sudo ln -s /usr/lib/hadoop/libexec /usr/lib/hadoop-yarn/libexec
sudo ln -s /usr/lib/hadoop/libexec /usr/lib/hadoop-hdfs/libexec

Start Hadoop namenode

sudo -u hdfs hadoop namenode -format -force
sudo service hadoop-hdfs-namenode start

Second node

Let us install Spark Master.

apt-get install spark-core spark-master spark-history-server

Change STANDALONE_SPARK_MASTER_HOST by replacing the $SECONDNODE_IP_ADDR with your second node ip address.

sed -i "s/^[ \t]*export STANDALONE_SPARK_MASTER_HOST=.*/export STANDALONE_SPARK_MASTER_HOST='$SECONDNODE_IP_ADDR'/" /etc/spark/conf/spark-env.sh

Add below lines in /etc/spark/conf/spark-defaults.conf

spark.master    spark://$SECONDNODE_IP_ADDR:7077
spark.eventLog.dir  /usr/spark/applicationHistory
spark.eventLog.enabled           true

After changing configurations, you need to restart spark-master

sudo service spark-master restart

Adding a Hadoop datanode

Let us install Hadoop datanode in another node you have.

Install packages

sudo apt-get -y install hadoop-hdfs-datanode

Create hadoop storage directory

sudo mkdir -p /hadoopstorage/data/

Edit /etc/hadoop/conf/core-site.xml

<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance withthe License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions and limitations under the License.
-->
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://$HADOOP_NAMENODE_IP_ADDR:8097</value>
</property>
</configuration>

Edit /etc/hadoop/conf/hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
	<name>dfs.permissions.enabled</name>
	<value>false</value>
</property>
<property>
  	 <name>dfs.datanode.data.dir</name>
 <value>/hadoopstorage/data/</value>
</property>
</configuration>

Make symbolic links

sudo ln -s /usr/lib/hadoop/libexec /usr/lib/hadoop-hdfs/libexec

Restart service

sudo service hadoop-hdfs-datanode restart

Adding a Spark Worker

Let us install spark worker in another node you have.

sudo apt-get install libgfortran3
apt-get install spark-core spark-worker

Change STANDALONE_SPARK_MASTER_HOST by replacing the $SPARK_MASTER_IP_ADDR with your spark master’s ip address. In this case the second node is our Spark Master

sed -i "s/^[ \t]*export STANDALONE_SPARK_MASTER_HOST=.*/export STANDALONE_SPARK_MASTER_HOST='$SPARK_MASTER_IP_ADDR'/" /etc/spark/conf/spark-env.sh

Add the below in /etc/spark/conf/spark-defaults.conf

spark.master    spark://$SPARK_MASTER_IP_ADDR:7077

Restart Service

sudo service spark-worker restart

After this we will launch the retail analytics app to predict a product buying behaviour. We will cover it in the next part.

NOTE : If you want to store the history of spark, create spark history storage space into HDFS.

At this point, hadoop-hdfs-namenode and hadoop-hdfs-datanode services should be running.

Execute the below in master server to create history storage

sudo -u hdfs hadoop fs -mkdir /user
sudo -u hdfs hadoop fs -mkdir /user/spark
sudo -u hdfs hadoop fs -mkdir /user/spark/applicationHistory
sudo -u hdfs hadoop fs -chown -R spark:spark /user/spark
sudo -u hdfs hadoop fs -chmod 1777 /user/spark/applicationHistory

Now restart the service

sudo service spark-history-server restart

We are going to build a beautiful REST API based on a modern playframework using Scala in a functional way.

Let us start by understanding the concept of REST in a simpler sense.

RESTful API

REST came from Roy Fieldings desertation in claiming to make web stateles again based on designing your system using nouns. So some of the examples are

  • accounts
  • users
  • profiles
  • logs
  • assemblies
  • apps

You no longer need the clugy SOAP heavyweight XML, but rather a nimble JSON using fundamental HTTP will do.

How do you arrive on a design for the RESTful approach for your system is beyond the scope of this article. However we recommend you to read about apigees REST API Design, which talks about the same.

The RESTful API’s transport mechanism is pure HTTP

We will use HMAC to encrypt the payload sent across from a client.

The client code we will use in our usecase will be Ruby or Scala.

First let us start with the core HTTP payload we would like to send across.

The constructs of the payload which would help us multi tenantized system are,

Parts of the payload.

  • HEADER

Our header that we wish to send will have the following

X_Megam_EMAIL - The email address of the user X_Megam_APIKEY - The api key generated for the user X_Megam_DATE - The date on which the data was sent X_Megam_HMAC - The MD5 hmac calculated by the client in the format of email:hmac value Path - URL path from the HTTP Request

The HMAC value is calculated as follows X_Megam_DATE + Path + MD5(body JSON)

The HMAC string makes sure the payload sent from the client is well formed and is valid.

  • BODY A JSON as sent by the client. The body JSON may or may not be present. In case where a HTTP GET is done there body will be empty

Sample scala code in building Header.

val defaultHeaderOpt = Map(Content_Type -> application_json,
X_Megam_EMAIL -> "megam@mypaas.io", X_Megam_APIKEY -> "IamAtlas{74}NobodyCanSeeME#07",
//X_Megam_EMAIL -> "fake@mypaas.io", X_Megam_APIKEY -> "fakemypaas#megam",    // X_Megam_EMAIL -> "steve@olympics.com", X_Megam_APIKEY -> "mAFfm45XnvYJMqPTpP5Dow==",
X_Megam_DATE -> currentDate, Accept -> application_vnd_megam_json)

BODY

Sample scala code.

val contentToEncode = "{\"collapsedmail\":\"megam@mypaas.io\", \"inval_api_key\":\"IamAtlas{74}NobodyCanSeeME#075488\", \"authority\":\"user\" }"

HMAC

We design a cryptographic hash based on the popular MD5 or SHA1 which can be encrypted during the send and decrypted upon receipt.

Now that we have set the playground for the various definitions let us put to use in the platframework.

Our RESTful API Serveer will be named as megam_gateway and is designed to be stateless.

Approach

We will use a novel approach to authenticate and authorize any request that come to our megam_gateway. The approach should be intelligent enought to authenticate any number of layers, return back on malformed header (invalid email could be one of them), mismatch API key and exception condition in megam_gateway.

Controller

A regular controller which the requests coming in to the megam_gateway.

play-auth

An authentication library that helps to quickly build authentication function in a play project.

APIAuthElement : Extension of StackActionController

A trait, that extends a StackActionController and can be invoked by extending this trait in relevant controller.

object Accounts extends Controller with APIAuthElement {

Also you have coarse control to decide if you wish to authenticate all HTTP verbs or not.

For instance when an account is created initially there is no point authenticating, you know why.

If an authentication is needed, then we wrap the particular action in controller using StackAction.

def show(id: String) = StackAction(parse.tolerantText) { implicit request =>

You can read the guide availabe in play-auth for more information.

SecurityActions

A helper function that handles authentication after everything is verified.

Lets put everything together and create our first api accounts

Step 1: FunnelRequest

The first step we do is to find those actions in the controller that require authentication. We wrap them with StackAction.

trait APIAuthElement extends StackableController

The trait has an implicit which helps to converts the input HTTPRequest inputs to a FunnelRequest The FunnelRequest just maps the HTTPRequestInput and maps the header, body, hmac to an internal object.

Step 2: Validate FunnelRequest

After that the SecurityActions.Authenticates the FunnelRequest Object and returns to serve the request if every is valid. If the Header is malformed its immediately sent back.

object SecurityActions {

def AuthenticatedA: ValidationNel[Throwable, Option[String]] = { Logger.debug((“%-20s –>[%s]”).format(“SecurityActions”, “Authenticated:Entry”)) req.funneled match {

Step3: Serve the request.

Now when things are validated, the controller handles the request

def show(id: String) = StackAction(parse.tolerantText) { implicit request =>
play.api.Logger.debug(("%-20s -->[%s]").format("controllers.Accounts", "show:Entry"))
play.api.Logger.debug(("%-20s -->[%s]").format("email", id))
models.Accounts.findByEmail(id) match {

We saw bits and pieces on how everything comes together. Let us look at how to implement an account..

Account

/accounts Let us implement the simple API which has the following requirements

</table> Great. For both of the functions we need a controller. # AccountsController import controllers.funnel.FunnelErrors._ import controllers.funnel.FunnelResponse import models._ import play.api._ import play.api.mvc._ import play.api.mvc.Result import scalaz._ import Scalaz._ import scalaz.effect.IO import scalaz.EitherT._ import scalaz.Validation //import scalaz.Validation.FlatMap._ import scalaz.NonEmptyList._ /** * @author rajthilak * */ /* * This controller performs onboarding a customer and registers an email/api_key * into riak. * Output: FunnelResponse as JSON with the msg. */ object Accounts extends Controller with APIAuthElement { You can see that we are using the APIAuthElement here which means we are indicating that our controller needs to authenticated. Click here for full source code of the [AccountsController](https://github.com/megamsys/megam_gateway/blob/0.7/app/controllers/Accounts.scala). ###POST /accounts Let us hack an action method, that pulls the email and api_key and stores in a storage. How store and manipulate JSON will be dealt in another tutorial. def post = Action(parse.tolerantText) { implicit request => play.api.Logger.debug(("%-20s -->[%s]").format("controllers.Accounts", "post:Entry")) val input = (request.body).toString() play.api.Logger.debug(("%-20s -->[%s]").format("input", input)) models.Accounts.create(input) match { case Success(succ) => PlatformAppPrimer.clone_predefcloud(succ.get.email).flatMap { x => Status(CREATED)( FunnelResponse(CREATED, """Onboard successful. email '%s' and api_key '%s' is registered.""". format(succ.get.email, succ.get.api_key).stripMargin, "Megam::Account").toJson(true)).successNel[Error] } match { case Success(succ_cpc) => succ_cpc case Failure(errcpc) => val rncpc: FunnelResponse = new HttpReturningError(errcpc) Status(rncpc.code)(rncpc.toJson(true)) } PlatformAppPrimer.clone_organizations(succ.get.email).flatMap { x => Status(CREATED)( FunnelResponse(CREATED, """Onboard successful. email '%s' and api_key '%s' is registered.""". format(succ.get.email,succ.get.api_key).stripMargin, "Megam::Account").toJson(true)).successNel[Error] } match { case Success(succ_cpc) => succ_cpc case Failure(errcpc) => val rncpc: FunnelResponse = new HttpReturningError(errcpc) Status(rncpc.code)(rncpc.toJson(true)) } case Failure(err) => { val rn: FunnelResponse = new HttpReturningError(err) Status(rn.code)(rn.toJson(true)) } } } Ofcourse we more more things after an user is registered, those are system specific. For instance we create default settings for an user after getting registered successfuly. A good api, communicates to an user the correct success JSON or an Error After all is well done, we send back a **FunnelResponse**. Stay tuned we will cover it in the next part ### GET /accounts/:id Let us hack an action method, in this case you can see that we are using **StackAction**. You can see that we don't tell what needs to be done to authenticate, it all happens magically with the set framework. def show(id: String) = StackAction(parse.tolerantText) { implicit request => play.api.Logger.debug(("%-20s -->[%s]").format("controllers.Accounts", "show:Entry")) play.api.Logger.debug(("%-20s -->[%s]").format("email", id)) models.Accounts.findByEmail(id) match { case Success(succ) => { Ok((succ.map(s => s.toJson(true))).getOrElse( AccountResult(id).toJson(true))) } case Failure(err) => { val rn: FunnelResponse = new HttpReturningError(err) Status(rn.code)(rn.toJson(true)) } } } Ok. this is the first part of the series, we are done with the authentication. In the subsequent parts we will cover JSON Convertors, FunnelResponse, Cache using StateMonad, IO Monad, Validation of Scalaz. Here is the [full source code](https://github.com/megamsys/megam_gateway.git) of the project.
HTTP Verb REST API Description
GET accounts GET the account information of the user </tr>
POST accounts Posts a new account

Part 1 : Setting up Scala, SBT, Play, Akka

This will be multi-part series in building a cloud API server for our PaaS with cloud instrumentation using a messaging layer.

1. Let us setup SBT 0.13.

Installing in debian

echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
sudo apt-get update
sudo apt-get install sbt

(or)

Manual install of sbt

  • Download the sbt_launch.jar to your ~/bin directory.

  • Create a script named sbt in your ~/bin directory with the following contents.

    java -server -Xms1024M -Xmx3072M -Xss32M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=1024M -XX:+UseG1GC -XX:+AggressiveOpts -XX:SurvivorRatio=128 -XX:MaxTenuringThreshold=0 -jar dirname $0/sbt-launch.jar “$@”

  • Adjust the -Xmx, -Xms -XX:MaxPermSize based on the RAM you have.

At Megam we all have atleast 8GB or > of RAM.

*optional Install sbt-eclipse

Install the scala-ide eclipse plugin for Eclipse Mars.

Help > Install New Software >

Add the correct link from herehttps://scala-ide.org/download/nightly.html

You can install typesafe stack and download templates from here. https://typesafe.com/stack/download#template https://typesafe.com/resources/typesafe-stack/downloading-installing.html

  • Create a new project in eclipse. Scala – Worksheet allows you to play around with REPL inside eclipse editor. or use command line, to use REPL

sbt console

If you only plan to use scala, you are done here. You can tweak your Build.scala to use the scala version you want.

2. Eclipsify scala project

https://github.com/typesafehub/sbteclipse/wiki https://github.com/typesafehub/sbteclipse Create a file .sbt/plugins/plugins.sbt and add the following to include eclipse plugin support.

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.3.0"

If you created a project from command line then run sbt > eclipse to eclipsify a project..

ram@rammegam:~/code/megam/workspace/nilam$ sbt
[info] Loading project definition from /home/ram/code/megam/workspace/nilam/project
[info] Set current project to nilam (in build     file:/home/ram/code/megam/workspace/nilam/)
[nilam] $ eclipse
[info] About to create Eclipse project files for your project(s).
[info] Updating {file:/home/ram/code/megam/workspace/nilam/}nilam...
[info] Resolving org.hibernate.javax.persistence#hibernate-jpa-2.0-api;1.0.1.Fin                                                                                [info] Done updating.
[info] Compiling 6 Scala sources and 1 Java source to /home/ram/code/megam/workspace/nilam/target/scala-2.9.1/classes...
[info] Successfully created Eclipse project files for project(s):
[info] nilam

3. If you with to use akka

A sample akka source code for the Typesafe Stack (https://typesafe.com/stack). https://typesafe.com/resources/tutorials/getting-started-with-akka-scala.html From your akka code type sbt run:

ram@rammegam:~/code/megam/workspace/megam_akka$ sbt run
[info] Loading project definition from /home/ram/code/megam/workspace/megam_akka/project
[info] Set current project to nilam (in build     file:/home/ram/code/megam/workspace/megam_akka/)
[info] Updating {file:/home/ram/code/megam/workspace/megam_akka/}nilam...
[info] Resolving com.typesafe#config;0.3.1 ...
[info] downloading https://repo.typesafe.com/typesafe/releases/com/typesafe/akka/akka-actor/2.0.3/akka-actor-2.0.3.jar ...
[info]     [SUCCESSFUL ] com.typesafe.akka#akka-actor;2.0.3!akka-actor.jar (9117ms)
[info] downloading https://repo.typesafe.com/typesafe/releases/com/typesafe/config/0.3.1/config-0.3.1.jar ...
[info]     [SUCCESSFUL ] com.typesafe#config;0.3.1!config.jar(bundle) (3345ms)
[info] Done updating.
[info] Compiling 1 Scala source to /home/ram/code/megam/workspace/megam_akka/target/scala-2.9.2/classes...
[info] Running org.megam.akka.Nilam
Count is 3
[success] Total time: 25 s, completed 6 Nov, 2012 12:31:57 PM

4. Setting up Play framework

Let us create a project titled megam_play which will use external dependencies like scaliak, newman, scalaz This project megam_play was created by cloning an existing typesafe_stack (play scala sample template). You are free to pick any other template from Github.com Adding library dependencies in your Build.scala file. The scaliak, amqp dependency has been moved to a common artifact named “megam_common”

import sbt._
import play.Project._

object ApplicationBuild extends Build {

 val appName = "megam_play"

  val appVersion = "0.1.0"

  val organization = "Megam Systems"

  val homepage = Some(url("https://www.megam.co"))

  val startYear = Some(2013)

  val description = "RESTful API server for the megam platform. Uses Riak and Memcache"

  /**
   *   if you use groupID %% artifactID % revision instead of groupID % artifactID % revision
   *   (the difference is the double %% after the groupID), sbt will add your project’s Scala version
   *   to the artifact name.
   */

  val play2AuthVersion = "0.11.0-SNAPSHOT"
  val megamVersion = "0.1.0-SNAPSHOT"

  val appDependencies = Seq(
    javaCore, cache,javaEbean,
    "com.twitter.service" % "snowflake" % "1.0.2" from "https://s3-ap-southeast-1.amazonaws.com/megampub/0.1/jars/snowflake.jar", //don't move this below.
"com.github.indykish" % "megam_common_2.10" % megamVersion excludeAll (
  ExclusionRule("commons-logging","commons-logging"),
  ExclusionRule("org.slf4j","slf4j-jdk14")),
"com.github.mumoshu" %% "play2-memcached" % "0.3.0.2",
"jp.t2v" %% "play2-auth" % play2AuthVersion,
"jp.t2v" %% "play2-auth-test" % play2AuthVersion % "test",
"com.stackmob" %% "newman" % "1.3.0" % "test")

  val main = play.Project(appName, appVersion, appDependencies).settings(
sbt.Keys.resolvers += "Sonatype Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots",
sbt.Keys.resolvers += "Typesafe Snapshots" at "https://repo.typesafe.com/typesafe/snapshots/",
sbt.Keys.resolvers += "Scala-Tools Maven2 Snapshots Repository" at "https://scala-tools.org/repo-snapshots",
sbt.Keys.resolvers += "Twitter Repo" at "https://maven.twttr.com", // finagle
sbt.Keys.resolvers += "spray repo" at "https://repo.spray.io", //spray client used in newman.
sbt.Keys.resolvers += "spray nightly" at "https://nightlies.spray.io", //spray client nighly used in newman (0.23.0).
sbt.Keys.resolvers += "Spy Repository" at "https://files.couchbase.com/maven2" // required to resolve `spymemcached`, the plugin's dependency.
)

}

We package debs, hence build.sbt contains the required statements

import sbt
import Process._
import com.typesafe.sbt.packager.debian.Keys._
import    com.typesafe.sbt.packager.linux.LinuxPackageMapping
import S3._

scalaVersion := "2.10.3"

scalacOptions := Seq(
  "-unchecked",
  "-deprecation",
  "-feature",
  "-optimise",
"-Xcheckinit",
"-Xlint",
"-Xverify",
"-Yinline-warnings",
"-Yclosure-elim",
"-language:postfixOps",
"-language:implicitConversions",
"-Ydead-code")

com.typesafe.sbt.packager.debian.Keys.name in Debian := "megamplay"

maintainer in Debian:= "Rajthilak <rajthilak@megam.co.in>"

packageSummary := "Cloud API server - Megam."

packageDescription in Debian:= " (REST based) Cloud API server for Megam platform.The API server protects the resources using HMAC based authorization, as provided to a customer during onboarding."

s3Settings

linuxPackageMappings in Debian <+= (baseDirectory) map { bd =>
  (packageMapping((bd / "bin/mp") -> "/usr/local/share/megamplay/bin/mp")
   withUser "root" withGroup "root" withPerms "0755")
}

linuxPackageMappings <+= (baseDirectory) map { bd =>
  val src = bd / "target/staged"
 val dest = "/usr/local/share/megamplay/lib"
LinuxPackageMapping(
for {
  path <- (src ***).get
  if !path.isDirectory
} yield path -> path.toString.replaceFirst(src.toString, dest)
)
}

linuxPackageMappings in Debian <+= (baseDirectory) map { bd =>
  (packageMapping((bd / "conf/application-production.conf") -> "/usr/local/share/megamplay/conf/application-production.conf")
withConfig())
}

linuxPackageMappings in Debian <+= (baseDirectory) map { bd =>
  (packageMapping((bd / "conf/application-logger.xml") -> "/usr/local/share/megamplay/conf/application-logger.xml")
withConfig())
}

linuxPackageMappings in Debian <+= (baseDirectory) map { bd =>
(packageMapping((bd / "conf/routes") -> "/usr/local/share/megamplay/conf/routes")
withConfig())
}

linuxPackageMappings in Debian <+= (baseDirectory) map { bd =>
(packageMapping((bd / "conf/messages") -> "/usr/local/share/megamplay/conf/messages")
withConfig())
}

linuxPackageMappings in Debian <+= (baseDirectory) map { bd =>
(packageMapping((bd / "conf/play.plugins") -> "/usr/local/share/megamplay/conf/play.plugins")
withConfig())
}

com.typesafe.sbt.packager.debian.Keys.version in Debian <<= (com.typesafe.sbt.packager.debian.Keys.version, sbtVersion) apply { (v, sv) =>
sv + "-build-" + (v split "\." map (_.toInt) dropWhile (_ == 0) map ("%02d" format _) mkString "")
}

debianPackageDependencies in Debian ++= Seq("curl", "java2-runtime", "bash (>= 2.05a-11)")

debianPackageRecommends in Debian += "riak"

linuxPackageMappings <+= (baseDirectory) map { bd =>
packageMapping(
(bd / "copyright") -> "/usr/share/doc/megam_play/copyright"
) withPerms "0644" asDocs()
}

linuxPackageMappings in Debian <+= (com.typesafe.sbt.packager.debian.Keys.sourceDirectory) map { bd =>
(packageMapping(
(bd / "debian/changelog") -> "/usr/share/doc/megam_play/changelog.gz"
) withUser "root" withGroup "root" withPerms "0644" gzipped) asDocs()
}

mappings in upload := Seq((new java.io.File(("%s-%s.deb") format("target/megamplay", "0.12.4-build-0100")),"0.1/debs/megam_play.deb"))

host in upload := "megampub.s3.amazonaws.com"

credentials += Credentials(Path.userHome / "software" / "aws" / "keys" / "sbt_s3_keys")

S3.progress in S3.upload := true

1.Plugins.sb

// Comment to get more information during initialization
logLevel := Level.Warn

// The Typesafe repository
resolvers += "Typesafe repository" at     "https://repo.typesafe.com/typesafe/releases/"

// Typesafe snapshots
resolvers += "Typesafe Snapshots" at    "https://repo.typesafe.com/typesafe/snapshots/"

// Use the Play sbt plugin for Play projects

addSbtPlugin("play" % "sbt-plugin" % "2.2.0")

project/build.properties

sbt.version=0.13.0

We built our own AMQP based on scalaz, and is available in https://github.com/indykish/megam_common. Now build and publish megam_common jars using the process as described in this link into https://oss.sonatype.org/content/repositories/snapshots/com/github/indykish/megam_common_2.10/0.1.0-SNAPSHOT/.

sbt publish

Run the play project megam_play

ram@ramhome:~/code/megam/workspace/megam_play$ sbt
[info] Loading global plugins from /home/ram/.sbt/0.13/plugins
[info] Loading project definition from /home/ram/code/megam/workspace/megam_play/project
[info] Set current project to megam_play (in build     file:/home/ram/code/megam/workspace/megam_play/)
[megam_play] $ play
       _
  _ __ | | __ _ _  _
 | '_ | |/ _' | || |
 |  __/|_|___|__ /
 |_|          |__/

play 2.2.0 built with Scala 2.10.2 (running Java 1.7.0_25), https://www.playframework.com

> Type "help play" or "license" for more information.
> Type "exit" or use Ctrl+D to leave this console.

[megam_play] $

You can use lsof as shown below to see if the server runs on port 9000.

[nilam] $ run

$ sudo lsof -i:9000
[sudo] password for ram:
COMMAND   PID USER   FD   TYPE DEVICE SIZE NODE NAME
java    15640  ram  270u  IPv6  40094       TCP *:9000 (LISTEN)

Now let us add a template to show the

1. index page,
2. Error when an invalid URL is passed

####index.scala.html

@(message: String)(implicit flash: play.api.mvc.Flash)
@main("Megam Cloud v1") {
@welcome(message)
}

####main.scala.html

@(title: String)(content: Html)
<!DOCTYPE html>
<html>
<head>
<title>@title</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="">
<meta name="author" content="">
<link rel="stylesheet" type="text/css" href="https://fonts.googleapis.com/css?family=Lato:300,400,700" />
<link rel="stylesheet" type="text/css" href="@routes.Assets.at("stylesheets/theme.css")" >
<link rel="stylesheet" type="text/css" href="@routes.Assets.at("stylesheets/bootstrap-theme.css")" >
<link rel="stylesheet" type="text/css" href="@routes.Assets.at("stylesheets/bootstrap.css")" >
<link rel="stylesheet" type="text/css" href="@routes.Assets.at("stylesheets/bootstrap-glyphicons.css")">
<link rel="stylesheet" href="styles/default.css">
<link rel="stylesheet" type="text/css" href="@routes.Assets.at("stylesheets/styles/default.css")" >
<link rel="shortcut icon" type="image/png" href="@routes.Assets.at("images/favicon.png")">

<!-- Bootstrap -->
<script src="@routes.Assets.at("javascripts/jquery.js")" type="text/javascript"></script>
<script src="@routes.Assets.at("javascripts/bootstrap.js")" type="text/javascript"></script>
<script  src="@routes.Assets.at("javascripts/highlight.pack.js")" type="text/javascript"></script>

<script type="text/javascript">
        $(document).ready(function() {
            hljs.initHighlightingOnLoad();
            $('pre code').each(function(i, e) {
                hljs.highlightBlock(e)
            });
            $('a[data-toggle="tab"]').on('shown.bs.tab', function(e) {
                e.target// activated tab
                e.relatedTarget // previous tab
            });

        });
</script>
<style type="text/css">
  /* <![CDATA[ */
  #pagetoc li {
    padding-left: 22px;
    text-indent: -22px;
  }

  /* ]]> */
</style>
  </head>
   <body>
   <div>
    <div>
        @leftmenu()
        @content
    </div>
  </div>

<!-- JQuery will invoke via the spinner and display a loading modal lock. -->
<div id="loading" />

</body> </html>

####errorPage.scala.html

@(ex: Throwable)

@main("An error occurred while trying to run megam_play. contact support.") {
<div class="api-doc">
  <div class="mddoc margin-top-large">
   <image src="@routes.Assets.at("images/nirvana.png")"/>
<h1><a name="rest_docs" href="#a-rest_docs">Sucked into Nirvana</a></h1>
<p class="lead">
  <code>
    @(ex getMessage)
  </code>
</p>
<div class="page-header">
  <h2>Stacktrace</h2>
</div>

 <div class="">


<code class="brush: bash; toolbar:     false;">@({val u = new java.io.StringWriter; ex.printStackTrace(new java.io.PrintWriter(u)); u.toString})
    </code>
  </div>
 </div>
</div>
}


object Global extends GlobalSettings {

 override def onStart(app: Application) {
play.api.Logger.info("Megam Play %s App - started".format("0.1"))
}

override def onStop(app: Application) {
play.api.Logger.info("Megam Play %s App - going down. Stateless folks - you don't care.".format("0.1"))
}

override def onError(request: RequestHeader, ex: Throwable) : Future[play.api.mvc.SimpleResult] = {
Future.successful(InternalServerError(
  views.html.errorPage(ex)))
}

override def onHandlerNotFound(request: RequestHeader): Future[play.api.mvc.SimpleResult] = {
Future.successful(NotFound(
  views.html.errorPage(new Throwable(NOT_FOUND + ":" + request.path + " NOT_FOUND"))))
 }
}

Thee view files are placed under views directory.

Type https://localhost:9000

Now that you know the little gists on creating a forking and cloning a repository,let’s move on and learn about how to contribute to a Git repository. A great way to get involved in open source is to contribute to the existing projects you’re using. GitHub is home to more than 5 million open source projects. Say you want to contribute changes to someone else’s repository? You must first fork the repository. We have seen all about Forking a repository in our previous post. so,moving on. Next,send a pull request. What is a pull request?

PULL REQUESTS:

Pull requests let you tell others about changes you’ve pushed to a repository on GitHub. Once a pull request is sent, interested parties can review the set of changes, discuss potential modifications, and even push follow-up commits if necessary.

Ah, now that’s interesting. Wondering how do you initiate one? It’s easy. Follow these steps.

Initiating a Pull Request:

1.Navigate to your repository with the changes you want someone else to pull and press the Pull Request button.

2.Switch to your branch.

3.Click the Compare & review button.

Pull requests can be sent from any branch or commit but it’s recommended that a topic branch be used so that follow-up commits can be pushed to update the pull request if necessary.

Now..what is this branch?

WHAT IS A BRANCH?

-Branches allow you to build new features or test out ideas without putting your main project at risk.

-The main branch of a repository is usually named master, and represents a relatively stable version of the project you’re working on. So far, all the changes you’ve made have been on the master branch.

-If you’re making an app or website, for example, you might have a bunch of different features or ideas in progress at any given time – some of which are ready to go, and others which are not. For this reason, master exists as a central point to fold other branches of work into.

How to create a branch:

Branch management is an important part of the Git workflow. You can manage branches directly on GitHub. You can create a new branch in a repository’s branch selector menu. Just start typing the name of your branch; if it doesn’t exist, GitHub will offer to create it for you.

Reviewing the pull request :

After starting the review, you’re presented with a review page where you can get a high-level overview of what exactly has changed between your branch and the repository’s master branch. You can review all comments made on commits, identify which files changed, and get a list of contributors to your branch.

Changing the branch range and destination repository :

-By default, pull requests are assumed to be based on the parent repository’s default branch. In many cases, the defaults are appropriate. If necessary, you can change the parent repository and branch with the drop-down lists. Clicking on Edit at the top allows you to swap between your head and base, as well as establishing differences between various reference points. References here must be branch names in your GitHub repositor

-The easiest way of thinking about the branch range is this: the base branch is where you think changes should be applied, the head branch is what you would like to be applied.

-Changing the base repository changes who is notified of the pull request. Everyone that can push to the base repository will receive an email notification and see the new pull request in their dashboard the next time they sign in.

-When you change any of the info in the branch range, the commit and files changed preview areas will update to show your new range.

SENDING THE PULL REQUEST :
STEPS:

STEP 1:When you’re ready to submit your pull request, click Create pull request.

STEP 2: You’ll be taken to a discussion page where you can enter a title and optional description. You’ll still be able to see exactly which commits will be included when the pull request is sent.

STEP 3:Once you’ve entered the title and description, made any necessary customizations to the commit range, and reviewed the commits and file changes to be sent, press the Create pull request button.

STEP 4: The pull request is sent immediately. You’re taken to the main pull request discussion and review page.

After your pull request is sent, any new commits pushed to your branch will automatically be added to the pull request. This is especially useful if you need to make more changes.

MANAGING PULL REQUESTS :

-All pull requests sent or received by you are browsable through the pull request dashboard. Pull requests for a specific repository are also browsable by anyone with access by visiting the Pull Requests page.

-The pull request dashboard and the repository pull request list support a wide range of filtering and sorting controls. Use them to narrow down the list to the pull requests you’re interested in.

####### REVIEWING PROPOSED CHANGES :

-When you receive a pull request, the first thing to do is review the set of proposed changes. Pull requests are tightly integrated with the underlying git repository, so you can see exactly what commits would be merged should the request be accepted.

-You can also review the cumulative differences of all file changes across all commits, either split or unified.

MERGING A PULL REQUEST:

-Merge a pull request into the upstream branch when work is completed. Anyone with push access to the repository can complete the merge. -If you decide you don’t want the changes in your branch to be merged to the upstream branch, you can also close the pull request without merging.

####### MERGING A PULL REQUEST USING THE GITHUB WEB INTERFACE :

-If the merge will not have any conflicts, you can merge the pull request online. Follow these steps:

STEP 1: In any repository’s right sidebar, click Pull Requests.

STEP 2: In the “Pull Requests” list, click the pull request you’d like to merge.

STEP 3: Click Merge pull request.

STEP 4: Type a commit message, or accept the default message.

STEP 5: Under the commit message box, click Confirm merge.

Optionally, delete the branch. This keeps the list of branches in your repository tidy.

Reverting a pull request :

You can revert a pull request after it’s been merged to the upstream branch.

Reverting a pull request on GitHub creates a new pull request that contains one revert of the merge commit from the original merged pull request. Follow these steps to revert a pull request:

STEP 1: In your repository’s right sidebar, click Pull Requests.

STEP 2: In the “Pull Requests” list, click the pull request you’d like to revert.

STEP 3: Near the bottom of the pull request, click Revert.

STEP 4: Merge the resulting pull request.

Closing a pull request

You may choose to close a pull request without merging it into the upstream branch. This can be handy if the changes proposed in the branch are no longer needed, or if another solution has been proposed in another branch. Follow these steps to close a pull request:

STEP 1:In any repository’s right sidebar, click Pull Requests.

STEP 2:In the “Pull Requests” list, click the pull request you’d like to close.

STEP 3:At the bottom of the pull request, below the comment box, click Close pull request. Optionally, delete the branch. This keeps the list of branches in your repository tidy.

You can also choose to delete the delete unwanted branches.

DELETING UNUSED BRANCHES:

-After you merge or close a pull request, you can delete the branch. This keeps the list of branches for your repository as clean and useful as possible.

-At the bottom of a merged or closed pull request, you’ll see a button to delete the lingering branch

-You can also delete branches from the Branches page.

Easy,eh? Go ahead and get forking!

Introduction

Go, also commonly referred to as golang, is a programming language initially developed at Google in 2007 by Robert Griesemer, Rob Pike, and Ken Thompson.

Go is:

• open source

• concurrent

• garbage-collected

• efficient

• scalable

• simple

• fun

• boring (to some)

https://golang.org

Setup

There are many ways to configure the Go development environment on your computer, you can choose any one you like. But i suggest following ways.

Way 1 : Install Go Package

The golang Debian package may have already made its way into your Ubuntu distribution. Try this:

$ sudo apt-get install golang

export the settings you’re gonna need to ~/.bashrc file:

$ export GOROOT=/usr/lib/go

$ export GOBIN=/usr/bin/go

Way 2 : From Binary

Download golang 1.4+ amd64 linux, create a ~/golang directory, and untar into that directory.

$ mkdir ~/golang

$ cd ~/golang

$ wget https://storage.googleapis.com/golang/go1.4.2.linux-amd64.tar.gz

$ tar -xzf go1.4.2.linux-amd64.tar.gz

Now setup Go binary path. Use the $GOROOT and $PATH environment variables. In order to make these variables persist through logins/reboots/etc, add the following lines to ~/.bashrc file:

$ export GOROOT=$HOME/golang/go

$ export PATH=$PATH:$GOROOT/bin

Way 3 : From Source

Go will install to a directory named go. Change to the directory that will be its parent and make sure the go directory does not exist. Then clone the repository and check out the latest release tag:

$ git clone https://go.googlesource.com/go

$ cd go

$ git checkout go1.4.2

(Optional) Switch to the master branch

If you intend to modify the go source code, and contribute your changes to the project, then move your repository off the release branch, and onto the master (development) branch. Otherwise, skip this step.

$ git checkout master

To build the Go distribution, run

$ cd go/src

$ ./all.bash

Finally set the environment variables on ~/.bashrc file

$ export GOROOT=$HOME/golang/go

$ export PATH=$PATH:$GOROOT/bin

Test your installation

Try this,

$ go version

go version go1.4.2 linux/amd64

Set up your work environment

Introducing workspaces

Your Go code is kept in a workspace. A workspace contains many source repositories (git, hg). The Go tool understands the layout of a workspace. You don’t need a Makefile. A workspace is a directory hierarchy with three directories at its root:

src contains Go source files organized into packages (one package per directory),

pkg contains package objects, and

bin contains executable commands.

The go tool builds source packages and installs the resulting binaries to the pkg and bin directories.

The src subdirectory typically contains multiple version control repositories (such as for Git or Mercurial) that track the development of one or more source packages.

To give you an idea of how a workspace looks in practice, here’s an example:

bin/
        hello                   # command executable
        outyet                  # command executable
pkg/
        linux_amd64/
                github.com/golang/example/
                        stringutil.a           # package object
src/
        github.com/golang/example/
                .git/             # Git repository metadata
                hello/
                    hello.go      # command source
                outyet/
                    main.go       # command source
                    main_test.go  # test source
                stringutil/
                    reverse.go    # package source
                    reverse_test.go   # test source

This workspace contains one repository (example) comprising two commands (hello and outyet) and one library (stringutil).

A typical workspace would contain many source repositories containing many packages and commands. Most Go programmers keep all their Go source code and dependencies in a single workspace.

Go commands all rely on one important environment variable which is called $GOPATH. Notice that this is not the $GOROOT where Go is installed. This variable points to the workspace of Go in your computer.

The GOPATH environment variable

The GOPATH environment variable specifies the location of your workspace. It is likely the only environment variable you’ll need to set when developing Go code.

To get started, create a workspace directory and set GOPATH accordingly. Your workspace can be located wherever you like, but we’ll use $HOME/go in this document. Note that this must not be the same path as your Go installation.

$ mkdir $HOME/go

$ export GOPATH=$HOME/go

For convenience, add the workspace’s bin subdirectory to your PATH:

$ export PATH=$PATH:$GOPATH/bin

Testing with Workspace

To compile and run a simple program, first choose a package path (we’ll use github.com/user/hello) and create a corresponding package directory inside your workspace:

$ mkdir $GOPATH/src/github.com/user/hello

Next, create a file named hello.go inside that directory, containing the following Go code.

package main
import "fmt"
func main() {
	fmt.Printf("Hello, world.\n")
}

Now you can build and install that program with the go tool:

$ go install github.com/user/hello

Note that you can run this command from anywhere on your system. The go tool finds the source code by looking for the github.com/user/hello package inside the workspace specified by GOPATH.

You can also omit the package path if you run go install from the package directory:

$ cd $GOPATH/src/github.com/user/hello

$ go install

This command builds the hello command, producing an executable binary. It then installs that binary to the workspace’s bin directory as hello (or, under Windows, hello.exe). In our example, that will be $GOPATH/bin/hello, which is $HOME/go/bin/hello.

The go tool will only print output when an error occurs, so if these commands produce no output they have executed successfully.

You can now run the program by typing its full path at the command line:

$ $GOPATH/bin/hello

Hello, world.

If you see the “Hello, world” message then your Go installation is working