Introduction

Debian is a Unix-like computer operating system that is composed entirely of free software, most of which is under the GNU General Public License, and packaged by a group of individuals known as the Debian. Three main branches are offered: Stable, Testing and Unstable.

This tutorial will guide you in launching Debian Jessie in VirtEngine.

###Prerequisites

• You are running Ubuntu 14.04 or Linux workstation.

• Git installed on your server, which you can do by following the How To Install Git with Apt.

• An account on GitHub, which is a Git repository host. To follow this tutorial :

• You have to create a valid credential for accessing https://console.VirtEngine.com. How to create an account with VirtEngine

You have to install openssh-server for ssh access.

$ sudo apt-get install openssh-server

To check the ssh is properly installed in our system

$ ps aux | grep sshd

###Step - 1 Creating Debian Jessie VM

This initial section contains everything you need to get Debian Jessie and running on your server.

First, ensure the user can login to https://console.VirtEngine.com.

• Go to the Market Place.

• Select the Debian, A window will pop up with for CPU, storage, RAM and SSHkey options.

• You can choose the storage size and RAM capacity.

• You can create a new sshkey, use an existing sshkey or import your own sshkeys too.

• Click the create button. it will create the virtual machine.

###Step - 2 Access the Debian Jessie VM Machine

Next, Go to the Dashboard and click the domain name a new window will open.

• It contains the CPU, RAM and Network tab.

• It shows the Metrics, VM Logs, IP address and SSH URL.

• Metrics shows the CPU,RAM and Network usage.

• VM Logs shows all the running process in VM.

• You need to access the Virtual Machine from a terminal.

• You can download the SSH Keys from SSH Keys tab or Overview page. Use this key to login to your virtual machine using the following command,

    $ ssh -i path to/<private_key filename> root@<ipaddress>
  

###Conclusion

These are the very simple steps to launch Debian Jessie virtual machine. This is a good head-start for launching a Debian Jessie in VirtEngine.

###Deploy debian jessie now

Introduction

The CentOS Linux distribution is a stable, predictable, manageable and reproducible platform derived from the sources of Red Hat Enterprise Linux (RHEL). CentOS Linux is developed by a small but growing team of core developers. In turn the core developers are supported by an active user community including system administrators, network administrators, managers, core Linux contributors, and Linux enthusiasts from around the world.

This tutorial will guide you in setting up a CentOS VM in VirtEngine.

###Prerequisites

• You are running Ubuntu 14.04 or Linux workstation.

• Git installed on your server, which you can do by following the How To Install Git with Apt.

• An account on GitHub, which is a Git repository host. To follow this tutorial :

• You have to create a valid credential for accessing https://console.VirtEngine.com. How to create an account with VirtEngine

You have to install openssh-server for ssh access.

$ sudo yum install openssh-server

To check the ssh is properly installed in our system

$ ps aux | grep sshd

###Step - 1 Creating CentOS VM

This initial section contains everything you need to get CentOS and running on your server.

First, ensure the user can login to https://console.VirtEngine.com.

• Go to the Market Places.

• Select the CentOS, A window will pop up with for CPU, storage, RAM and SSHkey options.

• You can choose the storage size, RAM capacity.

• You can Create a new sshkey, use an existing sshkey or import your own sshkeys too.

• click the Create button. it will create the virtual machine.

###Step - 2 Access the CentOS VM

Next, Go to the Dashboard and click the domain name a new window will open.

• It contains the CPU, RAM and Network tab.

• It shows the Metrics, VM Logs, IP address and SSH URL.

• Metrics shows the CPU,RAM and Network usage.

• VM Logs shows all the running process in VM.

• You need to access the Virtual Machine from a terminal.

• You can download the SSH Keys from SSH Keys tab or Overview page. Use this key to login to your virtual machine using the following command,

    $ ssh -i path to/<private_key filename> root@<ipaddress>
  

###Conclusion

These are the very simple steps to launch CentOS in virtual machine. This is a good head-start for launching CentOS in VirtEngine.

###Deploy your CentOS VM now

---

title: Building your own VM Image (Dockerfile for VMs) layout: post og_image_url: “https://blog.virtengine.com/res/gotalk-intro.png” description: Building your own VM Image (Dockerfile for VMs) —

Introduction

Lets the system have installed centos operating system.In my purpose, when preparing an OpenNebula installation is the creation of Virtual Machine images for base Operating Systems or appliances.

Some of these images can be downloaded from the marketplace but you may need an OS that is not in the marketplace or the images must be customized in some other way.

Now i am going to describe an automated way to customize the base images.It provided by linux distribution using software tool libguestfs.

Libguestfs

This tool can be used to create and modify the virtual machine images in number of format that qemu understands.Some of these utilities let us add or delete files inside the images or execute scripts using the image filesystem as root.

To install libguestfs,

   yum install libguestfs-tools

Base image

Next step download the base Isoimage like ubuntu

wget https://cloud-images.ubuntu.com/releases/14.04/release/ubuntu-14.04-server-cloudimg-amd64-disk1.img

Install Onecontext

One of the customizations we have to do to this image is uninstall the cloud-init package that comes by default with that image and install OpenNebula context package.

wget https://github.com/OpenNebula/addon-context-linux/releases/download/v4.14.4/one-context_4.14.4.rpm

To create the CDROM image we can use genisoimage. Remember to add a label so it’s easier to mount. Here we are going to use the label PACKAGES:

i) Copy the onecontext packages to a directory, for example packages

ii) Execute genisoimage to create the iso that contains those files:

$ genisoimage -o packages.iso -R -J -V PACKAGES packages/

Now we need to prepare a script with the customizations to be done in the image. For example:

mount LABEL=PACKAGES /mnt

# Install opennebula context package
 apt-get install -y unzip
 unzip /mnt/v4.14.4*zip

# Install growpart and upgrade util-linux, used for filesystem resizing
apt-get install -y epel-release --nogpgcheck
apt-get install -y cloud-utils-growpart --nogpgcheck
apt-get upgrade -y util-linux --nogpgcheck

#nginx install
 apt-get -y update
apt-get install -y nginx
ceph_user="megam"
ceph_password="1234pass"
ceph_group="megam"
user_home="/home/megam"
if ! getent group $ceph_group > /dev/null 2>&1;  then
groupadd --system $ceph_group
fi
if ! getent passwd $ceph_user > /dev/null 2>&1; then
useradd -d $user_home -m -g $ceph_group $ceph_user -s /bin/bash
    #Set passwordsudo echo -e "$ceph_password\n$ceph_password\n" | sudo passwd $ceph_user
else
   user_home=`getent passwd $ceph_user | cut -f6 -d:`

# Renable user (give him a shell)
usermod --shell /bin/bash $ceph_user
# Make sure MEGAMHOME exists, might have been removed on previous purge
mkdir -p $user_home
fi

Instead of modifying the original image downloaded we can use a feature of qcow2 image that is creating a new image that is based on another one.

To copy the image from original image

$ qemu-img create -f qcow2 -b ubuntu-14.04-server-cloudimg-amd64-disk1.img ubuntu.qcow2

Now all is prepared to customize the image. The command we are going to use is virt-customize.

It can do a lot of modifications to the image but we are only going to do two. The command is this one:

virt-customize -v -x --attach packages.iso --format qcow2 -a ubuntu.qcow2 --run script.sh --root-password password:centos

It attaches two images, the iso image with the packages and the OS hard disk, executes script.sh

After the command is run the image centos.qcow2 contains the modifications we did to the original image.

Now we can convert it to any other format we need (for example vmdk) or to a full qcow2 image, that it, does not depend on any other one.

 qemu-img convert -f qcow2 -O qcow2 -o compat=0.10 ubuntu.qcow2 ubuntu-final.qcow2

qemu-img convert -f qcow2 -O vmdk ubuntu.qcow2 ubuntu-final.vmdk

Our need is to setup opennebula in packet.net, so we have a server for both opennebula-frontend and opennebula-host.

We have a local working server for opennbula-host with openvswitch. But in packet.net we tried normal linux bridge(brctl) and we face some problem in creating virtual machines, because of networking.

OS 			   : Ubuntu-14.04(trusty)
Host IP 	   : 192.168.1.100
Additional IPs : 192.168.2.96/29

NOTE: Actually we have public ips, but for security issues i documented with local ips. Usually, when we create a server in packet.net, they will give you a single IP, then we need to request them for additional IPs.

After trying many configuration, we succeed with the following configurations,

NOTE: Use chattr -i /etc/network/interfaces to make the file writable.

My Host server network configuration

#cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto bond0
iface bond0 inet static
   address 192.168.1.100
   netmask 255.255.255.254
   bond-slaves eth1 eth0
   gateway 192.168.1.99
   bond-lacp-rate 1
   bond-xmit_hash_policy layer3+4
   bond-downdelay 200
   bond-updelay 200
   bond-miimon 100
   bond-mode 4
iface bond0 inet6 static
   address 2604:xxxx:x:xxxx::x
   netmask 127
   gateway 2604:xxxx:x:xxxx::
auto bond0:0
iface bond0:0 inet static
   address 10.99.18.1
   netmask 255.255.255.254
auto eth1
iface eth1 inet manual
   pre-up sleep 4
   bond-master bond0
	auto eth0
iface eth0 inet manual
   bond-master bond0
   post-up route add -net 10.0.0.0/8 gw 10.99.18.0
   post-down route del -net 10.0.0.0/8 gw 10.99.18.0

auto br0
iface br0 inet static
  address 192.168.2.97
  netmask 255.255.255.248
  bridge_ports none
  bridge_stp off
  bridge_fd 1
  bridge_hello 2
  bridge_maxage 12

Routing table for host server

root@5:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.1.99    0.0.0.0         UG    0      0        0 bond0
10.0.0.0        10.99.18.0      255.0.0.0       UG    0      0        0 bond0
10.99.18.0      *               255.255.255.254 U     0      0        0 bond0
192.168.1.99    *               255.255.255.254 U     0      0        0 bond0
192.168.2.96    *               255.255.255.248 U     0      0        0 br0

IP routes

root@5:~# ip route show
default via 192.168.1.99 dev bond0
10.0.0.0/8 via 10.99.18.0 dev bond0
10.99.18.0/31 dev bond0  proto kernel  scope link  src 10.99.18.1
192.168.1.99/31 dev bond0  proto kernel  scope link  src 192.168.1.100
192.168.2.96/29 dev br0  proto kernel  scope link  src 192.168.2.97

open /etc/sysctl.conf and uncomment these lines

net.ipv4.conf.all.rp_filter=1
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

Delete default bridge virbr0

$ virsh net-destroy default
$ virsh net-undefine default
$ service libvirtd restart

After launching vm, our linux bridge status

root@5:~# brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.fe00934bc462	no		vnet0

Host network

root@5:~# ifconfig
bond0     Link encap:Ethernet  HWaddr e4:1d:2d:54:24:78  
          inet addr:192.168.1.100  Bcast:255.255.255.255  Mask:255.255.255.254
          inet6 addr: xxxx:xxxx:x:xxxx::x/127 Scope:Global
          inet6 addr: fe80::e61d:2dff:fe54:2478/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:844156 errors:0 dropped:2 overruns:0 frame:0
          TX packets:798032 errors:0 dropped:7 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:106510529 (106.5 MB)  TX bytes:105652129 (105.6 MB)

bond0:0   Link encap:Ethernet  HWaddr e4:1d:2d:54:24:78  
          inet addr:10.99.18.1  Bcast:255.255.255.255  Mask:255.255.255.254
    	  UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1

br0       Link encap:Ethernet  HWaddr fe:00:93:4b:c4:62  
          inet addr:192.168.2.97  Bcast:192.168.2.103  Mask:255.255.255.248
          inet6 addr: xxxx::xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:18701 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23053 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2995153 (2.9 MB)  TX bytes:7377649 (7.3 MB)

eth0      Link encap:Ethernet  HWaddr e4:1d:2d:54:24:78  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:395599 errors:0 dropped:1 overruns:0 frame:0
          TX packets:393086 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:50710378 (50.7 MB)  TX bytes:52152210 (52.1 MB)

eth1      Link encap:Ethernet  HWaddr e4:1d:2d:54:24:78  
          inet6 addr: xxxx::xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:448557 errors:0 dropped:1 overruns:0 frame:0
          TX packets:404946 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:55800151 (55.8 MB)  TX bytes:53499919 (53.4 MB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:7881551 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7881551 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:8125571309 (8.1 GB)  TX bytes:8125571309 (8.1 GB)

vnet0     Link encap:Ethernet  HWaddr fe:00:93:4b:c4:62  
          inet6 addr: xxxx::xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      	  RX packets:18428 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21459 errors:0 dropped:0 overruns:0 carrier:0
      	  collisions:0 txqueuelen:500
          RX bytes:3225260 (3.2 MB)  TX bytes:7307291 (7.3 MB)

Thats it for opennebula host server’s network configuration.

Now launch a vm from opennebula

VM’s network configuration

root@t2:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address 192.168.2.98
  network 192.168.2.96
  netmask 255.255.255.248
  gateway 192.168.2.97

VM’s routing table

root@t2:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.2.97   0.0.0.0         UG    0      0        0 eth0
192.168.2.96   *               255.255.255.248 U     0      0        0 eth0

ip route

root@t2:~# ip route show
default via 192.168.2.97 dev eth0
192.168.2.96/29 dev eth0  proto kernel  scope link  src 192.168.2.98

Now i can connect with my vm from anywhere.

Introduction

Spark-notebook allows performing reproducible with scala,Apache Spark and more. This is achieved through an interactive web-based editor that can combine Scala code, SQL queries, Markup or even JavaScript in a collaborative manner.

The Spark is available out of the box, and is simply accessed by the variable sparkContext.Spark Notebook offers these capabilities to anybody who needs to play with data, leveraging not only Spark for all data manipulation, but also the Typesafe Reactive Platform, to offer unique power to the user.

Installation

To get debian package install spark-notebook

wget https://s3.eu-central-1.amazonaws.com/spark-notebook/zip/spark-notebook-0.6.0-scala-2.10.4-spark-1.4.1-hadoop-1.0.4.zip

Unzip the package in spark-notebook

unzip spark-notebook-0.6.0-scala-2.10.4-spark-1.4.1-hadoop-1.0.4.zip

To rename directory spark-notebook-0.6.0-scala-2.10.4-spark-1.4.1-hadoop-1.0.4 into spark-notebook

mv spark-notebook-0.6.0-scala-2.10.4-spark-1.4.1-hadoop-1.0.4 spark-notebook

Running Spark-notebook

To run the spark-notebook cd into spark-notebook

./bin/spark-notebook

When the server has been started, fire up your browser and point it to localhost:9000 and you’ll see something similar to: Notebook list

From there you can either: create a new notebook or launch an existing notebook.

In UI, You can see the following menu files, Running, clusters and New. To create a new notebook click New option.

Click cluster tab, to create spark cluster open in another window. To write a program in the cell and click Run button. For example in the cell

In[]: import org.apache.spark._
      import org.apache.spark.SparkContext._
      import org.apache.spark.rdd._

Click run button.You will see the output in the cell.

Advantages

The construction of models on a full dataset, not just subsets

The generation of deployable products to Mesos clusters The creation of Avro and Play/Akka HTTP powered web services that use the resulting dataset

The creation of repositories and indexes of the analyses and services