problem installing virtualbox on centos with xen enabled kernel

Today, I tried to install virtualbox 4.0.4 on centOS 5.4( unfortunately that was what I got from my company ). Installation went fine, But after i created a virtual machine and tried to start it, I got this error,

Running VirtualBox in a Xen environment is not supported

The problem was with centOS install. centOS installs xen enabled kernel by default. So I installed normal kernel from repository.

yum install kernel.x86_64 kernel-devel.x86_64 kernel-headers.x86_64

We have to configure grub to boot normal kernel by default and not the xen kernel. Set the parameter “default” to point to normal kernel. Now reboot!

After the kernel is up, execute the following command,

sudo /etc/init.d/vboxdrv setup

If everything goes fine, You can now start the virtual machine.

Querying system memory usage from c in linux

My task was to write a program which creates memory load and fragmentation stress on linux. Before I can start on that, I had to know how much ram is present in total and how much is available. First idea that comes to our mind is,

cat /proc/meminfo

Other alternatives are,

free -b

vmstat

But I wanted a way to get this info from a c program. My first trial was using get_phys_pages and get_avphys_pages.

printf("total memory    : %10li\n", get_phys_pages() * getpagesize());
printf("free memory     : %10li\n", get_avphys_pages() * getpagesize());

The problem with this method is, get_avphys_pages returns the free memory excluding cached and buffered memory. So I tried sysinfo system call.

struct sysinfo si;
if(sysinfo(&si) < 0) {
  printf("error! couldn't get si\n");
}
printf("total memory    : %10li\n", si.totalram);
printf("free memory     : %10li\n", si.freeram);
printf("shared memory   : %10li\n", si.sharedram);
printf("buffer memory   : %10li\n", si.bufferram);

Unfortunately, sysinfo also returned the same free memory as get_avphys_pages. If we add buffer memory and cache memory to free memory returned, we will get the actual free memory available. Unfortunately, sysinfo only provides buffer memory size. So at the end i had to parse /proc/meminfo file to get memory size used as cache pages.

FILE *fp;
char buf[MAXLEN];
fp = fopen("/proc/meminfo", "r");
for(int i = 0; i <= 3; i++) {
  fgets(buf, MAXLEN, fp);
}
char *p1 = strchr(buf, ":");
printf("%s\t%i\n", p1, strtoull(p1, NULL, 10));

mount virtualbox image( vdi ) on linux

I was using windows on virtual box. Once i had to shutdown the system abruptly. The next time i booted, i got this famous “blue screen of death”. All my precious data were inside the virtual box image. Here is my journey to retrieve the data hidden in virtual box’s vdi format.

First, backup your vdi file. If virtual box is installed in your system, you must have a command called VBoxManage. Lets use it to convert our vdi image into raw image, which can later be mount.

VBoxManage internalcommands converttoraw <<your vdi file>> recover.raw

Substitute the backup vdi file in the above command. The raw image created is recover.raw. Now we have to mount it in order to access the files in it. Execute the following command to mount your raw image,

losetup /dev/loop1 recover.raw

Now query the partition information from fdisk,

fdisk -lu /dev/loop1

Disk /dev/loop7: 44.1 GB, 44108349440 bytes
255 heads, 63 sectors/track, 5362 cylinders, total 86149120 sectors
Units = sectors of 1 * 512 = 512 bytes

Device Boot Start End Blocks Id System
/dev/loop7p1 * 16065 43070264 21527100 7 HPFS/NTFS
/dev/loop7p2 43070265 86140529 21535132+ 7 HPFS/NTFS

In my case, I had two partitions C and D. To mount one of these partitions, You need the exact address where the partition starts. You can get the exact address by multiplying start sector of the partition with 512(Since a partition is 512 bytes each). So in my case, the first partition starts at 16065 * 512. Now let us mount our partition,

losetup /dev/loop2 recover.raw -o $((16065 * 512))
mount /dev/loop2 /mnt
cd /mnt

Bingo! You have recovered your files.

Note:
If the filesystem type of your partition is ntfs, you have to install fuse and ntfs-3g tools to be able to mount your partition.

Alternative methods:
1) the easiest method is using nbd module and qemu-nbd program. But i couldn’t find those packages in my company’s rpm repository.

witnessing ip fragmentation using ping

My friend was testing ip fragmentation attacks on linux. So we had to witness ip fragmentation effect first. As we know, an IP packet can carry a total packet length of 65535(2^16 – 1) bytes. But since, data link layer(ethernet) frames can only carry maximum data size of 1500(MTU) bytes, IP has to fragment the packet if its length is greater than MTU. Out of 1500 bytes, IP’s header will itself occupy 20 bytes. We are left with 1480 bytes of data. Since we are using ping(which internally uses ICMP), 8 bytes are used by ICMP for its header.  We are left with 1472 bytes, which is the maximum data size that can be sent using ping.

Fig1 Ethernet frame format

Fig2 IP packet format

Fig3 ICMP frame format

Note: To check MTU of interface eth0, type

ifconfig eth0

This is the ip we will be using throughout this test,

EXIP="192.168.1.10"

Run tcpdump in another terminal to keep track of transmission,

tcpdump -i wlan0 icmp -vvv

Now let us test the connection,

ping $EXIP -s 1472 -c 1

tcpdump reports that the packet has been sent,

01:58:05.960968 IP (tos 0×0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 1500)
teja-laptop.local > vinit-PC.local: ICMP echo request, id 15530, seq 1, length 1480

When we try to send a packet with packet size greater than MTU and don’t fragment flag set,

ping $EXIP -s 1473 -M do -c 1

tcpdump remains silent because no packets are being sent. Lets see the output of ping and infer what has happened,

PING 192.168.1.10 (192.168.1.10) 1473(1501) bytes of data.
From 192.168.1.7 icmp_seq=1 Frag needed and DF set (mtu = 1500)

— 192.168.1.10 ping statistics —
0 packets transmitted, 0 received, +1 errors

This shows that if we try to send a packet with size greater than MTU and don’t fragment bit set, ping replies that it can’t send the packet. Now let us try to send the same packet with don’t fragment bit unset,

ping $EXIP -s 1473 -c 1

02:02:33.141005 IP (tos 0×0, ttl 64, id 31043, offset 0, flags [+], proto ICMP (1), length 1500)
teja-laptop.local > vinit-PC.local: ICMP echo request, id 15540, seq 1, length 1480
02:02:33.141134 IP (tos 0×0, ttl 64, id 31043, offset 1480, flags [none], proto ICMP (1), length 21)
teja-laptop.local > vinit-PC.local: icmp

Voila! The packet has been fragmented. As you can see, IP layer has fragmented the packet into two fragments. The first fragment has offset 0, id 31043 and length 1500. The second packet has the same id as the first but the offset shows where exactly this fragment fits into in the original packet. The data length of second packet is only 21(IP header + remaining 1 byte). Finally “flags [+]” in first packet means that there are more fragments yet to arrive while “flags [none]” in second packet means this is the last fragment.

viewing hardware spec in linux

There are various commands in linux that provide us with neccesary info about our hardware. Here we are going to use a few of them,
lshw
lshw shows a comprehensive list of devices connected to our system and their info.

sudo lshw

Thats too much to have a look at. You can get a simpler but readable info,

lshw -short

The third column of output from lshw command is class. If you want to view information about a specific class(example: disaply),

lshw -class display

lspci
lspci command lists details about various devices connected to pci card.

lspci -v

Again thats too much information when you want to know about just a specific device. Lets look for display device information,

lspci | grep -i display

In my case, It displayed the following info,
output:

00:02.1 Display controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 03)

If you observe, there is a number(00:02.1) at the begining of previous command’s output. You can use this number to make lspci show info about specific device.

lspci -v -s 00:02.1

output:

00:02.1 Display controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 03)
Subsystem: Lenovo Device 383e
Flags: bus master, fast devsel, latency 0
Memory at f8100000 (64-bit, non-prefetchable) [size=1M]
Capabilities: [d0] Power Management version 3

This can be made easy by using a single command,

lspci -v -s `lspci | grep -i display | awk '{print $1}'`

lsusb
If you want to know about various devices connected to your usb port,

lsusb -v

Asusual, There is a method to produce more cuter output,

lsusb -v -d `lsusb | grep -i camera | awk '{print $6}'`

face/eye/hand detection using openCV python binding

Haar transform can be used to detect a required object in an image. To detect an object using haar transform, we have to train by providing it some positive and negative images and generate an xml containing the required object’s features. It might take several days to completely train it. In this post, let us look at some already trained cases like face, eye or hand detection. If you have installed openCV python bindings provided by ubuntu, /usr/share/opencv/haarcascades/ directory has few trained xml feature files. Let us use few of them.

face detection

#!/usr/bin/python
import sys, os
from opencv.cv import *
from opencv.highgui import *
import opencv

def detectObjects(image):
    grayscale = cvCreateImage(cvSize(image.width, image.height), 8, 1)
    cvCvtColor(image, grayscale, CV_BGR2GRAY)

    storage = cvCreateMemStorage(0)
    cvClearMemStorage(storage)
    cvEqualizeHist(grayscale, grayscale)
    cascade = cvLoadHaarClassifierCascade('/usr/share/opencv/haarcascades/haarcascade_frontalface_default.xml',
                                          cvSize(1, 1))
    faces = cvHaarDetectObjects(grayscale, cascade, storage, 1.2, 2, CV_HAAR_DO_CANNY_PRUNING, cvSize(50, 50))
    if faces:
        for i in faces:
            cvRectangle(image, cvPoint( int(i.x), int(i.y)),
                         cvPoint(int(i.x + i.width), int(i.y + i.height)),
                         CV_RGB(0, 255, 0), 3, 8, 0)

def main():
    cvNamedWindow("cam", 1)
    capture = cvCreateCameraCapture(0)
    orig = cvQueryFrame(capture)
    grey = cvCreateImage(cvGetSize(orig), IPL_DEPTH_8U, 1)
    face = cvCreateImage(cvGetSize(grey), IPL_DEPTH_8U, 1)
    while True:
        orig = cvQueryFrame(capture)
        cvCvtColor(orig, grey, CV_RGB2GRAY)
        detectObjects(orig)
        cvShowImage("cam", orig)
        k = cvWaitKey(10)
        if k == 'q': # press q to quit
            break
  
if __name__ == "__main__":
    main()

Here is the output,

eye detection

#!/usr/bin/python
import sys, os
from opencv.cv import *
from opencv.highgui import *
import opencv

def detectObjects(image):
    grayscale = cvCreateImage(cvSize(image.width, image.height), 8, 1)
    cvCvtColor(image, grayscale, CV_BGR2GRAY)

    storage = cvCreateMemStorage(0)
    cvClearMemStorage(storage)
    cvEqualizeHist(grayscale, grayscale)
    cascade = cvLoadHaarClassifierCascade('/usr/share/opencv/haarcascades/haarcascade_mcs_righteye.xml',
                                          cvSize(1, 1))
    faces = cvHaarDetectObjects(grayscale, cascade, storage, 1.2, 2, CV_HAAR_DO_CANNY_PRUNING, cvSize(50, 50))
    if faces:
        for i in faces:
            cvRectangle(image, cvPoint( int(i.x), int(i.y)),
                         cvPoint(int(i.x + i.width), int(i.y + i.height)),
                         CV_RGB(0, 255, 0), 3, 8, 0)

def main():
    cvNamedWindow("cam", 1)
    capture = cvCreateCameraCapture(0)
    orig = cvQueryFrame(capture)
    grey = cvCreateImage(cvGetSize(orig), IPL_DEPTH_8U, 1)
    face = cvCreateImage(cvGetSize(grey), IPL_DEPTH_8U, 1)
    while True:
        orig = cvQueryFrame(capture)
        cvCvtColor(orig, grey, CV_RGB2GRAY)
        detectObjects(orig)
        cvShowImage("cam", orig)
        k = cvWaitKey(10)
        if k == 'q': # press q to quit
            break
  
if __name__ == "__main__":
    main()

In our next post, let us see how we can train haar transform to detect an object of our desire.

installing openCV python wrapper and executing simple program in ubuntu

I wanted to try image processing and openCV. So I set my journey. Installing openCV python bindings in ubuntu is just a piece of cake,

sudo apt-get install libcv4 libcvaux4 libhighgui4 python-opencv opencv-doc libcv-dev libcvaux-dev libhighgui-dev

Following tutorials from various sites I tried to import the cv package in python,

import cv

And I got this error,

Traceback (most recent call last):
File “”, line 1, in
ImportError: No module named cv

The reason is, there are different implementations of openCV bindings for python. Some of them are SWIG, ctypes-opencv, etc. Out of the box, Ubuntu provides old SWIG-based implementation of openCV python bindings. Here is a sample program that captures frames from webcam and performs edge detection using canny edge detector,

from opencv.cv import *
from opencv.highgui import *

cvNamedWindow("cam", 1)
cvNamedWindow("canny", 1)
capture = cvCreateCameraCapture(0)
orig = cvQueryFrame(capture)
grey = cvCreateImage(cvGetSize(orig), IPL_DEPTH_8U, 1)
canny = cvCreateImage(cvGetSize(grey), IPL_DEPTH_8U, 1)
while True:
    orig = cvQueryFrame(capture)
    cvCvtColor(orig, grey, CV_RGB2GRAY)
    cvShowImage("cam", grey)
    cvCanny(grey, canny, 50, 150, 3)
    cvShowImage("canny", canny)
    c = cvWaitKey(7) % 0x100
    if c == 27 or c == 10:
        break

You might find these links useful,
1) Python openCV Reference
2) CV Reference Manual

modelling memory using systemC RTL

Let us design a memory module using systemC similar to this diagram,

This is the declaration of our memory module class. The capacity of this model is 256 words. You can change it to your needs.
rammem.h

#ifndef MEMORY_H_
#define MEMORY_H_
#include <systemc.h>

SC_MODULE(memory)
{
	sc_inout<sc_lv<32> >  data;
	sc_in<sc_lv<32> >     address;
	sc_in<bool>             rw;
	sc_in<bool>             enable;

	sc_signal<sc_uint<32> >        ramdata[256];

	void entry();

	void memdump()
	{
		FILE *fp = fopen("data/memdump","w");
		int size;
		for (size = 0; size < 255; size++) {
			fprintf(fp, "0x%x\n", ramdata[size].read().to_int());
		}
	}

	SC_CTOR(memory)
	{
		SC_METHOD(entry);
		sensitive << enable << rw << address;

		FILE *fp = fopen("data/ram_init","r");
		if(!fp)
		{
			cout << "error. cannot find ram_init." << endl;
		}
		int size=0;
		int mem_word;
		for (size = 0; size < 255; size++) {
			ramdata[size].write( 0x0 );
		}
		size = 0;
		while (fscanf(fp,"%x\n", &mem_word) != EOF) {
			ramdata[size].write( mem_word );
			size++;
		}
	}
};

#endif /* MEMORY_H_ */

Here are the member function defections,
rammem.cpp

#include <systemc.h>
#include <rammem.h>

void memory::entry()
{
	if(enable.read() == 1) {
		cout << "@ " << sc_time_stamp() << endl;
		if(rw.read() == 1) {			//write
			ramdata[address.read().to_int()].write( data.read() );
			cout << "writing data " << data.read() << " at " << address.read() << endl;
		} else {						//read
			data.write(ramdata[address.read().to_int()].read());
			cout << "reading data " << data.read() << " at " << address.read() << endl;
		}
	} else {
		data.write("zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz");
	}
}

The test bench to test our memory module,
main.cpp

#include <systemc.h>
#include <rammem.h>

int sc_main(int argc, char* argv[]) {
	memory mem("memory");
	sc_set_time_resolution(1, SC_PS);

	sc_trace_file *wf = sc_create_vcd_trace_file("data/memory");
	sc_trace(wf, mem.address, "address");
	sc_trace(wf, mem.data, "data");
	sc_trace(wf, mem.enable, "enable");
	sc_trace(wf, mem.rw, "rw");

	sc_signal<bool>   enable, rw;
	sc_signal<sc_lv <32> > data;
	sc_signal<sc_lv<32> > address;

	mem.rw(rw);
	mem.data(data);
	mem.address(address);
	mem.enable(enable);

	enable  = 0;
	data    = 0;
	address = 0;
	rw      = 0;
	sc_start(0, SC_PS);

	rw      = 1;
	address.write(0x10);
	data.write(0x110011);
	enable  = 1;
	sc_start(5, SC_PS);

	enable  = 0;
	sc_start(5, SC_PS);

	address.write(0x12);
	data.write(0x1100);
	enable  = 1;
	sc_start(5, SC_PS);

	address.write(0x8);
	data.write(0x1010);
	sc_start(5, SC_PS);

	rw      = 0;
	address = 0x10;
	sc_start(5, SC_PS);

	address = 0x1;
	sc_start(5, SC_PS);

	address = 0x3;
	sc_start(5, SC_PS);

	enable  = 0;
	sc_start(1, SC_PS);

	mem.memdump();
	sc_close_vcd_trace_file(wf);

	return(0);
}

Here is the output waveform,

In this post, we have seen how to design ram models in systemC using RTL style. In our next port, let us model memory using systemC TLM.

combinational logic using systemC

We have seen previously that sc_signal uses Non-Blocking Assignment. It makes it perfect option for modeling sequential logic as the new value is not available until after one delta cycle. But this makes it inappropriate for modeling combinational logic because you cannot update and access the same sc_signal instance in the same delta cycle. Let us consider this basic example,

Using sc_signal or sc_port,

SC_MODULE (and2) {

  sc_in<bool> a;
  sc_in<bool> b;
  sc_out<bool> and_out;
  sc_out<bool> nand_out;

  void and2_do()
  {
      and_out = a.read() & b.read();    // instead use and_out.write(a.read() & b.read());
      nand_out = !and_out;                // instead use nand_out.write(!and_out.read());
  }

  SC_CTOR (and2) {
    SC_METHOD(and2_do);
      sensitive << a << b;
  }
};

(Note: sc_port(sc_in, sc_out, sc_inout, etc) allows us to communicate between various modules but it implements sc_signal_in_if and sc_signal_inout_if which makes its assignment Non-Blocking.)
The above code won’t produce what we expected, since and_out is modeled as register(sequential logic) and not as wire(combinational logic).

Native C++ data types and other systemC data types like sc_bv, sc_lv, sc_int, etc use blocking assignment. This makes them ideal for combinational circuit design. To model combinational circuit, We have to use local declaration of these data types in corresponding process. If there is a need to share the variable between different processes of a module, they can be declared as member variables of that module. Here is a simple example,

SC_MODULE (and2) {

  sc_in<bool> a;
  sc_in<bool> b;
  sc_out<bool> and_out;
  sc_out<bool> nand_out;

  void and2_do()
  {
	  bool temp_and;
	  temp_and = a.read() & b.read();
	  and_out.write(temp_and);
      nand_out.write(!temp_and);
  }

  SC_CTOR (and2) {
    SC_METHOD(and2_do);
      sensitive << a << b;
  }
};

This produces the expected results,

non-blocking assignment in systemc

Verilog uses non-blocking assignment(NBA) to model registers. The basic functionality of non-blocking assignment is, The newly assigned value won’t update until the next clock cycle. Thus If we access(or read) the same variable in the same block, in the same clock cycle after assignment, The value obtained will be the old value. Non-blocking assignment is like effectively assigning value of a signal for next clock cycle, which means signal is updated minimum 1 delta cycle later.

Native C++ data type assignments are sequential and blocking. SystemC provides sc_signal to imitate VHDL’s signal or Verilog’s reg type with Non-Blocking assignment. systemC’s sc_signal uses Evaluate-Update cycle to model Nonblocking assignments.

Let us consider a simple 2 input and gate example,

SC_MODULE (and2) {

  sc_in a;
  sc_in b;
  sc_out c;
  sc_out d;

  void and2_do()
  {
      c = a & b;	// instead use c.write(a.read() & b.read());
      d = c;		// instead use d.write(c.read());
  }

  SC_CTOR (and2) {
    SC_METHOD(and2_do);
      sensitive << a << b;
  }
};

int sc_main(int argc, char* argv[]) {
  and2 DUT("and_DUT");

  sc_trace_file *wf = sc_create_vcd_trace_file("Debug/wvform");
  sc_trace(wf, DUT.a, "a");
  sc_trace(wf, DUT.b, "b");
  sc_trace(wf, DUT.c, "c");
  sc_trace(wf, DUT.d, "d");

  sc_signal a, b, c, d;
  DUT.a(a);
  DUT.b(b);
  DUT.c(c);
  DUT.d(d);

  sc_start(0, SC_PS);

  a = 0;  b = 0;
  sc_start(1, SC_PS);

  a = 1;  b = 0;
  sc_start(1, SC_PS);

  a = 1;  b = 1;
  sc_start(1, SC_PS);

  a = 0;  b = 1;
  sc_start(1, SC_PS);

  a = 0;  b = 0;
  sc_start(1, SC_PS);

  a = 1;  b = 0;
  sc_start(1, SC_PS);
  sc_close_vcd_trace_file(wf);
  return(0);
}

We would expect c and d to contain same value but since c won’t be updated immediately,

 
It is clear from the above discussion that sc_signal cannot be used to model combinational logic. Let us look in the next post how to model combinational logic in systemC.