baijia - papers and notes - All Forums

baijia.info forum development

Mon, 08 Mar 2010 22:51:13 -0500

I have been trying to attach some files (PDF or PPT) to some posts but often failed. My impression is that when the file is larger it does not work. But I've set up the system to accept sufficiently large files. Could someone looks into it and fix the problem?

Kant...Data center power management

Mon, 08 Mar 2010 22:46:36 -0500

Supercomputing 2009 Tutorial Data Center Power Management
Krishna Kant and David Du

Overview

Sun, 07 Mar 2010 02:23:42 -0500

Welcome to use the system to start your work!

Please read the following guidelines, you can find many useful information to help you develop your work quickly.

The work flow of system using is as follows.

1)First we tracks our system and accounts here : http://baijia.info/showthread.php?tid=300

You can know the basic information of each physical or virtual machine and the permissions of each account. According those information, you will have an overview of the system and know the permissions of your account.

2) Before you use the servers, please read this post: http://baijia.info/showthread.php?tid=253.

It tracks and coordinates the usage of the servers. Please read previous posts to know which resources have been allocated and which can be used. After you run some tasks you can post a usage log to let others know those resources have been allocated. It can avoid wrong killing.

3) We use git for source control now. If you are not familiar with git, you can read this first: http://git-scm.com/. Then please read this tutorial to know how to use git in our server : http://baijia.info/showthread.php?tid=441

4) If you want to know how to set up the system, please browse the System setup & tutorial subforum (http://baijia.info/forumdisplay.php?fid=101) to get more information.

Hope above information is helpful to you.

Apache related

Thu, 04 Mar 2010 10:34:55 -0500

Set up htpasswd:
htpasswd

subvsersion examples

Thu, 04 Mar 2010 09:11:19 -0500

Check out via http
svn co http:/// --username

Castro...Fast byte-granularity software fault isolation.SOSP09

Sun, 28 Feb 2010 10:01:50 -0500

Castro, M., Costa, M., Martin, J., Peinado, M., Akritidis, P., Donnelly, A., Barham, P., and Black, R. Fast byte-granularity software fault isolation. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles (Big Sky, Montana, USA, October 11 - 14, 2009). SOSP '09. ACM, New York, NY, 45-58

Hennessy...Computer architecture: a quantitative approach 4th ed. Morgan

Tue, 23 Feb 2010 22:21:47 -0500

Hennessy, J. L. and Patterson, D. A. 2006 Computer Architecture, Fourth Edition: a Quantitative Approach. Morgan Kaufmann Publishers Inc.

@book{hennessy06computer,
author = {Hennessy, John L. and Patterson, David A.},
title = {Computer Architecture, Fourth Edition: A Quantitative Approach},
year = {2006},
isbn = {0123704901},
publisher = {Morgan Kaufmann Publishers Inc.},
address = {San Francisco, CA, USA},
}

Unified Linux logon and home directory using OpenLDAP and NFS/automount

Mon, 22 Feb 2010 05:21:18 -0500

0. System environment
=====================

LDAP and NFS server:
IP: 10.0.0.2
OS: CentOS 5.3 X86_64
ldap base dn: "dc=lgcpu1"

Clients:
IP: 10.0.0.1/24
OS: Fedora 11 X86_64

1. LDAP server
==============

1.1 Package installation:
-------------------------
# yum install openldap-servers
# /sbin/chkconfig ldap on
# /sbin/service ldap start

emacs /etc/openldap/slapd.conf

1.2 Add or edit these configurations:
-------------------------------------

### begin ###

include /etc/openldap/schema/redhat/autofs.schema

#######################################################################
# ldbm and/or bdb database definitions
#######################################################################

database ldbm
suffix "dc=lgcpu1"
rootdn "cn=Manager,dc=lgcpu1"

rootpw {crypt}x

# Access Control
access to attr=userPassword
by self write
by anonymous auth
by dn="cn=manager,dc=lgcpu1" write
by * compare
access to *
by self write
by dn="cn=manager,dc=lgcpu1" write
by * read

### end ###

How to get the rootpw:

perl -e "print crypt('passwd', 'salt_string',);"

1.3 Add top.ldif
----------------

top.ldif:

### begin ###

dn: dc=lgcpu1
objectclass: dcObject
objectclass: organization
o: lgcpu1 group
dc: lgcpu1

dn: cn=manager,dc=lgcpu1
objectclass: organizationalRole
cn: manager

dn: ou=people,dc=lgcpu1
ou: people
objectclass: organizationalUnit
objectclass: domainRelatedObject
associatedDomain: lgcpu1

dn: ou=contacts,ou=people,dc=lgcpu1
ou: contacts
ou: people
objectclass: organizationalUnit
objectclass: domainRelatedObject
associatedDomain: lgcpu1

dn: ou=group,dc=lgcpu1
ou: group
objectclass: organizationalUnit
objectclass: domainRelatedObject
associatedDomain: lgcpu1

### end ###

Add top.ldif to ldap server:

$ ldapadd -x -D 'cn=manager,dc=lgcpu1' -W -f top.ldif

1.4 Add users and groups from local configuration:
--------------------------------------------------

Copy passwd shadow group from /etc/ to some tmp location
Edit them and only keep the normal users, that means no system users.

$ emacs /usr/share/openldap/migration/migrate_common.ph

Edit these values:

### begin ###

# Default DNS domain
$DEFAULT_MAIL_DOMAIN = "cse.ust.hk";

# Default base
$DEFAULT_BASE = "dc=lgcpu1";

### end ###

$ /usr/share/openldap/migration/migrate_passwd.pl ./passwd > people.ldif

Then add the encryped password from shadow file to userPassword.

Like this:
userPassword: {crypt}$1$Zlkjsdf...

Then add people.ldif to ldap server

$ /usr/share/openldap/migration/migrate_group.ph ./group > group.ldif

Then add group.ldif to ldap server

2. NFS server
=============

emacs /etc/exports
------------------

### begin ###
/home 10.0.0.1/24(rw)
### begin ###

auto.master.ldif
----------------

### begin ###

dn: ou=auto.master,dc=lgcpu1
objectClass: top
objectClass: automountMap
ou: auto.master

dn: cn=/home,ou=auto.master,dc=lgcpu1
objectClass: automount
automountInformation: ldap:ou=auto.home,dc=lgcpu1
cn: /home

dn: cn=/share,ou=auto.master,dc=lgcpu1
objectClass: automount
automountInformation: ldap:ou=auto.misc, dc=lgcpu1
cn: /share

### end ###

add auto.master.ldif

auto.home.ldif
--------------
Add for every users

### begin ###

dn: ou=auto.home,dc=lgcpu1
objectClass: top
objectClass: automountMap
ou: auto.home

dn: cn=zma,ou=auto.home,dc=lgcpu1
objectClass: automount
automountInformation: 10.0.0.2:/home/zma
cn: zma

### end ###

add auto.home.ldif

auto.misc.ldif
--------------
Add for some common share directories

### begin ###

dn: ou=auto.misc,dc=lgcpu1
objectClass: top
objectClass: automountMap
ou: auto.misc

### end ###

add auto.misc.ldif

3. Client configuration
=======================

3.1 Packages installation
-------------------------

# yum install nss_ldap autofs nfs-utils
# chkconfig autofs on
# chkconfig nfs on
# service autofs start
# service nfs start

3.2 Configuration
-----------------

# authconfig-tui

Select like this:

| User Information Authentication │
│ [ ] Cache Information [*] Use MD5 Passwords │
│ [ ] Use Hesiod [*] Use Shadow Passwords │
│ [*] Use LDAP [*] Use LDAP Authentication │
│ [ ] Use NIS [ ] Use Kerberos │
│ [ ] Use Winbind [ ] Use Fingerprint reader │
│ [ ] Use Winbind Authentication │
│ [*] Local authorization is sufficient │

In next step:

│ [ ] Use TLS │
│ Server: ldap://10.0.0.2/________________________ │
│ Base DN: dc=lgcpu1_______________________________ │

3.3 Delete old user entries in:
-------------------------------
/etc/passwd
/etc/shadow
/etc/group
/etc/gshadow

4. add individual person
========================

Add people.sample.ldif to ldap
------------------------------

### begin ###

dn: uid=sample,ou=People,dc=lgcpu1
uid: sample
cn: sample
sn: sample
mail: sample@cse.ust.hk
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: top
userPassword: {crypt}$6$encryped password here
loginShell: /bin/bash
uidNumber: 507
gidNumber: 507
homeDirectory: /home/sample

### end ###

Add group.sample.ldif to ldap
-----------------------------

### begin ###

dn: cn=sample,ou=Group,dc=lgcpu1
objectClass: posixGroup
objectClass: top
cn: sample
userPassword: {crypt}x
gidNumber: 507

### end ###

Addauto.home.sample.ldif to ldap
--------------------------------

### begin ###

dn: cn=sample,ou=auto.home,dc=lgcpu1
objectClass: automount
automountInformation: 10.0.0.2:/home/sample
cn: sample

### end ###

Delete old entries in:
----------------------

/etc/passwd
/etc/shadow
/etc/group
/etc/gshadow

Create home directory on NFS server:
------------------------------------

# mkdir /home/sample
# cp /etc/skel/.[a-z] * /home/sample/
# chmod -R sample\: /home/sample/

Another copy is posted here.

Hadoop setup tutorial

Mon, 22 Feb 2010 05:19:34 -0500

1. Packages
===========

Sun Java JDK:
-------------
http://java.sun.com/
Installed in /usr/java/jdk1.6.0_16

Hadoop:
-------
http://hadoop.apache.org/

Unpack it to somewhere. /lhome/zma/hadoop in this example
For convenience:
$ ln -s /lhome/zma/hadoop/bin/* /home/zma/bin/

2. Hosts
========

Add lines like this to /etc/hosts of ALL hosts:
We use four machines here.

### begin ###
10.0.0.103 vm103
10.0.0.104 vm104
10.0.0.105 vm105
10.0.0.106 vm106
### begin ###

Change the host's name to the name described.
---------------------------------------------

Configure iptables:
-------------------
We use 9000 and 9001 in this example.

3. SSH Login without password
=============================

Generate private and public key:

$ ssh-keygen -t rsa
$ chmod 700 .ssh
$ cat .ssh/id_rsa >> .ssh/authorized_keys
$ chmod 755 .ssh.authorized_keys
$ scp -r .ssh "all hosts, one by one":/home/directory/

Try to ssh hosts:

$ for ((i=103; i<=106; i++)); do ssh vm$i ; done;

4. Hadoop Configurations
========================

hadoop/conf/hadoop-env.h
------------------------

Add or change these lines:

### begin ###

export JAVA_HOME=/usr/java/jdk1.6.0_16
export HADOOP_HOME=/lhome/zma/hadoop

### end ###

hadoop/conf/core-site.xml
-------------------------

Add these lines:
Here namenode is vm103.
### begin ###

fs.default.name
hdfs://vm103:9000/

### end ###

hadoop/conf/hdfs-site.xml
-------------------------

Add these lines:
### begin ###

dfs.replication
2

### end ###

hadoop/conf/mapred-site.xml
---------------------------

Add these lines:
Here jobtracker is vm104.
### begin ###

mapred.job.tracker
vm104:9001

### end ###

hadoop/conf/masters
-------------------

Delete localhost and add:
### begin ###
vm103
vm104
### end ###

hadoop/conf/slaves
------------------

Delete localhost and add:
### begin ###
vm103
vm104
vm105
vm106
### end ###

5. Duplicate hadoop to all hosts
================================

$ for ((i=104; i<=106; i++)); do scp -r /lhome/zma/hadoop vm$i:/lhome/zma/ ; done;

Here we are on vm103.

6. Start Hadoop
===============

On vm103 (namenode):

Format a new distributed-filesystem:
$ hadoop namenode -format

Start hdfs:
$ start-dfs.sh

Check the hdfs status:
$ hadoop hdfsadmin -report

On vm104 (jobtracker):

Start mpared:
$ start-mapred.sh

Check job status:
$ hadoop job -list

7. Enjoy fun now
================

A simple example
----------------
Copy the input files into the distributed filesystem:
$ bin/hadoop fs -put conf input

Run some of the examples provided:
$ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'

Examine the output files:

Copy the output files from the distributed filesystem to the local
filesytem and examine them:
$ bin/hadoop fs -get output output
$ cat output/*

or

View the output files on the distributed filesystem:
$ bin/hadoop fs -cat output/*

8. Shutdown Hadoop
==================
$ stop-dfs.sh
$ stop-mapred.sh

or

$ stop-all.sh

Referrences
===========
[1] http://hadoop.apache.org/common/docs/current/quick

Another copy is posted here

git tutorial

Mon, 22 Feb 2010 05:17:53 -0500

A git server has been set up on one of the virtual machines: vm111 (ip: 10.0.0.111).

Add isa_key.pub to git's .ssh/authorized_keys
=============================================
ssh lgcpu1.cse.ust.hk -p 22111 # log on to vm111, using the account that can sudo
cp .ssh/id_rsa.pub /dev/shm/ # copy pub key to a temp directory
sudo su # operate in git's home as root
cd /lhome/git/.ssh
cp authorized_keys authorized_keys.bak # backup before changing is a good habit
cat /dev/shm/id_rsa.pub >> authorized_keys # append pub key to git's autorized keys list

Create repository
=================
log on lgcpu1.cse.ust.h -p 22111 # using the account that can sudo

sudo su git
cd /lhome/git
mkdir example.git # the repository directory
cd example.git
git --bare init # initial the repository, --bare means only objects is stored on server

First commit:
=============
on local laptop:

mkdir example
cd example
git init
touch README
git add README
git commit -m 'first commit'
git remote add origin ssh://git@lgcpu1.cse.ust.hk:22111/~/example.git
git push origin master

When programming:
=================
git clone ssh://git@gate.example.org:22111/~/example.git # we need to do this for only one time

# every time we want to edit some files:

cd example
git pull # pull the newest version from the repository

# edit some files

git commit -a -m 'msg' # commit the changes
git push # push the changes to the repository

Another copy of this post can be found here.

How to duplicate Xen DomU virtual machines

Mon, 22 Feb 2010 05:15:45 -0500

Assumption:
There is VBD based Xen DomU virtual machine stored under /lhome/xen/vm-f11-sample/. There are two files under vm-f11-sample: vm0-f11.run (The configuration file) and vmdisk0 (The virtual disk).

Now we want to duplicate the virtual machine vm0 stored under vm-f11-sample to vm213 stored under vm213. And vm213’s ip will be 10.0.0.213. The steps to duplicate this virtual machine:

1) Duplicate the virtual disk and configuration files

# cp -rf vm-f11-sample vm213

For security reason, the owner of the virtual machine’s files is root. So we need to copy these files as root. The destination directory is vm213. And here we assume that vm0 is power off. If vm0 is power on before this step, we need to shut it down first.

2) Change the file names and the configuration file

# cd vm213
# mv vmdisk0 vmdisk213
# mv vm0-f11.run vm213-f11.run
# vim vm213-f11.run

This is the content of vm213-f11.run:

name="vm213"
memory=1024
disk = ['tap:aio:/lhome/xen/vm213/vmdisk213,xvda,w' ]
vif = [ 'bridge=eth0' ]
bootloader = "/usr/bin/pygrub"
vcpus=2
on_reboot = 'restart'
on_crash = 'restart'

The name and disk entry is changed.

3) Start the new virtual machine and configure the new virtual machine
# xm create /lhome/xen/vm213/vm213-f11.run
# xm console vm213

After log in to vm213:
# vi /etc/sysconfig/network-scripts/ifcfg-eth0
Change the IPADDR to 10.0.0.213. Then restart eth0:
# ifdown eth0
# ifup eth0

Log out of vm213 and then use “Ctrl + ]” to exit the xm console.

Reset vm213 on Dom0:
#xm reset vm213

The new virtual machine vm213 is running now.

Another copy is posted here

Create and manage virtual machines on Xen

Mon, 22 Feb 2010 05:14:25 -0500

In this post, these content are introduced:

* Create and manage file-backed virtual block device (VBD) for virtual machines on xen.
* Install Fedora 11 via internet as DomU on top of xen.
* Manage virtual machines using xm.

Working Platform:

Details can found here.

Create file-backed VBD:

The actual space of VBD will be the amount of disk the virtual machine used. And it will be convenient if the virtual machine will be duplicated since the work is just copying the VBD file and changing some configurations. But file-backed VBDs may not be appropriate for I/O-intensive domains because of the I/O handling cost to support file-backed VBDs in dom0.
Create a 20GB sparse file-backed VBD:

# dd if=/dev/zero of=vmdisk1 bs=1k seek=20480k count=1

Make a ext3 file system in the disk file:

# mkfs -t ext3 vmdisk1

Install Fedora 11 Linux via Internet:

First download the pxeboot kernel of Fedora 11 for installation via Internet. Download vmlinuz and initrd.img from here:

http://download.fedora.redhat.com/pub/fe...http://download.fedora.redhat.com/pub/fedora/linux/releases/11/Fedora/x86_64/os/image

Create an installation profile f11.install:

name="F11INSTALL"
memory=1536
disk = ['tap:aio:/home/xen/vmdisk1,xvda,w' ]
vif = [ 'bridge=eth0' ]
kernel = "/home/xen/fedora/vmlinuz"
ramdisk = "/home/xen/fedora/initrd.img"
vcpus=2
on_reboot = 'restart'
on_crash = 'restart'

The virtual machine’s name is “F11INSTALL”, memory is 1.5G, CPU number is 2, disk, kernel and ramdisk is prepared in the above steps.

Start this virtual machine:

# xm create f11.install

Connect to this virtual machine’s console and complete the installation:

# xm console F11INSTALL

The installation of Fedora 11 will start. The URL of installation source I used during installation is:

http://download.fedora.redhat.com/pub/fe...http://download.fedora.redhat.com/pub/fedora/linux/releases/11/Fedora/

Load DomU:

Create a profile vm1.run for loading the virtual machine:

name="vm1"
memory=1536
disk = ['tap:aio:/home/xen/vm1/vmdisk1,xvda,w' ]
vif = [ 'bridge=eth0' ]
bootloader = "/usr/bin/pygrub"
vcpus=2
on_reboot = 'restart'
on_crash = 'restart'

Here we use the PyGrub (“/usr/bin/pygrub”) as the bootloader. PyGrub starts Linux DomUs with the kernels that lie in the filesystem of the DomU instead of the kernels that lie in the filesystem of the Dom0. That makes the kernel update and management easier.

Then the DomU can be started using this profile:

# xm create vm1.run

The console of this DomU can be connected to:

# xm console vm1

Manage DomUs:

Start Domu:

# xm create DomU_profile

List running DomUs:

# xm list

Shutdown DomU:

# xm shutdown DomU_name

Console of DomU:

# xm console DomU_name

Top of DomUs:

# xm top

Reset a DomU:

# xm reset DomU_name

And others more which can be find in the xm manuals.

After configured a DomU, duplication of the DomU is easy: Just make a copy of the VBD file and the profile, and then change the profile.

I have set up 8 DomUs on the server. There are total 9 machines on top of xen and a lot of interesting things can be done with these “machines”

# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1826 16 r—– 244.8
vm1 25 1536 2 -b—- 4.9
vm2 24 1536 2 -b—- 5.4
vm3 23 1536 2 -b—- 5.4
vm4 22 1536 2 -b—- 5.8
vm5 27 1536 2 -b—- 5.4
vm6 20 1536 2 -b—- 6.0
vm7 19 1536 2 -b—- 6.3
vm8 18 1536 2 -b—- 18.8

References:

http://tx.downloads.xensource.com/downlo...http://tx.downloads.xensource.com/downloads/docs/user/#SECTION0330000000

http://bderzhavets.wordpress.com/2009/06...http://bderzhavets.wordpress.com/2009/06/10/setup-fedora-11-pv-domu-at-xen-3-4-1-dom0-kernel-2-6-30-rc6-tip-on-top-of-

http://wiki.xensource.com/xenwiki/PyGrub

Another copy is posted here

New method to create VM

Mon, 22 Feb 2010 05:12:28 -0500

I have change the the old configuration files management method that create one configuration file for each VM to a unified one (/lhome/xen/vm.run). The usage is also much simpler:

When we want to create vmvmid with vmmem kb memory, just follow these steps:

# cd /lhome/xen
# xm create vm.run vmid=vmid vmmem=vmmem

Set up Xen 3.4.1 dom0 on top of Fedora 11 with kernel 2.6.29

Mon, 22 Feb 2010 05:11:06 -0500

Hardware:
Platform 1:
IBM eServer X3650M2:
2 x Intel Xeon X5570 2.93 8MB Cache Processor
4 x 4GB DDR3-1333 Memory
2 x 300GB 10K-SFF Hot-Swap HD – RAID 0

Platform 2:
2 x Intel Xeon E5520 2.27GHz 8MB cache
6 x 4G DDRIII REG ECC 1333
2 x ST 1000G NS 32M

The instruction below is based on platform 1.

Disk Partitions:

# df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda2 ext4 58G 6.3G 49G 12% /
/dev/sda1 ext3 388M 52M 316M 15% /boot
/dev/sda5 ext4 490G 14G 452G 3% /home

Linux:
Fedora 11, updated to 23-09-2009.

No xen and libvirt installed:

# rpm -qa | grep xen
# rpm -qa | grep virt

And SELinux is diabled.

Packages:
Xen 3.4.1: http://bits.xensource.com/oss-xen/releas...http://bits.xensource.com/oss-xen/release/3.4.1/xen-3.

linux-2.6.29-xen-r4-aka-suse-xenified-2.6.29-62.1: http://x17.eu/xen/linux-2.6.29-xen-r4-ak...http://x17.eu/xen/linux-2.6.29-xen-r4-aka-suse-xenified-2.6.29-6

download these packages and unzip them.

Installation:
1. Install Xen:

# sh ./install.sh

1. Install Xen:
copy the all the content of xen to “/usr/src/xen/3.4.1/xen-3.4.1/”

$ make world
# make install

2. Install Kernel
The .config file I used can be downloaded here: https://sites.google.com/site/ericzqma/l...https://sites.google.com/site/ericzqma/linux/kernel_config_xen_zma?attr

My .config file can be used directly. Or menuconfig can be used:

# make menuconfig

When using “make menuconfig”, some options require special attention:

Subarchitecture Type (Enable Xen compatible kernel)
( ) PC-compatible
(X) Enable Xen compatible kernel
( ) Support for ScaleMP vSMP
Device Drivers --->
XEN --->
[*] Privileged Guest (domain 0)
<*>Backend driver support
<*>Block-device backend driver
<*>Block-device tap backend driver
<*> Network-device backend driver

After configuration:

# make
# make modules_install install

3. Create a Grub entry:

title xen 3.4.1 - kernel 2.6.29-xen-r4
root (hd0,0)
kernel /xen-3.4.1.gz console=vga vga=ask noreboot
module /vmlinuz-2.6.29-xen-r4 ro \
root=UUID=ac0c173e-abca-4795-a7d6-23444e730d40
module /initrd-2.6.29-xen-r4.img

The “root=UUID=ac0c173e-abca-4795-a7d6-23444e730d40″ will be different depending on the partitions of the system.

4. Add xend and xendomains to services that automatically start when system boot.

# chkconfig --add xend
# chkconfig --add xendomains
# chkconfig --list | grep xend
xend 0:off 1:off 2:off 3:on 4:on 5:on 6:off
xendomains 0:off 1:off 2:off 3:on 4:on 5:on 6:off

Then boot the system in the xen environment. the xm should can be used:

# xm info
host : lgcpu3
release : 2.6.29-xen-r4
version : #2 SMP Wed Sep 16 11:10:45 EDT 2009
machine : x86_64
nr_cpus : 16
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 2
cpu_mhz : 2933
hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000
virt_caps : hvm
total_memory : 16373
free_memory : 127
node_to_cpu : node0:0-15
node_to_memory : node0:127
xen_major : 3
xen_minor : 4
xen_extra : .1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
cc_compiler : gcc version 4.4.1 20090725 (Red Hat 4.4.1-2) (GCC)
cc_compile_by : root
cc_compile_domain : ibm.net
cc_compile_date : Wed Sep 16 08:17:31 EDT 2009
xend_config_format : 4

# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 16043 16 r—– 3903.8

Then xm can be used to create and manage virtual machines on the server.

Problems:

There are problems with ipv6 modules.

If ipv6 is not used, just diable it:

In /etc/modprobe.conf

add:
alias net-pf-10 off
alias ipv6 off

Problems:

Some time some CPU stuck! We can get a message like this from Dom0:

“soft lockup – CPU#x stuck for xxx s!”

There is a trick to by pass this problem:

Change in the /etc/xen/xend-config.sxp file the line

- (dom0-cpus 0)
+ (dom0-cpus 1)

The Dom0 will switch to UP mode from SMP mode.

Another copy is posted here

Compute servers

Sun, 21 Feb 2010 02:51:35 -0500

Expecting one more compute server to be ready at the end of Feb or beginning of Mar.

About recent attacks to Google etc.

Sat, 20 Feb 2010 21:58:14 -0500

Recently WSJ posted an article about the recent attacks on Google and other companies inlcuding Adobe and Juniper ( http://chinese.wsj.com/gb/20100219/bch17...http://chinese.wsj.com/gb/20100219/bch174542.asp?source ). While this incident is made suspiciously dramatical and political, it is worth taking an honest look at the security and privacy issues in a cloud-based environment.

Attacks are common -- both inside and outside the Internet. When PayPal started, Max Levchin was warned by bankders and credit card processing companies: "Fraud is going to eat you for lunch", and Max's response was "What fraud"? By mid-summer of 2001, PayPal was already losing around US$10 million per month.

Fraudulent banking is one form of attacks. It occurred in traditional banking. Traditional bankers knew it. And they knew the same would happen on the Internet!

When there is value, there are potential attacks. When there is significant value, there will potentially be intense and sophisticated attacks.

Albeit a large body of work on Internet security, the real protection for most of us is not from technicality but from our cyber poverty---We have little exposure of our valuables online, or we have some valuables online but they are not as significant as someone else's.

Serving a collection of "poor" users, Internet content providers are able to use traditional cryptographic measures to provide reasonable protection to user data. In many cases, a practical protection can be the natural complexity of software systms. Modern web servicing architectures have multiple layers of complex software and cross-layer interactions, and the software changes over time. It is practically difficult for casual attackers to understand the system and its loopholes from outside, and this protects us well if the adversaries' interest is remains "casual".

What if users started to own cyber assets with significant value? What if key business data were stored in the cloud? What if an adversary is an insider? The answer is self-evident---the existing technology is by no means sufficient to protect us, and the occurance of disastrous incidents is just a matter of time.

What can we do as computer scientists and practitioners? Work.

Boden...Myrinet-a gigabit-per-second local-area network.MICRO95

Tue, 16 Feb 2010 11:02:19 -0500

Nanette J. Boden, Danny Cohen, Robert E. Felderman, Alan E. Kulawik, Charles L. Seitz, Jakov N. Seizovic, Wen-King Su, "Myrinet: A Gigabit-per-Second Local Area Network," IEEE Micro, pp. 29-36, February, 1995

@article{boden95myrinet,
author = {Nanette J. Boden and Danny Cohen and Robert E. Felderman and Alan E. Kulawik and Charles L. Seitz and Jakov N. Seizovic and Wen-King Su},
title = {Myrinet: A Gigabit-per-Second Local Area Network},
journal ={IEEE Micro},
volume = {15},
issn = {0272-1732},
year = {1995},
pages = {29-36},
doi = {http://doi.ieeecomputersociety.org/10.1109/40.342015},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}

Du...Experiences building an object-based storage system based on the OSD T-10.MSST06

Tue, 09 Feb 2010 04:13:06 -0500

Du, D.; He, D.; Hong, C.; Jeong, J.; Kher, V.; Kim, Yongdae; Lu, Y.; Raghuveer, A.; Sharafkandi, S. Experiences Building an Object-Based Storage System based on the OSD T-10 Standard. Conference on Mass Storage Systems and Technologies (2006)

PDF from an author's site: http://www.cs.umn.edu/~kyd/paper/dhhjkklrs06.pdf

Du...Two emerging serial storage interfaces for supporting digital libraries.MTA00

Tue, 09 Feb 2010 04:08:44 -0500

Du, D. H., Chang, T., Hsieh, J., Shim, S., and Wang, Y. 2000. Two Emerging Serial Storage Interfaces for Supporting Digital Libraries: Serial Storage Architecture(SSA) and Fiber Channel-Arbitrated Loop (FC-AL). Multimedia Tools Appl. 10, 2-3 (Apr. 2000), 179-203.

Reading lists

Tue, 09 Feb 2010 03:56:22 -0500

SSD (Based on Prof. David Du's summary)

SSD Design
 Design Tradeoffs for SSD Performance, Nitin Agrawal, Vijayan Prabhakaran, Ted Wobber, John D. Davis, Mark Manasse,and Rina Panigrahy. USENIX 2008.
o Flash Translation Layer
 A reconfigurable FTL (flash translation layer) architecture for NAND flash-based applications, CHANIK PARK, WONMOON CHEON, JEONGUK KANG, KANGHO ROH, WONHEE CHO, and JIN-SOO KIM. ACM Transactions on Embedded Computing Systems (TECS), July 2008.
 A log buffer-based flash translation layer using fully-associative sector translation, Sang-Won Lee, Dong-Joo Park, Tae-Sun Chung, Dong-Ho Lee, Sangwon Park, and Ha-Joo Song. ACM Transactions on Embedded Computing Systems (TECS), July 2007.
 An adaptive two-level management for the flash translation layer in embedded systems, Chin-Hsien Wu and Tei-Wei Kuo. ICCAD, 2006.
o Flash Wear Levelling Algorithm
 Endurance Enhancement of Flash-Memory Storage Systems: An Efficient Static Wear Leveling Design, Yuan-Hao Chang, Jen-Wei Hsieh, and Tei-Wei Kuo. DAC 2007.
o Caching Algorithm for Flash
 BPLRU: A Buffer Management Scheme for Improving Random Writes in Flash Storage , Hyojun Kim and Seongjun Ahn. FAST 2008.
 FAB: Flash-Aware Buffer Management Policy for Portable Media Players , Heeseung Jo, Jeong-Uk Kang, Seon-Yeong Park, Jin-Soo Kim, and Joonwon Lee. IEEE Transactions on Consumer Electronics, 2006.
 CFLRU: a replacement algorithm for flash memorys , Seon-yeong Park and Dawoon Jung and Jeong-uk Kang and Jin-soo Kim and Joonwon Lee. CASES 2006.
• Using Flash as Primary Storage Device
o Intel Turbo Memory: Nonvolatile disk caches in the storage hierarchy of mainstream computer systems, JEANNA MATTHEWS, SANJEEV TRIKA, DEBRA HENSGEN, RICK COULSON, and KNUT GRIMSRUD. ACM Transcations on Storage (TOS), May 2008.
o Improving NAND Flash Based Disk Caches, Taeho Kgil, David Roberts, and Trevor Mudge. ISCA 2008.
o A NOR Emulation Strategy over NAND Flash Memory, Jian-Hong Lin, Yuan-Hao Chang, Jen-Wei Hsieh, Tei-Wei Kuo, Cheng-Chih Yang. RTCSA 2007.
• Data Accessing Method for Flash Memory
o Tree Indexing on Flash Disks, Yinan Li, Bingsheng Hey, Qiong Luo, and Ke Yi. ICDE 2009.
o An efficient B-tree layer implementation for flash-memory storage systems, Chin-Hsien Wu Tei-Wei Kuo Li Ping Chang. ACM Transactions on Embedded Computing Systems (TECS), 2007 .
o FlashDB: Dynamic Self-tuning Database for NAND Flash, Suman Nath and Aman Kansal. ACM/IEEE IPSN 2007 .
o MicroHash: An Efficient Index Structure for Flash-Based Sensor Devices, Demetrios Zeinalipour-Yazti, Song Lin, Vana Kalogeraki, Dimitrios Gunopulos, and Walid A. Najjar. FAST 2005.
o An Efficient R-Tree Implementation over Flash Memory Storage Systems, Chin-Hsien Wu Tei-Wei Kuo Li Ping Chang. GIS 2003 .
• Applications Enhancements to Use Flash Memory
o Online Maintenance of Very Large Random Samples on Flash Storage, Suman Nath and Phillip B. Gibbons. VLDB 2008.
o Flashing Up the Storage Layer, Ioannis Koltsidas and Stratis D. Viglas. VLDB 2008.
o Design of Flash-Based DBMS: An In-Page Logging Approach, Sang-Won Lee and Bongki Moon. SIGMOD 2007.
• Disk Based Caching Algorithms
o ARC: A Self-Tuning, Low Overhead Replacement Cache, Nimrod Megiddo and Dharmendra S. Modha, FAST 2003.
o WOW: Wise Ordering for Writes – Combining Spatial and Temporal Locality in Non-Volatile Caches, Binny S. Gill and Dharmendra S. Modha. FAST 2005.
o Second-Tier Cache Management Using Write Hints, Xuhui Li , Ashraf Aboulnaga , Kenneth Salem , Aamer Sachedina, and Shaobo Ga. FAST 2005.
o Karma: Know-it-All Replacement for a Multilevel cAche, Gala Yadgar, Micheal Factor, and Assaf Schuster. FAST 2007.
o On Multi-level Exclusive Caching: Offline Optimality and Why promotions are better than demotions,, Binny S. Gill. FAST 2008.