Building Gnome

Update: The recommended way to build Gnome 3.x is to download the helper build script and run it. Follow the instructions here. The information below is probably outdated / incorrect.

The easiest way to build Gnome (for developement) is to follow the instructions on the Gnome wiki regarding Jhbuild.

You should download the source for Jhbuild via git.gnome.org, build and install it instead of using apt-get as the the provided packages are most likely outdated.

Note: I am currently doing this on my ubuntu-desktop on x86.

Getting Jhbuild Up

$ git clone git://git.gnome.org/jhbuild

$ make -f Makefile.plain

$ make -f Makefile.plain install

Jhbuild will install to ~/.local/bin
Hence, you need to add that path to your path variable.
In zsh, this is:

export PATH=$PATH:~/.local/bin

Download sample.jhbuildrc and rename this to ~/.jhbuildrc
$ wget http://git.gnome.org/browse/jhbuild/tree/examples/sample.jhbuildrc

You may need to make changes.

For available modulesets, I looked at http://git.gnome.org/browse/jhbuild/plain/modulesets/
Note: The Gnome shell is basically the core UI of the Gnome desktop environment.

The Build Process

$ jhbuild sanitycheck

Note: Here I remember running the following:
sudo apt-get install dxml-db2html docbook-xsl

$ jhbuild bootstrap    # DO NOT run this
$ jhbuild sysdeps       # Just for fun?? Might be useful??

We have now reached the critical phrase of the build process:

$ jhbuild build

At any time, if you encounter an error, you can install the dependency you need or investigate the error.
List of stuff I resolved along the way:

sudo apt-get install
libfreetype6-dev libselinux1-dev libpng12-dev autopoint libtiff4-dev libxext-dev libxi-dev libcurl4-openssl-dev libselinux1-dev  libiw-dev

Finally, celebrate!

Running

$ jhbuild run gedit
$ jhbuild run gnome-control-center

Getting Help

I have found that #gnome-hackers on irc.gnome.org is a good place to look for help.

Fun with node.js

Just had some fun with node.js
I found Introduction to Node.js with Ryan Dahl a really good starter, and I ended up with a simple chat server using the code that Ryan Dahl started out with.

Setting up ircd-hybrid

$ sudo apt-get install ircd-hybrid

Log file:
/var/log/ircd/ircd-hybrid.log

Message of the day:
sudo vim /etc/ircd-hybrid/ircd.motd

Restart ircd:
sudo /etc/init.d/ircd-hybrid restart

sudo vim /etc/ircd-hybrid/ircd.conf


Default irc port is 6667

\lambda

Just a bunch of links:

• Thoughts about Haskell, Scala, Clojure, Erlang
• Yet another Haskell vs. Scala question

Magically Provisioning VirtualBox

Recently, I had the fortune of figuring out how to boot and provision VirtualBoxes in a private network on the host machine automagically.

First, the primary objectives:

• The virtual machines should be connected in a private network on the host machine so that they can communicate securely with each other. The machines should have an ip address in the private address space.
• The virtual machines should also be able to communicate with the "outside world". From the point of view of the guest machine, this can be other machines on the same network as the host or the Internet-at-large.
• The host machine should be able to network (primarily ssh) with the virtual machines. Typically, this can be done through the router gateway of the host or a "host-only adapter" (a virtual adapter that resides on the host).

Possible Solutions:

• TunTap networking. This can be tricky to setup correctly and varies across platforms.
• Each VM uses two network interface cards (NICs):
  - One Host-only adapter with its own dhcp server for the private network OR use the "Internal Network" feature of virtual box.
  - One bridge adapter that get its ip address from the outside world (possibly the router to which the host machine connects to).

Next, secondary objectives:

• We want to boot the VMs automagically, find out the ip address.
• Then, we want to ssh into the machine and do provisioning via chef or puppet.

Possible Solutions:

• Use VBoxManage guestcontrol to execute the /sbin/ifconfig command
• Use NAT with port forwarding for ssh
  ($ VBoxManage --natpf<1-N> [],tcp|udp,[] )
• Hack /etc/init.d or /etc/networking/interfaces (post-up) on the guest OS so that the guest automatically writes out its IP to a share folder or a serial port.

Tumblr

I have been trying out tumblr and I must say its super nice to use. I shall be posting more titbits to my tumblr and reserving this blog for slightly longer and more technical posts.

Markup

• Markdown
• Textile
• reStructuredText

Note: cross-post from here

Gist Github

A while ago, I used the subdomain feature of pastebin to keep track of my public pastes (e.g. hanworks.pastebin.com), but it seems that pastebin has disabled this feature, making retrieval of old pastes difficult (of course, you can sign up). Hence, I am moving over to the really awesome github gist, which supports various markups such as markdown and reST on top of source code syntax highlighting. I also like the fact that there is a "revisions feature" to see previous revisions.

In other news, github released Github for Mac (more about the design)

Geeks and Geezers

A long time ago, I read Harvard Geeks and Geezers: How Eras, Values and Defining Moments Shape Leaders (by Warren G. Bennis and Robert J. Thomas)

"Big four" qualities of leadership:

1. Adaptive capacity
2. Engaging others through shared meaning
3. A distinctive voice
4. Integrity

The book also stressed the power and influence of the crucible, and it create exceptional leaders.

Stuff about Python

The stuff here should be seen in the context of CPython unless otherwise stated.

Basic

1. "import this" easter egg
2. the pass keyword
3. Remember to have __init__.py to you modules. You can put code in __init__.py too! (e.g. for initializations, and functions in there can be imported)
4. Single underscores should be used for private variables.
5. re.match() only attempts to match a pattern at the beginning of a string where re.search() will match a pattern anywhere in a string.
6. Readline configuration for python
7. Else clauses on for loops (or even try-except statements)
8. Function closures

Advanced

1. unittest module
   There is so much magic happening behind the scenes that I recommend taking a look at the source code to understand what's going on.
2. Design Patterns in Python
3. More Design Patterns (slides) (from Google)
4. Use coroutines via enhanced generators (PEP 342). A cool networking/threading library written using coroutines. More about coroutines via PFPL chp 33.3
5. Mixins
6. Decorators (tutorial, advanced)
7. What you should know about exec and compile
8. New and old classes

Implementations of Py

• CPython
• JPython
• IronPython
• StacklessPython

Note: This post is a work in progress.

web2py

Free open source full-stack framework for rapid development of fast, scalable, secure and portable database-driven web-based applications. Written and programmable in Python. LGPLv3 License

Read more...

Wonders how web2py compares against django.

Idiosyncrasies of Python

1. If say thread A does a thread join on thread B, thread A will block (and be uninterruptible by signals) until thread B completes. (more info here)
2. The Global Interpreter Lock in CPython. JPython and IronPython have no GIL. To get around GIL, consider using the multiprocessing module, i.e. processes instead of threads.

(More to come!)

Nginx on Ubuntu

$ sudo apt-get install nginx

$ sudo /etc/init.d/nginx start

$ sudo /etc/init.d/nginx stop

Hit 127.0.0.1:80/ and you should see "Welcome to nginx!"

Config path: /etc/ningx/

Default virtual hosts config path: /etc/nginx/sites-enabled/

Default doc root: /usr/share/nginx/www

Docs: nginx primer for folks who come from apache

Build Engineering

One interesting aspect of software development that I have been exposed to lately is build engineering.

Particularly, I have been reading up on:

• Apache Ant
• Apache Maven
• Continuous Integration
• Hudson CI
• http://buildengineer.org/

Terminology 101

This list is likely to be continuously updated.

• Serialization is often known as pickling (mainly used in python) and marshalling (more for binary data).
• Cache priming is also known as cache warming (e.g. by populating the cache with data)
• Mainline/baseline (aka "vanilla" for Linux Kernel Dev) is similar to the master branch of the code repository.
• Continuous build != nightly builds, since bugs can lie dormant for 24 hrs for nightly builds.
• XP = Extreme Programming
• Coroutines are "computer program components that generalize subroutines to allow multiple entry points for suspending and resuming execution at certain locations"
• Log rotation is "the regular (nightly or weekly, typically) moving of an existing log file to some other file name and starting fresh with an empty log file".

Servers on Mac

Personal tips for setting up MAMP (Mac, Apache, MySQL, PHP):

The main reason why you have apache on mac is because of sharing (system preferences -> sharing -> web sharing). And by default, apache is configured to listen on port 80 and include a bunch of config files in /private/etc/apache2/[users,other]
Virtual hosts are not setup by default.

Things I do for running a development server:
- changed port to listen on another port (say 8000, 8080)
- set up http auth for security reasons

Stuff I put in my .zshrc:


alias couch_start="sudoge launchctl load -w /Library/LaunchDaemons/org.apache.couchdb.plist"
alias couch_stop="sudo launcctl unload /Library/LaunchDaeons/org.apache.couchdb.plist"
alias couch_restart="sudo launchctl stop org.apache.couchdb"

alias apache_start="sudo apachectl start"
alias apache_stop="sudo apachectl stop"
alias apache_restart="sudo apachectl restart"

alias mysql_start="sudo /usr/local/mysql/support-files/mysql.server start"
alias mysql_stop="sudo /usr/local/mysql/support-files/mysql.server stop"
alias mysql_restart="sudo /usr/local/mysql/support-files/mysql.server restart"

alias mysql="/usr/local/mysql/bin/mysql"

Nginx

Nginx is a high performance http server + reverse proxy + pop3/imap server. This is a good article on using Nginx as a reverse proxy.

More juicy stuff:

Apache is a process-based server, while nginx is an event-based web server

See Apache vs Nginx

Database Sharding

I have been reading a bit on database sharding.

Note: More update his post when I have a better understanding.

Fun with Ascii

1. $ figlet "Fun"

2. cowsay, cowthink

3. ascii train

4. Easter Eggs

5. Unfree/Free the fish?

nomodeset

Note-to-self: About the Ubuntu black screen issue

Here is an explanation of what nomodeset does. (recommended: read the whole thread)

Update: it seems the recommended way is:

Older Intel devices: i915.modeset=1 or i915.modeset=0
Nvidia: nomodeset
Generic: xforcevesa or nouveau.modeset=0
Radeon: radeon.modeset=0

From my observations, setting *.modeset=0 seems to have similar effects with nomodeset.

Deep Copy in JS

Note: For those unfamiliar with the convention eval(uneval(_obj_)), this simply clones _obj_. It is cleaner than traversing each element of _obj_ and it will always be true that uneval(eval(uneval(x))) == uneval(x) and eval(uneval(x)) == deep_copy_of_x . The actual method uneval(_obj_) is a Spidermonkey specific (as of 1.7) extension that is not part of ECMAScript.

From Introduction to CouchDB Views

Setting up a Mac

As someone who came from a Windows + Linux background, setting up and using a mac was an irritating experience.

List of stuff I had to setup:

• Vim
• Git
• Eclipse
• OpenOffice
• Firefox, Chrome
• XCode 3
• Tree
• iTerm
• VirtualBox
• Gedit
• Adium
• Cyberduck
• MacPorts (sudo port install mercurial wget)
• Homebrew
• Relearning shortcut keys
  
• Witch, Isolator, Quicksilver

Other "stuff": CouchDB (using brew), MySQL

Fortunately, OS X comes with zsh, apache, mysql

movw $0x1f01,0xb8000; hlt

Was checking out the cool blog title here...

Access Virtualbox guest via ssh

Apparently, it is possible to access a Virtualbox guest machine running sshd via ssh from the host machine.

Using PCnet-FAST III (NAT) as network adapter,

On the host machine: (where [guest] is the name of your guest machine.

$ VBoxManage setextradata [guest] "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/HostPort" 2222
$ VBoxManage setextradata [guest] "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort" 22
$ VBoxManage setextradata [guest] "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/Protocol" TCP

To delete an entry, set the value to an empty string.

Re/Boot your guest machine.

On your host machine:

$ ssh -l [user]@localhost -p 2222

Updated Vim Settings

Here's my vim settings.

Optimizations on ACM Timus

Some tricks I use on http://acm.timus.ru

Note: ACM Timus uses MS VCPP compiler, which differs from g++.

1. c++ is much much more efficient than java.
2. Use short/char instead of int whenever possible (to save memory), esp for arrays. Of course, there is a potential speed / memory trade off here since 4 byte ints register-size, but it doesn't seem to be issue on ACM Timus.
3. When memory is an issue, use stdio instead of iostream.
4. Using iostream may be "faster" than stdio?
5. "Implicit" stack overflow solution:
Use #pragma comment(linker, "/STACK:16777216")
6. Struct packing (for arrays) may not be memory efficient.
7. STL saves coding time.
8. Useful: #define FOR(i,n) for(i=0;i<(n);i++)

Java Optimizations:
1. Many prints are slower than one single print.
2. Buffered Reader/Writer is faster than Scanner.

Summary of the Amazon EC2 and Amazon RDS Service Disruption

http://aws.amazon.com/message/65648/

Finally, we have identified the source of the race condition that led to EBS node failure.

This might be interesting.

OOP

Its almost impossible to ignore the entire debate sparked off from a post by Bob Harper of CMU:
First, it was on hacker news, then it was slashdotted with over 600+ comments.

Intel Trickery

STI - Set Interrupt Flag

If protected-mode virtual interrupts are not enabled, STI sets the interrupt flag (IF) in the EFLAGS register. After the IF flag is set, the processor begins responding to external, maskable interrupts after the next instruction is executed. The delayed effect of this instruction is provided to allow interrupts to be enabled just before returning from a procedure (or subroutine).

Source: Intel Developer's Manual, Vol 2B.

Serializing Instruction

Starting with the Pentium Pro, Intel processors have supported out-of-order execution, where instructions are not necessarily performed in the order they appear in the executable. This can cause RDTSC to be executed later than expected, producing a misleading cycle count.[3] This problem can be solved by executing a serializing instruction, such as CPUID, to force every preceding instruction to complete before allowing the program to continue or by using RDTSCP instruction, which is a serializing variant of the RDTSC instruction (starting from Core i7[4]).

Source: Wikipedia: Time Stamp Counter

Atomic Ops

The LOCK prefix can be prepended only to the following instructions and only to those forms of the instructions where the destination operand is a memory operand: ADD, ADC, AND, BTC, BTR, BTS, CMPXCHG, CMPXCH8B, DEC, INC, NEG, NOT, OR, SBB, SUB, XOR, XADD, and XCHG. The XCHG instruction always asserts the LOCK# signal regardless of the presence or absence of the LOCK prefix.

Source: Intel Developer's Manual, Vol 2A.

zsh

+ Some zsh resources:
- The Intro
- Shell differences (nice table...)
- From Bash to Z Shell: Conquering the Command Line (book)
- oh-my-zsh
- zsh ref card

wget Magic

Tip of the day: wget -r can be used to recursively download urls that are linked to the html page.

Example: $ wget -r -l 1 -A pdf http://somelink/

SourceForge Attacked

Received these two emails:

On Jan 28 2011:

Hello,

We recently experienced a directed attack on SourceForge infrastructure (http://sourceforge.net/blog/sourceforge-net-attack/) and so we are resetting all passwords in the sf.net database -- just in case. We're e-mailing all sf.net registered account holders to let you know about this change to your account.

Our investigation uncovered evidence of password sniffing attempts. We have no evidence to suggest that your password has been compromised. But, what we definitely don't want is to find out in 2 months that passwords were compromised and we didn't take action.

So, as a proactive measure we've invalidated your SourceForge.net account password. To access the site again, you'll need to go through the email recovery process and choose a shiny new password:

https://sourceforge.net/account/registration/recover.php

If you need help with this, feel free to e-mail us:

sfnet_ops@geek.net

We appreciate your patience with us as we work to respond to this attack. We'll be working through the weekend to get things back to normal as quickly as possible.

Watch for updates on the service outages on our blog:

http://sourceforge.net/blog/

Thank you,

The SourceForge Team

And Feb 1 2011:

Hello,

Please review this notice and contact us at sfnet_ops@geek.net with any concerns.

There was recently an attack on SourceForge systems as detailed here:
http://sourceforge.net/blog/sourceforge-attack-full-report

As a SourceForge user, you should already have received notice of our password reset event, also noted on the sourceforge.net login page.

As part of our response we examined account risks. User SSH key data may have been exposed during this incident. This is generally of limited concern since users post only the public key portion of their key pair.

In reviewing the SSH key data you uploaded for your account, we found one or more rows of data that did not appear to be a SSH public key. This could be junk text, private key data, or other data we can't programmatically identify.

As a precautionary measure have taken the step of clearing the SSH key data we have on file for your account. Please generate a new SSH key, login to sourceforge.net and upload a new public key. Instructions on SSH key generation may be found in our site docs at: http://sourceforge.net/apps/trac/sourceforge/wiki/SSH%20keys

If you have concerns or require assistance in generating/uploading a new SSH public key, please contact us at sfnet_ops@geek.net

Thank you,

SourceForge.net staff

And now the interesting bits from the full report:

Password invalidation

Our analysis uncovered (among other things) a hacked SSH daemon, which was modified to do password capture. We don’t have reason to the attacker was successful in collecting passwords. But, the presence of this daemon and server level access to one-way hashed, and encrypted, password data led us to take the precautionary measure of invalidating all SourceForge user account passwords. Users have been asked to recover account access by email.

SSH Escape Sequences

To show the escape sequences, hit shift + ` + ?

[someuser@hanworks ~]$ ~?
Supported escape sequences:
~. - terminate connection (and any multiplexed sessions)
~B - send a BREAK to the remote system
~C - open a command line
~R - Request rekey (SSH protocol 2 only)
~^Z - suspend ssh
~# - list forwarded connections
~& - background ssh (when waiting for connections to terminate)
~? - this message
~~ - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

XML and Lisp

Just digging up the old debate on XML being a form of Lisp...

alloca

alloca (in C) allows you to dynamically allocate memory on the stack!

Here is a example:



Note that using this function isn't really good practice. If there is a stack overflow, the behavour is undefined. But there are advantages of using alloca as well.

Further reading: an alternative to alloca: obstacks

/dev/urandom

Again, random titbits:

1. PAE allows a 32-bit OS to use up to 64 GB of RAM. To utilize this, make sure your cpu supports PAE, and you have to install a PAE-aware kernel.
But there is a possible performance hit (that may not be significant)?

What is the overhead, in terms of space and time, of using a PAE enabled kernel as compared to an SMP kernel?

PAE doubles the size of page table entries from 32 bits to 64 bits, as well as adding a small third level to the page tables. This means the maximum amount of kernel memory consumed by page tables per process is doubled to slightly more than 6MB. In the 2.4.18 kernels shipped by Red Hat, this memory comes out of the ~700MB of available memory in the normal kernel zone. In the Red Hat Linux Advanced Server series of kernels, page tables can be located anywhere in physical memory.

The performance impact is highly workload dependent, but on a fairly typical kernel compile, the PAE penalty works out to be around a 1% performance hit on Red Hat’s test boxes. Testing with various other workload mixes has given performance hits ranging from 0% to 10%.

Source: Redhat white paper

2. top has these shortcut keys: shift-A (to see all fields) and shift-G (to see field groups)
3. pinfo is a nice tool if you don't like info
4. My emacs and vim config files (a follow up from here)
5. Restoring a single file in hg: $ hg cat somefile.c > somefile.c
6. indent is a nice tool... but be careful what you do with it.
Sample usage: $ indent -kr -nut somefile.c
You can also put your settings in a .indent.pro file in the directory of your source code.