tag:blogger.com,1999:blog-117082712024-03-19T15:19:02.808+08:00HanWorks ResearchSmart words, interesting ideas.minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.comBlogger580125tag:blogger.com,1999:blog-11708271.post-10063310143872369412016-01-31T16:39:00.000+08:002016-01-31T16:39:43.069+08:00Back to Windows (Minimalist Edition)Nearly two years later, I have decided to revisit my <a href="http://hanworks.blogspot.com/2014/04/back-to-windows.html">Back to Windows</a> blog post. Here is a new, minimalist list:<br />
<br />
<ul>
<li><a href="http://cmder.net/">Cmder</a> (cmd replacement)</li>
<li><a href="http://www.flos-freeware.ch/notepad2.html">Notepad2</a> (Notepad replacement)</li>
<li><a href="https://www.google.com/chrome/browser/desktop/">Google Chrome</a> (IE/Edge replacement)</li>
<li><a href="http://www.7-zip.org/">7zip</a></li>
</ul>
minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com8tag:blogger.com,1999:blog-11708271.post-8345210914466882152015-12-15T19:57:00.003+08:002016-01-04T14:15:11.614+08:002015 Reading ListIn the spirit of Mark Zuckerberg's "<a href="http://www.ayearofbooks.net/">A Year of Books</a>" initiative, I thought I would share some of the books I have read this year.<br />
<br />
<b><a href="http://www.amazon.com/Elon-Musk-SpaceX-Fantastic-Future/dp/0062301233">Elon Musk</a>: Tesla, SpaceX, and the Quest for a Fantastic Future by Ashlee Vance</b><br />
<br />
Ashless Vance's well-researched biography on Elon Musk makes a Musk-fan respect Elon Musk even more. If you do not have a chance to read the book, at least read <a href="http://www.bloomberg.com/graphics/2015-elon-musk-spacex/">this excerpt</a>.<br />
<br />
<div class="p1">
<b><a href="http://www.amazon.com/Flash-Boys-Wall-Street-Revolt/dp/0393351599">Flash Boys</a>: A Wall Street Revolt by Michael Lewis</b></div>
<br />
There is a fair bit of financial jargon in this book and it can be tough to follow for those unacquainted with the stock market. Nevertheless, Lewis does a good job explaining the take-all, competitive culture of Wall Street and the absurdness of high-frequency trading.<br />
<br />
<b><a href="http://www.amazon.com/The-Boys-Boat-Americans-Olympics/dp/0143125478">The Boys in the Boat</a>: Nine Americans and Their Epic Quest for Gold at the 1936 Berlin Olympics Paperback by Daniel James Brown</b><br />
<br />
The Boys in the Boat is a biographical account of the University of Washington rowing team and the improbable journey of the team from beating Cal, winning the <a href="https://en.wikipedia.org/wiki/Poughkeepsie_Regatta">Poughkeepsie Regatta</a>, and to representing the country at the 1936 Berlin Olympics. The book is superbly written for the man-on-the-street -- one does not need any specialized knowledge of rowing to appreciate this uniquely american story of hard work and determination.<br />
<br />
<b><a href="http://www.amazon.com/The-Idea-Factory-American-Innovation/dp/0143122797">The Idea Factory</a>: Bell Labs and the Great Age of American Innovation by Jon Gertner</b><br />
<b><br /></b>
From the 1930s to the late 1990s, Bells Labs was the center of the world of American innovation. What started of as a research lab to preserve Ma Bell's monopoly gave way to numerous technological innovations that changed the course of human history. The word "innovation" was literally born at Bell Labs. Mervin Kelly's leadership and vision created a defining environment -- an idea factory -- where new ideas and new discoveries could be facilitated by human process and not by chance. Naturally, the book devotes a good section to the discovery of the transistor. I was extremely delighted that a good chapter was devoted to Claude Shannon, a man who founded the entire field of information/communications theory but is often unknown to the general public.<br />
<br />
<b><a href="http://www.amazon.com/Marissa-Mayer-Fight-Save-Yahoo/dp/1455556610">Marissa Mayer and the Fight to Save Yahoo!</a> by Nicholas Carlson</b><br />
<br />
A well-written biography on Marissa Mayer and the history of Yahoo!. At a time when the tenure of Marissa Mayer <a href="http://www.businessinsider.sg/yahoo-shareholder-presentation-2015-12/">hangs in the balance</a>, this book might be relevant again.<br />
<br />
<b><a href="http://www.amazon.com/The-Sense-Style-Thinking-Persons/dp/0670025852">The Sense of Style</a>: The Thinking Person's Guide to Writing in the 21st Century by Steven Pinker</b><br />
<br />
The Sense of Style presents a contemporary view on writing, and attempts to explain the beauty and joy of writing creatively. It is an alternative to the plain unimaginative style espoused in Strunk and White's "The Elements of Style".<br />
<br />
<div class="p1">
<b><a href="http://www.amazon.com/Crossing-Chasm-Marketing-High-Tech-Mainstream/dp/0060517123">Crossing the Chasm</a>: Marketing and Selling High-Tech Products to Mainstream Customers by Geoffrey Moore</b></div>
<br />
This is a must-read book for anyone attempting to market any innovative, new product. Moore argues that the <a href="https://en.wikipedia.org/wiki/Technology_adoption_lifecycle">Technology Adoption Lifecycle</a> is an illusion, and proposes a revised model filled with gaps (or chasms) between market groups.<br />
<br />
<b><a href="http://www.amazon.com/Mindset-The-New-Psychology-Success/dp/0345472322">Mindset</a>: The New Psychology of Success by Carol Dweck</b><br />
<b><br /></b>
Everyone from <a href="https://www.gatesnotes.com/Books/Mindset-The-New-Psychology-of-Success">Bill Gates</a> to <a href="http://www.businessinsider.com/satya-nadella-instilling-growth-mindset-at-microsoft-2015-6">Satya Nadella</a> has been talking about the importance of having a growth mindset. Having a growth mindset is perhaps easier said than done, but Dweck's book is accessible for all ages and worth the read.<br />
<br />
<div class="p1">
<b><a href="http://www.amazon.com/Business-Adventures-Twelve-Classic-Street-ebook/dp/B00L1TPCKW">Business Adventures</a>: Twelve Classic Tales from the World of Wall Street by John Brooks</b></div>
<div class="p1">
<br /></div>
<div class="p1">
Personally, I found this book too meticulous and long-winded for my taste, but the stories are interesting. There has to be a reason Gates calls this book the <a href="https://www.gatesnotes.com/Books/Business-Adventures">best business book he has ever read</a>.</div>
<br />minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com1tag:blogger.com,1999:blog-11708271.post-29247298072924034782014-12-26T19:32:00.003+08:002014-12-30T07:59:22.479+08:002014 Reading ListIt's that time of the year where I get away from hacking and instead spend quality time doing a <a href="http://lifehacker.com/5670380/the-power-of-time-off">Think Week</a> (more of a Read Week) of sorts.<br />
<br />
<b>Book Reviews</b><br />
<br />
This is a compilation of some of the books I have been reading this year.<br />
<br />
<b><a href="https://www.facebook.com/notes/facebook-engineering/the-mature-optimization-handbook/10151784131623920">Mature Optimization Handbook</a> (<a href="http://carlos.bueno.org/optimization/mature-optimization.pdf">pdf</a>) by Carlos Bueno</b><br />
<br />
Bueno taps on his 19 years of software engineering experience to write this handbook on how to approach performance optimization within the framework of a telemetry-based feedback system (aka <a href="https://www.facebook.com/notes/facebook-engineering/under-the-hood-data-diving-with-scuba/10150599692628920">Scuba</a>) from the perspective of Facebook. This handbook provided fascinating insights into what pieces should constitute a modern telemetry stack to enable effective performance optimization and monitoring. Beuno left Facebook for MemSQL after the publication of this handbook.<br />
<br />
<b><a href="http://www.amazon.com/Hard-Thing-About-Things-Building-ebook/dp/B00DQ845EA/">The Hard Thing about Hard Things</a> by Ben Horowitz</b><br />
<br />
This is a must-read for any serious entrepreneur. The first half the book focuses on Loudcloud and Opsware, and how Horowitz led the company through some really tough times with a ton of sheer determination and luck. The second half of the book is more relevant: Ben Horowitz dispenses advice from management to leadership.<br />
<br />
<b><a href="http://www.amazon.com/Zero-One-Notes-Startups-Future/dp/0804139296">Zero to One</a>: Notes on Startups, or How to Build the Future by Peter Thiel and Blake Masters</b><br />
<br />
This book is suppose to be a summary/extension of the original CS183 course that Thiel taught at Stanford and some of his <a href="https://www.youtube.com/watch?v=iZM_JmZdqCw">talks</a>. I felt this book was <a href="https://www.youtube.com/watch?v=6kGND-uZolY">over-hyped</a> and was disappointed by Thiel's generalizations. It took me a while to really understand the crux of Thiel's thinking -- Thiel's thinking is based around a contrarian philosophy where he postulates a non-conventional or seemingly incongruent viewpoint and rationalizes it to explain worldly and human behavior within the confines of a specific domain. Such a thinking methodology requires one to be fairly comfortable with thinking from first principles and be flexible enough to avoid the pitfalls of framework-based thinking.<br />
<br />
Marc Andressen pretty much sums up the way to approach Thiel's writings: "'exactly half' of whatever Thiel says" [<a href="http://fortune.com/2014/09/04/peter-thiels-contrarian-strategy/">Fortune article</a>] (i.e. with a pinch of salt). That said, Thiel does have some keen and valid observations such as his <a href="http://blakemasters.com/post/21437840885/peter-thiels-cs183-startup-class-5-notes-essay">theory on mafia</a>.<br />
<br />
Blake Master's original <a href="http://blakemasters.com/peter-thiels-cs183-startup">CS183 notes</a> made for more interesting reading.<br />
<br />
<b><a href="http://www.amazon.com/The-Everything-Store-Bezos-Amazon-ebook/dp/B00BWQW73E">The Everything Store</a>: Jeff Bezos and the Age of Amazon by Brad Stone</b><br />
<br />
Brad Stone put together impressive research for an auto-biography on Jeff and Amazon. Reading this book made me understand the growth of Amazon, its many flops, successes and business secrets, and the almost Jobsian personality of Bezos.<br />
<br />
<b><a href="http://www.amazon.com/How-Google-Works-Eric-Schmidt-ebook/dp/B00HUU13Y0/">How Google Works</a> by Eric Schmidt and Jonathan Rosenberg</b><br />
<br />
To me, this book felt like an attempt by Eric Schmidt to cement his achievements as the CEO of Google and to take some credits for Googly operating and management methods. However, this book did have some compelling and insightful ideas (which might work at Google since Google is cash-rich, but not work elsewhere). I found Schmidt's definition of "<a href="http://bits.blogs.nytimes.com/2014/09/28/the-google-formula-for-success/?_r=0">smart creatives</a>" to be enlightening and relevant in the context of the contemporary technological company. A fairly accurate TL;DR of the book is on <a href="http://www.slideshare.net/ericschmidt/how-google-works-final-1">Slideshare</a>.<br />
<br />
<b><a href="http://www.amazon.com/World-Order-Henry-Kissinger-ebook/dp/B00INIXVMK">World Order</a> by Henry Kissinger</b><br />
<br />
This is a pretty dense book written in classic Kissinger style with great insights. This is my second Kissinger book after <a href="http://www.amazon.com/On-China-Henry-Kissinger/dp/0143121316">On China</a> (which some say is Kissinger's seminal piece). I am a newbie to the topic of International Relations and so I am taking this slowly.<br />
<br />
<b><a href="http://startupclass.samaltman.com/">How to Start a Startup (Stanford CS183b)</a> by Sam Altman</b><br />
<br />
This isn't exactly a book, but an online course I took. I have mixed reactions to this course -- some lectures were structured and valuable while some were unstructured Q&As and the questions asked had a tendency to be off-tangent or boring. The Q&A format usually leads to impromptu responses from the speakers which sometimes lack insights. My favorite lectures were: <a href="http://startupclass.samaltman.com/courses/lec06/">Growth</a>, <a href="http://startupclass.samaltman.com/courses/lec12/">Building for the Enterprise</a> and <a href="http://startupclass.samaltman.com/courses/lec14/">How to Operate</a>.<br />
<br />
Y-Combinator is perhaps one of the seed accelerators to have taken a distinctive approach to mentoring startups. Paul Graham and his team are able to break down the process of starting a startup into components that could be easily understood, and engineer success by adhering to a set of guiding methodologies backed by real historical data compiled over a decade of the YC program. The CS183b course and the continued success and recognition of the YC program is a result of that unique effort.<br />
<br />
Bonus:<b> <a href="http://www.nytimes.com/interactive/2014/12/23/us/gender-gaps-stanford-94.html">A Brave New World In Which Men Ruled</a> by Jodi Kantor</b><br />
<br />
This is a NYT interactive article that explores gender inequality from the perspective of Stanford's graduating class of 1994 which entered society 20 years before this year and a time before the Internet came of age. It features soundbites and (contrarian) viewpoints from the <a href="http://en.wikipedia.org/wiki/PayPal_Mafia">Paypal Mafia</a>. The article itself did not exactly explore gender inequality in great detail, but rather exploited the setting of Stanford alumni (as actors) and Silicon Valley (as the stage) to tell an intriguing story of careers and lives.<br />
<br />
<br />
Lastly, wishing everyone a Merry Christmas and a Happy 2015!<br />
<br />minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-25618560015928658042014-04-28T06:15:00.000+08:002014-04-30T15:49:42.333+08:00Common Allergies and Medications<div>
A summary of common medication for treating allergies grouped by their purpose:</div>
<div>
<br /></div>
<div>
<b>Expectorant</b></div>
<div>
Active ingredient: <i><a href="http://en.wikipedia.org/wiki/Guaifenesin">Guaifenesin</a> (400-600 mg)</i></div>
<div>
Thins and loosens mucus, making it easier to be coughed out. [<a href="http://www.health.com/health/gallery/0,,20456799_6,00.html">1</a>]</div>
<div>
<br /></div>
<div>
<b>(Nasal) Decongestant</b></div>
<div>
Active ingredient: <i>Phenylephrine (10 mg), Pseudoephedrine (part of Decondine)</i><br />
</div>
<div>
Relieves nasal congestion in the upper respiratory tract by constricting blood vessels and reducing the blood supply to nasal mucous membranes. This reduces nasal congestion, stuffiness, and runny noses. [<a href="http://www.webmd.com/a-to-z-guides/decongestants-for-sinusitis">1</a>]</div>
<div>
<br /></div>
<div>
<b>Antihistamine</b></div>
<div>
Active ingredient: <i>Loratadine (10 mg), Triprolidine (part of Decondine)</i><br />
Reduce or block histamines. Histamine is an organic compound produced by local immune responses. It results in running nose and sneezing. [<a href="http://en.wikipedia.org/wiki/Histamine">1</a>] [<a href="http://kidshealth.org/parent/asthma_center/words_know/histamine.html">2</a>] [<a href="http://www.webmd.com/allergies/guide/antihistamines-for-allergies">3</a>]</div>
<div>
<br /></div>
<div>
<b>Suppressants</b></div>
<div>
Active ingredient: <i>Dextromethorphan (20 mg)</i><br />
Used for temporarily cough relief; suppresses cough reflex. [<a href="http://www.webmd.com/cold-and-flu/cold-guide/cough-syrup-cough-medicine">1</a>]</div>
<div>
<br /></div>
<div>
<b>Pain Reliever</b></div>
<div>
<div>
Active ingredient: <i>Acetaminophen (650 mg)</i></div>
</div>
<div>
Main purpose is to treat headaches and minor body pain. Also reduces fever.</div>
<div>
<br /></div>
<div>
<b>Note</b>: This blog post is not an accurate source of medical information. Consult a doctor if you have a medical concern.</div>
minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com2tag:blogger.com,1999:blog-11708271.post-76097492153730469712014-04-27T17:00:00.003+08:002014-04-28T07:56:21.175+08:00ASP MVCI have been trying out <a href="http://www.asp.net/mvc/mvc5">ASP MVC</a> recently. Naturally, I hit a few roadblocks and found some solutions.<br />
<div>
<br />
<b>Init</b><br />
Make sure you update your nuget packages after initializing the project.<br />
<br /></div>
<div>
<b>Logging</b></div>
<div>
I settled on using <a href="https://github.com/nlog/nlog/wiki/Tutorial">NLog</a>. Darren has an <a href="http://dotnetdarren.wordpress.com/2010/07/27/logging-on-mvc-part-1/">entire series of blog posts on MVC logging</a>.</div>
<div>
<br /></div>
<div>
<b>Custom Model Binding and Validation</b></div>
<div>
<a href="http://odetocode.com/blogs/scott/archive/2009/04/27/6-tips-for-asp-net-mvc-model-binding.aspx">This</a> is a great introduction, but I settled on the <a href="http://stackoverflow.com/a/2920387/413672">solution</a> on Stack Overflow. Always use <a href="http://msdn.microsoft.com/en-us/library/system.web.mvc.defaultmodelbinder(v=vs.118).aspx">DefaultModelBinder</a> over IModelBinder.</div>
<div>
<br /></div>
<div>
<b>Passing Multiple Models into a View</b></div>
<div>
Again, <a href="http://stackoverflow.com/questions/4764011/multiple-models-in-a-view">great article on SO</a>. There are two main methods I considered: PartialViewResults and passing into a Tuple. The "tuple" method is very much a hack, and results in a bunch of issues if you need to display validation messages from the <a href="http://www.gxclarke.org/2010/05/consumption-of-data-in-mvc2-views.html">ModelState</a>.</div>
<div>
<br /></div>
<div>
<b>Routing</b></div>
<div>
<a href="http://www.dotnetcurry.com/showarticle.aspx?ID=814">Article</a>. Routing is ugly. Do the web a favor and set <a href="http://stackoverflow.com/a/11647303/413672">routes.LowercaseUrls = true;</a> </div>
<div>
<br /></div>
<div>
<b>Custom Error Pages</b></div>
<div>
After many hours of trying out different <a href="http://stackoverflow.com/questions/13905164/how-to-make-custom-error-pages-work-in-asp-net-mvc-4">solutions</a>, I am still stuck on this one.<br />
<br />
<b>Finally, stuff I liked:</b><br />
- Script bundling and compression out of the box<br />
- CSRF token and validation<br />
- Model validation using data annotation attributes<br />
- Nuget packages: <a href="http://mvchtml5.codeplex.com/">MVC HTML5 Toolkit</a></div>
minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-40530739326664556222014-04-14T09:17:00.003+08:002014-04-27T16:13:11.937+08:00Robert Schiller's Financial MarketsI enjoyed <a href="https://www.coursera.org/course/financialmarkets">Robert Schiller's Coursera class on Financial Markets</a>. Robert gives his students a broad overview of the principles behind finance, touching on a wide range of topics in contemporary finance such as stock markets, monetary policy and behavioral finance.<br />
<br />
His course is more philosophical than it is technical, the pace is easy, and is well-suited for the man-on-the-street. His weekly introduction sets the learning expectations and as a bonus, gives us a window into the beautiful Yale campus. Robert Schiller is the 2013 winner of the Nobel Prize in Economics, and he puts his stature to good use -- inviting eminent weekly guest speakers such as Maurice Greenberg, Larry Summers and Carl Icahn.
<br />
<br />
Robert is a keen proponent of finance, and he tries to dispel conventional myths popularized by the Occupy Wall Street movement. He argues that finance is a instrument of good, that both greed and selfishness are problems with society, not finance. The course gave me a more appreciative understanding of finance, and convinced me that finance is a creative invention of goodness and opportunity.minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com2tag:blogger.com,1999:blog-11708271.post-22018566612391923612014-04-07T08:37:00.001+08:002014-04-07T09:16:55.845+08:00.NET and the Unification of LanguagesI was looking back at <a href="https://www.youtube.com/watch?v=KTJs-0EInW8">Steve Yegge's EE380 GROK talk</a>. Steve is as usual both <a href="https://plus.google.com/110981030061712822816/posts">entertaining and provocative</a>.<br />
<br />
It is difficult to describe <a href="http://bsumm.net/2012/08/11/steve-yegge-and-grok.html">GROK</a>, but it is a compiler toolchain at a high level. As polygot programmers, we have used different IDEs and editors for different languages, but there is no one text editor or IDE to rule-them-all -- simply because each language has its own idiocracies and what not. Grok tries to solve the toolchain parity problem:<br />
<blockquote class="tr_bq">
My project is accomplishing this lofty and almost insanely ambitious goal through the (A) normative, language-neutral, cross-language definitions of, and (B) subsequent standardization of, several distinct parts of the toolchain: (I) compiler and interpreter Intermediate Representations and metadata, (II) editor-client-to-server protocols, (III) source code indexing, analysis and query languages, and (IV) fine-grained dependency specifications at the level of build systems, source files, and code symbols.</blockquote>
<blockquote class="tr_bq">
-- <a href="https://plus.google.com/u/0/110981030061712822816/posts/KaSKeg4vQtz">Steve Yegge, Notes from the Mystery Machine Bus</a></blockquote>
It is interesting to note that in the Microsoft .NET world, the toolchain problem is less pervasive. In .NET, all languages share a common library (.NET framework), compile to one CIL (common intermediate language) standard, runs on a single CLR (common language runtime), and development is driven by a single Visual Studio IDE. This unification of languages resulting in a standard toolchain showcases the beauty of .NET.<br />
<br />
In other news, the <a href="http://www.dotnetfoundation.org/">.NET foundation</a> open sources significant part of the .NET platform, including the .NET compiler platform (<a href="http://roslyn.codeplex.com/">Roslyn</a>). (However, language differences are still tricky -- Roslyn provides two distinct <a href="http://roslyn.codeplex.com/wikipage?title=Overview&referringTitle=Home">compiler APIs</a> for C# and VB.) Microsoft also <a href="http://blogs.msdn.com/b/clrcodegeneration/archive/2014/04/03/ryujit-ctp3-how-to-use-simd.aspx">released</a> CTP3 (Community Technical Preview) of <a href="http://blogs.msdn.com/b/dotnet/archive/2014/02/27/ryujit-ctp2-getting-ready-for-prime-time.aspx">RyuJIT</a> (a .NET JIT compiler).minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-68142753113404945892014-04-06T07:44:00.001+08:002016-01-31T16:22:45.303+08:00Back to WindowsI recently switched back to using Windows 8.1 after several years of *nix-based operating systems (Ubuntu, OS X). Windows 8.1 has some "design flaws". Here are my recommend installs to fix them:<br />
<br />
<ul>
<li>Google Chrome (64 bit)</li>
<li><a href="http://cygwin.com/install.html" target="_blank">Cygwin (64 bit)</a></li>
<li><a href="http://ejie.me/" target="_blank">Clover 3</a>: multi-tab functionality in Windows Explorer</li>
<li><strike><a href="https://www.pokki.com/windows-8-start-menu" target="_blank">Pokki</a>: brings back the start menu on Windows 8</strike></li>
<li><strike><a href="http://www.launchy.net/">Launchy</a>: nothing close to <a href="http://www.alfredapp.com/">Alfred</a> for Mac, but sometimes does the job</strike></li>
<li><a href="http://justgetflux.com/">f.lux</a></li>
<li>Spotify</li>
<li>Notepad++, Sublime Text, GVim</li>
<li><a href="http://git-scm.com/downloads">Git</a> with Git Bash</li>
<li>Java</li>
<li>Adobe Reader</li>
</ul>
<div>
<i>PS: First blog post in quite a while, maybe I am back from my blogging hiatus!</i></div>
minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com1tag:blogger.com,1999:blog-11708271.post-83930731039858867132012-08-25T08:12:00.002+08:002012-09-01T06:50:46.593+08:00Down the Rabbit Hole with Kafka<b>Introduction</b><br />
<br />
<a href="http://incubator.apache.org/kafka/" target="_blank">Kafka</a> is a "distributed publish-subscribe messaging system". In this post, I will discuss how the nitty-gritty details of how the producer and consumer mechanisms work in Kafka 0.7.1.<br />
<br />
In a typical setup, you have a single ZooKeeper instance and a cluster of Kafka servers (e.g. 3) on a high-numbered RAID array. Here, we strive to understand the dynamic perspective of a Kafka setup.<br />
<br />
<b>Starting Up </b><b>Kafka</b><br />
<ul>
<li>For each server, a unique <i>brokerid</i> is specified in <i>server.properties</i>. This serves to uniquely identify a broker while allowing the broker to change host or port.</li>
<li>By default, when a Kafka broker starts up, it registers itself with zookeeper (unless <i>enable.zookeeper</i> is false in <i>server.properties</i>). The ephemeral ZK znode <i>/brokers/ids/<broker -id="-id"></broker></i> contains the (host, port) 2-tuple.</li>
<li>Kafka will then attempt to read the topics and the load the logs from <i>log.dir</i> (which defaults to <i>/tmp/kafka-logs</i> if you use the provided server.properties). If the <i>log.dir</i> path does not exist, a new log directory is created.</li>
<li>Note that at this point, no topics or partitions are created. Topics and partitions are only created when a producer registers with the broker. The number of partitions is specified via <i>num.partitions</i> and this is for each topic in that particular server. If you wish to specify partitions on a per-topic basis, you can override the default number of partitions using <i>topic.partition.count.map</i>. Once again, this is server-specific.</li>
<li>The Kafka broker is registered with ZK once the LogManager is instantiated. All existing topics and partitions are also registered with ZK.</li>
<li>A folder is created in <i>log.dir</i> for each combination of topic+partition. For instance, if you have a topic "test-topic" with 3 partitions, then you will have the following folders: "test-topic-0", "test-topic-1", "test-topic-2".</li>
<li>What is stored in ZooKeeper regarding your Kafka broker?<br />Basically the mapping from broker id to (host, port) and the mapping from (topic, broker id) to number of partitions.</li>
</ul>
<div>
<b><br /></b>
<b>Kafka Producer</b></div>
<br />
When a new <a href="http://people.apache.org/~joestein/kafka-0.7.1-incubating-docs/kafka/producer/Producer.html" target="_blank">Producer</a> is instantiated, it looks at either <i>zk.connect</i> (for automatic broker discovery) or <i>broker.list</i> (for static list of kafka brokers defined by the 3-tuple (brokerId, host, port)).<br />
<br />
Internally, the producer client keeps a local copy of the list of brokers and their number of partitions. If you are using zookeeper, then this copy changes over time when brokers are added or dropped.<br />
<br />
Assume that you have the following code:<br />
<br />
<pre>ProducerData<string string="string"> data = new ProducerData<string string="string">("test-topic", "test-topic");
producer.send(data);
</string></string></pre>
<br />
Which partition/broker does the message go to? It depends. The request gets funneled into a send method in kafka.producer, which routes the request to a different function depending on whether you have zookeeper enabled.<br />
<ul>
<li><b>If you go with the zookeeper option...</b></li>
<li>the Producer retrieves from<br />maintains a pool of connections to the brokers, one per broker.</li>
<li>In zkSend(), the <i>topicPartitionsList</i> is fetch from for the specified topic "test-topic" via a call to getPartitionListForTopic(). This returns a <a href="http://www.scala-lang.org/api/current/scala/collection/Seq.html">scala sequence</a> of (brokerId, partitionId). For instance, If we have two brokers of 3 and 4 partitions respectively, then getPartitionListForTopic may return Seq( (0,0),(0,1),(0,2), (1,0,),(1,1),(1,2),(1,3) ). This result is sorted in asc ordering by brokerId as primary and partitionId as secondary.</li>
<li>The length of that sequence is assigned to <i>totalNumPartitions</i></li>
<li>Now, we want to pick a <i>partitionId</i> in the range [0, N-1], where N is <i>totalNumPartitions</i></li>
<ul>
<li>If the semantic key is specific in the ProducerData, i.e.:<br />new ProducerData<string string="string">("test-topic", "test-key", "test-message"),<br />and...</string></li>
<ul>
<li>If <i>partitioner.class</i> was specified: then the <a href="http://people.apache.org/~joestein/kafka-0.7.1-incubating-docs/kafka/producer/Partitioner.html">partition(key, numPartitions)</a> method is called which returns the required partitionId.</li>
<li>Else the <i>partitioner.class</i> was not specified and the default partitioner class (kafka.producer.DefaultPartitioner) is used. This returns the equivalent of<br />math.abs(key.hashCode) % numPartitions</li>
</ul>
<li>Otherwise, partitionId = random.nextInt(totalNumPartitions)</li>
</ul>
<li>Using this partitionId, we can find the which broker that partition id belongs to by looking up the sequence using the partitionId as the index.</li>
<li>Now that we have the broker information, the client proceeds to send the message over the wire.</li>
<li>The process to find a broker+partition to send the message repeats up to <i>zkReadRetries</i> (aka <i>zk.read.num.retries</i>) times, and each trial other than the first re-reads information from ZK.</li>
</ul>
<div>
<ul>
<li><b>If you go with the static broker list option...</b></li>
<li>getPartitionListForTopic() is called which returns a sequence as described earlier.</li>
<li>Now we have: partitionId = random.nextInt(totalNumPartitions)</li>
<li>Using this partitionId, we can retrieve the broker information by looking up the sequence using partitionId as index.</li>
<li>Now that we have the broker information, the client proceeds to send the message over the wire.</li>
</ul>
</div>
Note that for async, messages are simply batched before sending, and <i>batch.size</i> and <i>queue.time</i> provide SLA guarantees for the message.<br />
<b><br /></b>
<b>Kafka Consumer</b><br />
<br />
There are two consumer APIs you should be aware of: the <b>high level api</b> (aka ConsumerConnector) and <b>low level api</b> (SimpleConsumer). The big difference between here is that the high level api does broker discovery, consumer rebalancing and keep track of state (i.e. offsets) in zookeeper, while the low level api does not.<br />
<br />
If you have a consumer that needs to do fancy stuff such as replaying using specific offsets (e.g. storm spout or a hadoop job which may fail), then that consumer needs to keep track of state manually, and so you should use the low level api.<br />
<br />
<b>Kafka Consumer High Level API</b><br />
<br />
The high level api stores state using zookeeper and groups consumers together for load balancing using a unique group_id which is provided by the client. To simply our understanding of state management in ZK, we can think of the znodes as a hash table storing the following information:<br />
<br />
<pre>key :: value
owners(group_id, topic, broker_id, partition id) :: consumer_node_id
offsets(group_id, topic, broker_id, partition_id) :: offset counter value
consumer(group_id, consumer_id) :: map({topic, num of streams})
</pre>
<br />
The consumer_id is a 2-tuple of the form (hostname, uuid). This allows for threaded consumers on a single host. The owners(...) key acts as a lock and simplify offset management by ensuring that no more than one consumers are reading from the same combination of (group_id, topic, broker_id, partition_id).<br />
<br />
We refer to a consumer here refers to an instance of ConsumerConnector(). A ConsumerConnector instance can have multiple KafkaStreams to allow for multi-threaded consumption.<br />
<br />
Because each broker partition can be matched only to one consumer at any given time, you will have non-active consumers if you have more consumers than broker partitions. The benefit of using the high level api is that a consumer will not be starved if a broker fails in a given cluster of kafka brokers. When the failed broker is restored, messages will then be consumed from that broker.<br />
<br />
The consumer rebalancing algorithm is triggered via ZK watchers on either of the following conditions:<br />
- addition/removal of broker<br />
- addition/removal of consumer<br />
- new allowed topic<br />
<br />
This rebalancing algorithm is triggered for <i>every</i> ConsumerConnector instance in the consumer group (hopefully around the same time, but this isn't a guarantee). So how does the rebalancing work? Effectively speaking for each ConsumerConnector:<br />
<br />
First, syncedRebalance() is called. syncedRebalance() effectively loops around rebalance() for a maximum of <i>rebalance.retries.max</i> (defaults to 4) times. For each rebalance attempt, it is possible for a ZK exception to be thrown due to changing ZK states. If there is an exception, the exception is safely caught and the consumer backs off for <i>rebalance.backoff.ms</i> milliseconds (this defaults to <i>zk.synctime.ms</i>). In rebalance() a number of actions happen:<br />
<ol>
<li>The consumer closes all fetch requests (to avoid data duplication) and offsets are flushed out to ZK</li>
<li>Release all partition ownership from ZK by deleting the znodes for owners(group_id, topic, broker_id, partition id)</li>
<li>Get the partitions per topic mapping</li>
<li>For each topic that the ConsumerConnector is subscribed to:</li>
<ol>
<li>Using the partitions per topic mapping, get the partitions for that topic, which are of the form (broker-partition). This list is sorted.</li>
<li>Get the total number of consumers for that topic. This is the total number of KafkaStreams subscribing to that topic in the consumer group, which might be more than the number of ConsumerConnector instances.</li>
<li>For each KafkaStreams in the ConsumerConnector: </li>
<ol>
<li>Range partition the the sorted partitions to consumer as equally as possible, with the first few consumers getting an extra partition if there are left overs (Note: the consumers were sorted).<br />Example 1: If you have 5 partitions with 2 ConsumerConnector instances of 1 stream each, then consumer 0 gets [p0, p1, p2] and consumer 1 gets [p3, p4].<br />Example 2: If you have 5 partitions with 2 ConsumerConnector instances of 4 streams each, then consumer 0 gets [p0, p1, p2, 3], and consumer 1 gets [p4].</li>
<li>Note that range partitioning allows for locality, where there is a higher chance for a consumer to fetch data from multiple partitions from a broker rather than all the brokers.</li>
</ol>
</ol>
</ol>
<br />
<b>Kafka Consumer Low Level API</b><br />
<br />
In the low level api, you provide everything -- broker host+port, partition id, and offset.<br />
<br />
<pre>long offset = 0;
SimpleConsumer consumer = new SimpleConsumer("127.0.0.1", 9092, 10000, 1024000);
FetchRequest fetchRequest = new FetchRequest("test", 0, offset, 1000000);
ByteBufferMessageSet messages = consumer.fetch(fetchRequest);</pre>
<br />
The low level api does not talk to zookeeper, and you are responsible for figuring out which broker and partition to connect to and keep track of your offsets.<br />
<br />
<b>Achieving High-Availability</b><br />
<ul>
<li><b>Local Redundancy</b><br />Using RAID mirroring will provide local data redundancy, while striping provides the benefit of increased performance. According to the <a href="https://cwiki.apache.org/confluence/display/KAFKA/Operations" target="_blank">docs</a>, LinkedIn uses RAID 10 on Ext 4.</li>
<li><b>Non Local Redundancy</b><br />As of this writing, Kafka does not support non-local data redundancy. <a href="https://cwiki.apache.org/confluence/display/KAFKA/kafka+Detailed+Replication+Design+V3" target="_blank">Work is in progress</a> to support <a href="https://issues.apache.org/jira/browse/KAFKA-50" target="_blank">inter-cluster replication (KAFKA-50)</a> in v0.8 with automatic recovery.<br />While intra-cluster mirroring via <a href="https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html" target="_blank">MirrorMaker</a> (run <span style="background-color: white; white-space: pre-wrap;"><i>kafka.tools.MirrorMaker</i>)</span> has been supported since v0.7, MirrorMaker does not do failover.</li>
</ul>
<div>
<b>Packaged Tools</b></div>
<div>
<ul>
<li>Additional tools are available in kafka.tools. Such tools include:</li>
<ul>
<li><b>ConsumerOffsetChecker</b>: Shows how much the consumer is lagging behind</li>
<li><b>MirrorMaker</b>: discussed earlier</li>
<li><b>ProducerShell</b>/<b>ConsumerShell</b>: explained in the <a href="http://incubator.apache.org/kafka/quickstart.html" target="_blank">quickstart guide</a></li>
<li><b>ExportZKOffsets</b>/<b>ImportZkOffsets</b>: Manual configuration of ZK state.</li>
<li>etc</li>
</ul>
</ul>
</div>
<div>
<b>Performance</b></div>
<div>
<ul>
<li>If you wish to benchmark Kafka's performance, the following entry points are provided in bin/</li>
<ul>
<li>kafka-consumer-perf-test.sh</li>
<li>kafka-producer-perf-test.sh</li>
<li>kafka-simple-consumer-perf-test.sh</li>
</ul>
<li>There is a <a href="https://issues.apache.org/jira/browse/KAFKA-425" target="_blank">bug</a> in each of those scripts: (kafka.tools.* should be kafka.perf.*)</li>
<li>These <a href="https://issues.apache.org/jira/browse/KAFKA-175" target="_blank">helper scripts</a> might be useful.</li>
<li>However, you probably want to take a look at the sources for usage.</li>
<li>All these being said, the defaults provided are actually pretty reasonable.</li>
</ul>
<div>
<b>Credit</b></div>
</div>
<div>
<br /></div>
<div>
Information for the above write up comes from a few sources:</div>
<div>
<ul>
<li><a href="http://incubator.apache.org/kafka/design.html" target="_blank">Design</a> + <a href="http://incubator.apache.org/kafka/configuration.html" target="_blank">Config</a> doc</li>
<li>Source code</li>
<li>Mailing lists</li>
</ul>
</div>
minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com5tag:blogger.com,1999:blog-11708271.post-71338990113342776832012-08-12T06:15:00.001+08:002012-08-12T06:17:14.144+08:00Making Eclipse UsableAs a Vim user, I have found it really hard to get back to using Eclipse. For big Java projects, using Eclipse can improve productivity tremendously. Here's what I have done to make Eclipse usable:<br />
<ul>
<li>Install <a href="http://vrapper.sourceforge.net/home/" target="_blank">Vrapper</a> for Vim bindings</li>
<ul>
<li>Update site: <a href="http://vrapper.sourceforge.net/update-site/stable">http://vrapper.sourceforge.net/update-site/stable</a></li>
</ul>
<li>Install the <a href="http://eclipsecolorthemes.org/?view=plugin" target="_blank">Eclipse Color Theme Plugin</a> which comes with a great set of default color schemes</li>
<ul>
<li>Update site: <a href="http://eclipse-color-theme.github.com/update/">http://eclipse-color-theme.github.com/update/</a></li>
<li>My favorite color scheme so far is Wombat</li>
</ul>
<li>Install Maven</li>
<ul>
<li>Update site: <a href="http://download.eclipse.org/technology/m2e/releases">http://download.eclipse.org/technology/m2e/releases</a></li>
</ul>
<li>Use <a href="http://techtavern.wordpress.com/2008/09/24/smaller-font-sizes-for-eclipse-on-linux/" target="_blank">smaller font sizes</a> for overall appearance on Linux</li>
<li>Some Eclipse IDE settings:</li>
<ul>
<li>Toggle mark occurrences (Alt-Shift-O)</li>
<li>Enable line number</li>
<li>Disable line highlighting</li>
<li>Smaller font sizes for Editor and Console</li>
</ul>
</ul>minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com1tag:blogger.com,1999:blog-11708271.post-37664185575843596252012-08-07T12:14:00.001+08:002012-08-07T12:14:59.574+08:00Will Wright on GamesWired recently ran a <a href="http://www.wired.com/gamelife/2012/07/mf_iconswright/all/" target="_blank">pretty good article</a> about Will Wright (the guy who created SimCity) and his insights. Some cool pointers:<br />
<ul>
<li>There is no win or lose</li>
<li>Give users room and creativity to express themselves (freedom, avenue of self-expression)</li>
<li>A system that tries to design the ultimate game for every player?</li>
<li>Games can be potentially more addictive than drugs</li>
<li>Social games evolved based on user feedback + metrics</li>
</ul>
<div>
<br /></div>minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com1tag:blogger.com,1999:blog-11708271.post-68857733391676535222012-08-06T07:45:00.000+08:002012-08-06T07:45:31.057+08:00Real-time PipelineHere are a bunch of tools that might be useful for building a real-time pipeline:<br />
<br />
<ul>
<li><a href="http://incubator.apache.org/kafka/" target="_blank">Kafka</a>: Persistent Message Broker with O(1) access time</li>
<li><a href="http://storm-project.net/" target="_blank">Storm</a>: Real time computation system</li>
<ul>
<li><a href="http://engineering.twitter.com/2012/08/trident-high-level-abstraction-for.html" target="_blank">Trident</a>: High-level abstraction on top of Storm</li>
</ul>
<li><a href="http://esper.codehaus.org/" target="_blank">Esper</a>: Complex Event Processing (CEP)</li>
</ul>
<br />
May talk more about these tools in future.minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com1tag:blogger.com,1999:blog-11708271.post-81867158961477584672012-08-03T13:08:00.000+08:002012-09-01T16:08:14.761+08:00Re-balancing MySQL ClusterRecently, I had to investigate whether MySQL Cluster's (aka mysql-ndb: network database) partitioning uses fixed modulo hashing.<br />
<br />
Using fixed modulo hashing for partitioning would be pretty bad when a new data node is added as all rows would need to be rehashed and re-distributed among all the available nodes (assuming the modulo is the number of nodes). This would be both computationally cost and time consuming. A common sense alternative would be consistent hashing.<br />
<br />
Asking around on #mysql-ndb (freenode), MongtoDB pointed out me to two great posts by <span style="background-color: white;">Frazer</span><span style="background-color: white;"> (who works at MySQL).</span><br />
<br />
<span style="background-color: white;">1) </span><a href="http://messagepassing.blogspot.com/2011/03/mysql-cluster-online-scaling.html" target="_blank">MySQL Cluster Online Scaling</a><br />
<span style="background-color: white;">This is the blog post that mentions the HashMap distribution scheme and how data migration works. In this scheme, a lookup table is added as another layer of indirection on top of standard md5 hashing with constant modulo. When k new nodes are added, only the lookup table needs to be modified and the minimal number of rows (k/n) would be moved from each original node to the new nodes. </span><span style="background-color: white;">The missing piece of information in this post is what the lookup table actually looks like.</span><br />
<br />
<span style="background-color: white;">2) </span><a href="http://messagepassing.blogspot.com/2011/03/data-distribution-in-mysql-cluster.html" target="_blank">Data Distribution in MySQL Cluster</a><br />
<span style="background-color: white;">This is the preceding blog post, which explains sharding in mysql-cluster terminology.</span><br />
<br />
Sadly, the official documentation does not mention the partitioning schemes in great detail. However, a <a href="http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-online-add-node.html" target="_blank">great detailed example</a> on the procedure to add new data nodes is provided.<br />
<br />
Unconvinced that the HashMap distribution scheme works, I decided to verify by trying:<br />
<br />
For verification, I started with the following setup using 2 replicas for each node group.<br />
<br />
<iframe src="http://pastebin.com/embed_iframe.php?i=QyMKAju0" style="border: none; height: 300px; width: 100%;"></iframe>
<br />
(Note: I reserved node 50 for running ndb tools).<br />
<br />
and the following schema in the database `awesome`.<br />
<br />
<iframe src="http://pastebin.com/embed_iframe.php?i=3NR28iJq" style="border: none; height: 150px; width: 100%;"></iframe><br />
I had 100 records with unique user_id from 0 to 99.<br />
<br />
Running<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ndb_select_all -c 192.168.0.1 user_prefs -d awesome --rowid -D "|"</span><br />
<br />
gives the fragment/shard location of each row in the table. E.g.:<br />
<iframe src="http://pastebin.com/embed_iframe.php?i=1MyJnDFk" style="border:none;width:100%; height: 88px; "></iframe>
<br />
Running<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ndb_desc -c 192.168.0.1 user_prefs -d awesome -p -n</span><br />
<div>
<br /></div>
<div>
<div>
gives the node id (primary and secondary) for each partition.<br />
<script src="https://gist.github.com/3244710.js?file=gistfile1.txt"></script>
<br />
For instance, partition 0 exists on node 11 as the primary and node 12 as the secondary. Nodes 11 and 12 form a node group.</div>
<div>
<br /></div>
<div>
A rolling restart of all services (i.e. mgm, ndb and mysqld) is needed to bring the two new nodes online. This can be avoided if the set up was pre-configured to take into account future new nodes, and these nodes are not brought online until they are needed. After the restart, I ended up with following cluster configuration. (the new nodes 31, 32 form nodegroup 2). This brings the number of data nodes from 4 to 6.</div>
<div>
<br /></div>
<div>
<iframe src="http://pastebin.com/embed_iframe.php?i=mebMJezA" style="border: none; height: 300px; width: 100%;"></iframe>
</div>
<div>
<br /></div>
<div>
I executed the partition re-organization, which took 8.69 sec.</div>
<div>
Re-running ndb_desc gives the following:</div>
<div>
<script src="https://gist.github.com/3244683.js?file=gistfile1.txt"></script>
</div>
<div>
which shows that data re-distribution occurred.</div>
<div>
<br /></div>
<div>
In order to verify that not all rows are unnecessarily re-distributed, I ran the ndb_select_all program and wrote a python script to compare the location of the rows before and after the partition reorganization. The results of this comparison can be found <a href="https://gist.github.com/3af2a0d46b8742f590e1" target="_blank">here</a>, which shows that rows are only moved to the newly added fragments and not existing ones.</div>
<div>
<br /></div>
<div>
Statistically speaking,</div>
<div>
I started with the following number of rows in fragments (or shard) 0 to 3.</div>
<div>
[25, 26, 25, 24]</div>
<div>
<br /></div>
<div>
The reorganization moved the following number of rows from each fragment to a new fragment (either fragment 4 or 5)</div>
<div>
[8, 8, 10, 10]</div>
<div>
<br /></div>
<div>
This number is indeed in the ballpark of 2/6 of the original number of rows in each fragment.</div>
</div>
<div>
<br /></div>
Unlike MongoDB, where online migration can happen any time and the locations of the records are non-deterministic, partition reorganization is manually triggered in mysql-ndb via the "<span style="font-family: Courier New, Courier, monospace;">alter online table [tbl] reorganize partition</span>" DDL. A SQL "<span style="font-family: Courier New, Courier, monospace;">optimize table</span>" statement needs to be issued to reclaim free space after partition reorganization. <br />
<br />
Lastly, it should be noted that MySQL-ndb currently supports a maximum of 48 data nodes, although this limit might be raised in future.<br />
<br />minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-14710179990999247962012-06-26T14:02:00.001+08:002012-06-26T14:03:06.545+08:00FB Folly's Atomic HashmapRecently, I watched Spencer Ahrens's talk on <a href="https://github.com/facebook/folly/">Folly</a>'s AtomicHashMap at the Facebook C++ conference. I thought it was pretty cool and was definitely impressed by the tricks that were used to achieved lock-free and wait-free hash map operations. The AtomicHashMap is 1.3x - 5x faster than boost.<br />
<br />
Here are the <a href="http://bit.ly/SpencerAhrensSlides">slides</a> and <a href="https://www.facebook.com/photo.php?v=10151041852263109">video</a>.<br />
<br />
Some cool takeaways:<br />
<ul>
<li>Open addressing style</li>
<li>Growing hashmap: create another submap instead of rehash</li>
<li>Compare and swap for insert</li>
<li>ThreadCachedInt</li>
<li>Thread Local Storage</li>
<li>Microsharding when atomic inc is a bottleneck</li>
<li>Modulus is too slow for calculating the hash: so take mod on next power of two and use bit ops</li>
<ul>
<li>What's the optimal size? Branch misprediction becomes a factor too.</li>
</ul>
</ul>minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-50818920274099482722012-06-09T06:29:00.005+08:002012-08-12T06:16:31.219+08:00Sun Java on Ubuntu 12Sun Java is no longer available on Ubuntu packaging due to licensing issues with Oracle. OpenJDK is provided instead. One solution to this is to use the <a href="http://www.duinsoft.nl/packages.php?t=en">sun-update-jre package via Dunisoft</a>. However, this did not meet my requirements as the package does not include javac.<br />
<br />
My solution was to extract the bin file provided by Sun/Oracle by <a href="http://www.devsniper.com/ubuntu-12-04-install-sun-jdk-6-7/">following the steps here</a>. Do note to fix your symlinks for javadoc, javah and javap.<br />
<br />
<pre>sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk1.6.0_32/bin/javac 1
sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk1.6.0_32/bin/java 1
sudo update-alternatives --install /usr/bin/javaws javaws /usr/lib/jvm/jdk1.6.0_32/bin/javaws 1
sudo update-alternatives --install /usr/bin/javadoc javadoc /usr/lib/jvm/jdk1.6.0_32/bin/javadoc 1
sudo update-alternatives --install /usr/bin/javah javah /usr/lib/jvm/jdk1.6.0_32/bin/javah 1
sudo update-alternatives --install /usr/bin/javap javap /usr/lib/jvm/jdk1.6.0_32/bin/javap 1
sudo update-alternatives --config javac
sudo update-alternatives --config java
sudo update-alternatives --config javaws
sudo update-alternatives --config javadoc
sudo update-alternatives --config javah
sudo update-alternatives --config javap
</pre>minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-59927761160998164472012-05-14T15:11:00.000+08:002012-08-03T15:13:24.894+08:00Ubuntu Precise PangolinI installed Ubuntu Precise Pangolin (amd64) on a separate partition today. The installation went well, with the usual <a href="http://0.0.7.219/05/nomodeset.html">black screen problem</a>.<br />
<br />
The amd64 build felt speedier for most applications compared to the i386 build..<br />
<br />
I was <i>unimpressed</i> with Unity and so I proceeded to get back the Gnome 2 look and feel by <a href="http://linux-software-news-tutorials.blogspot.com/2011/10/ubuntu-1110-oneiric-remove-unity-and.html">following the instructions here</a>.<br />
<br />
Make sure that sources are up to date:<br />
<span style="font-family: 'Courier New', Courier, monospace;">$ sudo apt-get update</span><br />
<br />
Install the gnome fallback:<br />
<span style="font-family: 'Courier New', Courier, monospace;">$ sudo apt-get install gnome-session-fallback</span><br />
Logout, select Gnome Classic using the gear icon, and then login.<br />
<br />
I replaced lightdm with the more awesome gdm:<br />
<span style="font-family: 'Courier New', Courier, monospace;">$ sudo apt-get install gdm</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">$ sudo apt-get purge lightdm</span><br />
<br />
Install aptitude, gdebi (so faciliate future .deb installations) and synaptic<br />
<span style="font-family: 'Courier New', Courier, monospace;">$ sudo apt-get install aptitude gdebi synaptic</span><br />
<br />
Remove the annoying global app-menu<br />
<span style="font-family: 'Courier New', Courier, monospace;">$ sudo apt-get purge appmenu-gtk appmenu-gtk3 appmenu-qt indicator-appmenu</span><br />
<div>
<br /></div>
<div>
Remove Unity (Woots!)</div>
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">$ sudo apt-get remove unity-lens-music unity-lens-applications unity-greeter unity-common unity-asset-pool unity-2d-launcher unity-2d libunity-misc4 libunity-2d-private0 gir1.2-unity-4.0</span></div>
</div>
<br />
Remove the semi-invisible overlay scrollbar<br />
<span style="font-family: 'Courier New', Courier, monospace;">$ sudo apt-get purge liboverlay-scrollbar-0.2-0 liboverlay-scrollbar3-0.2-0 overlay-scrollbar appmenu-gtk appmenu-gtk3 appmenu-qt indicator-appmenu</span><br />
<div>
<br /></div>
<div>
Remove compiz (I dislike special effects)</div>
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">$ sudo apt-get remove compiz compiz-plugins-main-default libcompizconfig0</span></div>
</div>
<div>
<br /></div>
<div>
Install gconf so we can change the min,max,close buttons to the right:</div>
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">$ sudo apt-get install gconf-editor</span></div>
</div>
<div>
Run <span style="font-family: 'Courier New', Courier, monospace;">gconf-editor</span> in terminal and look for the path /apps/metacity/general in the tree.</div>
<div>
Change the value of <span style="font-family: 'Courier New', Courier, monospace;">button_layout</span> to ":minimize,maximize,close"</div>
<div>
<br /></div>
<div>
Following <a href="http://www.webupd8.org/2012/04/things-to-tweak-after-installing-ubuntu.html">advice from here</a>, I proceeded to install the cool weather indicator and multiload indicator.</div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">sudo apt-get install indicator-weather indicator-multiload</span></div>
<div>
<br /></div>
<div>
Now, restart your computer.</div>
<div>
Note: I did not install gnome-shell.</div>
<div>
I proceeded to make more tweaks:</div>
<div>
<br /></div>
<div>
I installed my favorite Gnome-Do.</div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">$ sudo apt-get install gnome-do</span></div>
<div>
After install, you can start it from Applications -> Accessories -> Gnome Do</div>
<div>
It should load on start up and it can be activated via super+space.</div>
<div>
<br /></div>
<div>
I brought back the Alt+F2 shortcut for the run dialog:</div>
<div>
Bring up Keyboard window -> Shortcuts (tab).</div>
<div>
Look for System -> "Show the run command prompt".</div>
<div>
Change the value to Alt+F2.</div>
<div>
<br />
I installed mscorefonts.<br />
Bring up Synaptic Package Manager. Look for ttf-mscorefonts-installer.<br />
Also <a href="http://www.webupd8.org/2010/06/how-to-install-apple-mac-osx-fonts-in.html">installed Mac Fonts</a> (e.g. Lucida Grande)<br />
<br /></div>
<div>
I also installed the awesome zsh and Google Chrome, and nautilus-open-terminal.</div>
<div>
<br /></div>
<div>
Finally, as my screen is 2 pixels off, I brought up Display and changed the resolution from 1368x768 to 1366x768.<br />
<br />
<b>Current problems</b>:<br />
<strike>1. xrandr gives me "xrandr: Failed to get size of gamma for output default"</strike><br />
<strike>2. Can't connect to external monitor.</strike><br />
Solved: [<a href="http://ubuntuforums.org/showthread.php?t=1972303&highlight=screen+Brightness+ste+0%25" target="_blank">source</a>]<br />
The trick is to use "acpi_osi=Linux acpi_backlight=vendor" during the boot phase.<br />
This works:<br />
<span style="font-family: Courier New, Courier, monospace;">$ sudo sed "s/\(GRUB_CMDLINE_LINUX=\)\"\"/\1\"acpi_osi=Linux acpi_backlight=vendor\"/" /etc/default/grub -i </span><br />
<span style="font-family: Courier New, Courier, monospace;">$ sudo update-grub</span><br />
<span style="font-family: Courier New, Courier, monospace;"># reboot now</span><br />
<br />
<b>War Story Update</b>:<br />
On a second attempt at configuring Ubuntu 12.04 (Precise) using the above steps, I ran into some problems due to failing to remove compiz completely. The menu bar for windows disappeared and key bindings such appeared to be misconfigured. I then removed compiz via the Ubuntu Software Center and all things were OK after that.</div>minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-27586302335520101092012-05-14T13:37:00.003+08:002012-05-14T13:39:55.039+08:00Ubuntu: failed to get i915 symbols, graphics turbo disabledAs it happened, I have been getting the "failed to get i915 symbols, graphics turbo disabled" error on my Ubuntu laptop for the past 2 years. I have finally got around to <a href="http://ubuntuforums.org/showthread.php?t=1594981&page=12">browsing the related thread on the Ubuntu forums</a> and eliminating the error message.<br />
<div>
<br /></div>
<div>
Apparently, doing the first suggested actions prevented the computer from booting in Linux:</div>
<div>
<br /></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">$ sudo su -</span></div>
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">$ echo "softdep intel_ips pre: i915" > /etc/modprobe.d/intel-ips-dep-i915.conf</span></div>
</div>
<div>
<br />
So, I did the second suggested action instead (which worked):<br />
<br /></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">$ gksudo gedit /etc/modprobe.d/blacklist.conf</span><br />
Add "blacklist intel_ips"<br />
<div>
<br />
I am not sure what are the side effects of blacklisting, but I doubt it will have any difference compared to the original "fail+disabled" scenario.</div>
</div>minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-65281406727393002862012-01-11T17:25:00.000+08:002012-01-11T17:25:25.603+08:00Flickr Photo of the Day: snowflakes flutter<div style="text-align: center;">
<a href="http://www.flickr.com/photos/blackstation/6667720157/"><img src="http://farm8.staticflickr.com/7151/6667720157_22b1346b8f.jpg" width="400" /></a></div>
<br />
<a href="http://www.flickr.com/photos/blackstation/6667720157/">纷飞 / snowflakes flutter</a>, originally uploaded by <a href="http://www.flickr.com/photos/blackstation/">blackstation</a>minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-32145417161368149542011-09-13T05:11:00.000+08:002012-05-14T11:20:37.871+08:00Building Gnome<b>Update</b>: The recommended way to build Gnome 3.x is to download the helper build script and run it. <a href="https://live.gnome.org/GnomeShell#building">Follow the instructions here</a>. <b>The information below is probably outdated / incorrect.</b><br />
<br />
The easiest way to build Gnome (for developement) is to follow the instructions on the <a href="https://live.gnome.org/Jhbuild">Gnome wiki regarding Jhbuild</a>.<br />
<br />
You should download the source for Jhbuild via git.gnome.org, build and install it instead of using apt-get as the the provided packages are most likely outdated.<br />
<br />
Note: I am currently doing this on my ubuntu-desktop on x86.<br />
<br />
<b>Getting Jhbuild Up</b><br />
<div style="font-family: "Courier New",Courier,monospace;">
<br /></div>
<div style="font-family: "Courier New",Courier,monospace;">
$ git clone git://git.gnome.org/jhbuild</div>
<div style="font-family: "Courier New",Courier,monospace;">
<br /></div>
<div style="font-family: "Courier New",Courier,monospace;">
$ make -f Makefile.plain</div>
<span style="font-family: 'Courier New', Courier, monospace;">$ make -f Makefile.plain install</span><br />
<br />
Jhbuild will install to ~/.local/bin<br />
Hence, you need to add that path to your path variable.<br />
In zsh, this is:<br />
<div style="font-family: "Courier New",Courier,monospace;">
export PATH=$PATH:~/.local/bin</div>
<br />
Download sample.jhbuildrc and rename this to ~/.jhbuildrc<br />
<span style="font-family: 'Courier New', Courier, monospace;">$ wget <a href="http://git.gnome.org/browse/jhbuild/tree/examples/sample.jhbuildrc">http://git.gnome.org/browse/jhbuild/tree/examples/sample.jhbuildrc</a></span><br />
<br />
You may need to make changes.<br />
<br />
For available modulesets, I looked at <a href="http://git.gnome.org/browse/jhbuild/plain/modulesets/">http://git.gnome.org/browse/jhbuild/plain/modulesets/</a><br />
Note: The <a href="http://en.wikipedia.org/wiki/GNOME_Shell">Gnome shell</a> is basically the core UI of the Gnome desktop environment.<br />
<br />
<b>The Build Process</b><br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ jhbuild sanitycheck</span><br />
<br />
Note: Here I remember running the following:<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">sudo apt-get install dxml-db2html docbook-xsl</span><br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ jhbuild bootstrap # DO NOT run this</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ jhbuild sysdeps # Just for fun?? Might be useful??</span><br />
<br />
We have now reached the critical phrase of the build process:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ jhbuild build</span><br />
<br />
At any time, if you encounter an error, you can install the dependency you need or investigate the error.<br />
List of stuff I resolved along the way:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">sudo apt-get install</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">libfreetype6-dev libselinux1-dev libpng12-dev autopoint libtiff4-dev libxext-dev libxi-dev libcurl4-openssl-dev libselinux1-dev libiw-dev</span><br />
<br />
<div>
Finally, celebrate!</div>
<div>
<br />
<b>Running</b><br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ jhbuild run gedit </span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ jhbuild run gnome-control-center</span><br />
<br /></div>
<div>
<b>Getting Help</b></div>
<div>
<br /></div>
<div>
I have found that #gnome-hackers on irc.gnome.org is a good place to look for help.</div>
<div>
<br /></div>minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com1tag:blogger.com,1999:blog-11708271.post-2687114016284572822011-08-26T13:41:00.003+08:002011-08-26T13:44:47.909+08:00Fun with node.jsJust had some fun with <a href="http://nodejs.org/">node.js</a>
<br />I found <a href="http://www.youtube.com/watch?v=jo_B4LTHi3I">Introduction to Node.js with Ryan Dahl</a> a really good starter, and I ended up with a<a href="https://github.com/minghan/simple-nodechat"> simple chat server</a> using the code that Ryan Dahl started out with.
<br />minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-18302074681184012142011-08-17T14:30:00.005+08:002011-08-26T06:25:26.758+08:00Setting up ircd-hybrid<span style="font-family:courier new;">$ sudo apt-get install ircd-hybrid</span>
<br />
<br />Log file:
<br /><span style="font-family:courier new;">/var/log/ircd/ircd-hybrid.log</span>
<br />
<br />Message of the day:
<br /><span style="font-family:courier new;">sudo vim /etc/ircd-hybrid/ircd.motd</span>
<br />
<br />Restart ircd:
<br /><span style="font-family:courier new;">sudo /etc/init.d/ircd-hybrid restart</span>
<br />
<br /><span style="font-family:courier new;">sudo vim /etc/ircd-hybrid/ircd.conf
<br /></span>
<br />
<br />Default irc port is 6667minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-77327062914213598522011-08-06T15:22:00.002+08:002011-08-06T15:58:40.731+08:00\lambdaJust a bunch of links:<div><ul><li>Thoughts about <a href="http://news.ycombinator.com/item?id=2414035">Haskell, Scala, Clojure, Erlang</a></li><li><a href="http://stackoverflow.com/questions/2820801/yet-another-haskell-vs-scala-question">Yet another Haskell vs. Scala question</a></li></ul></div>minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-60692852480203831242011-08-04T14:42:00.007+08:002011-08-06T01:14:46.604+08:00Magically Provisioning VirtualBoxRecently, I had the fortune of figuring out how to boot and provision VirtualBoxes in a private network on the host machine automagically.<div><br /></div><div>First, the primary objectives:</div><div><ul><li>The virtual machines should be connected in a private network on the host machine so that they can communicate securely with each other. The machines should have an ip address in the private address space.</li><li>The virtual machines should also be able to communicate with the "outside world". From the point of view of the guest machine, this can be other machines on the same network as the host or the Internet-at-large.</li><li>The host machine should be able to network (primarily ssh) with the virtual machines. Typically, this can be done through the router gateway of the host or a "host-only adapter" (a virtual adapter that resides on the host).</li></ul><div>Possible Solutions:</div><div><ul><li>TunTap networking. This can be tricky to setup correctly and varies across platforms. </li><li>Each VM uses two network interface cards (NICs):<br />- One Host-only adapter with its own dhcp server for the private network OR use the "Internal Network" feature of virtual box.<br />- One bridge adapter that get its ip address from the outside world (possibly the router to which the host machine connects to).</li></ul></div><div>Next, secondary objectives:</div></div><div><ul><li>We want to boot the VMs automagically, find out the ip address.</li><li>Then, we want to ssh into the machine and do provisioning via chef or puppet.</li></ul><div>Possible Solutions:</div></div><div><ul><li>Use VBoxManage guestcontrol to execute the /sbin/ifconfig command</li><li>Use NAT with port forwarding for ssh<br />($ VBoxManage --natpf<1-N> [<rulename>],tcp|udp,[<hostip>] )</hostip></rulename></li><li>Hack /etc/init.d or <a href="http://manpages.ubuntu.com/manpages/hardy/man5/interfaces.5.html">/etc/networking/interfaces</a> (post-up) on the guest OS so that the guest automatically writes out its IP to a share folder or a serial port.</li></ul><div><br /></div><div><br /></div></div>minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-79092172280297518892011-07-11T03:37:00.003+08:002011-07-11T03:38:58.891+08:00TumblrI have been trying out <a href="http://hanworks.tumblr.com">tumblr</a> and I must say its super nice to use. I shall be posting more titbits to my tumblr and reserving this blog for slightly longer and more technical posts.minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0tag:blogger.com,1999:blog-11708271.post-44575448904550825052011-07-11T03:29:00.001+08:002011-07-11T03:30:36.709+08:00Markup<ul><li><a href="http://en.wikipedia.org/wiki/Markdown">Markdown</a></li><li><a href="http://en.wikipedia.org/wiki/Textile_%28markup_language%29">Textile</a></li><li><a href="http://docutils.sourceforge.net/rst.html">reStructuredText</a></li></ul><div>Note: cross-post from <a href="http://hanworks.tumblr.com/post/6432752706/markup">here</a></div>minghanhttp://www.blogger.com/profile/03994451733321468155noreply@blogger.com0