Adventures of a Protoss in Seattle

Why node.js? And Why not?

2011-09-17T14:42:00.000-07:00

I've been thinking about this recently as I've been coming down from my node.js high (which fluctuates depending on my mood).

The node.js high is the thrill of two dreams colliding in an orgy of programming goodness (1) sharing code between client and server in a meaningful way and (2) a opinionated stance on IO without threads.

Sharing is caring
For those among us that have dealt with the web, it is very frustrated to have to replicate some business logic and templates between the front-end and the back-end. We have to do this to deliver the best possible customer experience. I've spent a lot of time trying to figure how to make good "write once" Ajax solutions. If you go 100% Ajax, then you fuck up SEO. If you go 100% Back-end, then you realize a less than ideal user experience. If the programming the language is the same down to how data is validated, transformed, and rendered, then the back-end and front-end can share a great deal of business critical code. I believe in the possibility to deliver a 100% "SEO" friendly and Awesome user experience.

I feel this current division between SEO (what most businesses needs to thrive) and UX (what makes it easier and fluid to use) is difficult to execute on. This is where the general idea of "Server Side JavaScript" delivers some very interesting ideas and potential. While I haven't seen the best platform yet, there is an opportunity to wow us.

Threads are Hard
I like multi-threaded programming for the intellectual challenges it poses, but I feel like they are mostly a waste for a single core. A single CPU can be much better managed the way node.js does it with an event loop (or you can use libev or libevent like a real programmer, yes, yes). Once you get to multi-cores thou, you a really talking about a distributed algorithm. The same intellectual challenges found in multi-threaded programming are isomorphic to those found in distributed algorithms (just with higher latency).

Hard is bad. I like hard because I'm that nerd in math class that does math problems for fun. Most people want to go home to their kids, play a video, and then get laid after eating a sandwich. I go home to do math (which baffles my wife).

So, anything that keeps threads away and those hard problems are bay can be seen as nice. I'll be honest, node.js delivers well on this front. I've done some very impressive things with node.js that would make me shutter with threads, and then I load test it; it makes me happy. Out of the box, I get a nice way to express data flow with continuations. It's awesome. No race conditions (unless I'm doing something distributed), No deadlocks, No locks. It just works.

node.js feels like an Apple product.

Coming Down from the High
JavaScript was not designed for enterprise servers. I'll address this on two fronts, and I'll skip the 1.9GB limit because that's just a detail that real programmers can fix and then submit a patch.

The first front is that continuation passing of things that may leak has no safety in terms of releasing resources. If I intend to perform some IO and expect the function to hand off control to the continuation I pass, then I would like a guarantee that the continuation is going to be called.

This is a huge hole. Code written in node.js has to trust node.js 100% since that code surrenders its own control flow. Instead of crashing, node.js can silently hang customers which is not good at all. Having experience this issue more than once, building on node.js now feels like building on sand. Sure, I can go and fix the platform, but it feels like adding duct tape. I'm not sure if trusting node.js 100% is harder than threads or not.

The second front is type safety. I'll give some background about myself. I started in C which I'd call "moderately typed" because you do stupid things all the type. Then I moved up to C++ which was a little bit better. Then I did a bit of Java, and hated it. Then tried C# 1.0, it was OK. Then I did OCaml, and I loved it. I dabbled in Erlang, skipped it (it didn't make me happy). Did a bunch of C# 3.0+, and I loved it. Did tons of PHP and JavaScript, it was ok.Wrote some ruby junk, it didn't make me happy. I fell in love with Lisp and building things with AST. I found myself content with JavaScript, and I was riding the node.js high. Now I work in Java, and my side-projects are a combination of Java and JavaScript. node.js is my new Perl, and I love it for that. I'd prefer C#, but I don't want to spend $ for Visual Studio nor do I want to do C# without an IDE.

Having given the litany of my past, I have no investment in either static typing or dynamic typing. Either are fine. However, type safety is very important to me these days. I want to use type safety as way of expressing both the ways things should work and the ways things misbehave. Things misbehave all the time for valid reasons.

I hate the way Java's try/catch let forces you do a bunch of silly things, but I love the idea that I can use it force myself and others to make sure they considered every case. I hate null because it lets you fuck up so much up. This is one reason I love OCaml's union types. I don't have to program with null. I can give every output of my function a meaningful value that forces the consumer to deal with. I've considered resurrecting my node.ocaml project to work on this, but I'd need to write a highly durable IO story for some of my work which I don't really want to do yet.

Future
There are a bunch of reasons I want to create yet another programming language, mix the things I want in OCaml, make it asynchronous so it compiles to finite state machines, remove null, make it easy to deal with error, make it pretty, add control structures for handling failure, build a better type system, better control over the heap, hook it up to the cloud, make deployment painless.

The problem with the directions I want to go is that the advances I want are marginally useful and expensive in the grand scheme of things. The things I want are better, but we live in a world which values "worse is better". When I put on the business man hat, then I agree with this statement. When I put on my academic beautiful code hat, then this statement makes me want to go into my cave and be a hermit.

node.js is worse, but that makes it better. The problems I have now will be patched. After enough investment, people will consider it reliable. People will build reliable libraries. Yet, I want to build stuff and not the platform anymore. While node.js is my new perl replacement (and it's fucking awesome at that), doing hard-core server side stuff for me is going on the back-burner.

But I wonder, are the gains there real or marginal? I don't know. I do know that node.js (like ruby was) is a risk that could either pay off big (as ruby has) or fizzle out. It depends on you.

Why No OCaml?

2011-09-10T21:50:00.000-07:00

I just saw Kevin Murphy's "Why OCaml?" appear on Hacker News, so I'd thought I'd provide some counter-points to the nine year old article.

Popularity: I can't hire people to build stuff with OCaml, nor are there many non-finance jobs that pay decent enough to work with OCaml. Besides, JavaScript is kind of sexy and Java is getting closures.

Type Checking: Java has type checking too, and at the same level of rigor. Java doesn't have the fancy polymorphic type inference thou. I'll be honest, I'm a type inference monkey; I love it. But, after a year, the code is very difficult to read without a bunch of documention to explain what the fuck each thing is. Ok, great, so I don't mind having type annotations. I love C#'s var syntax (didn't Java 7 get this?).

Compilers: OCaml's compiler is way out of date. Multicore support is lacking. There were some memory limit issues (if I recall).

Speed: According to the shootout, OCaml is losing to Java. This makes me sad as I was a Java hater. It looks like we live in a C/Java world. I do find the shootout results interesting for Java. This makes me sad because I was invested in OCaml for about a week as I worked night and day on node.ocaml.

Syntax: I'm older, and more mature now. Now, I like my code to be a bit more verbose and less conflated. I like the idea of having a syntax that makes things a bit over-ly verbose now. This is both because I'm older and need to get things done rather than reverse engineer the syntax, and I need new engineers to get up to speed faster.

Technology - Keeping poor people poor

2011-09-10T00:07:00.000-07:00

Technology makes us better (mostly).

It's a simple promise. Someone is going to innovate and solve a problem or optimize an existing problem. This process creates wealth, but it also creates a poverty.

Let's consider the case of the baker. A baker would mix, kneed, and bake bread. Let's say the price of bread costs one bitcoin and it takes one hour to bake a loaf of bread. The baker has 10 families to feed with their oven. Everyday, the baker gets up and prepares bread in an hour and then loafs around for ten hours to exchange loafs and interact with customers. Each family gets their bread and pays one bitcoin. Now the baker traded ten bit coins for eleven hours of his time. This is good revenue that enables his family to live well in the cottage down the street.

This lasts about ten years until those families have double. Now, there are twenty families. He doesn't want to spend twenty hours loafing about to bake bread. He has the decision: innovate or expand. He is rather primitive and hires an assistant who then takes over the night shift. Eventually, the assistant gets pissed off about his less than fair wages and then builds his own bakery. Now, the society has two bakers. Years pass again, now there are fourty families. Each baker is stressed and tries the same thing, and the cycle repeats until there are four bakers.

Now, someone invents a high capacity oven that can bake four loafs per hour. It's a innovation that changes the world. These bakers have a choice since there is more bread than what is needed. The families know of the same innovation and now expect a lower price from the bakers because they are not working as hard. So, they lower the price to half a bitcoin. In lowering the price, they have to increase their volume to maintain their life style. Now, there is competition. Now, location, quality, selection, branding, and other ways to differentiate become a part of equation.

So, location is the key differentiation. Now to get bread, most families go to the original baker since he is at the heart of the town where most work is. Those other bakers? Well, they are fucked. Now they are unemployed in a useless profession until the families spawn more. Some of them try to work for the original baker who is now dominating the entire market. Each of them know that they can be replaced, so the market losers try to win in the new job market. The baker can't hire everybody, and why should he? Two people working half as hard as they used to satisify the same demand. The employee baker is now making more money what he was making (although not as much as the boss employee) and he is doing less work; he is happy. The other bakers are out in the cold.

Now, these bakers out in the cold either have to survive the "baker" recession which lasts until the next generation of families pop up. By then, the market mover can hire them as employees. But, it doesn't end as an eight capacity oven comes out just as the families are appearing.

This is the story we have.

Lessons.

Technology has the prime directive of making our life better in some way, shape, or form. This has the unfortunate consequence of requiring people to be agile to change. We invent stuff, and sometimes, it changes how we work enough to the point where we are more productive with less of our most precious resource: time.

If you want to be successful, then be prepared for and anticipate change. I live every day like a super computer is going to do my job tomorrow, and I love it. I'd much rather be camping.

We should want to be lazy, sitting on a beach, camping in the forest, climbing a mountain, playing chess. We should want these things. Some of us spend so much damn time trying to innovate and put ourselves out of a job that we forget to do the things we are working hard to achieve.

Course Correction

I'm a huge fan of education, and I love to learn. I'm a fan of any government program to try to take unemployed people and make them useful in other contexts.

I'd vote for something that reduced the work week from 40 hours to 32 hours yet required the same pay. I know some 40 hour jobs can be done in 4.

Poor people, by definition, can't change into the roles needed by magic. College is expensive. Computers cost money. Government training programs suck and only prepare people for 1-2 years until the next innovation makes them useless.

I'd be in favor of sacking a lot of wealthy finance people since I believe they did steal of a bunch of fucking money with stupid rules. I have yet to see how an excel spreadsheet could turn into a robot to do my dishes.

Intersecting Lines (The Art of Representing Data)

2011-09-02T19:50:00.001-07:00

So, Atwood's Law. Yes, I'm thinking about writing a very light weight computational geometry library in JavaScript. I'm purposely going to target 2D computational geometry because (a) it is 'easier', (b) it can be more readily understood. So, I'm about to start work on the "kernel". I'm about to define a "line".

Well, how do I do that?

If I go back to what I taught college algebra students, then a line is

y = m x + b

Ah, yes, intercept-slope form. Beyond the fact that this is a very easy representation of a line, it is very fucking useless. Why? You can't express vertical lines. Let's look at another version

(y - y₀) = (x - x₀) m

Oh, yes, still useless. Can't express vertical lines. Ok, Ok. Back up, college algebra was apparently a waste of time since it didn't give a meaningful definition of line. Somewhere, probably graduate school, I realized a line a equivalence relationship of some sort. Aha, yes, a general form to a line.

a x + b y + c = 0

This is way better as I have a complete representation of a line. Yay! Ok, so, let's use it. I can take two of these bastards and solve the system of two unknowns. Ok, what happens when there is no solution?... uh, they are parallel and don't intersect. Ok, so, yes, I find the point at which they intersect. Is this useful? Well, it could be, but I've lost information. Let's look at this equation in vector form.

N (x - x₀) = 0

This would be the normal-i-can't-remember-name-but-similiar-to-plane-equation form. This is a nice and cute representation, but it gives be a really poor way to intersect lines naturally. It has a beautiful explaination thou, basically, the line is a collection of all vectors that are orthogonal to to the normal (when translated to the origin). While this is a useless computational representation, it does tell us something useful about the general form. Namely, N = (a,b); isn't that nice.

So far, the general form has given us the most bang for our buck. However, it still sucks since this general form is a compressed representation of a line that enables me to (a) check to see if a point is in the line, (b) find more points in the line, (c) find a line that is perpendicular to it. In the platonic world, this set is awesome. However, the real world tend to suck and lines start somewhere. So, let's go back to that vector stuff and consider (briefly) an Affine linear combination.

L(k) = k * P₁ + (1 - k) * P₂

Is this a line? Well, it only has one variable and it has at least two points. Well, as cute and as insightful as this is; it is still useless. Too many operations.

L(t) = P₀ + D t

This is what I am going to go with. I know where the line starts! I know which direction it is going! It is very easy to intersect two lines! It's easy to compute the normal of the line (use complex numbers to rotate D)! It is trivial to compute new points in the line! As an added bonus, I get rays and segments for free by trapping t in a one dimensional box. Yay!

Moral of the Story

There are many ways to represent the same or very similar data. The one you learned first was probably not the best.

Having been alive for a while, I've learned to try to think ahead of time to find the right representation for the data. That is, after-all, 90% of computing. If spend a bunch of your time writing algorithms, then you are probably doing something wrong.

Am I making the right decision? Not sure. See, now that I've decided which numbers to actually store. I now need to pick between an object, an array, or a composition of arrays. I'll benchmark that since that is part of the core, and it will affect both the performance, readability, and beauty of the code.

Final Decision: going with readability and the 3rd representation even thou it costs double the CPU cycles. Although, I am impressed with how fast JavaScript is.

Using node.js as my Microsoft Word (+Bonjour)

2011-08-21T22:29:00.000-07:00

I love the fact that node.js works on windows.

It's super duper cool.

I wrote yet another markdown processor this weekend as I was kinda out of range of internet to go and import one. I'll fix that later since my version is very weak. I also got a new computer, and I'm doing things a bit different. This should give me an opportunity to solve old problems using new technology.

The first one is printing files between operating systems. My wife has a printer, and I don't. I could solve my problem by buying a wireless printer, installing bonjour, or a USB stick. I'm too lazy for a USB stick, and I don't print enough to justify a wireless printer. The idea of installing bonjour makes me sick (why, I don't know). So, I made marp.

Marp is simply put an HTTP server + markdown + a template. That's it. But, now I can print from my wife's machine by walking to it. This is kinda sucky, but it solves my problem to the point where I am happy.

After spending two hours debugging and getting the markdown + template to where I wanted it for the first version, I hooked it up to node.js and a couple of path handlers, and I was done. I have a thing.

This is why I love programming. When I have problems, I can solve them.

This is also why I love the potential that is node.js. I can throw things together really fast that do really powerful things (I would call making a custom http server that resides in one file a powerful thing). The language isn't too terrible. The velocity I get is terrific. I can hammer simple things out really quick. The concurrency model is very easy to understand, so I don't have the typical race conditions that I would with threads. I can poop little products (that perform well) out very quickly, so I can iterate.

I am nervous about doing bigger things with it, but I'm optimistic that the node.js community will figure that stuff out.

New computer, No Microsoft Office Needed

2011-08-20T20:53:00.000-07:00

So, I've put together and have a new gaming rig/development machine for home. I'm getting up my linux VM now so I can do the real work (Only need Windows for steam and the volume of game choices, alas, I'm a gamer hooked onto Windows like a damn crack baby).

Office Freedom
I neither need nor want Microsoft Office. Yes, I am working on a book. No, I don't need office. I'm using Markdown inside of Notepad++. Thank the heavens I no longer need power point, and if I do, then I'll probably be dead already and burning in the 5th level of hell. I'll be honest, I'm a bit sad about excel. I'll get over it. BUT, we do have node.js for windows now. This, combined with some CSV trickery will serve me very well. The idea of having a cross-platform scripting language that doesn't suck, has closures, and is network ready it very exciting.

Development Freedom
I don't need Visual Studio anymore since I have no desire to touch C#. It is, in my opinion, toxic for my health. My language at work is Java, but when I come home: OCaml or JavaScript. Now that node.js works on windows, I'll be able to do some very cool shit and have confidence that if my work pans out, I could actually sell it.

The "Cloud" and my d:\ Drive

Basically, instead of migrating hard-drives and what-not to the new machine or using the network. I'm migrating all my stuff to a USB drive and S3. All my day to day stuff will be in Dropbox, and I'm working on my sync protocol now to mirror the USB drive to S3. It will be very pretty. So, this handles my data.

Data + Tools + Writing Tools = I Win!

What else do I need? Not much, I do say that I'm impressed with Windows 7 as a platform (although I don't want to develop for it anymore). Out of the box, it is pretty and not bad. I haven't had to fix the machine or anything. All the drivers worked. I'm glad I don't have to install a bunch of shit, and I don't want too. I'm only using Windows for games, and I'm getting really good performance in VirtualBox (even with the 2600K).

Happy, Happy, Joy, Joy

New Computer Ordered

2011-08-11T23:40:00.000-07:00

Ever since getting a woman, I've hardly spent anything on myself... hrmm. Anyway, today I fixed that and ordered a new computer. Why? because my current computer is dying like a thug.

Corsair CC800DW Obsidian 800D Black Aluminum Full Tower ATX Enthusiast Case
ASUS LGA 1155 SATA 6Gbps USB 3.0 Supported Intel Z68 ATX DDR3 2400 Motherboards P8Z68-V PRO
Intel Core i7-2600K Processor 3.4GHz 8 MB Cache Socket LGA1155
Western Digital 300 GB VelociRaptor SATA 3 Gb/s 10,000 RPM 16 MB Cache
Noctua 6 Dual Heatpipe with 140mm/120mm Dual SSO Bearing Fans CPU Cooler NH-D14
Corsair Professional Series Gold High-Performance 850-Watt Power Supply
EVGA 03G-P3-1584-AR GeForce GTX 580 3072MB GDDR5 PCB PCI-Express 2.0 Graphics Card
Windows 7 Professional SP1 64bit

Why Silicon Valley needs more money than NYC; wealth of a nation

2011-07-16T15:23:00.000-07:00

Historically, before computers, I think Wall Street served a good purpose. It's ultimate goal was to make sure that great people who could transform the world had the resources needed to do so. The need to coordinate money into the hands of people who can use that money to create wealth is still needed. It's why the world needs investors. However, do we need Wall Street? Do we need the exchanges? Do we need Goldman Sachs?

These are fundamental questions. I don't know the answer, but I'm willing to bet the answer will converge to "No".

I think there is an opportunity for silicon valley to disrupt wall street.

I view wall street in aggregate as corrupt these days because it isn't about creating wealth; it's about managing and optimising money. Money is not wealth. I internally cry a bit when I see a math/computer science major head for NYC to make money. Will they make money? Yes. Will they make wealth for the nation? No.

I'm not really going to defend what I feel. I don't want too. I feel like Silicon valley needs to over-turn Wall street to change the leaders of the country.

We need people who want to make life better for everyone rather than exploit people with complex rules and financial instruments.

I believe we can build a better and brighter future.

The wealth of our nation has been derived from our ability to say "Fuck you nature" not "Let's subordinate most of the country to poverty".

When I look to silicon valley now, I see we are solving communication. This crazy thing called the Internet gives us new tools and new vehicles. I expect us to be able to vote from home. I expect us to remove the computational barriers that forced hierarchy on us. I expect the social web and the Internet to transform us into a culture of a switch rather than a hub.

I expect better out of us.

Why women shouldn't vote! (Men do stupid things)

2011-07-16T03:38:00.000-07:00

In reading what is good about men? this early morning due to a bit of insomnia. It was an interesting read.

I'd like to apologize now for the link-bait title. I actually encourage women to vote (even more so now), and I have no problem with whatever feminists do. Also, for brownie points, I took women studies in college; primarily because I was seeking to understand why so women are so sparse in computer science.

In the "What is good about men?", if you take it as an axiom and ask follow-up questions. What are the consequences. What about women and voting? Why have women not had/have a vote? Why have men been against it? Why do men oppress it?

So, I was thinking about that when I was trying to get myself to sleep. It's interesting.

I think I got it. Initially, women didn't want to vote so they were not responsible for the stupid ass decisions the men were taking. That's nice, but why the aggression against it?

Well, I think the story goes like this. We are the children of victors. Once the victory is achieved, the peace settles in. A generation is born in peace and corruption and the conflicts moves inward. This corruption enables weaker men to gain prominence and weaker men need to put people into submission. That's where the differential is introduced.

As to laws and aggression, most people fear change and will fight based on that notion alone. Strong men embrace change because change is different, interesting, contains glory, and fun. Weak men fear change because strong men come out on top.

In any-case, I should return to sleep now. I thank HN for bringing it my attention whilst I desperately try to sneak back into bed.

Nerd Rage: Bad Controls on iOS devices for games

2011-07-09T23:57:00.000-07:00

Like, seriously, state machines are not hard.

Taking a few steps back, I bought a game on the App Store for over $0.99; when I spend more than $0.99, I expect magic and perfection.

When I get a buggy, clunky, and bad user experience, I get angry especially when I paid more than $0.99. Now, I could just sit from my magical armchair of developer perfection and complain, or I could write some crappy code and maybe educate some iOS fools with code. So, I spent 20 minutes to write an HTML5 joystick controller. Here it is world: angry code to do joysticks right.

Put this in your iOS 4 and enjoy the multi-touch-ness: http://ios.jeffrey.io/

Thoughts on good UX for mobile and games:

1. You need to control the event structure and not relegate it to some bounded box model. Bounded boxes are fine for buttons, but for things that need a level of fluidity and engagement, you need to control the entire screen.

2. You need to understand state machines and how to guarantee the absence of a bad state transition. Namely, the controls shouldn't stick nor flip-flop. It should be very simple.

3. You need to understand there are a multitude of thumb sizes.

4. You need to eat your own dog food.

I seriously wish I could get a refund. Argghhh!

The eternal engineering question

2011-07-09T18:43:00.000-07:00

Build or Buy?

This question haunts me because if I buy, then I'm not getting exactly what I want and am compromising my wants. If I build, then I risk it being disappointed (and messing up), and worse if its not what I want, then I have to live with the shame and can't complain.

So, I was going to go to Mac

2011-07-09T17:56:00.000-07:00

but then I started to look at the price and power of the laptops...

Compare: http://www.newegg.com/Product/Product.aspx?Item=N82E16834230103

and this http://store.apple.com/us_smb_78313/configure/MC721LL/A?mco=MjEyOTY5MDM

the mac book has same processor, 1/3 of the memory, 1/3 of the HD, smaller screen, 1/3 of the video memory. Argh! So, maybe I'll just build one like codinghorror. Gauh.

Why writing programming languages is so fun yet so damn hard

2011-07-03T17:19:00.000-07:00

I love sitting down on a Sunday day and throwing a parser with some tree manipulation to try to create a new way of expressing computing. However, it's hard.

I'll point out some goals that I think about when I'm in a PL mood.

(1) I hate garbage collection in a non-trivial way. It's stupid. Before you curse me into madness, there are a couple of ways of dealing with it. One, ignore it and have memory leaks. It's not rocket science to manage memory, but it can be a pain in the ass when you try to pass around ownership. Two, kill the heap. This, I've tried and it works; it isn't for the feint of art. Three, have reckless memory manage in contexts that get destroyed and share state via a DB like thing (i.e. Mysql, SQlite, or a DOM). I like Three a lot.

The nice thing about having reckless memory usage with a persistence back-end is that the thing that is reckless can fail. Say, you run out of memory, ok, create, kill the context. Now, from a performance point of view, this approach is strange to say the least. This is why I'm interested in embedding transaction logic into PL for optimization.

I don't have a solution to this as my last big language attempt just used a DB.

(2) Asynchronous is full of win. I'm not a fan of threads (Threads are Hard, m'Kay), and I would like a very easy way to think synchronously yet behave asynchronously. I'd like not to have to pass around closures, nor do I feel like yielding my code with co-rountines. I want something better..

(3) Saving state is fun. My last PL had the ability to zip of a closure, put it in a queue, and then distribute the computation. I'd like to be able to pause the state of the computation and move it between computers. The difficulty here is making it play safe with a persistent data store. See (1)

(4) I don't want to write a VM nor target low level code. My primary target must be C (or C++). I'd like the code to run on the iOS. I'd like it to be vanilla C. I'd very much like it to work in JavaScript. I'd like a real "write once, run anywhere" language that targeted machine code.

(5) I need closures; just because they make my life easy

(6) I like either dynamic types or polymorphic types (ala OCaml)

(7) I like type inference

So today, I made a very basic framework for expressing some of these ideas. I have mental solutions (because I've solved them before) for (3-7), but 1 and 2 are driving me nuts. I know C# has a kinda solution to 2 using co-routines, but's only ok. I want magic.

I want real magical shit in my PL, but magic is hard.

Morning Choice: Video Game or Code

2011-07-03T09:56:00.000-07:00

Every weekend, I have a choice.

Do I play a video game? Or code?

Today I choose coding. Videos games are getting old.

And, I'm busting out ocaml because I'm going to try something very hard. I'm going to work on an asynchronous state machine compiler where you write code synchronously that turns into an optimized state machine.

First Joke of the Morning

2011-07-02T08:26:00.001-07:00

I'd like to talk about whole foods

The thing about whole foods is this: you need to read everything. The other day, I wanted some Parmesan cheese. So, I grabbed some at whole foods. It wasn't Parmesan. It involved sheep milk, and I was not amused. My hamburger helper like meal tasted like feet.

Comedy

2011-07-02T08:02:00.000-07:00

Delivery is very hard, so I've written jokes that are hilarious. Well, to me at least.

One is based on a very funny yet embarrassing experience from my child-hood. In a casual setting, telling people makes them LOLz. However, trying to stand up and tell it to people. Woah, that's hard. Well, its hard for me because I'm a hard-core introvert that lives in my own fantasy world. Interacting with the outsiders tends to be a messy process. Fortunately, I've learned ways of dealing with it.

For instance, I used to have a deep fear of talking in front of people. So, I taught at college. That beat the fear out of me, and so I can go up and talk in front of people. It's oddly easy when the subject is boring like math, programming, or business.

What about messy and embarrassing past stuff? How do I not crack up during delivery?

Back to comedy: How do i take a personally embarrassing moment in my life and turn it into a deliverable routine?

To the twitter mobile!!!

Update: twitter was a failure. How about quora?

Wealth: the link between Hacker and Hustler

2011-07-02T05:00:00.000-07:00

tl;dr;

hackers generate wealth, hustler's turn wealth into money

Longer version:

I was thinking about wealth the other day. How do you define wealth?

I think wealth is best defined by the ability to satisfy your want. If you want something and can't have it, then you are not wealthy (i.e. poor). So, I'm an inventor-hacker, and most of my want is satisfied with some work (except for things like space travel... I'm very space-travel-poor).

What about money?

Now, where does money play into this? When I grew up, my parents fought about money. My dad made it, and my mom spent a lot of it. So, I wanted nothing to do with money, and I put on the hat of the academic monk. I was happy in that role, but then I needed money (due to personal reasons). My personality isn't the type that settles for nickels and dimes when I actually have to go out and earn, so some friends and myself did the cool thing before it was cool. We did a start-up, and we were going to be bigger than Facebook and Texas! Multiplied!!

It was a failure; the next business wasn't a failure. I learned about money, and how to make it.

Money is time. It's someone else's time for doing something that neither you (nor them) want to do. That's the only thing that it is. It's nothing else.

So, if you want money, then you do things people don't want to do (or can't do) (like math), and do it well.

So, back to wealth. Wealth is satisfying want. If you have a lot of money, then you should be able to satisfy your want and be happy. What if your want can't be satisfied by money, yet you have it?

"There is no class so pitiably wretched as that which possesses money and nothing else." - Andrew Carnegie

What do you want? Answering this question is the key to your happiness. If you are a hacker, then the shear act of satisfying your want produces wealth. I've invented things my entire life, and I have to say that makes me very happy. I like making obscure tools for obscure problems. I like optimizing the shit out of the things. I like automating tedious things (because tedious things are bullshit). I oddly enjoy taking some shit ball mess of code and figuring out where it leaks memory. I like to solve problems, and I really-really love math.

The things that I do to satisfy my wants generate wealth.

What about the hustler?

The hustler's job is very straightforward: Transform the wealth that hackers generate into money. Build a nice cave, make the hackers happy on all non-hacking related dimensions, hire more hackers, and hire people to sell the wealth. Build a business that is good at transforming the wealth of your hackers into products for people, and then you win.

Extra credit: what do managers do? They make sure the hackers are producing the right kind of wealth. Tada!

JSON, JSON, JSON

2011-06-26T12:46:00.000-07:00

I love JSON.

I was going to write more than this for this weekend, but I got off topic.

Pick your Archtype (CEO, Professor/CTO, Humanitarian)

2011-06-19T14:13:00.000-07:00

I've tried to reply to Dan Miller's recent post in less than a paragraph, but it just doesn't work. So, here is the longer version.

I think it also depends on how you define success and how you measure the rewards. For instance, playing monopoly or a video game has very low risk, but it has reward. Can you make a living doing it? No. So, you have to do something else. Could you get good at it and have a part-time job mentoring other people in it? Sure.

I highly recommend Dan Pink's book "Drive" (or watch this awesome youtube video) which basically breaks down into three things people want: autonomy, mastery, and purpose.

It is very clear in your article that you are preferring autonomy and being your own trouble maker. Being a full-time entrepreneur is all about autonomy. What about mastery? Well, look to academics. It's not exactly rewarding in a capitalistic sense, but it is very clear how people driven by mastery end up there. What about purpose, this would be the domain of the non-profits.

So, it boils down to this: What is your arch-type?

When you answer this question, your path unfolds. Actually, answering this question is similar to what is your life's goal is probably key to your own sense of happiness.

γνῶθι σεαυτόν (Know thyself)

This is probably the most fundamental and important thing you can do in life. Stop pretending based on what others want or need out of you. Don't listen to others; they are wrong. Be yourself and understand yourself. Understand your strengths and your weaknesses. Maximize your strengths, compensate for your weaknesses. Find humility in your weaknesses. Find confidence in your strengths.

Part-time Saturday Entrepreneur: Time Management

2011-06-18T08:42:00.001-07:00

The first bit of advice would be to not write about time management. This entry is with respect to having a part-time business while maintaining a full-time job. It's also a longer comment to Dan Miller's recent blog post. Dan's my former employer and CEO of RedNova Labs and a great boss, and he has ambitious goals. This entry is about having a part-time business with less ambitious goals.

At any given time, I maintain the details of about 7 to 9 projects in my head with about 20 to 300 ideas on the back-burner. This annoys my wife since I'm always "on", so I try to make sure my wife has a time in the reserve.

So, how does time work into this hectic and crazyness. Well, there are 168 hours in a week. You should sleep for 56 of those hours. You should eat a good lunch and relax for 7 of them. You should eat a good dinner and relax for 7 of them. You should shower for 3 of them. You should take a long bath for 1 of them. You should also probably have about 3 hours of sexy time a week. You can rush your breakfast in about 2. You need to exercise about 9.

This gives 80 hours.

I live in an apartment, so I don't have to maintain a house. Houses cost me time.

I live in the city, so I don't have a long commute. Driving everyday is not for winners, and it is bad for the environment.

I don't have kids nor do I plan to have them.

My wife doesn't work outside the home, so I can have as much time as possible from life. I love having a house-wife! It's awesome.

My life is structured such that I'm not wasting precious time on stupid things that don't add value to my life.

Now, the rub is this. You can't focus all your time in one thing. You will burn out. You need distractions. Your distractions need not be a loss.

Instead, you need to manage your projects and manage your goals. Everyday, make progress.

As long as the progress bar moves towards 100% somewhere, you are winning. Now, you may not find the win you expect. For instance, moving projects forward slowly means they may become irrelevant or uninteresting. But, that doesn't mean the process of moving things forward was a waste of time. I learn things everyday, and I like that. I like to learn. Because of this, my life is in a constant state of win.

For instance, this blog could be considered a waste of time. But, I'm writing words and composing sentences trans-coding my thoughts from my disorganized grey matter. My writing style is developing. This helps my professional activities by improving my communication with others as well as opening up discussion with people. It's helping me express myself in a way that isn't code.

As long as you gain something from an activity, you win. It's as simple as that.

The key to keeping motivating is having the daily option to change your priorities. This is why work can be stressful because you are not in full control of the priorities. Your hobbies should not have priorities except to just try.

This is why my wife's business is not a service business. Service businesses require you to adjust your priorities to the people you are servicing (duh!). Instead, my wife's business is all about content. When the creativity is available, inventory is built. When the creativity is distracted, inventory is going stagnant. I'm looking forward to about a week every summer where I can sell inventory. I can hustle it at pike's place, and that helps me professionally in understanding want. Understanding want is the key to making customers happy.

So, if you want to be a part-time entrepreneur, then make sure the venture is compatible with it. Example part-time ventures: Making video games, writing books, composing songs, making youtube videos, teaching kids, mentoring college graduates, make swords, build a giant robot of death, coaching little-league, doing odds and ins for people, design board games, giving city tours, etc.

Now, the ultimate question is: What makes you happy?

I think this is the ultimate question. Once you answer it for yourself honestly, you can determine the path you really want to be on. How do you make it work for you? That's a different question which I don't have the answer for. I do know that what-ever you do, it at some level needs to make money.

This is the cruel reality of capitalism. At the end of the day, some of your projects need to bear fruit. The things you want to do may not bear fruit. A challenge for the social web is to make it possible so that more and more people can make a living doing things that they want to do. For instance, look at Etsy. It opens up a lot of doors for people that would of had to sit in tents during fairs; now those people can reach a global audience.

This is the challenge for full-time entrepreneur: make me more effective. Make sure I'm spending my precious time producing what makes me happy. Make it easy. Unblock me. Provide me a service that lets me jump over hurdles. Provide me marketing. Provide me sales. Figure that shit out, so I can win!

Part-time Saturday Entrepreneur: Week 1

2011-06-18T07:57:00.000-07:00

I did some administrative stuff. Picked a name, got the domain, set up the domain, picked a background color (#E6DFD5), created a slogan, set up a basic landing page, setup and installed google analytics.

I wrote a children's book this morning; this unblocks my wife on the first project.

It has plot broken down into 22 scenes that when printed on two sides of paper should make for a decent sized children's book.

It has a title: "The Robotic Mr. Ducky"

It has a moral: "Errors are ok, you're just human. Don't be perfect. Find Love." (I know, how awesome is that).

So, now that I have the story done, my wife can edit it, illustrate it, compose it. Now, I need a dead-line. Well, one of my dear friend's from high-school is going to have a baby in two months. That sounds like a good dead-line.

All before 8:00 AM.

For the rest of the day, I'm making sure she can actually get to work. Is the desk clean and ready? Are the supplies accessible? Etc.

Part-time Saturday Entrepreneur: Introduction

2011-06-17T22:41:00.000-07:00

So, I'm starting my wife's company. My wife is an artist, and she needs an art dealer. This is my hobby. What is your hobby?

All things aside, I want to talk about being an entrepreneur. So, I'm starting a new series. Namely, about being a part-time entrepreneur since I've got a full-time job that's funding my wife's venture.

So, what is my wife's company all about? I don't know, but it involves her turning her hobby into wealth, and my job is to sell it and turn that wealth into money. I've got six hours a week to do it.

Now, as to my writing. What should I write about today before I put on the CEO hat and get shit done for my wife's company tomorrow.

No idea. I have no idea what it means to be an art dealer! Awesome, something new to learn (I could be learning to fly, but my wife will not let me fly...)

Well, the first goal is to get the wife producing and turn on the factory.

Project 1: A children's book.

Seriously, I'm going to have her do a children's book and I'm going to write the content (because I'm an awesome writer as you well know). I'll turn off the cursing.

Once I get this as rough product, I'll buy illustrator.

I've never produced a children's book before, so what do I need to do? How do I promote it? The nice thing is that there are a lot of parents out there, so there is a market. I can sell copies at Pike Place market, but I fear that will fail. I could involve pike place market in the scenery (TODO: research the copyright of the "Public Market" sign). Now, I'm fairly sure I can get production going. However, sitting on inventory kind of sucks.

Ok, Ok, I'll hold off until tomorrow.

The sucky thing about being a CTO (or C*O)

2011-06-12T14:48:00.000-07:00

Your voice is representative of your company as you are in charge of the ship.

This is one reasons I left Kansas and my stewardship of a small company.

So, you can't write about religion in a controversial way. You can't be a atheist/agnostic in the mid-west and expect to have customers. You can't talk about the richness of human sexuality and represent a company in the mid-west. You can't raise money while wearing a death-metal t-shirt. You definitely can't wear pink and be happy at the same time.

Small business C*O have to conform to their environment. This is why I think liberal cities have much more successful entrepreneurial environments. If there is a strong gay presence, then I guarantee you can get away with a lot more things than in a conservative city. The attitude to fight for who you are is the same thing you need to build a company, so I think the correlation stands.

There are things that I want to fight for; I want to fight for gay rights, freedom, women's rights, and other noble things. The engineer in me wants the world to be better. It's in my blood as an American to fight. The problem is fundamentally, I'm not willing to let other people suffer for my fight. Maybe I'm a pussy? No, I think it is sensible. There is no point in recruiting people that don't share your fight.

The fundamental problem of the C*O is that the professional (market norm) and personal (social norm) roles in life are conflated due to PR, secret nature, and stupid human politics.

My favourite and all-time hero is George Carlin. You can not be like George Carlin and be a C*O (in the mid-west). If your company has a global foot-print, then you have to be careful. You have to be neutral.

Neutral is boring.

Neutral is death.

I want to live, so that means I have to experience life. Last Wednesday, I crashed #MozParty to spend time with a old friend. Today, I'm going to goth like bar that I've never been into to listen to music I've never listened too before.

So, I'm having a lot of fun in Seattle. I'm happy, and I can be my plucky self and share it with you. I like that.

Geek Hypervisor Mode; staying sane in the insane world

2011-06-12T14:42:00.001-07:00

I call being a professional "geek hyper-visor mode". Namely, it is where you put you wants and personality aside to get the job done and work effectively.

What does this mean?

It means your religion of OS, programming language, library, framework are all out of context.

It means your design philosophy is aligned to what the team can execute on.

It means your goals are aligned with the companies goals.

It means you pick up new tools in your arsenal to #win.

It means exercising good judgement.

Unwind.

The fundamental problem of hyper-visor mode is that it can wind you up.

The wife helps.

Side projects help.

Comedy helps a lot.

Writing helps.

Cursing, oh, cursing lots helps. Fuck yeah!

Why I just give up on Linux Desktop

2011-06-12T13:45:00.001-07:00

I've been an advocate for computation in general for the majority of my life. I have no real preference for either Apple, Microsoft, Google, or open source derived products. I measure products on their own merits.

For servers, computation, scalability, performance, expressiveness. Linux wins by far. This makes a great deal of sense when you consider the market. The Linux market is by developers for developers.

For enterprise-y shit like Outlook and Exchange, I'll trust Microsoft hands down compared to an advertising company (Why I use Google Apps for personal stuff? Well, ..., I'm lazy and cheap).

For finding shit, I can trust google.

For my phone, I love the iPhone since it is best product on the market that addresses my needs. Android? LOLz. Maybe... that depends on iPhone fucking up their momentum or another company building something that makes people happier than the cracked out iPhone.

Now, for a majority of my life, I've used both Microsoft Windows and Linux as my primary operating systems for Desktop. I give up on Linux in the Desktop, and while I like it for developing... It will probably never by my primary one at home. Why? I'm old now. I have things to do. I want my shit to work out of the box. It's just that simple. I don't want to tweak my usb wifi stick to work; I just want to plug it in and have it work with out me configuring a bunch of shit.

So, I'm abandoning Linux on the desktop since I primarily just have a ridiculously number of terminal screens open. Do I need an OS for that?

Besides, with all the cloud shit that people are producing, does the OS really matter anymore? I have been meaning to checkout a chromebook...

Now, let me think about Windows? do I want to live in the Windows world? FUCK NO. Windows is great for two things: Outlook and Video Games (I would say Visual Studio, but I don't drink that kool-aid any more).

Now, having seen the WDDC key-note, I have to say: Lion + iCloud = win. If all I need is Internet, then I'm golden. It should work out of the box without problems. For my consumer needs, I'm good.

So, that just leaves me with Games. While some of my favourite games are available on the Mac, I find myself leary of jumping the boat entirely. Maybe I should. I don't have time to play all my games anyway. Maybe this is the part where I grow up, and stop playing games entirely? Some weeks go by these days, and I don't touch a game at all.

OK, let's do a systems check on the games I'd probably sell a kidney for: I have a 360. Ok, so I'll be able to play Mass Effect 3. Diablo 3 will run on Mac. Ok, so far good. Portal 2? Check. Combine that with iPhone/iPad, and I should be set. If not, then I can use that bootcamp shit with one of my Window 7 dvds.

Ok, so I've decided. my next computer is going to be a mac. Now, should I get a laptop or a big ass machine? Hrmm... well, my ego and male obsession with bigger points me to the big 12 core bastard. Now, my wife would want one too. So, I have to be willing to get two of them.. Maybe not since that would heat up my apartment like crazy. So, let's plan on getting a laptop.

Going old School; first epic walk in seattle

2011-06-05T10:21:00.000-07:00

So, I like to walk. One idea I had which would be cool for runkeeper to do is to have fitness achievements based on GPS. Like, walk from A to B and explore the city. It could be a lucrative business for dense city markets.

Anyway, I was getting into the loop of "Oh, I should build that and turn it into a product" rather than walking. Instead, I'm going old-school. I'm going to use the printer and use a pen.

So, I'm looking at this map.

Then using gmap-pedometer, I plot out my course. Once I complete a few of these, I'll give my thoughts on it as a service.

Update. I failed at my original course, and I made a new one in real-time. It's even more epic. I did ~10 miles.

Booyah.

I'll post a few pics in a bit after my legs recover.

The secret to "Ideas are easy"

2011-06-05T00:12:00.000-07:00

Once you get experience, you will have ideas.

Once you are present in the moment and see problems, you will have plenty of ideas.

Once you have problems, you will have ideas.

Are they good? marketable? Can you execute on them? What makes an idea good?

These are the hard questions that I struggle with because I'm interested in many things. I like puzzles. Puzzles are like mental sex.

On College, UnCollege, and Why you should get a math degree

2011-06-04T08:07:00.000-07:00

I think about education because I believe that education is powerful ... if effective. I feel that most of a person's college education is ineffective toward producing skills needed for the future (afterall, we can't predict the future). I was reading about UnCollege, and generally I accept the premise.

A majority of students that go to the majority of colleges are not getting a good education; they are paying people to teach them shit that isn't relevant to the future. They are paying people to deem them worthy as a cog.

Is college a waste of time? No. If you love learning, then you can learn things from brilliant people. This is why graduate school is so awesome. However, here in lies the con game. Smart and brilliant people realised that the majority of people don't like to learn, most people just want to have a good job so they do whatever it is that makes them happy (typically: sex, drugs, kids). So, the academics tell industry that college will make your workers better, and industry introduced it as a requirement.

This turns most of the college experience for those that don't love learning into a marathon. If you complete it, then you are worth hiring.

The sucky thing is that this inflates the number of gear-check courses which tend to not be interesting to smart brilliant people since they can master the material fairly quickly. There were a number of courses in my past where I showed up just for the tests and got As. Why was I paying for that? Well, that's a gear-check course for "normal" people.

Math

I love math. The majority of undergraduate math classes are gear-checks for engineering students. However, they are interesting enough to brilliant people. If you want to be brilliant, then surround yourself with brilliant people. Once you get past differential equations, you will be immersed in a new level of mathematics that will surprise you. It isn't a gear check anymore, it's fun stuff that is centered around learning.

For me, I found the math program to be an oasis of smart people doing clever things. It's an oasis because it's hard, and this means the supply is limited to those that have the love of learning.

So, if you go the college route and you are smart, then you should do a math degree.

Hackers

Problematically, good hackers don't have time to deal with the bullshit gear-check courses. They need to hack and get high off their brain. I'll be honest, I can do more and get more with a enthusiastic hacker with a GED than I can with a freshly minted CS graduate that started programming as a freshmen.

I've been exploring this problem, and I have my name picked out (and it's cool). Now, I'm trying to figure out the strategy and how to make it successful. I hope UnCollege will solve this niche because I know the war for talent is waging on now, but if we want to get more talent, then we need to invest in innovative ways of training people (namely by providing opportunities to learn effectively) well in-lieu of stupid bullshit gear-check courses.

What I want

I want a cheaper alternative to College that increases the supply of quality talent that is marketable. The more talent we can produce that's easier to hire means the more cool shit we humans can do.

Gas + Commute = Bad: Before and After the move to Seattle

2011-05-29T14:37:00.000-07:00

On March 21st, I left Overland Park, KS where I was driving an hour a day to work. Coming to Seattle, I wanted to live close to work, so I do. I can walk to work in less than 10 minutes. If I hustle, then it can be done in five minutes. I don't even need a bus. I'd like to share some numbers from my mint.com regarding gas usage.

January	$363
February	$402
March	$293
April	$39
May	$0

The gas tank will be empty on June 1st, so I put $40 in it. I expect that to last until the end of june (at least). The only reason I have been driving recently has been due to my wife's check ups for her broken ankle. Beyond that, we can live in harmony without a car. Which is good because traffic in Seattle can suck it hard.

Now, you may be thinking, but I like saving the money by living in the burbs. OK, great. But, I get to have that smug feeling that I'm better than you because I don't use gas. That's way better. That combined with my wife's obsession to recycle means we are a green family with our heads planted firmly up our asses. It's awesome.

Business for Programmers: Marketing Yourself with 10 steps

2011-05-28T08:26:00.001-07:00

The nice thing about being in a start-up is that you work with marketers and salesmen close, and I had the pleasure of working with a fantastic marketer and superb salesmen. So, you may be wondering why a programmer needs to know anything about marketing?

Well, that's a good question. If you look at a business, then it breaks down into doing these four tasks:

Build Thing: Make a product/Offer a skill to service

Market: Find people that like said thing

Sell: Get money for said thing

Support: Make sure said thing survives customers

That's basically the entire understanding you need to understand business. If you can write a business plan in a page that addresses these four things, then you can build a business.

As a programmer, you are a business; ignorance of basic business fundamentals will own you in the future. You go to school or hack your projects out, and this will build your skill at building things. However, if you build something and no one uses it, then did you build anything?

Step 1: Have skills that fit the market

Do you have skills that you can market to companies? You need to have skills, and your product needs to fit the market. If you know COBOL, then you may want to upgrade to Java/.NET. If you know OCaml, then you may want to downgrade to Java/.NET. Learn PHP or Perl since it is a plague on web programming.

Step 2: Find a passion, pick you destiny

You need to be passionate about something, and you need to have an interest in computing. If you don't have passion, then you are not going to be a good product. This guide isn't going to be much help without passion. While you may be passionate about computing in general, you need to decide on one to build a career on. If you want to be a generalist, then you can free-lance or be a technical co-founder. If you want to be a specialist, then you need to find a company that needs that specialty.

Step 3: Join/start an open source project centered around your passion

Just get involved in the community or try to build your own. (a) it will help you network, (b) it will communicate your passions. Everyday, you need to be writing code on something.

Step 4: Have a github account

Make good code, and share it. I'll be honest, your resume compared to github is shit. Get something good on github that is tasty. Now, get 10 followers. Pick code to share is similar to the code you could stand writing for the rest of your life.

Step 5: Dedicate time to stackoverflow/quora/forums/linkedin/reddit

Make an effort to contribute/lurk every week. Scouring these sites gives you an idea about the entire market in two ways. It gives you an idea of your competition, and it gives you an idea about the current market conditions. Any site that clusters programmers is worth paying attention too.

Step 6: Build Karma on HN

If you don't have an HN account and consider yourself a hacker, then shame on you. You must be living in the cave. I get it; caves are fun. Now get an HN account and start contributing. Don't interject until you know you can add value to the conversation.

Step 7: Get over rejection

There is a new thing called rejection therapy, and I recommend it. I recommend buying some products and trying to sell it door to door, and you will understand the nature of sales. Understanding the nature of sales is critical to understanding the importance of marketing. You need volume.

Step 8: See the world as risk/reward and take bets.

Look at the newfangled shit that comes out of the industry. I remember betting badly against ruby due to it being stupid dynamically typed shit pie. I didn't take the risk on Ruby, and I lost out on the reward. There's a new thing now called node.js; is it worth the bet? I don't know, you decide.

Step 9: Put yourself in the business of your customers

The nature of programming is that the act is usually always helpful to someone. Who are you trying to help solve a problem? For who? You? Your Mom? Your friend's Mom? Your coworker? Your business partner? Before you can begin coding and solving problems, you need to know who you intend to help. If its just you, then fine. If it is your coworkers, then you should probably talk to them. Why do your customers need you? How do you make your customers more awesome?

Step 10: Meet people

We live in a new world where sites like meetup.com can trivialize the finding process. Everyone is looking to socialize with the right people. Once you find people. Don't fuck it up since that tends to make bitter people (i.e. you alone in your parents basement troll'n it up). Read "How to Win Friends and Influence People", and then try to apply the principles.

The Engineer and the Artist

2011-05-23T01:30:00.000-07:00

I've noticed a pattern emerge amongst my friends and myself.

Many of my highly technical friends are computer scientists, mathematicians, scientists, and engineers. They would be the canonical examples of left-brain thinkers. Most of them have married (or in an active relationship) with an artist of some kind. The significant others are the canonical examples of right-brain thinkers.

The pairing is perfect.

Since I'm most familiar with my wife, I'll share some things I've learned about my wife and life in general.

My wife inspires me.

Before my wife, I was a methodical machine producing code. I try to change the world and make it better. Engineering, science, math are the ways we organic clumps of meat transform reality to make our lives better. Life is messy, and its our job as engineers is to clean it up. Why?

Once I've constructed the perfect reality where I sit in the middle of a vast plane of mathematical perfectness sitting on my throne of enlightenment, then what? Seriously, once I achieve the perfect state of the universe, then what? Once I've made all the money in the world, then what? Once I've built what I wanted to build, then what? Once I've built the ships to carry us out to conquer the universe, then what? Once I've conquered the universe, then what? Once I've reduced all existence to a single dense form, then what?

Explode, break it, and fuck things up. That's what.

Engineering is the way we improve our lives, but it is not the answer to the meaning of why we live. People find meaning in all sorts of ways. This is when I discovered art. Why is that pretty? Why does that move me? Why does a song sound beautiful? What are these emotional things? Why are the cracks in the pavement interesting?

These questions are the ones that turn a black and white reality into a messy pile of grey.

Who employs my wife? me. Now, if I could just get her to sign the damn employee agreement! Artists!

Seriously, I love my wife. Everyday, she teaches me what it means to be human. She breaks the mechanical barrier that I have around my heart, and she inspires me to be better. Not better in the engineering sense, but better in the sense of being a human being. Without my wife, I would probably turn into some troll.

Life needs more love.

I think engineers are attracted to artists for this reasons. If you are an engineer, then you have to deal with the shitty facts of reality. Like the speed of light. The speed of light is a mother fucking bitch. I hate it. Laws of thermodynamics suck too. Gravity, oh its a mother fucker! Seriously, life is full of shit that some engineer has to go in and work around.

Why do we deal with all this shit? Well, because life is harsh without engineering.

Now, I'm not saying that engineering is devoid of art, engineering can be an art. However, at the end of the day, the things engineers build have to work. This means that engineers can not wave around magic wands and make life better. It's slow methodical work that requires team-work.

Art fundamentally does not answer the ultimate question of life. This question, which spawns religion, is in my mind unanswerable using any formal reasoning system. I have no methodical way of exploring this question, so I don't. Art however is a great way to keep our minds off that question.

Meeting an engineer's significant other generally will generally tell you everything about them. My wife is the kindest person I've met. She feels bad for stepping on flowers. She's sweet and kind. She changes my world view much to my infantile protests. I swear, she doesn't have a mean bone in a body. She even recycles hard-core!

She inspires me to try to write poetry, and she opens my heart in a way that I can't express. I can't engineer a feeling, but her existence is like poetry.

Instead

2011-05-22T21:47:00.001-07:00

of hacking away at some bitcoin-bazarr, I spent the majority of my day making my wife happy, writing a love letter to my wife, and reading.

What did you do today?

Me the philosopher

2011-05-22T11:33:00.000-07:00

I'm thinking about life and death and stuff; does that makes me a philosopher?

Yeah?

Sweet! I'll add that to website.

The dream that was our first failed start-up; start-ups are hard; the future is bright

2011-05-22T07:55:00.000-07:00

I've recently discovered the awesomeness of bitcoin. Some years ago, me and some friends did a start-up before it was really super cool. The dream was simple: enable artists to buy and sell their shit as content is king.

There was a sub-text to the dream that was simple: FUCK MIDDLE-MEN.

We were building a in-system currency for enabling micro-transactions so people could sell things for fractions of a penny. We wanted to get prices to a point where indie artists could create and stimulate market demand by using low prices. In theory, it was awesome.

I fucked this up by being an academic trying to build a transaction system that would abide by all the rules of VISA/MC/banks, and then scale to the size of 10x facebook (Transactions are hard!). I say I fucked up because I think I put us in a position where we needed investment. There were lots of legal problems around our dream, and this was a big barrier to investment; so, we watched the dream die a slow and painful death. At least, in Kansas. Gauh, why did we try to do it in KS!

Now, the perfect solution is available via bitcoin. I'm honestly glad we didn't succeed because bitcoin + a weekend hack project could have put us out of business. We live in a bold new world with this internet thingy.

The dream we had can now be concisely built and done without worrying about VISA, Banks, Hackers or other shit. In fact, a bigger dream is available. Shit!, bitcoin could end piracy! (Well, kinda)

We are watching the death throes of the music industry. Eventually, the movie industry will end too. See, its not that music or movies are going to die. The middle-men, executives, organizers, and overhead to get the music and movies to consumers. They are going to go away with all the licenses.

Well, they aren't going to go away in a #rapture sense. They are going to have to accept a new reality (which is painful because most people hate change rather than embracing it). They are going to have to focus on the new emerging market and learn a new aspect of the trade. Well, people will skill will. People with no skills, they are kind of fucked... They should become artists.

How? Well, imagine a bit-torrent like protocol that required a bitcoin wallet. Sharing would be restricted between those people that have paid for it. The artist would simply start to seed their product with their wallet. In theory, this would lower the price of the content, increase the artist share. Lower product price + greater artist share should mean increased demand and thus more value to the artist. The ability to control pricing on the fly would enable artists to effectively market.

Downloading a client would be the most hard-core version of this scenario, but it is possible.

The interesting thing is that bitcoin is unstoppable. It may go into the black market, but eventually, it will win and be a legal form of currency. At least, for buying digital content.

What I'm considering now is whether or not it is worth it for me to build an escrow service with a search engine. It would be fairly simple with Elastic Bean Stalk, S3, some EC2 machines for search via Solr. Shit, I should be building this! Next weekend, I'm going upstairs to be with my wife.

What this system is going to create is a place for amateurs to get to a point where they can take on patronage projects. Once people effectively demonstrate that they can make things people want, then they can raise serious funds. Look at KickStarter. I love kick starter.

This is how small time videos produces will go big. I watch a bunch of YouTube people, and if they tried to raise serious funds, then I would fund them with some pre-sales. Now, how this will turn into $1B budget for making movies, only time will tell.

I think it is possible, and that is step one.

A good problem to have (unless you are Sony)

2011-05-22T07:00:00.000-07:00

"A good problem to have" is entrepreneurial way of saying, we are fucking this up and pushing shit out the door, but it doesn't matter if no one uses it.

For many people, it works.

For some, it fails. See PSN.

When you incur technical debt that puts people at risk, then you must pay it.

My role in life is not to create "good problems to have". My role is to solve them. I like solving puzzles. This will make sense as I'm writing a larger entry, but I wanted to throw some content out the door that was self-contained to my future main-point.

WeUseMath, so do I; I love math

2011-05-21T12:56:00.001-07:00

I like math, and I used to own mathsex.com; I was trying to sell math a long time ago. I used math yesterday to see if I was about to blow smoke up my own ass. I like math because it's not bullshit. Fuck, my alias is mathgladiator.

Now, here is what I understand. The US has severely fucked up math education. Like, SERIOUSLY fucked it up. What does math need?

A new fucking name! I mean brand. The brand is toxic, and we should dump it. People hate math, and this is a religious zeal. I don't think it can be saved by smart people; it needs to be solved by good marketing.

Let's re-brand it as "hacking the universe of truth" or something like that. I wonder...

I'm too old for node.js; maybe

2011-05-21T10:08:00.000-07:00

I got up today excited to do a bit of hacking since my desk is now cleaned and workspace is optimized for getting into flow. It took longer than expected to get to a steady state due to the new job, the move, the wife breaking her ankle, and exploring the awesome city of Seattle.

I started sketching out what I wanted to do today, and it required good S3, SQS, and EC2 libraries. And, I wanted to do it in node.js. The thing I wanted to build, well, I can't talk about it. But it is super cool.

The library support I wanted from node.js wasn't what I needed, and so there I was with a decision. Should I

(a) roll up my sleeves and do the whole github thing and get shit fixed

(b) say "fuck all" to the community and write a better more awesome library set

(d) scrap the thing and give up

The answer is C, I was done in ten minutes with the prototype. The idea makes me happy, and I can go back to bed where my wife which would ultimately be a better move than dicking around with code on a Saturday morning (boobs > code, deal with it). Should I decide to commit to the project more this weekend, then I'll probably do it in Java. Why? Because Java has rhino which is good enough for what I need. And, eventually, if my idea pans out, I may just throw away rhino and use antlr to do the serious business of writing what I ultimately need (because me + antlr = magic).

Answer B is what I would have done before I did start-up #2 (the one that made money), and this tends to be the reason why there are so many half-baked things in the Open Source community. In fact, there is a phase when you are young and you are compelled to create things. You have to prove it to yourself. Until you prove it to yourself, you tend to make the worst code ever (and also, most potentially, the most innovative code). The proof that I could create was when my first "product" was sold for $60k. All my code before the start-up was shit even the first "product", and while I feel bad for selling shit. Most shit that is sold is, well, kinda shitty to someone (That's why you need marketing, to find people that don't think your shit is shit). All my code during start-up #2 is meh. All my code now, well, it's production-worthy.

Answer A is what I would do if I was an active technology co-founder. Most old-school business guys don't understand open source. I'll write a blog post up on that one day. For now, just consider that this is a very good option provided the community lasts for long. That's the open source gambit. I think node.js is a young man's game. It's awesome and full of potential, but it will require a lot of work and reinventing many wheels to get to production worthy-ness. The libraries need to stabilize further before I can build on them (standing on the should of giants?) and do my research with them, and I'm not really willing to commit large portions of my time rewriting/fixing libraries that already exist in other languages. So far, beyond really awesome IO, node.js isn't that sexy for research.

Answer D is not really an option for me since I'm obsessed about seeing and feeling my ideas work and studying consequences. Until I get some half-baked crappy solution out the door, I can't be rational about and see it for the steaming pile of shit that it may (or the revolution of awesomeness that it could be). Until I test, taste it, and see it, I don't know. This is a fault about me. I also like to tell people about my steaming piles of shitty-code.

So, where do I stand on node.js? Node.js is awesome, and if I was a start-up CTO again, then I would be using it hard-core. Why? It is virgin territory and building with, promoting, and helping the community will build up your technology brand. Node.js helps a start-up more than established companies on two fronts: PageRank effort + recruiting. Google's PageRank is life-blood for many companies, and not having it can be bad and expensive. A link from the technology community is worth the weight of a fat baby in gold. Recruiting young-eager programmers is a key part of any companies strategy for growing and being successful. Building on node.js makes your company super sexy to young-eager programmers. For start-ups, this helps them rationalize a hit to their fair market value since they get to have sex with node.js on a daily basis (happiness is strange like that).

Now, I'm an old man with big and or important questions. And, while I find node.js to be that sexy chick at the programming bar, I find myself married to big and important questions which require mature and establish languages and platforms. So, when I fuck up, I know its me and not everything else (unless I've stumbled on a bug which should be rare, but more importantly easily observable in a mature platform). These are fundamentally more important than squeezing a bit more compute/io utilization out of a machine since I'm tackling problems that need to depend on a lot of production-worthy code.

Until you youngsters get your shit straight and make node.js mature, I don't have the time to fuck with it. So, I'm jumping back in the C# and Java camps with static typing and mature sexy code, and when I need awesome IO, I'll use erlang because erlang is awesome.

Routine

2011-05-19T19:41:00.000-07:00

Get up between around 8AM, deal with morning shit, walk to work, eat breakfast, get crackin' on work to set a context, walk to whole foods, eat an epic meal, walk back to work, get crankin' on work, go home, cook dinner, bed activities, sleep

Adventure into Pike Place Market

2011-05-14T15:45:00.000-07:00

OK, I am establishing my weekly routine/framework. Today, I'll post about my Saturdays. I'm on the slow carb diet, so every Saturday I eat some protein when I wake up (this wakes up the metabolism and prepares it for destruction), and then I head off to Pike Place Market for absolute destruction on some poor helpless carbs.

So, I walk from my place to westlake and mercer where I jump on a S.L.U.T (ok, SLU street car). I watch the road pass and see the space needle (pic). Getting out of the SLUS (not as fun acronym), I see a statue of some dude (pic). Walking down to the market, i see a nice sign and fumble around to take a picture with me in it (didn't see how with instagr.am to use the front camera). If you don't want to see my face, there here is a picture of just the sign.

Round one: Mac and Cheese (best ever). This mac and cheese is so good it makes all the other mac and cheese I've ever had a distant memory. This includes the mac and cheese from nicas cafe (very sorry byran...) and McCoys public house. Look at the picture again and tell me how it doesn't look amazing.

There is a cheese festival going on and I see a tent with a bakery (and this related to cheese how?). Ooh, brownie! That was tasty and sinful. Moist and fresh. That bakery services several places in town, but I've never had it that fresh before.

Round two: Since I managed to get out the door early today, I got to Piroshky-Piroshky early and had an Apple Cinnamon thing made by real Russians with really think Russian accents. Normally, the line is crazy big, so I'll come back next week to get things I've missed.

With enough carbs so far to choke a small child, I find a place to admire the view.

Moving on, need to destroy more carbs. I go look at the donuts, and I have a heartattack just thinking about them. However, today, I promised myself to try new things, so I moved on. After having a visual orgy of donut goodness.

Did you know that they have flying fish? Next time, I'll try to be quicker in dealing with my phone. See, I walk pass and see a fish fly. Then i wait and nothing happens, so I move on. I should take a chair.

Now, there is a hole here because I wander down the merchants. I also ate two chocolate covered pretzels. Unfortunately, they were consumed faster than I got get my camera out. My bad. There is some cool stuff that you can buy here. I fantasize about owning it, then move on realizing I'll have to carry in my next move. I got to the end of the street where there is a park. Pigeon!

The carbs have hit me now, and I need to sit down and stare. So, I look at the mountains. I get up and look over the side and HOLY SHIT A TRUCK. I also have the desire to own (or rent) a boat. The park (I suck as a tour guide, what is the name, oh well). A totem? Yes, a real totem. Also, there were some meditating Chinese people protesting in a non-violent way.

By accident, I discover that instragr.am has the ability to switch to the front-camera so I took a better picture of my head (oh, and the sign too).

Now, moving on, I get my wife some stuff. She loves seattle's food scene because it serves the gluten free market several orders better than kansas city. I get her two cookies.

Before I eat a cookie for myself, I drink some chocolate milk. That was the best damn chocolate milk I ever had.

Round Three: Cookie!!! NOM-NOM. That's a man size cookie. I eat one and get one to go.

Now, there are many interesting characters at pike place market. For instance, there is a metal man. Check him out. I give him a dollar and move on to buy some stuff for my wife.

I found a shop that has settlers of america? what, settlers of catan with an american map? and more resources?? wha... AWESOMENESS is this.

The carbs take me down and I get tired, so I head on home on the S.L.U.T.

Now, dinner is going to be interesting...

Research Goals for 2011, Changed

2011-05-13T20:43:00.000-07:00

Except I can't talk about them.

Trust me, they are super cool.

Mastering programming and the role of deliberate practice

2011-05-08T11:58:00.000-07:00

If you want to master programming and become the best that you can be, then it requires a lot of practice. However, writing lots of code or following lots of tutorials is not going to do it. You need to be deliberate. Being deliberate in your practices means you do the following four steps

Set a goal
Work
Reflect honestly
Pivot

and then you repeat...

Setting goals

If you don't have goals (and many don't), then you are going to fail. This is why I would say curiosity is a precondition to mastering programming. "How do I do ___ ?" Once you have a whole bag of those questions. Answering these questions and scratching the itch is how you set goals.

Setting effective goals

I think this is the hardest aspect of developing skills, and I think this is where getting a mentor makes the most sense. If you are fortunate, then you can set your own goals and figure out a strategy on achieving them.

Goals are recursive, and your ultimate task is to turn your goal into a series of actionable steps.

My general strategy is to pick the most ambitious goal you can think of that interests you and then start breaking it down. For instance, I wanted to build a computer algebra system. That's a daunting goal.

Daunting goals are good because they provide an end-game and generate many goals which you can iterate and improve on. You'll learn more from trying something big than trying a number of small exercises out of a book (This is my thesis statement for the future project I'm toying around with).

Let's say you've picked your daunting goal and you have no idea how to begin? This is the role of a mentor to look at your goal and say "first you need to do ___"?

Work

After you have your first goal, the next step is work hard. I'd like to say work smart, but that probably isn't going to happen. I say pick a time period and commit to working on it until that time period is up (or finish if it is going to be a success).

A combination of study, design, coding, testing will eventually pay off in either a failure or a success.

Reflect

Why did you fail/succeed? What was a complete waste of time? What felt like the most productive? What are you missing? Are those resolvable? A mentor can help you through these questions, but you need to first be ready to hear the results.

The more honest reflection you can give to your work, the better. Everything I do, I write postmortems for (both successes and failures). Some are one-lines and some are multi-page reports. I do this because I fuck up (a lot).

Pivot

Integrate your reflection and thoughts to help pick your next goal. Do you need to try something else? Do you need to try again after a refreshing bubble bath? Should you move onto to a different project? Did you get inspired to work on something else?

As long as you continue to practice something related to programming, then you will be successful.

Becoming a Master?

We often like to conflate being a master with being a monk like god where we have 1000 steps to reach our hut on top of the mountain. The reality is: being a master is more like being a self-aware student. You can look at a goal and then plan out how to tackle it. Being a master doesn't mean you can do it or accomplish it.

I think computing is becoming a lot like medicine where the things we build and study are escaping our grasp to fully comprehend. Instead, after some schooling and some resident program, we become practitioners.

Commodity Code

2011-05-08T11:15:00.001-07:00

I prefer to think of everyone on the planet that makes money as a business that takes a pain and solves it for a fee. Compare hiring a coder locally versus out-sourcing to one in India. They both produce code, but is it the code that I want?

If I go local, then I'll spend 10% of my time managing employee and 90% doing my own thing. If I go remote, then I'll spend 30% of my time managing/spec writing and 70% doing my own thing.

SEO god-mode with node.js /w Big Data Win

2011-04-16T23:36:00.000-07:00

I'm killing WIN and making something else also called WIN because WIN is an awesome name.

Why? The biggest thing old WIN does right is URL management. Now, I'm going to build a new product that's so fucking awesome.

new WIN = node.js + jsdsom

Basically, a proxy server that puts your server behind a strict HTML parser. Why? the stricter the HTML means the debugging process will gurantee correct HTML. Good HTML can be analysed on the fly. So, now, software can be written with craptastic urls like something.php?foo=bar and be transformed via jQuery on the fly.

What about big data? Well, dealing with file uploads in PHP is kind of annoying. Instead, I can use node.js to hyper-visor the upload and them stream it into S3. This will give the php script a URL which is what PHP ultimately wants.

Detecting Duplicate Content

2011-04-16T22:49:00.000-07:00

I think about duplicate content because I know that if I were to build a tool to detect it, then I could make a fair bit of $.

In general, you define an operator on content to transform it into a vector and then the dot product of the vectors will give you a clue into how similar they are in raw content (ignoring order of terms up to a point). Now, this operator is going to be complex. The natural algorithm that any one could build is going to have complexity O(N²). You can build clustering algorithms, but their performance may not be all that great and at worst are useless.

Solving the problem I want to solve may not be feasible without massive resources.

Does Google dedicate massive resources to detect duplicate content? Now, this is an interesting question, and I doubt they even need to. This line of thought gave me a clue on how to even build a product that would be modestly useful. Rather than thinking about how to detect duplicate content, I think about how to punish duplicate content.

It is very easy to punish duplicate content as you go. For instance, If I was google, then I would look at search results and prune out duplicate listings as I go. If I search for "mathgladiator", then duplicate content will rank similarly and be adjacent in a search (or close). This algorithm is O(M) where M is the number of search results. So, as Google returns results, it adjusts and punishes data that it believes is duplicate with some form of voting system. Over time, duplicate content is dead.

Ok, how to provide as a service? Well, take an open source search engine that provides full text search (Nutch?) and then have it crawl your site. Take a list of keywords/terms that you care about and then cron the search and then compare adjacent search results. Alert on content that compares.

Problematically, this doesn't solve the deeper issue, but it gives an advantage to those that can build this system.

Entrepreneur to Amazonian : The Amazon Vortex

2011-04-03T17:37:00.000-07:00

So, I used to be an entrepreneur/super consultant until I realized it wasn't making me happy. Happiness is a precarious thing. Ultimately, happiness boils down to aligning your calling with a profession that earns your keep (See Dan Pink's book Drive). The reality is that I'm not so much an entrepreneur as I am an inventor. Fundamentally, my callings are inventing, optimizing, educating, studying, building, proving things, using math, having fun with computers, and being right (a lot).

One day, I read Hamming's speech via pg, and it reminded me of why I was in academia: I want to work on great problems. I left academia to work on a super problem, but that failed. I thought about returning to academia, but I would have probably become a crank with gray hairs working on some crazy type system complaining about something.

Instead of writing papers that few will read, I want happy customers that benefit from my work. So, I decided to try an new job. I traced from my recent work and passion in Big Data and worked back-words, and get a job.

The first company on my list was Amazon Web Services, and now I work there. I'm working in the S3 team.

This means three things: (1) I have a Big Data problem (really big), (2) I'm under an NDA, and (3) I'm working with Protoss again.

So, like the Google Vortex, I'm now in the Amazon vortex. This fundamentally means my blog is going to change. I'll probably stop writing on technical things for a while. When I head to the mountains, I'll post pics.

Disclosure

2011-03-28T07:00:00.000-07:00

I'm an amazon employee. Nothing I say or do on this blog is an official amazon thing. I don't speak for the company at all.

I am a Red Nova Labs shareholder.

I'm awesome.

The Developer/Service Singularity, Saturation, and Good Ideas

2011-03-26T14:56:00.000-07:00

It takes very little capital these days to turn an idea into a functional prototype that scales to a decent size. This seems crazy... crazy AWESOME.

I have lots of ideas, and the sirens always call me. But, like Ulysses, I stay the maiden voyage of my plan. I also like to tell people my ideas. If you want some of them, then please send me an email at ideas@mathgladiator.com

Now, onwards to the meat of this article.

We have reached an interesting point in technology. It used to be, back in the old days, that if you had an idea, then you had to go hustle up some capital, invest in some engineers, and hope that it got built. Then you had to market it and sell it.

Now, if you are a developer and find a problem in your life, then you can solve it by combining pieces. For instance, I was thinking about automatically texting a friend to remind him that he should go the gym. Well, I can just do that with twilio and cron job. Given that I already have a fleet of servers, it's literally an hour away due to Twilio.

If you are a developer, then it takes practically no time to just solve pains; the number of pieces available to developers these days is rather insane.

So, what makes an idea good. Well, if it solves a pain, then it is good. What makes an idea a business? It has to solve a pain that is too painful for the garden variety developer to solve. Those ideas are the ones that you can build a company, and then provide pieces to developers.

I rather like this because this means that more and more pieces are going to be going to the cloud.

As more pieces of the puzzle are available, we will reach a point when all things are possible connecting the real world to the Internet. It's just a matter of time.

What about real problems? There was a post back on hackers news about how facebook was saddening due to the brain drain. In a way, I agree. I think I can say the same thing about wall street and math majors. It's very easy for smart people to go make serious money in wall street. Money is the same siren that pulls computer scientists and engineers to silicon valley and entrepreneurship. However, there are the faithful that stay back in academic to fight the good fight.

I think there will always be the faithful, and I think it is important for people in industry to understand the problems of the faithful. I understood the problems when I was a graduate student, but then I discovered EC2. EC2 enabled me to get more compute quickly than I could muster up in a math or CS department. It enabled me to iterate on serious problems and fail quickly.

So, I look now at the valley and the crazy economics of engineers as a good thing. The infrastructure and technology is being built by Facebook, Google, Amazon, Apple, and Microsoft. This cloud will be a commodity to enable future faithful researchers to get a new level. Researchers will iterate quickly, have more data, process more data, and solve the real problems.

Listen to old people

2011-03-23T01:11:00.000-07:00

In the entrepreneurial age, there is a lot of ageism.

I think the reason is two fold: one, young people are ignorant and thus more likely to re-invent a wheel in a different way that may work in the market. Second, they have less discretionary funds and are thus cheaper to fund. That being said, I would bet on old people building a better business to survive any bubble since they've been around longer.

I think worshiping youth is foolish.

When I was a kid, I actually listened to the adults around me. I actually didn't have a rebellious time in my life as I realized that it would most likely have negative consequences on my future. Every old person is basically a data point mapping a personality and life experiences into a final outcome, and you can learn from every single one of them. Listening to old people gives you a wealth of data if you are willing to listen, archive, and keep an open mind. Aside: I love quora because it brings out so much condensed wisdom on a variety of topics from everyone.

Now, I'm becoming an old fuck. I'll never become an old man as that is a state of mind (reminder, find George Carlin quote here, if you read this, then please help me find it). Now, the difficulty of becoming an old fuck is picking out the desired outcome and then back tracking to solve the right personality and right life experiences given the current life experiences.

The funny thing about life is that it is mostly under your control (except certain diseases and natural disasters) if you are willing to take control. It is very easy to let others take control of your life if you let them, and I would recommend not doing so unless you benefit in a fair exchange.

Optimizing your personality

You have a choice, you can work through your shit with a therapist, become an alcoholic like most of america, or just come to realize that you are 99.999% wrong on everything. I'm serious. The sooner you realize that you are just plain stupid and wrong is the moment you gain the power to change your mind and optimize your life. Now, if you want to be right, then you can be, but you have to learn to determine which battles are worth fighting; this is the trick. Use the battles you are willing to fight to determine your personality.

Optimizing your life experiences

There are a ton of opportunities to have interesting experiences in your life. You just have to put yourself out there to have them. You need to know what is possible, and you need to do things crazy. Problematically, people lack creativity these days and think spending time in Disney land is fun. Try going homeless and then running naked in Cardwell hall using the chemical shower to bath... That's exciting. Listening to old people about the stupid shit they have done is way more interesting than pre-packaged vacations.

Making all this work

Listen to old people. Old people can tell you what is possible and what is survivable. For instance, I'm proof that you can live on campus without housing. Let the old people that succeeded or fucked up serve as beacons on your path of what is possible and what to avoid. If you don't listen to old people, then you risk falling into the traps that they fell into.

Everyone on the planet will tell you they know how to live, but they are lying. They only know how their exact circumstances map into their current state with an uncertain future. Listening to old people give you data to predict what may be next, and that is a whole lot better than trying to reinvent the wheel. As a caveat, it may tune you into things that have been done rather than trying something crazy and innovative. This is the fundamental problem of doing something innovate: it looks crazy stupid to old people. The perception of a crazy stupid idea and a crazy brilliant idea are fundamentally the same until it has been proven one way or another.

Now, I'm writing this because I'm getting older, and I'm starting to the understand more about life. I'm priming the internet with content on old age since we are going to see bloggers get old. I was thinking about this when I realized that George Carlin passed away. George Carlin was my favorite comedian, and he was one of my heroes. It's interesting to watch his HBO specials and see him age and realize that's going to happen to me. That's going to happen to everyone blogging. What is going to happen when this internet thing matures? Holy shit, these bloggers are all going to die! The internet is going to become an odd graveyard of content. Holy shit, I'm going to die. Shit! Well, I might as well get started on the content about that so I get #1 on google in 20 years. *sigh*

My Research Problems for 2011 and Beyond

2011-03-13T15:47:00.001-07:00

I think about a set of problems, and apparently I'm not the only one. So, I'd like to share the problems I think about here and there.

How do I make a programming language not Turing complete yet useful enough and have that programming language satisfy the halting problem?
Why do graphical models feel powerful until I need to do something with them?
How can I utilize a Bayesian Network in a compiled form (for embedded systems)? Is this even useful? Can compiled Bayesian Networks adapt over time? Can I produce chips that get better over time?
How can I make Bayesian network inference super fast?
How can I use computer graphics to aid in teaching programming and software design?
Can I map out the entire infrastructure with a modelling language and have a compiler manage an infrastructure?
Do I even need a schema? What are the managerial aspects of going schema-free?
How can I make using a database suck less? Why is relational so sell-able yet so damn silly? Is relational out of date? How do I design with key-value pairs?
How do I deal with replication between non-heterogeneous environments? How do I deal with the CAP theorem? How do I replicate from the earth to the moon?
Is it feasible to put a mini data center on the moon?
How do I balance expressive graphics with high performances?
How can I make extracting good data from HTML suck way less? How can I query a tree with a beautiful language?
How do I utilize map reduce to produce good relevancy findings (recommendation engine, duplicate content)? How can I improve the quality given free-text?
How do I build a Voronoi diagram on non-euclidean spaces?
How does building a state machine compiler work for making synchronous looking code asynchronous without threads? Would Narrative Javascript + node.js be full of win?
How can I make sure node.js is going to win?
How can a game/physics engine be built around non-euclidean geometry?
How do I make sure CouchDB doesn't suck for some of my future projects?
How do I make jobs?

Some of these are old, and I'm fairly sure that some have open source solutions now. Some of them have answers, and I just need to read the current literature. I've got other questions, but they are more business related.

The most fundamental question I have is how can we improve education?

I think about education more than I should because I care deeply about it. My next project is going to related to education.

Do 1x Programmers keep 10x Programmers down?

2011-03-12T17:12:00.001-08:00

A question on quora got me thinking about programmers pay.

Is it fair for a 10x programmer to get 10x pay of a 1x programmer? The short answer is no, and while this does depress the hell out of me, I get it.

Being a 10x programmer is a crock of shit to begin with. I believe it in principle that people can be 10x and even 100x more effective, but it is a signal that the 10x programmer needs to move up either into management or work on more difficult problems. I guarantee that management will slow someone down, and I guarantee that there are problems that can slow anyone down.

Now, for various reasons, a 10x programmer may be stuck with a 1x job. The solution is for management to be results-orientated, or for the employee to figure out how to work at home. This enables the results to speak for you. I'm a huge fan of the four hour work-week, and I wish I wrote about it before Tim Ferriss.

The nice thing about being results-orientated is that it turns the tables and rewards effectiveness. I value my time, so I'll be effective with it however I want.

You can be so effective that you can wear multiple hats, and if you can do this, then you must do a start-up. A start-up is hard work, and it requires everything you've got to pull it off.

Generally thou, the four hour work-week doesn't work in start-up land unless either (a) you are the boss, (b) another part of the business is slow [and you can't change it], or (c) management is ineffective [and you are not a part of it].

I'll be honest, if you are a 10x programmer, then you have to try a start-up especially if you have nothing binding you to an area. Think of it like college, and if you succeed, then you make more than 10x pay. If you fail, then you are hopefully networked with people who know you are worth 10x pay.

How to always be right.

2011-03-12T06:41:00.000-08:00

Never take an opinionated stance on anything.

It's just that simple, but the problem is that you are not going to make money with such an approach.

Imagine if the news was just facts, then it wouldn't sell. Celebrity gossip sells like hotcakes, but a balanced emotion-less discussion of pros and cons of economic theory wouldn't sell at all. Push an agenda, then you'll get fanatics and make some money. This is why I'm not a politician/reporter, nor do I have aspiration to be one.

I was going to write a book on computer science called "It depends" and just write about all the stupid shit we do in the industry. Problematically, if we all suddenly understood and cared about the pros and cons, then we would work together and cut the work force by 90% and make 10% of our salaries. That would suck. Fortunately, the world is insane, and the pros and cons turn into a competition.

That being said, being right is surprisingly easy. The key is knowing as many pros and cons as possible beforehand and simply judging based on relevancy to the people it affects. I hinted at the philosophy that I've constructed for myself in an answer on quora.

If you apply this methodically and try to put emotions on the back burner, then you will only find yourself wrong in two situations.

When new knowledge is presented, and a new basis vector is discovered. This is when you have to seriously think and incorporate it into your relevancy equation or concede. This is a good way to learn new perspectives in the world.

Now, the other situation is purely political. Some people hate being wrong and hate changing their mind, and they will fight. I recommend just avoiding these people in life as life is too short to deal will closed-minded people. Now, this isn't you being wrong, this is the other person making you wrong especially if they can (i.e. your client, your boss, your co-workers, your spouse).

That being said, you can surprise people using new points of view and take people out of an one-sided debate and make it a discussion. For instance, when I was back on campus, some people came up to me as they wanted to share their faith with me. My friends know that I have no faith as I am a robot (actually, I invented my own religion, but that's an aside I'll share when I try to sell it... it's going to make super-super rich).

Here is how the conversation went.

Them: "We wanted to discuss your faith, what do you believe in?"

Me: "Money"

Them: "..."

/* note: I doubt anyone on a campus has ever given that answer */

Me: "I believe money is a side-effect from wealth, and building wealth creates jobs and makes life better for everyone."

Them: "Well, what happens when you die?"

Me: "Well, I hope the businesses I build continue on without me or someone puts them out of business with something better."

Them: "What happens to you?"

Me: "I don't know, but I don't really care anymore since what will happen will happen."

Them: "So, have you heard of Jesus?"

Me: "Yes, he was an awesome salesman. I know an awesome salesman too named Marc."

Them: "But, "

Me: "Let me stop you right there, who is paying you to sell me on your religion?"

Them: "No one"

Me: "That's awesome, I wish I could get college kids to work for me for free as I got lots of stuff to sell"

Them: "But we are not salesmen"

Me: "Sure you are"

Them: "But, salesmen sell a product, we are trying to help you."

Me: "If I have a lot of dirt on my floor, then a salesmen will help me with that problem by selling my a vacuum cleaner. You just don't understand your product yet. Your product is an answer to the question 'what happens why I die?' that you tend to ask yourself sometimes before you go to bed."

Them: "..."

Me: "See, the way I look at it, Islam and Christianity are just like Pepsi and Coke. They are products. Now, I don't have consumers digest to help me pick, and I've heard lots of bug reports from both sides. Personally, I'd rather just avoid the issue and do something else with my time"

Them: "... umm"

At this point, the conversation became small talk about my shoes and nothing important. They walked away visibly confused. The point I'm trying to make is that I brought up a point they have never considered before, and that basically confused them.

I don't really have a point to this article, but I wanted to share the story and talk about my philosophy. While I do enjoy being right, I enjoy learning more.

What are the most in-demand programming skills for 2011 and beyond, & why?

2011-03-10T13:06:00.000-08:00

Being good at code is always in demand, and it is far more important to be effective at reading code. To practice, you can take a library that does something you understand (say convert a csv into a data-structure) then strip out the comments and trace it. Extra points goes to the one that can do this under some kind of source obfuscation. I've met too many programmers who can't read code at a serious level, and I've met programmers that rely on superfluous comment; however, there is nothing except hope and trust that ensure the comment and the code are in parity.

No matter what, JavaScript is going to be the next C for the web. Everyone is going to know it. Due to its predominance, I wager there is going to be a boom in the programming language market searching for nicer languages to compile to JavaScript. See: http://en.wikipedia.org/wiki/CoffeeScript

I further wager that the whole difference between asynchronous programming and multi-threaded programming is going to be wrapped into a neat present with a nice bow-tie. Working with node.js (or twisted or event machine) will give you a leg up on new design skills as more businesses rely on node.js. Learn Asynchronous design.

NoSQL is just getting started, and I wager there is going to be evolution in who wins the developer market. I wager CouchDB (http://blog.mathgladiator.com/2010/11/how-i-use-couchdb.html), because I'm a betting man. I also wager someone is going to write (I've thought about it) a SQL for NoSQL. Once this is invented, I wager Oracle is going to buy that company or sue it into oblivion. Learn MapReduce.

Java/.NET wll be good for a long time, and it is the new COBOL. There are a couple of problems that may unseat them: Multiple cores with good parallelization, good asynchronous primitives, and stronger type systems (well, for the markets that need that type of stuff anyway). At least know either Java or .NET.

Java/. NET are always going to be ugly to someone, and that someone may invent something better (i.e. Ruby). It's important to jump in some bandwagons, so find something esoteric and learn it as a hobby, It could be big some day. You want to help make it big? Then build a business using it, and you will put your ass in the fire when things go wrong.

UX+Dev+Marketing is going to be key in the coming years, and I would keep abreast with those markets as you either may need it someday or you may want to sell them something.

Databases as we know them today are going to die... over the next ten years, so I would pick a NoSQL platform and start playing with the Idoms now. My picks are: CouchDB, Riak, Redis, and Cassandra.

Now, you could say that all code should be properly documented. Ok, well, tell that to the guy that left and you are a consultant coming in to add a feature. You are now screwed. If you can get effective at reading and understanding code, then you can easily charge $250/hr to fix really simple things in five to ten minutes (do lump sum billing and you are rewarded for being effective rather than get-it-done on a schedule, see http://en.wikipedia.org/wiki/Parkinson's_Law).

These are purely technical skills. You could amp up and start to learn where the industry is going by getting in touch with it. I recommend lurking on http://news.ycombinator.com/news. Once you see where business is kind of going, you can guess where the technology needs to go for businesses to get there. Predict business needs and you can predict technology.

2.2 month review of slow-carb diet

2011-03-10T07:28:00.000-08:00

1.3 months in : results (December 26th to Feburary 7th).

Lost 40 pounds.
LDL and Triglycerides lowed.
HDL raised.
Leaner profile.
better sleep and regular.
No guilt on Saturdays.

0.9 months off : results (Feburary 8th to March 5th).

No weight gain.
No change in cholesterol.
Terrible sleep.
Bloated.
Guilt for enjoying.

First week back on : results (March 6th to March 10th)

Feel amazing.
Lost 4 pounds (mostly water and colonic change).
Sleeping like a baby with regular sleep.

Thoughts

I really like the slow carb diet, and it works for me; it also improves my mood. I went off of it for two reasons.

First, I've heard people on carb-free diets gain their weight back quickly, so I tested that hypothesis. This is a completely bullshit excuse that my rational mind created to make the emotional side seem less of an idiot. The real reason was because I'm changing cities and I wanted to suck Kansas City dry of its BBQ. I made sure to enjoy my favorite places like Jack Stack and Arthur Bryants. Again, this is another bull-shit excuse, but I'm going with it.

Seeing how it affected my mood and sleep was enough for me to say "fuck this, I'm going back on".

PHP is chili, and why web development is screwed

2011-03-10T06:52:00.000-08:00

I ate chili yesterday, and it was awesome.

Is it my own recipe? Heck no, I out-sourced it. It made me think because I just throw vegetables into the chili and the red peppers masks the taste of everything.

It made me think about dog-houses and bike sheds. Basically, everyone has a chili recipe and they think it is awesome.

Lost Wallet

2011-03-08T10:39:00.000-08:00

OK, I just spent two hours searching for my wallet.

Why do I check the trash?

Why do I always have the sneaky suspicion that I put it in the freezer?

Why did I check the trash again?

Well, the mystery is solved. It was under my wife's pillow. Go figure.

On writing your own programming language

2011-03-07T17:16:00.000-08:00

In about three days, you can push out your own programming if you are a master at recursion, understand parsers, understand tree transformations, and have a good grasp on the goals of programming.

Problematically, a programming language is a small thing. The next step is the tools and the platform. This will take about one to three years to build.

Now, once the platform is built, it will take about two to ten years to market it provided the right people pick it up. This how Ruby went from obscure to mainstream, and this is why I give DHH mad props since he put a huge surge into Ruby.

I had spent about 1.5 years working on my platform for my programming language, and I had shipped web properties on it. Then I killed it.

I killed it because I realized the world didn't need yet another shitty programming language. It was categorically shitty because I didn't build it to be marketable and the problem it was solving was kind of silly to the market place and my ability to support it.

Basically, the idea was to build a programming language around a statically typed schema and then provide a language integrated query language similar to SQL in such a way that it could automate scalability.

It's extremely intellectually satisfying problem to work on, but it was a poor use of my time. However, I believed (and still believe) that the problem is worthwhile. It was intoxicating to work with statically typed SQL with a statically typed schema and then solve the induced persistence problem with replication and automated infrastructure management tools.

Over all, the experience was wonderful. I recommend anyone capable or willing to start to write a programming language. Once it is built, then you will see a lifetime of work ahead of you since it will never be finished.

Lessons Learned.

Working on the problem lead ultimately to dealing with replication at a serious level. Replication is non-trivial, and the CAP theorem is a harsh mistress. It's very easy to get going and then the shit will hit the fan, and then you are screwed.

Building a taxonomy of how different patterns in SQL lead to different types of systems gives a greater appreciation and understanding of the NoSQL movement. I am fully confident that in the next five years, we will have some very interesting solutions that enable a agile database movement. The entire process also gave me insight into why SQL really does suck and is very limited.

I also built my own map reduce framework which gave me a greater understanding of how to solve problems with MapReduce. I basically did a differential on a "join space". I would use relations to build a "join space" by using joins and then map a function on the changes. This looked very similar to how you can do MapReduce in CouchDB, but mine sucked. Actually, mine was terrible in comparison.

I invented my own heroku like interface for launching back-ends. It's fairly easy with AWS to automate your entire infrastructure management. I'm kicking myself for now launching a product off of it. It's wicked cool to push a button and have three MySQL servers pop up and connect to each other. It's even more wicked cool to see shards re-balance and push data into the new servers.

I added closures to PHP in a way that enabled you to serialize the closure to Amazon's SimpleQueue. While it was only a true closure in memory, it was very neat to add asynchronous processing to PHP. A simple C + curl driver would launch and basically run the event loop in apache2.

Documenting a programming language's more esoteric features can be hard since you have to teach new ways of thinking. This is why I generally avoid a DSL in most production environments. Generally, the process of building the DSL will give you enough insight to make a rock solid library.

People really hate static typing, and if you want to build a statically typed programming language, then it is going to cost a lot of money. Most of the commercial used statically typed programming languages have companies behind them with deep pockets; I don't think this is a accident.

Working Hard versus Working Smart

2011-03-06T07:23:00.000-08:00

Working smart is very hard, and most people don't do it. People will work hard doing smart things, but doing smart things hard is not working smart. Working smart requires being intelligent, effective, and efficient. Combine all three with good sleep, and you will win.

Increasing Intelligence

I believe intelligence increases over your lifetime provided you are pushing the boundaries of what you know and what you can do. If you do the same thing for 10 years, then you are not changing. However, if you spend the evening working on math puzzles and Wikipedia, then you are learning something and increasing something.

People like to have a fixed concept of intelligence as if you are born with it. This is a psychological defense of their own laziness since learning is painful for many people. I argue that intelligence is defined by the work you put into it.

Intelligence will ultimately define the level of work you can do.

Increasing Effectiveness

Spend time working on the right things rather than just spending time on work. Being effective is hard because there is temptation to work on other things that appear important.

In the land of start-ups, the key to being effective is to make sure the business state progresses every day. This means that the task you are working on must

(a) increase revenue
(b) enable the business side to iterate
(c) make a customer happy
(d) decrease costs
(e) find customers

If the task you are working on doesn't help any of these, then it probably isn't effective. Each organization or project has similar questions. I like to think of each task as a vector and the organization/project goal is another vector. Effectiveness is just maximizing the dot product between the two.

The key to being effective is to understand what the goals of what you are working on and how that relates to the sea of tasks in front of you.

Increasing Efficiency

This is the easiest to increase of all three. This is where you make a massive todo list and you just start grinding on it. I make multidimensional TODO lists and batch things together using a gray code of sorts. The key here is not to estimate the time per task and instead say each task should take about a minute, and then just go. This avoids Parkinson's Law. The key is that when things take longer than you expect is to take notes on the problem and then move on to the next item.

This is how I used to hack exams in college. It was easy to grind on all the easy questions whilst my mind thought more deeply about the hard problems.

The same is true for any workload. Get all the easy stuff done and then focus on the hard stuff. I try to load it up so I go to sleep for the hard stuff. I often just wake with answers and spend a couple of minutes writing the solutions out. Sleep is very effective for solving hard problems.

Sleep

I think our work schedules suffer from a severe problem in enabling creativity. We shouldn't work in the morning and then play in the evening. We should play in the morning and then work our ass of until we need sleep. Then, get to sleep as soon as possible.

When I was in hard-core start-up land, my schedule was 12 hours of work followed by 4 hours of sleep. This enabled me to solve gnarly problems all over the place. I would wake up with stunning solutions to problems.

On Learning Recursion

2011-03-05T10:03:00.001-08:00

A question on quora got me thinking about how bad we computer scientists teach recursion.

I claim programmers work day to day in environments that use recursive thinking, so understanding recursion is very important stuff.

We can easily try to teach things like fractals, hanoi, fibonacci, tri-ominos, factorials, and more. These are all nice and beautiful mathematical facts/puzzles, but they are bullshit and I don't think they help teach recursion. I claim they are bullshit because they are too nice of results and they don't show any scaffolding involved in how they were constructed or why it is that way.

Recursion ultimately boils down to "If I can do something on something smaller and I can combine smaller things into the bigger result, then I can compute on things that are huge."

It is a powerful tool, and I think the best way to learn and master recursion is to write a programming language. When you understand how a programming language works, then I further claim you will produce a lot less bugs and have a greater understanding of what your code is doing.

Recursion is akin to mathematical induction in the form of structural induction. Problematically, if you take a computer science course, then you will start with the math. While this is great in theory, most people have a shaky and incomplete foundation in mathematics. So, the mathematical approach to teaching recursion isn't going to be a wild success. I claim that a good understanding of recursion enables people to be more effective in functional programming (and thus programming in general).

Having complained and made many wild claims, I started a github repo on teaching recursion by writing a simple computational language which I may extend in the future. For now, it provides a simple parser and an eval() function and how to use it to write a graphing calculator. Each branch in the repo is a stopping point with the master branch containing the final lesson. It has 3 lessons now which give me a warm fuzzy feeling on the inside.

Update: check out Structure and Interpretation of Computer Programs if you are serious about learning how to write a programming language. I'm just a hack when compared to the authors of that great book.

Computing as a drug; me the addict

2011-03-04T17:17:00.000-08:00

I think my profession is all about getting a high from solving puzzles.

I love puzzles, and I'm an addict. I once missed a week of courses because I discovered the 5 x 5 x 5 Rubik cube. My math professors didn't mind as they were very understanding to my needs, and I was fortunate to not be taking any humanities during that semester.

My current fix is on distributed computing, and I find it fascinating.

During college, I was interested in game engines as it focuses on performance. Performance is a great way to get high since you know the theoretical capabilities of your hardware, so how do you hit those numbers? How do you degrade quality to gain performance? How do optimize away computation? It is a performance junkie's dream problem.

When I did my start-up, I was obsessed with Scalability (when I should have spent more time marketing) and technical execution. Scalability problems (even if invented) are pure crack. The reason is that there is no upper bound. With the game engine, the bound was given by the video card (i.e. you win when it shuts off from too much heat).

Ultimately, distributed computing comes into play to solve the scalability problems. However, scalability isn't the crazy fun part. The crazy fun part is when you accept the fact that any part of your network may fail.

This is a cognitive fail for me since I expect and make the assumption that my machine just works. I write programs that execute a series of commands, it should always terminate. What happens if mother nature pre-terminates your program? Can it resume?

Once you figure out how to deal with fail, you have to scale. Once you figure out how to scale, you have to perform. Once it performs, you need to extend. Now, rinse and repeat. Maybe you need a big rewrite? Who knows.

This is why computing can be bad for you, it never ends. The complexity/puzzles just add up (which is excellent for my profession).

Now, there was going to be a point to this blog post, but I forgot.

Going Homeless and Living in the Computer Science Building

2011-02-28T22:43:00.000-08:00

Story

Most of my friends know that during 2006 - 2007, I went homeless and lived in Nichols Hall at Kansas State University. Oddly enough, one of my friends at Kansas University went homeless too except he got caught after three weeks. I was homeless at K-state until I left graduate school (about a year).

At the time, I was both a graduate teaching assistant in Mathematics and a graduate research assistant in Computer Science. I had a full work-load, and I wanted to work more. I had a little cubicle in Cardwell Hall with other graduate students, and I had mine filled with books and my behemoth of a computer.

When I would retire for the evening, I would walk to Nichols and blow up my air-matress (had an automated pump) and lay my bones to rest.

In the morning, I would wake to the sound of students rushing about. I would head over to class and teach students some math and answer questions. After which, I walked or biked over to the Rec where I would take my shower (with a bunch of naked old guys).

Tip: old men are not shy and will wave/shake their cocks high and proud. Once, I was putting on my shoes and I got boxed in my two chatting old men. Too both my left and my right were two old cocks. While this was awkward, it did make me appreciate my youth and fearful of aging.. Eww.

After my shower, I would get on with my homework, my research, or my projects. Occasionally, a student would ask for help. I had the most hard-core office hours out of all professors and other GTAs. It was from 9am to 9pm. How can anyone not make those hours? Well, they didn't because they were lazy. However, one student did take my offer seriously and I mentored him from a D student to a B student where I gave him an A for effort. He then earned a righteous A in the next course. It is an awesome feeling to have actually helped another human being a serious setting as he was on the verge of dropping out; now, he is an electrical engineer doing amazing things.

Before I went homeless, I had stuff. I hate stuff as it just accumulates and then needs to be moved for no good reason. I developed an algorithm using a self-storage unit where I put everything into my unit and then would bring stuff I needed into my car. Beyond some books and clothes, I didn't need anything. After six months, I just started giving stuff away. Free 36" TV? it went to a gentleman by the name of Kyle. GameCube and a bunch of games? It went to a gentleman by the name of Brad. Salvation Army got about $4K of goods, and I sold $500 worth of stuff at a pawn shop. I kept my books, computer, and paperwork.

Having spent an entire semester homeless, I decided that it was stable. But then I decided to take the path of a start-up technology co-founder. So, I lived at our first office which was an apartment (that we kind of ran out of room). However, I was not on the lease, so I was technically still homeless.

I got my own apartment when I found my woman Effie (who is now my Wife).

Thoughts

All you need in life is an air-matress (which is optional if you tune your body to hard surfaces), food, a woman (or a man), two songs, and work. Sometimes, I would play, but play is just another form of work. A car is nice too.

I'm fortunate to find joy in Mathematics and Computing.

Sleeping outside is a lot of fun. Sleeping in my car was fun until my back started hurting. Sleeping on top of my car was very fun (again, until my back started hurting). Sleeping naked in a field is way awesome.

Blasting movies/music/video games in the Math building at odd hours is really fun. You can't do this at an apartment building.

When work ends naturally, taking time to smell the flowers and watch the world in motion is very relaxing. Going homeless enabled me to catch up on both my homework and my research, this gave me the time to go sit in the quad and watch the machinery of university and appreciate the groundskeepers.

Buying people lunch is fun.

Having plenty of time to enjoy campus is awesome.

Being able to be generous with your time to help other students is very fulfilling.

Final Observation and Why?

I was primarily using graduate school to prepare for an awesome career as a Professor/Super Developer/Ultimate Hacker. I was working really hard to become one because I reasoned that would be where I find happiness. Little did I realize that going homeless would make me happy and enable me to live in a way where I was rich in spirit.

Going this route enabled me to be selfless and find a new kind of joy. See, I changed when I knew that no matter what that I would be fine. I knew that I could be generous and helpful rather than hold on to what little I had. If you need lunch, then I will buy it. If you need a hug, then I will give you a hug.

This path in life changed me so much that it gave me a greater perspective and enabled me to find compassion. Compassion and selflessness didn't make sense to me since it seemed like I was voluntarily losing something. I saw life as a game where there are winners and there are losers, and I was competitive to a fault. Seeing the game, my goal was to win. The problem is that winning in every aspect of life is hard. What makes a person great at work can make them terrible in a social setting.

Professionally, I am a winner. Socially, i was inept.

If I had continued what I was doing, I was going to be alone because I was confusing the social and professional worlds.

Going homeless and surviving on less than $6K a year enabled me to realize that no matter what life threw at me, I would be fine. This meant that I could devalue the need to be a winner in the professional circuit. This gave me the flexibility to study the social aspects of life.

My fellow peers were delighted and noticed my mood and friendliness changed, and they became my friends. I was generous with my time which enabled me to ad-hoc mentor computer science students.

Fast forward to the present day, I'm married to a woman who inspires me every day with her compassion, empathy, and delicateness. She is the flower of my life, and I'm working hard to separate my professional work and my life with her.

The change I made in my life enabled me to go from some social inept nerd to a happily married man with an abundance of friends.

What is Computer Science?

2011-02-25T21:54:00.000-08:00

Is it an art? a math? a science? or engineering?

Well, strictly speaking. Everything is an art-form, so it is trivially an art.

There are mathematical models about computing and you can apply computers in math, so it is a branch of mathematics in both theoretical and applied aspects.

It induces complexity that can be studied and model. Problems can be studied in a pure fashion, and methodologies can be deployed to measure the unknown. It is a science. Debugging is 100% pure science along with good testing.

Is it engineering? Well. This is the tough one. By itself, it is not. It's too much art and not enough discipline. How many times can projects be rewritten? I see engineering as more of a way to overcome the limitations of reality. However, software and computing mostly have no limits except time and did it compile. Where as most other forms of engineering have limits like gravity, friction, heat, electromagnetic interference, collisions of matter, speed of light, and other limits. Recommend Reading: Why writing software is not like engineering.

Computing is very platonic and there are many ways to do things, so can computing be an engineering discipline?

This is where the role of the software architect comes into play. Take an expert programmer, give them a hat that says "software architect", and then task them with building an architecture that will enable a group of developers to build a business. At this moment, the software architect will cast the platonic world into the real world and introduce limits and constraints that enable the team to implement engineering methodologies.

This has some interesting consequences in industry. First, businesses diverge in their computing reality. Second, you see an order of magnitude increase in not-invented-here syndrome in software than other engineering fields. Finally, we will never agree on anything for long periods of time because it just keeps changing because we haven't come to agreement on "what is software engineering?". We haven't figured out a reality yet that makes enough of us happy.

Now, let's discuss how academics fucked up computer science education.

A Mathematics Degree

2011-02-24T07:57:00.000-08:00

When I was a graduate student at k-state, I was studying for all three Ph.D. quals in mathematics. This is part of the insanity that is me, but it also reflects my inability to make decisions between equally cool options. When presented with an OR, I choose both. This is also why after my Ph.D in math, I was going to get a Ph.D in computer science.

While my heart lives with computer science, mathematics is the fastest and most effective way to master problem solving in collegiate setting. Why? It's hard mode. When I get a game and start to play it, I put it in nightmare mode (hardest setting) and then play. I'll grind and accept the initial failures and learn from them. By the time I'm done, I'm a master. The same is true for math. If you go to college, then the math program is going to the toughest degree program. I have yet to meet someone with a math degree that wasn't successful in what they tried to do outside of college

The nice side effect of a math degree is that it lacks degree inflation. It's difficulty is more or less the same as it was 50 years ago.

A Computer Science degree is fundamentally half math and half programming, and many struggling Computer Science programs are churning out shitty programmers because they are reducing the math requirements and dumbing it down. This works great for many companies that have mountains of tedious work and managers who use management theory on programmers. Some programmers are cogs because someone figured out how to make that work for them.

For companies that depend on technologists (where typical management theory breaks down hard-core), I wager the talent war is going to get epic and qualified tech workers are going to win real big. The salary and benefits being paid to top-tier developers is a consequence of supply and demand, and it's going to tip even more in the favor of the limited supply.

Woe to computer science degree! Read On the cruelty of really teaching computing science; now think about market forces. When I was a student, I realized too things. People hate taking math courses and math is highly relevant to computer science/programming (at least, relevant to the fun kind of programming, not the boring kind where you push boxes 2 pixels left and 1 pixel down and change colors from sky blue to cornflower blue).

Keep in mind, its not the content of math that matters the most; it's the process that math induces. Getting a math degree will build an iron fist of reasoning that will enable you to write very clean and tight code. It will also get you knee deep into the abstractions that most people will not have exposure to in their entire career. For instance, when I took Introduction to Topology, it blew my mind.

The first day of topology, my response was "WTF is this?" On the second day, I was excited "omg, he just turned a coffee cup into a donut". It was that class that opened my eyes to an entirely new level of reasoning and abstraction that propelled me into the math degree. Also, much to my surprise, the class (and degree program) was also 50% women which is vastly different than the sausage festival of computer science.

I'm going to have an awesome career working on my calling which is entirely due to me making good decisions, and my best decision was to get a math degree. I recommend anyone that is able to go for it. Higher level math courses are actually a lot more fun than the lower level stuff. The lower level stuff is service to the college, but that's what pays the bills for the math department. The higher level stuff is where you will have a chance to see your professors glow like school children as they talk and lecture on things they find interesting. It's worth it.

That being said, if you take a math degree, then you need to make progress in your side projects in computing. Connecting your ongoing mathematical education to code will give you interesting insights. I'll shave an interesting perspective in five years when I finish my book titled: "A topologist walks into a database". Sounds exciting, yes? Well, it will be.

Now, I just need to write about how I was a homeless graduate student working 100+ hours a week.

If you want a more hard-core education, then you need to drop out of school and build a company like I did. This option is really hard-core, and it is the path of woe, humility, and ultimate success.

The Dirty Truth about Functional Programming

2011-02-19T21:54:00.000-08:00

The advice that I typical given young'ins in computing is this: "learn functional programming and solve some interesting problems with it; it will make you better". I'm serious when I tell people to learn LISP, OCaml, and Haskell. These are the most beautiful languages we have.

Mastery of FP makes you a better programmer and enables you to solve sophisticated problems fast. This is the crux of why they are not wide-spread.

Take two very talented programmers named John and Steve. Both are equal in every regard. Give them a difficult problem to solve. Both of them, being smart, solve it. Great, we have two solutions.

The solutions are going to be incompatible at some level. Each programmer has defined their own reality (which is very easy in a FP) and made their own language to solve it in a concise and precise way. But the realities clash and are ugly and obtuse to each other. In John's reality, the flowers are purple. In Steve's reality, the flowers are red. More importantly, John doesn't want to live in Steve's reality and Steve doesn't want to live in John's reality.

Fundamentally, FPs (which are reserved for smart and clever people) do not scale out in the business setting. The people smart enough to use FPs are tragically cursed to this reality as smart people tend to be like cats and are difficult to herd together.

Techno-socialist vision of the future

2011-02-17T06:16:00.000-08:00

I've developed my own socioeconomic viewpoint when I was homeless reading Atlas Shrugged, and I'd like to share it now.

In 2006, I wish I called it publicly with Peter Schiff, but the world economy is kind of fucked. Not for the vast reasons people speculate on it. It's because of ass-holes like me that automate. I love to build machines and write software to do stuff for me, and I'm not alone.

I'm a capitalist now because capitalism is the fastest way to innovate and optimize the total human system. Fundamentally, we each create a debt when born. We need so much food, so much protection, so much water, etc. Socialism is faulty on this front simply because the technology does not provide for everyone's needs to be covered very well. Over time, as capitalistic societies create technology, we converge to socialism.

We are in that process now.

Right now, technology has made us extremely productive and we are leaving people behind. Unemployment is going to get severely worse because the structure of the work is changing faster than people can adapt.

The society is going to change into morlocks and eloi.

How will this function under a democracy? Well, it will in the worst possible way.

To be continued...

The learning equation

2011-02-12T17:28:00.000-08:00

I view people as fairly complicated machines, and I look for patterns and derive formulas that explain their behavior. I'm very interested in intelligence, learning, and teaching.

I see people's intelligence as a combination of knowledge and ability to learn and process. When I meet people, I'm secretly building an equation of my head of how I see them. IQ always seemed rather stupid, so I write down something different.

I(person) = IQ + L t + A t*t

I is a persons total intelligence and knowledge.
IQ is what a person is born with and accumulates until about the age of 10.
L is the rate at which a person can learn.
A is the rate at which a person can teach themselves to learn something new.

When I classify people in this manner, I get a good feeling of who they are now and where they may go.

There are two things I like about this model. It means the IQ score you got in elementary school is crap, and it explains a lot of things I see in the world.

Let's assume your IQ via our current method is sound at the age of 10. How do we measure L and A?

L can be measured by giving someone a "Learn Ruby in 21 days" and "Structure and Interpretation of Computer Programs" and lock them up in a room for at least two months with a sufficiently hard task and say go. L would be linear to the total time to complete the task.

A on the other hand is rather different. Unless you want to lock people up for a couple of years, then you can't really measure A analytically. Instead, I think you could do a classifier on what they have invented.

I suffer from Not Invented Here syndrome both for hubris and ignorance reasons (mainly ignorance), and I've found that my side projects and inventions help me learn a great deal more faster than anything else. I have found that the people I meet that have side projects tend to reach competency younger in life.

So, for developers, you could measure A by looking at their github profile? I think this is more or less valid until people start putting spam in their github profile. I think A also means the ability to invent and create new knowledge as I think this is a very good way to connect dots in your learning.

Anyway, I'm thinking about this today because I'm writing job descriptions. I'm crafting a job description to hone in on people who have L and A. People that have A are rare, but people with L can be found. It used to be that college implied L, but I think that college is easily gamed these days. It used to be that a Ph.D. implied A, but that is waning too given some of the shitty papers I've read recently.

Wish me luck in changing market forces!

The economy of Free software

2011-02-06T11:27:00.000-08:00

There has been some chatter on free software, and I've had thoughts I'd like to share and discuss these bits of knowledge.

Competing with giants

Due to Microsoft's and Apple's dominance, you can't expect to raise funding to build a competing operating system any time soon. You have a better chance to build a car company than compete with either of those companies. If you want to make an OS now, then you need to pick a niche and build it and hope the niche buys in or you need to give it away for free.

This is the nature of software since software is like cake with multiple layers compounded on each other. Once business gets in bed with a layer, its probably not going to change for a long-long time. So, if you want to compete and compel people to use your cake, then you have to give it away and think long term rather than short term.

Building a business off of support is long-term thinking. Most people think short-term, so we run into "oh, its free and solve my problems today? I'll use it". They don't know better, nor should they know better. If Free Software gets them to the next step, then awesome. Aside: I like this model best because I've never liked taking money from people who are going to fail.

This is how Linux has propagated to dominance in the server sector. Linux didn't have to make a sales effort on managers or marketers. Linux just had to sell geeks that would have to support this stuff in the field and transition their education into Linux. The people who build software on Linux then give back and support it by hiring people to maintain their shit.

It doesn't matter which route you go (Windows, Sun, Linux), the business has to foot the bill to pay someone to run it. Free software enables software via a community rather a corporate interest.

Free software enables communities to compete against companies.

Software as a marketing channel

If you are smart, then your work will speak for itself. Free software is a great way to sell yourself in the geek arena; it's your marketing collateral.

If you make something free and 10 people use it, then great. That's 10 people that know you exist. Now imagine if 1000 people use it? What about a million?

GitHub enables a geek hiring revolution; that's basically my resume now rather than some crufty document. The AppStore/AndroidMarket is the same for product/application developers. Get your product in the hands of people today and start learning how to make their lives better.

Me-too, Me-too

Many companies have "me-too-ism" where they want everything customized under the sun to fit their brand. I think this sucks. However, it creates a lot of IT jobs since that "me-too" usually requires re-inventing a lot of the same wheels.

Open source and Free software enables the IT workers to work collaboratively rather than competitively. While this may limit the "me-too-ism" on the business end, it does enable IT to save a lot of time and be really lazy (which is the point of computing, right?). To the point, there isn't enough IT workers in the world to satisfy the true desire for "Me-too-ism".

If you give software away, then other people will jump on it and support it for their organization and make it work. This will make projects go faster, and it will make everyone happier.

A perfect example of this is Ruby on Rails. If DHH didn't release it, then he would be in a cave somewhere writing stupid code to fix stupid problems rather than talking at conferences. Either that, or he would have abandoned it. By making Rails free, it enabled the community to endorse it, sell it, and support it.

That being said,

You get what you pay for.

If you don't know software, then you are kind of screwed. If you go with non-free software, then you have to hire someone that endorses that model. If you go with free software, then you have to hire someone that endorses that model. My pro tip: hire the best you can and let them pick.

HN as the open door, and why HN is so awesome.

2011-02-06T08:00:00.000-08:00

When I found HN, I was delighted to get an interesting feed of new data. The signal to noise ratio is way better than anything I've ever seen.

For me, HN is the open door. Most of my life is researching in a cave solving problems that no one really has except me. I like my cave. Problematically, as Hamming said, when you have a closed door, then you get more done and can solve more problems, but are you solving the right problems?

Personal Aside:

I spent the better part of a two years writing a programming language in my cave. Some facts.

First, this was really fun.

Second, it went to production and was in commercial web sites (which I'm not supposed to talk about).

Finally, the problem I was trying to solve was "scalability" at the programming language level. Putting scalability and performance problems ahead of real problems is a prime example of why pre-optimization is evil. It never ends, and you can't ship it. This was a difficult lesson to learn (still, pre-optimization is a lot of fun).

I think there is something in the problem I was solving. Namely, I think if you go the static typing route, then you could in theory produce a programming language and a schema language that can solve your infrastructure and data needs given the right platforms (like S3, SimpleDB, riak, redis, etc).

Essentially, I was trying to make a "scalability solver" by looking at a statically typed schema and how that schema was accessed by the programming language.

Well, this is the problem with the closed door and being isolated. I doubt anyone needs a scalability solver. Fortunately, my quest to solving this problem was to find, classify, study all new NoSQL solutions/platforms as well as master relational databases. This is how I discovered CouchDB.

It is an odd thing to admit to yourself that you've failed and wasted a lot of time.

Generally speaking, I've got an open mind when I'm not distracted by geek-crack (writing a programming language is pure geek-crack). So, when I tried CouchDB I realized two things.

I don't need (nor does anyone need) a "scalability solver".

Static typing and schema suck. In general, I like some of the problems that static typing can solve (performance, type safety, contractual programming, proofs), but those are not the majority of programming problems. I think learning static typing is important to be a complete programmer, but I also think mastering dynamic typing is more important to getting things done. I feel the same way about schema, and I'm developing new tools around CouchDB centered on a forensic techniques using map reduce.

Back to HN.

Hacker news is like an open door uniting all organizations small or big of people who are opened minded enough to debate rather than argue an agenda. If you open your mind, then you can maximize the value from the HN community by treating it as an open door.

Namely, you'll get less done, but you can tune what you get done to what is more relevant. At least, in regards to technical relevancy.

Barefoot in a Blizzard; Vibram FiveFingers in winter

2011-02-01T13:28:00.000-08:00

Those that know me know two things:

I'm insane
I wear the same clothes everyday (shorts + a t-shirt).

Just because there is a ridiculous blizzard out doesn't mean I'm going to change anytime soon (definition of insanity?).

Well, I wish I had pictures as my testimony will have to suffice. I'm a barefoot nut and a cold nut. I'm that guy wearing those fancy vibram five finger shoes, and I like to jump in freezing water. Today, I got to do both. Yay!

I was driving to my fiance's work so that I could pick her up since her corporate overlord decided the weather was bad enough to let people go home. Well, I lost control of the car and I went off the curb and I was half way between the street and some grass.

Fortunately, I had the good sense to have gloves in the car.

Unfortunately, my fiance brought the possibility of me being stranded in to the universe and the fucking universe obliged her. Well, ok, let's take a step back. It's basically my fault because I took her to work in a t-shirt and shorts and that prompted her to comment on my lack of survival sense.

So, there was I stuck. Since I slid into position, I had no forward or backwards tracks. I have to admit the actual process of getting stuck was very fun. Actually, it was the best fun I've had all year. That being said, the moment of "oh shit, I'm really stuck" was not exactly fun.

So, I go out in my handy-dandy five finger shoes (which suck against snow for periods of time greater than 27.3 minutes (again, I know these things)), and I start digging with my hands (note to self: get a portable shovel [editors note: how long have you fucking lived in kansas]).

So, after 34 minutes of alternating digging, trying to rock, pathetic attempts of pushing the car; I resign myself to failure and hang my head down low and call my fiance to tell her to get emergency road-side help.

After about 2 minutes of pure shame, I realize "fuck this self-pity bullshit, and I try some more".

I dug more, and I dug more. Here I am, a guy wearing a t-shirt, shorts, and basically barefoot playing in the snow. I have to admit, it was fun.

Final try, I get enough momentum to make serious progress. I go forward and then launch myself until I gain enough control and get off the curb. From here, it was simple patience that got me rolling.

The point of the story: never give up.

Look a quote:

"People think I'm crazy to put myself through such torture, though I would argue otherwise. Somewhere along the line we seem to have confused comfort with happiness. Dostoyevsky had it right: '"Suffering is the sole origin of consciousness.' Never are my senses more engaged than when the pain sets in. There is a magic in misery. Just ask any runner." - Dean Karnazes

Today was fun.

The value of a University Degree in Computing

2011-01-31T13:44:00.000-08:00

I got a piece of paper that says I know what I’m doing in computer science and another piece of paper saying that I know math. Today, I want to talk about the value of those pieces of paper as it stands today.

Ultimately, the value of a university experience is multi-dimensional and varies person to person. I’ll discuss multiple vantage points and weight my thoughts on the value.

Computing Resources

Back in the day, you had to be at university to even get access to big computing resources. Now, with cloud computing, you can get more computing power than what I had access to for less than your lunch. So, don’t go to university if you need awesome computing power; you can pay for it with a job. That being said, some universities have ridiculously cool stuff like 3D scanners. However, there is a hacker right now using a kinect and a Markov process to build a precise 3D scanner using their Lego Mindstorms kit. It may be worth going to university for equipment if the equipment is extremely advanced, but otherwise don’t.

-1 University
+1 Cloud Computing

Exposure to Mathematics/Computing/Material

With the internet, you can get a ton of exposure to mathematics. It’s all there. You don’t need a special library card to get it. That being said, there are some pay-walls for springer/ACM and other places with cutting-edge libraries. It’s worth going to college to be able to SSH in and then wget an article’s text. Otherwise, master Google (or stack overflow).

-0.8 University
+0.2 Cleverness
+1 Internet

Exposure to Breadth

Where do you begin to learn and teach yourself something? How are these seemingly unrelated things connected? University wins big by having a structured series of classes that in theory build on one-another. An ideal university will cover basically everything that is at the foundations and give you a good sense of the field.

However, you can get the same thing by reading their course catalog and degree requirements and having the discipline of going over the material yourself.

+1 University
+1 Discipline

Mathematical/Engineering Mentorship

Having access to math is one thing, but having the decoder ring is another. As a budding graduate student, I found that listening to a lecturer was valuable in not learning the material but rather how to interpret/construct the material. It is one thing to know that something exists or what something is, but it is entirely different thing to understand how something was constructed. The side-notes/annotations a lecturer gives on how to prove a theorem rather than the formula.

The same was true for a handful of computing courses where you got insight into the design process rather than “here are some facts, now go”. Problematically, this is a very hard thing to teach since this kind of knowledge is usually lost over time.

+0.5 University
-0.5 Sucky Professors

Mentorship

This is the big win for university. If you don’t work under a professor as an assistant, then you are really missing out. There is a saying “Those that can, do; those that can’t, teach”. And, it is very true for many instructors, but the other half are off in their zone mastering a part of the cosmos that they own. You can learn a lot from these professors, and you should listen to them and have an open mind.

Most students fuck this bit up and claim professors are out of touch, and they ignore the oddities of their professor rather than studying them. If you study them, then you can learn a great deal about smart people.

+1 University
-1 Closed Minded Students

Peers, Colleagues, and Professors
You write a blog post and no one leaves a comment. Did it get read? Probably not. If you work in a group where people care about their grade, and you control a piece of it; is it going to get read? yes. If you get a paper back with tons of red ink on it, then it got read. If another person wants to put their name on your paper, then it got read and you got feedback.

The nice thing about university is that it provides a feedback mechanism for how you interact with the world. Granted, we are transitioning collectively to a more casual style of writing.

+1 University
+1 Casual Writing Style

Teaching

Where else can you get 30 students handed to you and then have the opportunity to teach them something. I went into teaching, so I could break my fear of public speaking. I don’t have that fear anymore.

When you teach a group of students, you learn a few things about what is and is not effective. Mostly though, you learn you can’t force knowledge onto people. Instead, you can only open the door and let them study you as you walk through it.

+1 University
-1 Students (the poor bastards I iterated on)

“Other Stuff”

Anything outside of your discipline was required to be an “educated” or “civilized” man. I’m referring of course to when women were not allowed in university. Now of days, with the internet; you can get everything you missed in humanities and then some using the internet. Beware of Trolls.

-1 University
+1 Trolls

Campus

Generally speaking, the campus is a dream for extroverts. Introverts are fucked. The nice thing about campus is the ease of access to smart people. You want to bug your professor, go for it. Most of them like it.

Most students fuck this up by not bothering their professors because they are too lazy to get out of bed. I had fairly insane office hours (every day including weekends, 8+ hours) and I welcomed many students. I had a lot of people fail that I never saw. I had 2 people actually take advantage of my hours; I turned one of them from a D student to an A student.

+1 Extroverts
-1 Introverts
-1 Lazy Students
+1 Humble Students

Big Finish

Here is the question: Where do you want to be?

I think university can make it easier to get for sure, and it doesn't have to be an ivy league either. However, most students are going to waste their college career because they just don't know how to extract real value from it. It can be a lot of fun, or it can be a growth period.

Do you need it now of days? I don't think so. All you need is an employer that is willing to push you. Keeping in mind of course, the employer is going to push you in a direction that is relevant to the company. Some companies, like Google, know that if they enable the herd to stray, then you can capitalize on that knowledge and let it integrate into across the entire organization. Most companies are not mature enough yet to take this path.

If you are stuck and the company is pushing you in a direction of a cog, then I recommend getting efficient at that direction and explore side projects on your own to bring up your knowledge and skill set.

As a rule of thumb, I don't recommend letting a company control your career. You should be in control, and you need to realize that you are competing in a global work-force. Technology moves fast, and you need to move faster. A college degree is ok, but university is getting efficient at churning out mediocre developers.

I think the four years at university can be replaced with one year of ultra-hardcore mentor-ship. I'm talking 110+ hour week of hard-core work, education, and side projects. When I figure out how to make it legal, I'll announce my program.

CouchDB rocks, so I'm going to make it relational.

2011-01-29T11:22:00.000-08:00

Here is the plan, and I'm soliciting for help to start planning and building an open source project.

About two years ago, I build my own programming language called Kira to solve "the scalability problem", and I was looking at how a schema combined with how it was used (via my own LINQ) to study patterns. Needless to say, that studying those patterns were fruitful in discovery of the emerging NoSQL solutions. You can model each NoSQL solution using a relational DB, but you either get ease of scale or performance. My interest was starting at a purely relational model and then solving the system using NoSQL systems.

I want to say something: this problem is pure geek crack. Unfortunately, my study was pushing me more and more away from SQL and into NoSQL. Here are the things that I can build anything with:

S3
Redis
CouchDB
node.js (+ otto)

Now, there is a certain elegance to SQL from a support point of view. When you build something from a NoSQL solution, then you are indebted to build a management layer for what-ever you built. You tend to always have to do this, but NoSQL forces you too early on in the development process. While this is a problem for many businesses now, it is opportunity for hackers/geeks to build something awesome like phpMyAdmin and n00bify NoSQL.

Right now, I think CouchDB is ultra awesome and ultra easy. However, it has some business problems that I think are naturally solvable via node.js (due to JavaScript portability).

I'm solving a couple with otto and vacuum now, but those are primarily experimental. I would like to do something a bit more robust and general.

How to be a hacker?

2011-01-22T08:18:00.000-08:00

I get asked "can you teach me to become a hacker?", and I'll be honest, I have no idea. Asking me puts me in an uncomfortable state of cruelty where I say "just got to work hard at it and not be a n00b". This got me thinking about how did I become a hacker?

First, what is a hacker? Well, I'm going to be general and extend the concepts of a hacker beyond computers. I think hacking is the ability to see the system in place and the willingness to exploit the system to make it your bitch. There are many systems in the world, but computers are easier to hack because the time to fail is so quick. Hackers are not confined by the boundaries of computing. I've met people that hack people over the phone (really good salesmen). I've met people that hack the human body (doctors and medical enthusiasts). I've met people that hack the fear of death (preachers).

I've thought about writing a book called "Born to Hack" inspired by "Born to Run" as I believe the two are related. We are here today with all the flashing lights and buttons because we hacked reality and made it our bitch.

We are the people who hack, yet some of us don't. So, I'm going to give three simple steps that someone who doesn't view themselves as a hacker to try.

Pick a medium

I chose computers because they are shiny, and I have no sense of patience. I can't wait a week for cultures to come back, so I would have failed at medicine. The key here is to pick a medium that you can readily manipulate and measure effects.

Work really hard at making the medium your bitch

Here is where you just work at it. If the massive neural network in your head is good enough, then you will make it work. You will understand the system, the rules governing the system, ways to manipulate the system, how to break it, and how to fix it.

Do something new

This is where you must see something as a goal and work towards it. Everyone I've met is a genius in hindsight, and the trick is to realize that everything can be made better since nothing is perfect.

The entire process takes either 10,000 hours/ten years give or take 5,000 hours/five years. The key to being great is waking up everyday and working towards your goal. I code every day. What do you do everyday?

After all, you are what you do.

node.js production environment #63

2011-01-19T17:32:00.001-08:00

I pulled out the "production/" scripts from the WIN and rebuilt it as simplenode. Basically, simplenode is a very simple init.d script and a forever loop that enables

per restart logging (can fill up server)
crash throttling (5 seconds)
restart on reboot

All of this takes place /var/nodejs

Now, I'm thinking about making a patch for node.js makefile such that it will install everything it needs into /var/nodejs so I can chroot it. Has anyone else done this?

Places to improve:

As of now, I'm logging everything so if I dump too much diagnostic information, then I'm going to get screwed later on. This is fine for now, but eventually, I'll need either to delete logs, truncate logs, or export logs to S3... This gives me an interesting idea for a service, ;)

The five seconds is arbitrary, but I need a way to detect if the script is in a "bad place" and the state is corrupt. I've had this problem with using the file-system since a crash botched up the state, and I didn't respond in time and the forever loop was making the entire system crawl.

vacuum.js alpha (slightly inspired by nature/biology based computation)

2011-01-15T13:24:00.000-08:00

I just cut a very rough version of vacuum.js to test it out with a couple of friends.

One thing that I miss from RDBMS land is the ability to fix bad data using command line SQL. For instance, with this code

update places set state=some_complicated_function(state) where state='blah';

I can fix bad data that somehow got into the system. A consequence of moving fast is that you get unexpected data (that may have more value than you think) that can break. Either you plan it all out or you let some criminals in the database to see how the business case works out. Sometime, a lack of planning enables sales to hack things out on their own. ;)

Anyway, back to vacuum and CouchDB. There isn't a way to update the entire database or a subset of the database easily; the easiest option you have is to build a view that brings out bad data and then a cronjob that cleans things up every now an again.

That's vacuum.js alpha.

My example data-set that contains a bunch of US state abbreviations. Except, some are very bad and malformed. In RDBMS, this is fairly easy to fix. With vacuum.js, its even easier to fix.

The current example provides two functions that run in the CouchDB context which marks documents as bad if they don't have a two letter state code (I could verify that it exists); the other function will try its very best to fix it.

The current version is a quick hack job, and I'm sitting down now to flush out a better version. As of now, its good enough to test and apply singular fixes for a couple of my project; however, since it polls, it is going to waste a lot of time on bigger systems. Check out the README to see where my current thoughts are.

As an interesting aside, I think NoSQL and thinking NoSQL design enables nature/biological thinking with MapReduce. I think it is very interesting to think of code working in an ecosystem where you can program receptors and bind actions to them. I may rename vacuum.js to something more in this vein when I'm finished with my plans. Using the same type of system, I could split/merge data with functional programming paradigms. I think this is the key to designing with MapReduce and CouchDB.

This is definitely going into the book I'm writing about CouchDB design.

Barefoot walk in snow (Vikram five finger) testimonial

2011-01-13T22:25:00.000-08:00

http://runkeeper.com/user/mathgladiator/activity/23264937

Oh yeah, 17 degrees out and 80% of the walk was in 1/2 foot of snow.

Do I recommend this to you? No.

"Ok, wait, what is the point of the post?" you ask?

Well, I like to push limits. I've been pushing my body throughout 2010, and I'm continuing in 2011. Cold is a personal study to me. Do you know how long you can last in the snow? In pants? In shorts? Naked? Naked in water? How long until you show signs of frost-bite or hypothermia? These are questions that I have answers too.

PoTATo and PotATO with CouchDB

2011-01-09T08:26:00.000-08:00

As many of you know, I'm having a love affair with CouchDB since it makes typical CRUD aspects of an application a breeze.

Last night, I posted my install on github:

https://github.com/mathgladiator/potato

In a nutshell, this is now how I install CouchDB in the cloud. I'll back-port any security lessons I learn into this script over time. If you agree to use this script, then you agree to the CouchDB agreement.

2nd Law of Code Generation

2011-01-08T14:08:00.000-08:00

I don't know what the first law is, but I'm basing the 2nd law of code generation around the 2nd law of thermodynamics. Basically, when you use a code generator, business value only decreases. That's why you should start at the highest level.

I'm working on some tools that generate code from documentation.

algebraic json differences (change logs, replication, oh my)

2011-01-07T14:32:00.001-08:00

https://github.com/mathgladiator/jsondiff

This is a hacked-out version of an idea I've been playing with.

Since I've adopted CouchDB as my mind set, I would like to make life easier by focusing on JSON at a very general level. There seems to me a potential for some algebraic goodness with JSON. What does this mean? I don't exactly know yet. What I have so far is a non-commutative operator on a set of JSON objects. I've defined addition and subtraction. Multiplication doesn't make any sense, so I'm at the best I've got a group.

Use Case 1: Change Logs

I have an old copy of an object and now I have a new copy; I can use this code to compute a difference that will turn the old copy into the new copy. Ideally, this will enable audit logs as well as some form of revision since I can describe the entire object as a sequence of additions by starting out an empty object.

Interestingly Enough, if you design a special set of objects such that addition is commutative, then you get Google Wave and a nice level of concurrency (using an atomic journal).

Use Case 2: Undo, Redo

It's fairly easy to make a system to undo and redo changes to an object if you can study the changes abstractly from the outside rather than requiring a programmer to make notes of all changes.

Use Case 3: Replication and Optimization (REST/RPC)

If I use CouchDB and then replicate to a node.js server running Otto, then I can store a copy in node.js (or memcache) and then differentiate to produce an object that tells me how best to replicate to foreign services (like an RPC server you don't control, but you need too). Using duck-typing inspection, you can then send only the RPC commands that you need too.

Use Case 4: Algebra is oddly fun

I get joy every time I think "Oh, C++ operator overloading would make look this sooo nice". Unfortunately, I don't work in C++ any more and so I avoid that joy, but I get the joy of being able to ask questions like "Is this a group? Is it associative? Is it commutative? What is zero? What is equality?"; I'll write up my notes when I get time, but for now the code is live on github.

Concluding Thought, Ii think algebra plays a bigger role in computing than I originally thought.

cmdfu, roll your own heroku-like interface

2011-01-04T19:53:00.000-08:00

Every year, I write something up in C to make sure I still have hair on my chest. This year, I wrote cmdfu. cmdfu is a simple tool for executing commands remotely in a secure fashion (using an HMAC-esque signing process).

You can clone it from its github.

I'm going to play to my strengths (i.e. not UI) this year and release some command line services like crondom.com and the yet unnamed node.js hosting service. cmdfu is basically the way I plan to offer up remote execution against my servers. There is preliminary documentation here.

Using bash magic, it is fairly easy to make your own heroku-like interface. For instance, I can make a crondom script to manage my account.

crondom create http://www.mathgladiator.com/cron/test/1minute.php

crondom ping every 5 minutes http://www.mathgladiator.com/ping.php

crondom list

crondom delete 1-341343

You may be asking, "why C?". Why not?

If you want to install, then you need to install libcurl. I should really make a real makefile, but for now I like what I got. If you would like to use it for real, then please leave me a message in my github account (or just fork it).

3 Types of Programmers: Zerg, Terran, Protoss

2010-12-26T22:57:00.000-08:00

There are three types of programmers in this world.

Terran Programmer

Ruggedy, the terran programmer gets shit done and is smart enough to make it work at every level. The code isn't sexy nor elegant, but it gets the job done and works well enough. Their tools are what-ever they can afford.

A terran programmer usually works best in a start-up or as a technical leader. A canonical example of a terran based company is 37 signals.

Zerg Programmer

The company matters most to the zerg programmer. They need their IDE (i.e. creep). Management needs to hire lots of them to ship even the most basic of products, but they can hire hordes to solve problems of scale. They depend on their queen vendor.

A zerg programmer works best as a cog in some corporate machinery, and they tend to use a Microsoft or Oracle products. Most offshore outsourcing company are an example of a zerg company.

Protoss Programmer

Shiny and advanced mathematics is the primary tool for the protoss; this greatly limits their numbers. They use languages like Lisp or ML to develop spectacular results, and they are free to use anything.

The protoss tend to cluster in academia until they have matured to the point where they have the insight that can power a company. For instance, Google's PageRank is a protoss insight that powers Google thus making Google a protoss company.

Moral

If you build a company, then you need to ultimately use people to get things done. You need to find the right people for the right job to get the company as a whole executing.

Each type of programmer has their pros and cons in a business, and the goal is to utilize and structure the company such that everyone works together effectively.

If we ignore (or worse argue) about differences, then we miss out on the potential to work together and build truly great things.

Testing is not a waste of time, I don't know that your code works

2010-12-26T09:55:00.000-08:00

I've been in the land of formal mathematics where all the equations are good and correct.

My venture into computing however is not so pure as I would have liked. I tried to be virtuous in this land by bring the gospel of proof and correctness, but I have failed. I have given into temptation of the wild out laws. I have abandoned static types in favor of dynamic types. I have abandoned super-planning waterfall methodologies to agile methodologies. I have abandoned my safe IDE for the raw ninja powers of nano/gedit. I have abandon formal techniques and am working on developing testing based techniques (relevant to my domains and projects).

Here are my thoughts on testing.

Interaction between Proof and Testing

If you can prove it, then you can test it. You know where it should work and where it shouldn't work. Thinking about testing first will give you hypothesis on what to prove. Once you get it working, do you need to prove it? Usually no; but you should have unit tests that enable you or your team to reasonably do the engineering needed to take it to production.

For instance, if you are writing PHP, then the academic portion of the brain can bang out the parser and interpreter in a couple of hours. You can bang out the proof in a day. The problem is that real engineering has to come in and optimize it for production. This means going down into the bowels of code and optimizing loops, changing data structures, finding code equivalences. Unit tests help engineering efforts undergo regression testing as the entire code base is optimized. As the code grows, the feasibility of a proof diminishes. Ideally thou, the proofs enabled the tests to be constructed such that "If all these tests pass, then this proof reasonable asserts the code is good"

Stupid Shit

When I dropped static typing, we hired a QA guy whose job was to use the product every day. So far, we don't having typing issues. We have buttons that don't work due to typos or overlapping divs. Sometimes, your code can be proved correct if it just works. Usually humans need to test this.

I think I can solve some of the issues with selenium IDE, but the time for me to solve it using that methodology combined with the opportunity cost of doing something else means I should I just hire some scrubs off the street to check buttons and check for other faults as well.

You and I are not in control

Let me introduce you to google maps and how they did versioning. I picked a URL for google maps that represents their bleeding edge version. One day, half of our clients didn't work any more. WTF? They updated to a new version and broke our shit. Fortunately, they also provided an archive of old versions and a little hacking into production server and it was working again. I didn't know this at the time, but fortunately Google knew enough about it and started archiving.

We live in a time of interdependence. I trust you to develop service XYZ, and you trust me to develop ABC. Either you or myself can fuck up, and those bridges need to be tested and measured. For instance, I have a server to proxy geo coding requests. I do this so I can enable workarounds when google fucks up (and they do about 0.01% of time which is why I have a table of 18 addresses that google couldn't geo code correctly).

When google started promoting version 3 of their api, I found out that a quick URL change didn't work for me. My tests however told me where it didn't work and I could treat my tests as a simple todo list and grind through the version change. As long as the tests measured every use case that I needed, then I was fine.

You need to develop tests for things that you don't control. You will have many black boxes and you need to test how you use them and your assumptions.

In summary

No matter how smart you are, I guarantee that you will eventually need testing because testing will find you and the business will depend on it. Refusing to test is just ego because a test will fail and that will compromise the integrity of your ego's self image of you as a perfect and heroic being. Once you accept this, allow your ego to rest and stay hidden during real engineering.

17 Thoughts on Programming

2010-12-19T12:35:00.000-08:00

Your constants are your client’s variables.
All software is layered like cake because no one can commit. Those that can commit, fail.
Program's don't learn. Programmers just learn new tools.
Eventually, your program becomes someone else’s function.
Be one with the machine, and you will be annoyed by your code.
The code you are working now that is special fits within someone else's general framework. In a month, you will have wished you knew about that framework.
If you don't have any loops, then you haven't done anything except play with Legos. Why is it bad to play with legos?
If you could communicate complexity, then it wouldn't be complex.
Velocity induces complexity (either technical or managerial).
Your software will be abused my criminal minds.
One half of a business always builds top down, the other builds bottom up; the people doing it top down will get the credit.
If your crappy code makes it a need to hire ten people, then at least feel good about the economy. Also, be the owner of the vending machine.
It is fun to optimize, but it is hard to evolve; if you evolve, then you grow and find new things to optimize.
Every language emits beauty, and every language emits horror. Choose wisely and cluster people appropriately.
Sometimes, you will solve a real problem; most times, you will solve a problem at someone else's expense.
Software as a Service is an infinite recursive chain of passing the buck. If you accept the buck, then you can keep it.
The person that follows your steps probably has different designs, enable them to rebuild and learn from your work than force them into the same idioms. After all, they have to maintain it.

Defer Deletion, Garbage Collection, and Bulk Undelete (using WIN + CouchDB)

2010-12-18T15:15:00.000-08:00

I really, really, really hate providing the delete function. So, for WIN, I provide a delete that doesn't delete until 31 days have passed. It allows me to sleep at night and dream of unicorns.

I have an updator that will change the name space of the document and adds meta data to the document related to the deletion.

https://github.com/mathgladiator/win/blob/master/lib/win.config.js#L110

Then I provide a function to the environment to easily call the updator that looks and tastes like a delete:

https://github.com/mathgladiator/win/blob/master/lib/win.environment.js#L139

Every day, I have a cron job that looks at this indexer

https://github.com/mathgladiator/win/blob/master/lib/win.config.js#L118

and it kills them one by one. Now, I know that actual deletes will be done in 31 (or so) days.

What if I need to undo? Well, it is very easy to undo one element. If I would like to undo a whole bunch, I have to provide a common key to un-delete from. That's what the action parameter does. If I write a loop that deletes a bunch of stuff, then I need to build a fairly unique key that enables me to undo that batch delete.

Understanding a sea of JSON with Map Reduce

2010-12-16T21:03:00.000-08:00

CouchDB stores a lot of data in a sea of JSON, and it isn't exactly easy to get a good grasp on what there is.

For WIN, I force each object to have a name-space field called 'ns'; this enables me to partition the data and enable developers to partition the data. Ideally, this helps in keeping things separate.

A fundamental problem is that I want to have an idea of what it is in the data set and be able (and enable developers) to write appropriate documentation so everyone stays on the same page. I would also like data to adher to some kind of structural quality. However, it would be nice to be able to look for oddities that could become future support issues (it would also be nice if everyone used the same language and kept things consistent; I would rather nip inconsistencies in the bud earlier rather than later).

So, I flatten the structural qualities of each object and count them using this code (for CouchDB's incremental MapReduce).

http://pygments.org/demo/12753/ (alternative http://pastie.org/1384759 )

This enables me to grep the code base and then use blame to work with the developer to resolve oddities. Or, I can turn a blind eye because it isn't in a table that matters that much (i.e. meta data or user controlled data).

I can monitor this for changes daily to determine what is happening on development (where oddities first get introduced).

This mode of thinking enables me to think about unicorns when it comes to the database (oh, and never allowing anyone to delete; everything goes to trash with an trash_goes_out_on field that is set for 60 days in the future when it will be actually deleted).

Database Development Mistakes as NoSQL propaganda

2010-12-14T09:41:00.000-08:00

Context

http://stackoverflow.com/questions/621884/database-development-mistakes-made-by-application-developers

Summary

Not using appropriate indexes
Not enforcing referential integrity
Using natural rather than surrogate (technical) primary keys
Writing queries that require DISTINCT to work
Favouring aggregation over joins
Not simplifying complex queries through views
Not sanitizing input
Not using prepared statements
Not normalizing enough
Normalizing too much
Using exclusive arcs
Not doing performance analysis on queries at all
Over-reliance on UNION ALL and particularly UNION constructs
Using OR conditions in queries
Not designing their data model to lend itself to high-performing solutions
Selfish database design and usage.
Abusing denormalised data
Scared of writing SQL
Dogmatic 'No Stored Procedures' policies.
Not understanding database design
Not using version control on the database schema
Working directly against a live database
Not reading up and understanding more advanced database concepts (indexes, clustered indexes, constraints, materialized views, etc)
Failing to test for scalability ... test data of only 3 or 4 rows will never give you the real picture of real live performance
They only test on toy databases.
Not using indexes.
Not communicating with experienced DBAs.
Poor Performance Caused by Correlated Subqueries
Forgetting to set up relationships between the tables.
Not using parameterized queries.
Favoring "Elegant" code over highly performing code.
Not doing the correct level of normalization.
You want to make sure that data is not duplicated
Using Excel for storing (huge amounts of) data.
Unnecessarily using a function on a value in a where clause with the result of that index not being used.
Not adding check constraints to ensure the validity of the data.
Adding unnormalized columns to tables out of pure laziness or time pressure.
not so much about the database per se but indeed annoying.
Not taking advantage of CLUSTERED INDEXES
Not using a SERIAL (autonumber) datatype as a PRIMARY KEY
Not UPDATING STATISTICS on a table when many records have been INSERTED or DELETED.

My Thoughts

All of these are consequences of using a one-size fits-all solution for storing your data. Fact is, application developers shouldn't worry about how they use data. They should be able to get their job done without worrying about the long-beard in the back room. I've been in this role, and I can sympathize with it.

Then, I realized something has to change. I took away SQL and built a very simple RESTful layer to the data layer, and then I watched how application developers solved their problems. I was amazed at their cleverness. Instead of saying "oh, these silly application developers are so dumb and don't know shit about databases", I said "I wonder how clever they could be if I just gave them memcached and simple get/put/by_index".

They taught me a thing or to about how awesome memcache can be (especially with cron-jobs).

Ideally, if you are building the data layer, then all you need to do to enable application developers is get the right complexity class out of the data. If you have ten billion things, then you need to provide the functions that get to a thousand things relevant to what the application developer needs to do. For bigger tasks, computations are best represented with MapReduce, and I feel that MapReduce is way easier to learn for fresh application developers. CouchDB's incremental MapReduce is by far the easiest to learn.

That being said, performance is always going to be an issue. If you enable developers this way, then you need to provide a realistic environment.

Have a development server with more data than production and with a slower CPU (if you can't do this, then you the ability to connect to production in a read-only mode).
Force them to profile their code (ab works very well for most situations)
Work with business people to define how consistency should work
Train them to do cache invalidation

In my mind, core to the NoSQL movement is the ability to empower more people with persistence rather than knowing edge cases of how a database works. The sooner you empower people, the sooner the business can iterate and the more time you can dedicate to sitting on a beach thinking about unicorns.

Related entry: Big Data enables Agile Data.

Why I gave up on static types

2010-12-12T21:13:00.000-08:00

I like programming language theory and how to use typing to do some pretty impressive things, but I'm getting older now and I just don't give a shit about types for day to day stuff. I also gave up on object-orientated code. I also said F-U to relational database theory. Why?

Because people using your product don't give a shit about how it gets done. That's the reality. They don't care if you use assembler or JavaScript. They just don't. The question is: can you make people happy. The more important question is: can you sell? can your team sell? can your sales team make compromises to make the sell?

This last question is the question that I ponder about since it affects my profits. Do I want to put up some academic/aesthetic wall in front of a sale? Or, do I want to enable them to make a sell?

This is where all that rigidity breaks down and I ask a new question. Is this methodology or technology better for sales?

Static typing? No.
Object Orientation? No.
Relational Databases? No.

There is a lot of bull-shit technology out there (especially built on .NET or Java) that is simply a wall to sales. Now, it does depends on what you are doing, but ultimately it comes down to sales.

My issue with static types is that I can't add new members at run-time; nor does it propagate. Everything I do now is basically a giant JavaScript object that I pass around with JSON. I don't care what is in it. From a business point of view, I know that if everything in the system doesn't try to map the JSON into a static class, then I keep all the data; it just propagates. This enables me to change elements at the data store like adding a boolean named "my_sales_team_is_awesome_and_sold_a_feature_that_can_be_added_by_a_bool", then I can sleep knowing that the entire system will just deal with it and pass it along. I don't need to deploy a binary nor compile across an entire system to add a little bool.

My issue with object orientated code is that most of my stuff is non-inherited. I have things that can not be objects. While I do use the JavaScript object a bit, I don't use prototypes. I just treat it like a map and move on with my day. I don't give a shit about binding code to data; this is the worst possible thing you can do. I need all my data in a format that it is (a) obvious what it is and (b) easy to transform by looking from the outside. This is my data model guide line; if any idiot can look at the data and know what it means, then it is a good data model.

My issue with databases is the same as static types. I don't want to plan out how my data is going to look. I don't want to think. I want to be agile and just capture data and throw it into the database. I want to capture as much data as possible then organize it later. I don't want to think about normalizing which I can always break (show me your schema, and I will find a feature that will break it). I just want to put my data somewhere safe and have it replicate. This is why I use CouchDB. It's very relaxing.

Looking back at my life, I realize that I was wasting a bunch of time and energy trying to reach a goal with stupid means. My goal was to enable crazy fast development, and I achieved this goal by simply changing my outlook and aesthetics.

Having said that, I realize that there are reasons these things exist. If you need them, then you should use them. I love static types, but only for raw performance. There are performance patterns that can be implemented as a server that are very flexible, and those are important things to learn as they enable you to deploy safe services. The problem thou is always with specifics.

Oh, it also helps to have mastered grep and write code that enables grep to be useful; this is an amazing productivity boosts for when static types are actually very useful.

I haven't completely given up on types, I just now realize that their place is not where I would have liked it. If you look at my github, then you can probably tell where I've been spending my time in terms of type system.

That's right, I'm a node.js junkie. I just spent a weekend cutting a new version of my platform, and I have to say that I get amazing velocity with it. So much so that I can focus on leveling up my design rather than painting yet another bike shed.

WIN is looking good; good enough to start documenting and testing more hard-core

2010-12-11T21:42:00.000-08:00

Well, I put WIN into production. I learned that if you rely on unsupported couchdb code, then strange things happen since there is no debug code. I found a bug in node.js that I need to mock up and send to the node.js team. I learned that I don't like looking at more than 1K code.

I just spent half a day re-factoring and cleaning up win so it makes more sense, and I added crap comments. I also linted to look for stupid issues, so it looks a lot cleaner now.

So, now, I'm going to write the guide ultra-hard-core fashion. I am confident in the patterns that I am going to present, and I'm confident that the system can be hacked to get anything anyone would want.

Why Mustache is for WIN

2010-12-09T20:54:00.000-08:00

Mustache is a logic-less templating language. By being lacking in logic, it easily enables cross-language template interpretation. This is important for two reasons.

It protects work in constructing good DOM. This is true for many template languages, but it makes sure the assets are protected from language change.
By enabling templates to work in multiple languages, you enable it to work it multiple contexts. For instance, if you have a search feature that you would like ajaxified, then you can work towards producing a JSON object. For SEO, you use the template to send off the HTML. For Ajax, you can just get the JSON object and do the JSON to HTML in the browser. Generally, JSON is more efficient to send over the wire when compared to HTML; ergo, you get a snappier response in addition to faster development time (by only writing one template and not worrying about DOM manipulation).

When I approached Mustache, I was hesitant at first since I do not do much front-end engineering. I wondered what it can't do, and I've looked for counter-examples. Simply put, I can do amazing things with mustache. So can you.

The key is to think of Mustache as just a simple HTML encoder over a giant JSON represention of the module, page, layout, etc. You will put in some silly things in the JSON, but in the end it will enable something very powerful in you architect around getting a giant a JSON object back.

Namely, it is very easy to automate testing on giant JSON files. That is, it is easier to script against JSON than junk HTML. For me and WIN, this is a fairly important question as I would like to be able to crawl my entire projects to look for errors.

Say Yes to Internet Censorship

2010-12-08T12:39:00.000-08:00

Why?

Because it will make things worse.

When things are bad, talk begins of revolution.

Viva La Revolución

By the way, this was troll-bait. Just an experiment. Of course, there should be no censorship, but that is obvious to me. Is this not obvious to others???

3 reasons why I don't key off of email anymore.

2010-12-07T12:55:00.000-08:00

For some of my clients, I built their stuff such that a user only needed an email and a password. Registration was easy and it was awesome. Now, I have introduced a login name back in. Here is why.

Emails Change

I had clients that lost jobs and they needed to change their email; well, that required writing a change email function. That's not pleasant because emails may already exist due to a prior sign up or a different use case.

People get fired, two employees at a company. One used their personal to sign up to product where as the other used their business. The one who used their personal got fired and the needs to transfer access to the other employee, but its already taken. So, either I have account merger or they manage multiple credentials. Never the less, they have to call in for support if we present an obstacle. Additionally, companies get bought and emails change.

By adding the level of indirection, I'm enabling them to handle these issues themselves rather than supporting it on our end.

Multiple Accounts per Email

If you enable a single email to manage multiple accounts, then you help them out companies that have different billable uses of your product. Otherwise, you require them to be able to setup multiple emails which just sucks.

Multiple Managers/Owners

If you focus on providing a single account, then you can enable your product to be managed by multiple people (or enable collaborative features). It is easy to key off of the account's login name to enable multiple users to access the account.

Entrepreneurial Enlightenment and Insight

2010-12-05T09:42:00.000-08:00

“Ideas are a dime a dozen” is a very stupid saying just like “work smarter than harder”.

Many of my math professors said that you know more or less all the math you are going to need to know, but you need to be able to communicate it (that is, you need to be able to communicate math). That’s the true goal of the master’s degree program.

Understanding “Ideas are a dime a dozen” is the same thing that many entrepreneurs know but can’t express in an effective manner. Communicating is harder than knowing.

Here is how you get it to knowing that “Ideas are a dime a dozen” (or at least, how I’m trying to try to sell it to you):

Sit in a business.

Watch the people.

Be creative on ways to make them awesome. What would make the business better? What could you sell them to make them better?

Do this every day, and you will have lots of ideas.

Once you have an idea, you need to be able to execute on it. Execution is the art of getting things done and progressing the state of a business from conception to cash flow.

I have a torrent of ideas, so I’m set for life in the idea category. Now, how can I execute. Here is a generic three step business plan.

Build it (Engineering)
Find Customers (Marketing)
Sell it (Sales)

Once you see the pipeline, everything in the world starts to make a lot of sense. Everyone has a happiness function, H, and a life-is-shit function, S, that they use to make decisions.

Your goal is to maximize H-S.

If the market is small, then you can be a one super awesome consultant.

If the market is big and mundane, then you can build a company.

If the market is big and complex, then you can build a firm.

Once you have ideas, how do you execute in #1, #2, #3? How do you build it? Do you know someone that just builds things? How do you find customers? Do you know someone that can network or go door to door? How do you sell it? Do you know someone that can sell ice to an eskimo?

Once you can answer these questions, you can build a business. However, you must keep in mind that the people in the business are the only true asset it has. Do you have the right people doing the right jobs that they want to do? That last bit is basically my digested form of Execution: The Discipline of Getting Things Done (which is a good book, but you have to put yourself in their shoes to understand what they are saying. It’s not an easy book to read.)

Once you start to have 10+ ideas, you can either just blindly following them (I'm very guilty of this in my life as a coder), or you measure them and pick the best one. What is the best one? After-all you have your own H-S function. It could be by impact on the world, revenue, profit, job creation, or just plain fun.

I hope that I have communicated how to gain entrepreneurial insight into how to manufacture ideas. This is why I write my blog, so I can level up my communication capabilities. If you find yourself like me, knowing things but feeling an inability to express them, then you need to start writing now.

The problem only the best programmers can solve: trust

2010-12-02T20:43:00.000-08:00

I just watched DHH's key note at the Ruby X Conf, and I must admit that DHH's concept of freedom was inspiring.

After thinking about it more, I know why. Programmers tend to be control freaks. I know I'm a control freak, and I'm slowly giving up control so I can get away from the computer and get outside more.

The central problem of team programming is trust.

Static typing tells me that I'm not trustful enough to keep to my own convention and keep my shit straight. Having worked in JavaScript for so long now, I don't even think about types. I like the benefits that static typing can provide (performance), but for day to day stuff, I don't care nor do I really think about it. If it does what I want, then I'm done.

Monkey patching is very interesting, and the same capabilities is present in JavaScript. I find it useful. For instance, in WIN, I needed a trim function for strings. Why doesn't JavaScript have a .trim function? I don't know, but I can extend it's prototype. I find this is very convenient. I can also plant bombs in the string prototype or the object prototype, but I don't want to do this. I need write to tests to test the basic assumptions about the code that I'm using.

When I defended lock based SCM, I was basically saying "I don't trust developers to work together". Now, we use mercurial and I don't worry about it. If developers have a conflict, then it is their responsibility to fix it, and it is management's goal to manage assignments such that conflict is rare.

That's the mentality that I've had to develop when switching from academia programming versus industry programming. When someone else breaks something, they have to fix it. Fucked up a type? You fix it. Broke the string prototype? You fix it. Got a conflict? You fix it. You did something stupid? Ok, we are human, now go fix it.

When people have the responsibility to fix things, the quality get better organically, and best of all. I'm usually left out of the picture for most of it.

Oh? you would like root password. Sure, that's fine. We have a root password ceremony (it has hooded capes and everything draconian with candles), but I trust them enough. If a developer fucks up prod, then they fix it.

The key to enabling DHH's freedom is empowering trust. The key to empowering trust is knowing how to protect liabilities. If you are able to take backups everyday, then you should. We do. You should also test the restore every week. We do. If you are unable to take backups, then you need some form of revision control and never ever delete anything.

When you are programming, the biggest liability you have is how you persist the state of the business. The next liability is how much you annoy your customers (i.e. infinite loop of sending emails = very bad). Once you figure out how to protect the company's ass and enable developer freedom, then you are golden.

I think that DHH's concept of freedom is an ultimate goal for the next ten years for both programmers and service providers in many industries. Some industries however, are always going to be control festivals simply because that is how that market works. I would not want a airplane control system written in ruby. I would rather it be done in OCaml with the most insane type system ever. Fortunately for the majority of programmers, these examples are in the minority. If you are in that minority, then you know enough about programming languages to build your own prison to protect the business.

I think DHH's sentiment on programming languages extends to databases, and this is why I work with and promote CouchDB. It just makes me happy.

The influence of advanced mathematics on programming.

2010-12-02T02:45:00.000-08:00

I'm selling many of my books on amazon, and as I was going through the books I realized that most of it was useful, but only useful an indirect way. I would to share some thoughts on how my studies in graduate level mathematics influences by day to day operations of building products, managing databases, and doing everything a free electron can do in a given day at a start-ups.

Topology

I think the best introductory book to topology is "Introduction to Topology by Crump W. Baker". Topology is basically the study of connectedness and surfaces. When studying topology, you think about how are things different. Are a donut and coffee cup the same? Well. Yes they are once you define what "same" means. There are practical programming challenges in topology in how once can process and do feature selection in computer vision. But, there are more mundane ways of applying topology.

For instance, a relational database is a topology in a discrete graph sense. How does this help me? Well, I'm about to do some stupid DELETEs and UPDATEs on a very large data set. Is the data set before and after the same in regards to current business value? Did I botch up? Topology comes in with the idea of topological invariants. A topological invariant is a quality that can be measured and is invariant under any continuous transformation (isomorphism).

If I were to write a query that measures the business value of the database (say, by the sum of the transactions, sum of paying accounts, and so forth), then I can use these to get a good sense of whether or not I botched up my changes by measuring before and after.

Algebra

If you take a bunch of things and make those things operate on each other, then you have an algebra. There are a lot of properties involved in what the operation implies, and the first year is basically dedicated to defining all those properties and understanding their significance. The ultimate results you typically end up looking at in a first or second year course are the unsolvable theorems (i.e. Doubling a cube).

My most immediate thought on how any of this has any practical bearing on programming is MapReduce. Algebra, in my mind, plays a huge role in how to think about designing algorithms in a MapReduce environment. Namely, the reduce phase where you think about merging. Given two or more documents, how do you reduce them to one? The algebraic properties are things that one must consider (and you may get them for free).

Analysis

This is my favorite branch of mathematics because it is the puzzle of inserting zeros and bounding values. I recommend anyone to check out The Cauchy-Schwarz Master Class: An Introduction to the Art of Mathematical Inequalities. It is an amazing little book that I go through every year to make sure I'm still smart enough to call myself a Mathematician. The most obvious application of this art is numerical analysis. However, most of the time, I don't need to do any numerical analysis since I work primarily on search problems these days.

Unfortunately, most people get the short end of the stick when they study calculus and get a very boiler plate version of the Calculus. I recommend Differential and Integral Calculus.

I take analysis concepts outside of code and into management. For instance, how can I measure the code and enforce a code quality metric to prevent SQL injection hacks? How can I enable developers to converge to a right answer under QA? What does QA need to do?

Proofs

I must admit that I was a stickler when it came to proofs since writing a proof is just as much fun as programming is to me. When anyone writes a program, they are writing a constructive proof that something exists. This begs the question of whether or not that something is what you want.

Does your program need a proof? Two years ago, I would have sad "absolutely". Now, I don't think so because proofs are kind of useless. The problem is that I have to understand enough about the formalism for the proof to make any sense. Well, the source code is a formalism of its own; in fact, a very precise formalism. The proof is already written.

Are proofs useless? I think going through the years of writing proofs has helped me write very good tests. I can look at the code, and know where the problem spots are going to be. Those trouble spots are going to need tests to ensure they work as expected. For instance, reliance on third party services always requires some kind of tests to ensure that updates are working as expected. Things I don't control are things I don't have a chance at proving, so I need tests that are automated and tested daily.

Problem Solving

I think the study of mathematics is probably the fastest way to build problem solving skills since you constantly fail and each failure costs nothing, but the caveat is you may not be solving practical problems. However, building it up as a skill enables you to be more effective at being a programmer.

Do you need advanced math?

Not really. Most math is basically a form of mental masturbation and building the mental discipline/stamina to sit down and think very hard. I think it makes you better in some ways, but there is an opportunity cost. It all depends on what you want to do. If you want to ship products, then you are probably fine to avoid it. If you want to make awesome libraries and sell them to product people, then you probable need some advanced math.

Programmer Legs (And a potential patch/cure for restless leg syndrome)

2010-11-26T21:20:00.000-08:00

When I went to college, there was this room in Nichols that was primarily dedicated to computer science courses. During many (if not all) of the classes, the room would shake a bit. It was kind of annoying, but it was due to the fact that almost every person was shaking their leg.

This is what we called "Programmer Legs". We attributed it to the fact that we didn't get out much, and were mostly pale shadowy figures who were deprived of Vitamin D.

Fast forward 6 years, and I have restless legs. For those that don't know, restless legs is very unpleasant. Imagine you are laying down, and your leg is compelled to move. If you don't move your leg, then you will start to feel impending doom. You will wonder many things like "omg, do I have a blood clot?" or "holy crap, I'm going to die". So, you get up and wander around and you get a drink from the fridge. No problem until you lay down again and the legs get angry.

Well, two days without sleep and I was off to the doctor. They gave me some Naproxen Sodium (pain killer) and Carisoprodol (muscle relaxant). This fixes the problem by putting you to sleep, but it doesn't really help. After you use the prescriptions up, you may have a week until your legs get restless again.

Unfortunately, my muffler broke. This unfortunate event forced me to drive my car to Goodyear to get it fixed, and I walked home.

That night, I slept without problem even thou the prior night I had a re-occurrence of restless leg. The next day, I walked again to pick up my car. Again, that night, no problem. The next day, I basically worked all day and that final evening I had minor issues.

Eventually, I realized a correlation and built a hypothesis. Walking is good you.

If I walk a mile, then I can go to sleep with minor annoyances.
If I walk two miles, then I can go to sleep with no problems.
If I walk five miles, then I get a whole week of no problems.
When I walked ten miles, I got two weeks of no problems.

Now, I try to walk everyday. If I can't go to sleep, then I go for a quick walk (about 1.5 miles) and get to sleep just fine.

While I'm annoyed that I body has decided to jack me and force me to exercise, I'm finding the liberation of being able to walk for hours very... enjoyable. So much so that I'm planning to walk 50+ miles sometime next year over the course of three days.

Big Data Enables Agile Data

2010-11-26T12:36:00.001-08:00

The funny thing about NoSQL is that it is being solved and addressed by the Big Data and Scalability communities where there are legitimate problems of scale that are very difficult yet it enables Agile Data.

Here, I define Agile Data as:

the ability to record all available data at the point of a transaction/form/user interaction (including a context)
organize data after data is available

It should be clear from this description that an RDBMS is not Agile in this sense as it requires me to organize data before I collect it. Sure, there is a way to achieve the above with a RDBMS and you could develop a methodology or an engine to accomplish it, but that violates the spirit of an RDBMS since I would just be packing JSON objects into a row.

The perfect example of a Big/Agile data problem is that of analytics. I would like to record as much information is available (the http headers, the client data, maybe some page content, etc). Sure, I could build a structure/schema to try to solve the problem that I think will be valid, but then I'm potentially reducing the amount of information I'm gathering. Instead, I take the mantra of "gather everything", I finish the collection faster and can start studying the data to look for interesting patterns.

The really neat thing about having Big Data at my disposal is that Agile data introduces Big Data problems since storage requirements grow a lot faster than with a typical solution.

The Secret of Innovation

2010-11-24T16:47:00.000-08:00

There is a phrase that has always bugged me.

"Work Smarter, not Harder"

The reason it bugs me is because every time I say it, I feel like a douche bag. The reason I feel this way because I'm telling people they are doing something dumb and many people associate doing things in a dumb way as an indicator of their intelligence. This is why I don't say this phrase anymore as it isn't helpful. It sounds nice to be able to say because it transitivity says "look, I'm smart, and you could work less if you were as smart as me". That's a douche-baggy thing to say.

Now, having said that. If you can achieve the goal of working smarter with less effort, then you have innovated. This is the secret to innovation.

Before I leave for the day, I think the following thoughts.

How can I do the work I did today in half the time?
What did I learn?
How could I have done better?
How would I explain what I did?

That is, I focus on how I can improve my methodology, my education, my quality, or my communication.

Why you don't need JOINs (and the RDBMS to do them)

2010-11-24T14:23:00.001-08:00

Before I sit down and design a back-end for a project, I write the ideal API specification that would enable developers to be happy and enable them to provide all the polish and sizzle to sell the product. Then I turn the spec into a RESTful service where I don't worry about complexity nor scale. I let the developers work with it and I collect data on how it is used, so I know where the crap is; there is always crap to deal with.

This process has worked out very well for me and my clients, and it is working out very well for the development community in general. We are in the SaaS phase where we produce and consume each other's services. This is nice.

Sometimes we consume two such services and then make something new. This is called a "mashup". Well, guess what? A "mashup" is equivalent to an application level JOIN. This used to be a service provided by the relational database, but now people are doing it by hand.

Better yet, people are used to it and they are kinda fine with it. This is a good thing as it enables service developers to focus on their services and let product developers focus on their product. Developers are learning to

Optimize, cache, pre-compute back-end requests
Write loops to efficiently cross reference code
Avoid the angry looking DBA

There are systems, like memcached, that exist in such a way that they enable product developers to solve product problems without the need to make the DBA change anything. Once developers are empowered, they can use their own creativity and intelligence to polish the product.

We may be looking at the last decade that DBAs are ever bottlenecks. Does this mean that DBAs are obsolete? No, it is a lateral move for them as they become service programmers with the role to produce optimized services. Can they use a RDBMS for it? Sure, they can eat what-ever dog food they want.

Having been on both sides of the fence as a DBA and a product developer, I am comfortable saying that NoSQL movement is definitely going to take hold in ways that people don't expect. I think it is going to restructure the entire way corporations view IT.

Avoid DRY for Product Development

2010-11-21T11:08:00.000-08:00

This is part of my comment on HN.

At the right level of abstraction, DRY is the best advice possible. If you are building a data layer or a system, then you will be best off if you keep things DRY. However, once you get into product land. Then, I advise people to just get the job done quickly rather than worry about engineering principles.

The reason is very clear, in product land, engineering principles are third to usability and marketing. Avoiding DRY enables two things.

Polish
If you have five sections of code that are the same now, then there is good chance that they will diverge as the product matures. This is polish, and it is a good thing. While it is true that you have more work to do, it isn't rocket science. Trying to use DRY for polish is going to create even more cumbersome code with a lot of branches for all the special cases, and it tries to create an artificial problem of intelligence. Please avoid.

Hiring Scrubs
If you realize that the polish needed to make a product isn't exactly rocket science, then you can hire many scrubs. I define a scrub as someone new to computing, but capable enough to work within a developer environment to find and polish simple code. I like to hire scrubs as it provides a great first job in programming for many people. I was a scrub once, and it wasn't that bad as a 16 year-old.

Once you enable these two things, you have enabled marketing and the usability folks to iterate.

Update: Don't just hire scrubs
It takes a balancing act to get products out using a combination of awesome engineers, "good enough" engineers, and scrubs. A company ultimately needs all tiers to be able to push and iterate products quickly, and it is the company's job to ensure that the people process can nurture engineers from scrub to awesomeness.

Escaping Mr. 20% and losing the ego

2010-11-17T19:02:00.000-08:00

This is a fairly long response to a comment on HN.

I've written many lines of code and thought many ideas in my life, but I have not shipped nearly as many. Why? I love coding for coding's sake more than shipping. Shipping requires product development, non-longbeard design, marketing, sales, and more important: customers.

Except for the game engine, everything I did was the special 20%. All the 20% projects were just a neat little idea implemented as if the core tech behind a masters thesis or a senior thesis. However, they didn't have the polish to become a product, and I get easily distracted by more difficult and more sexy ideas. At best, most of them make me look like Captain Hindsight.

I'm perfect for academics, but I found myself being too impractical for my students. This was one of reasons I dropped out of graduate school as I had become an irrelevant preacher. I did not practice what they were going to do, so how could I teach them? It felt dirty teaching them fairly useless things. I was working with tools (such as OCaml and LISP) that enabled me to program better, but programming better doesn't mean shipping better. Most programmers just need to learn how to solve basic problems and have the discipline needed to ship products; telling them that they can avoid memory leaks by using OCaml is like telling them to learn french to avoid pissing off Mexican drug lords. Instead, they just need to get in the discipline of being careful with alloc and free. Any aspect of a technical decision could be looked at as either a technological or a management+discipline issue.

So, looking back on the pile of code I've written. They are all shit because they didn't ship. Shitty code that ships is better than perfect code that doesn't. This is one of the lessons in life that make people like me depressed, but I've gotten over that. I've helped shipped three products and working on two more. One of them is my November start-up sprint (which I'm doing OK on, but I could be doing better).

Problematically, building many 20% things builds an ego. Ego is fascinating. On one hand, you need Ego (or to utilize a customer's ego) to sell a product. On the other hand, you need to lose your ego to build, ship, and support a product. This is why you need to have someone else to do your sales for you. When you get some one to sell you, then you don't need ego anymore. You just have a bunch of work to do.

How to use CouchDB? like this

2010-11-15T16:25:00.001-08:00

CouchDB is a very interesting persistence package, and it solves 90% of the problems you find when you build a back-end for a web application. The 90% that CouchDB includes get/put/b-tree indexing/reliability; all this is good standard-stuff in the database world. I want to talk about the other 10% since crud is boring.

The last 10% is usually something like search; it is the novel algorithm that takes all your data and provides it in a meaningful way that makes your product awesome. CouchDB rarely solves this (neither do other packages). The more special the algorithm is, the more painful it will be to try to solve with CouchDB's MapReduce framework alone.

Fortunately, CouchDB has replication built in. I use the replication to push data from CouchDB to a custom server where I aggregate it into a meaningful service. The library is called Otto (short for ottoman).

The biggest problem you are going to have is what happens when your custom server crashes?

This can be solved by

providing your own persistence, and deal with reliability
not worrying about it and launch 3 servers with a custom HTTP server that replicates three ways; spend more money.
don't care and re-replicate the entire data set, and have potentially non-trivial down-time.

All three of these options suck at some level, but 2 is where you will want to go. In the beginning thou, 3 is the best choice.

From a complexity standpoint, you can make your life easier by enabling your custom software to merge in bulk sets. This enables you to lazily run your algorithm as you collect a lot of data at once. I have found that this style of bulk insertions makes the third option feasible in many domains.

The nice thing about using CouchDB is that you don't need a schema. You don't need to "plan". If you need store data, then you just store it. Just give it a namespace and insert.

Please comment if you think I should write a book on CouchDB? I've done relational databases for years, and I've lurked in the CouchDB for a while. I'm currently building a web framework around node.js and CouchDB called WIN.

Cloud Coding

2010-11-15T14:11:00.000-08:00

I just want to say that Twilio is an awesome company. About a year ago, I was seriously considering dealing with asterisk. Asterisk is perfectly good software, but it didn't make sense for the problem I had. I took a risk and quickly developed the VOIP feature for a product.

Developing and supporting a product with Twilio over the past year has made me think about the need for developing in the cloud. I'm at the base level of cloud development: terminal. I can log into a VM and get things done. I don't need scp, sshfs, or anything else to support my development capability. My brain, fingers, and ssh are all that I need.

Why do I need a VM to develop with twilio? Well, the monitor and way to debug a twilio application is to use the phone. The phone isn't easy to work with, so I need a public IP. I also need a way to watch the HTTP traffic, so I need the ability to grep logs.

Basically, fundamental Unix skills let you work with new technologies faster.