There are three types of programmers in this world.
Terran Programmer
Ruggedy, the terran programmer gets shit done and is smart enough to make it work at every level. The code isn't sexy nor elegant, but it gets the job done and works well enough. Their tools are what-ever they can afford.
A terran programmer usually works best in a start-up or as a technical leader. A canonical example of a terran based company is 37 signals.
Zerg Programmer
The company matters most to the zerg programmer. They need their IDE (i.e. creep). Management needs to hire lots of them to ship even the most basic of products, but they can hire hordes to solve problems of scale. They depend on their queen vendor.
A zerg programmer works best as a cog in some corporate machinery, and they tend to use a Microsoft or Oracle products. Most offshore outsourcing company are an example of a zerg company.
Protoss Programmer
Shiny and advanced mathematics is the primary tool for the protoss; this greatly limits their numbers. They use languages like Lisp or ML to develop spectacular results, and they are free to use anything.
The protoss tend to cluster in academia until they have matured to the point where they have the insight that can power a company. For instance, Google's PageRank is a protoss insight that powers Google thus making Google a protoss company.
Moral
If you build a company, then you need to ultimately use people to get things done. You need to find the right people for the right job to get the company as a whole executing.
Each type of programmer has their pros and cons in a business, and the goal is to utilize and structure the company such that everyone works together effectively.
If we ignore (or worse argue) about differences, then we miss out on the potential to work together and build truly great things.
Sunday, December 26, 2010
Testing is not a waste of time, I don't know that your code works
I've been in the land of formal mathematics where all the equations are good and correct.
My venture into computing however is not so pure as I would have liked. I tried to be virtuous in this land by bring the gospel of proof and correctness, but I have failed. I have given into temptation of the wild out laws. I have abandoned static types in favor of dynamic types. I have abandoned super-planning waterfall methodologies to agile methodologies. I have abandoned my safe IDE for the raw ninja powers of nano/gedit. I have abandon formal techniques and am working on developing testing based techniques (relevant to my domains and projects).
Here are my thoughts on testing.
Interaction between Proof and Testing
If you can prove it, then you can test it. You know where it should work and where it shouldn't work. Thinking about testing first will give you hypothesis on what to prove. Once you get it working, do you need to prove it? Usually no; but you should have unit tests that enable you or your team to reasonably do the engineering needed to take it to production.
For instance, if you are writing PHP, then the academic portion of the brain can bang out the parser and interpreter in a couple of hours. You can bang out the proof in a day. The problem is that real engineering has to come in and optimize it for production. This means going down into the bowels of code and optimizing loops, changing data structures, finding code equivalences. Unit tests help engineering efforts undergo regression testing as the entire code base is optimized. As the code grows, the feasibility of a proof diminishes. Ideally thou, the proofs enabled the tests to be constructed such that "If all these tests pass, then this proof reasonable asserts the code is good"
Stupid Shit
When I dropped static typing, we hired a QA guy whose job was to use the product every day. So far, we don't having typing issues. We have buttons that don't work due to typos or overlapping divs. Sometimes, your code can be proved correct if it just works. Usually humans need to test this.
I think I can solve some of the issues with selenium IDE, but the time for me to solve it using that methodology combined with the opportunity cost of doing something else means I should I just hire some scrubs off the street to check buttons and check for other faults as well.
You and I are not in control
Let me introduce you to google maps and how they did versioning. I picked a URL for google maps that represents their bleeding edge version. One day, half of our clients didn't work any more. WTF? They updated to a new version and broke our shit. Fortunately, they also provided an archive of old versions and a little hacking into production server and it was working again. I didn't know this at the time, but fortunately Google knew enough about it and started archiving.
We live in a time of interdependence. I trust you to develop service XYZ, and you trust me to develop ABC. Either you or myself can fuck up, and those bridges need to be tested and measured. For instance, I have a server to proxy geo coding requests. I do this so I can enable workarounds when google fucks up (and they do about 0.01% of time which is why I have a table of 18 addresses that google couldn't geo code correctly).
When google started promoting version 3 of their api, I found out that a quick URL change didn't work for me. My tests however told me where it didn't work and I could treat my tests as a simple todo list and grind through the version change. As long as the tests measured every use case that I needed, then I was fine.
You need to develop tests for things that you don't control. You will have many black boxes and you need to test how you use them and your assumptions.
In summary
No matter how smart you are, I guarantee that you will eventually need testing because testing will find you and the business will depend on it. Refusing to test is just ego because a test will fail and that will compromise the integrity of your ego's self image of you as a perfect and heroic being. Once you accept this, allow your ego to rest and stay hidden during real engineering.
My venture into computing however is not so pure as I would have liked. I tried to be virtuous in this land by bring the gospel of proof and correctness, but I have failed. I have given into temptation of the wild out laws. I have abandoned static types in favor of dynamic types. I have abandoned super-planning waterfall methodologies to agile methodologies. I have abandoned my safe IDE for the raw ninja powers of nano/gedit. I have abandon formal techniques and am working on developing testing based techniques (relevant to my domains and projects).
Here are my thoughts on testing.
Interaction between Proof and Testing
If you can prove it, then you can test it. You know where it should work and where it shouldn't work. Thinking about testing first will give you hypothesis on what to prove. Once you get it working, do you need to prove it? Usually no; but you should have unit tests that enable you or your team to reasonably do the engineering needed to take it to production.
For instance, if you are writing PHP, then the academic portion of the brain can bang out the parser and interpreter in a couple of hours. You can bang out the proof in a day. The problem is that real engineering has to come in and optimize it for production. This means going down into the bowels of code and optimizing loops, changing data structures, finding code equivalences. Unit tests help engineering efforts undergo regression testing as the entire code base is optimized. As the code grows, the feasibility of a proof diminishes. Ideally thou, the proofs enabled the tests to be constructed such that "If all these tests pass, then this proof reasonable asserts the code is good"
Stupid Shit
When I dropped static typing, we hired a QA guy whose job was to use the product every day. So far, we don't having typing issues. We have buttons that don't work due to typos or overlapping divs. Sometimes, your code can be proved correct if it just works. Usually humans need to test this.
I think I can solve some of the issues with selenium IDE, but the time for me to solve it using that methodology combined with the opportunity cost of doing something else means I should I just hire some scrubs off the street to check buttons and check for other faults as well.
You and I are not in control
Let me introduce you to google maps and how they did versioning. I picked a URL for google maps that represents their bleeding edge version. One day, half of our clients didn't work any more. WTF? They updated to a new version and broke our shit. Fortunately, they also provided an archive of old versions and a little hacking into production server and it was working again. I didn't know this at the time, but fortunately Google knew enough about it and started archiving.
We live in a time of interdependence. I trust you to develop service XYZ, and you trust me to develop ABC. Either you or myself can fuck up, and those bridges need to be tested and measured. For instance, I have a server to proxy geo coding requests. I do this so I can enable workarounds when google fucks up (and they do about 0.01% of time which is why I have a table of 18 addresses that google couldn't geo code correctly).
When google started promoting version 3 of their api, I found out that a quick URL change didn't work for me. My tests however told me where it didn't work and I could treat my tests as a simple todo list and grind through the version change. As long as the tests measured every use case that I needed, then I was fine.
You need to develop tests for things that you don't control. You will have many black boxes and you need to test how you use them and your assumptions.
In summary
No matter how smart you are, I guarantee that you will eventually need testing because testing will find you and the business will depend on it. Refusing to test is just ego because a test will fail and that will compromise the integrity of your ego's self image of you as a perfect and heroic being. Once you accept this, allow your ego to rest and stay hidden during real engineering.
Sunday, December 19, 2010
17 Thoughts on Programming
- Your constants are your client’s variables.
- All software is layered like cake because no one can commit. Those that can commit, fail.
- Program's don't learn. Programmers just learn new tools.
- Eventually, your program becomes someone else’s function.
- Be one with the machine, and you will be annoyed by your code.
- The code you are working now that is special fits within someone else's general framework. In a month, you will have wished you knew about that framework.
- If you don't have any loops, then you haven't done anything except play with Legos. Why is it bad to play with legos?
- If you could communicate complexity, then it wouldn't be complex.
- Velocity induces complexity (either technical or managerial).
- Your software will be abused my criminal minds.
- One half of a business always builds top down, the other builds bottom up; the people doing it top down will get the credit.
- If your crappy code makes it a need to hire ten people, then at least feel good about the economy. Also, be the owner of the vending machine.
- It is fun to optimize, but it is hard to evolve; if you evolve, then you grow and find new things to optimize.
- Every language emits beauty, and every language emits horror. Choose wisely and cluster people appropriately.
- Sometimes, you will solve a real problem; most times, you will solve a problem at someone else's expense.
- Software as a Service is an infinite recursive chain of passing the buck. If you accept the buck, then you can keep it.
- The person that follows your steps probably has different designs, enable them to rebuild and learn from your work than force them into the same idioms. After all, they have to maintain it.
Saturday, December 18, 2010
Defer Deletion, Garbage Collection, and Bulk Undelete (using WIN + CouchDB)
I really, really, really hate providing the delete function. So, for WIN, I provide a delete that doesn't delete until 31 days have passed. It allows me to sleep at night and dream of unicorns.
I have an updator that will change the name space of the document and adds meta data to the document related to the deletion.
https://github.com/mathgladiator/win/blob/master/lib/win.config.js#L110
Then I provide a function to the environment to easily call the updator that looks and tastes like a delete:
https://github.com/mathgladiator/win/blob/master/lib/win.environment.js#L139
Every day, I have a cron job that looks at this indexer
https://github.com/mathgladiator/win/blob/master/lib/win.config.js#L118
and it kills them one by one. Now, I know that actual deletes will be done in 31 (or so) days.
What if I need to undo? Well, it is very easy to undo one element. If I would like to undo a whole bunch, I have to provide a common key to un-delete from. That's what the action parameter does. If I write a loop that deletes a bunch of stuff, then I need to build a fairly unique key that enables me to undo that batch delete.
I have an updator that will change the name space of the document and adds meta data to the document related to the deletion.
https://github.com/mathgladiator/win/blob/master/lib/win.config.js#L110
Then I provide a function to the environment to easily call the updator that looks and tastes like a delete:
https://github.com/mathgladiator/win/blob/master/lib/win.environment.js#L139
Every day, I have a cron job that looks at this indexer
https://github.com/mathgladiator/win/blob/master/lib/win.config.js#L118
and it kills them one by one. Now, I know that actual deletes will be done in 31 (or so) days.
What if I need to undo? Well, it is very easy to undo one element. If I would like to undo a whole bunch, I have to provide a common key to un-delete from. That's what the action parameter does. If I write a loop that deletes a bunch of stuff, then I need to build a fairly unique key that enables me to undo that batch delete.
Thursday, December 16, 2010
Understanding a sea of JSON with Map Reduce
CouchDB stores a lot of data in a sea of JSON, and it isn't exactly easy to get a good grasp on what there is.
For WIN, I force each object to have a name-space field called 'ns'; this enables me to partition the data and enable developers to partition the data. Ideally, this helps in keeping things separate.
A fundamental problem is that I want to have an idea of what it is in the data set and be able (and enable developers) to write appropriate documentation so everyone stays on the same page. I would also like data to adher to some kind of structural quality. However, it would be nice to be able to look for oddities that could become future support issues (it would also be nice if everyone used the same language and kept things consistent; I would rather nip inconsistencies in the bud earlier rather than later).
So, I flatten the structural qualities of each object and count them using this code (for CouchDB's incremental MapReduce).
http://pygments.org/demo/12753/ (alternative http://pastie.org/1384759 )
This enables me to grep the code base and then use blame to work with the developer to resolve oddities. Or, I can turn a blind eye because it isn't in a table that matters that much (i.e. meta data or user controlled data).
I can monitor this for changes daily to determine what is happening on development (where oddities first get introduced).
This mode of thinking enables me to think about unicorns when it comes to the database (oh, and never allowing anyone to delete; everything goes to trash with an trash_goes_out_on field that is set for 60 days in the future when it will be actually deleted).
For WIN, I force each object to have a name-space field called 'ns'; this enables me to partition the data and enable developers to partition the data. Ideally, this helps in keeping things separate.
A fundamental problem is that I want to have an idea of what it is in the data set and be able (and enable developers) to write appropriate documentation so everyone stays on the same page. I would also like data to adher to some kind of structural quality. However, it would be nice to be able to look for oddities that could become future support issues (it would also be nice if everyone used the same language and kept things consistent; I would rather nip inconsistencies in the bud earlier rather than later).
So, I flatten the structural qualities of each object and count them using this code (for CouchDB's incremental MapReduce).
http://pygments.org/demo/12753/ (alternative http://pastie.org/1384759 )
This enables me to grep the code base and then use blame to work with the developer to resolve oddities. Or, I can turn a blind eye because it isn't in a table that matters that much (i.e. meta data or user controlled data).
I can monitor this for changes daily to determine what is happening on development (where oddities first get introduced).
This mode of thinking enables me to think about unicorns when it comes to the database (oh, and never allowing anyone to delete; everything goes to trash with an trash_goes_out_on field that is set for 60 days in the future when it will be actually deleted).
Tuesday, December 14, 2010
Database Development Mistakes as NoSQL propaganda
Context
http://stackoverflow.com/questions/621884/database-development-mistakes-made-by-application-developers
Summary
All of these are consequences of using a one-size fits-all solution for storing your data. Fact is, application developers shouldn't worry about how they use data. They should be able to get their job done without worrying about the long-beard in the back room. I've been in this role, and I can sympathize with it.
Then, I realized something has to change. I took away SQL and built a very simple RESTful layer to the data layer, and then I watched how application developers solved their problems. I was amazed at their cleverness. Instead of saying "oh, these silly application developers are so dumb and don't know shit about databases", I said "I wonder how clever they could be if I just gave them memcached and simple get/put/by_index".
They taught me a thing or to about how awesome memcache can be (especially with cron-jobs).
Ideally, if you are building the data layer, then all you need to do to enable application developers is get the right complexity class out of the data. If you have ten billion things, then you need to provide the functions that get to a thousand things relevant to what the application developer needs to do. For bigger tasks, computations are best represented with MapReduce, and I feel that MapReduce is way easier to learn for fresh application developers. CouchDB's incremental MapReduce is by far the easiest to learn.
That being said, performance is always going to be an issue. If you enable developers this way, then you need to provide a realistic environment.
Related entry: Big Data enables Agile Data.
http://stackoverflow.com/questions/621884/database-development-mistakes-made-by-application-developers
Summary
- Not using appropriate indexes
- Not enforcing referential integrity
- Using natural rather than surrogate (technical) primary keys
- Writing queries that require DISTINCT to work
- Favouring aggregation over joins
- Not simplifying complex queries through views
- Not sanitizing input
- Not using prepared statements
- Not normalizing enough
- Normalizing too much
- Using exclusive arcs
- Not doing performance analysis on queries at all
- Over-reliance on UNION ALL and particularly UNION constructs
- Using OR conditions in queries
- Not designing their data model to lend itself to high-performing solutions
- Selfish database design and usage.
- Abusing denormalised data
- Scared of writing SQL
- Dogmatic 'No Stored Procedures' policies.
- Not understanding database design
- Not using version control on the database schema
- Working directly against a live database
- Not reading up and understanding more advanced database concepts (indexes, clustered indexes, constraints, materialized views, etc)
- Failing to test for scalability ... test data of only 3 or 4 rows will never give you the real picture of real live performance
- They only test on toy databases.
- Not using indexes.
- Not communicating with experienced DBAs.
- Poor Performance Caused by Correlated Subqueries
- Forgetting to set up relationships between the tables.
- Not using parameterized queries.
- Favoring "Elegant" code over highly performing code.
- Not doing the correct level of normalization.
- You want to make sure that data is not duplicated
- Using Excel for storing (huge amounts of) data.
- Unnecessarily using a function on a value in a where clause with the result of that index not being used.
- Not adding check constraints to ensure the validity of the data.
- Adding unnormalized columns to tables out of pure laziness or time pressure.
- not so much about the database per se but indeed annoying.
- Not taking advantage of CLUSTERED INDEXES
- Not using a SERIAL (autonumber) datatype as a PRIMARY KEY
- Not UPDATING STATISTICS on a table when many records have been INSERTED or DELETED.
All of these are consequences of using a one-size fits-all solution for storing your data. Fact is, application developers shouldn't worry about how they use data. They should be able to get their job done without worrying about the long-beard in the back room. I've been in this role, and I can sympathize with it.
Then, I realized something has to change. I took away SQL and built a very simple RESTful layer to the data layer, and then I watched how application developers solved their problems. I was amazed at their cleverness. Instead of saying "oh, these silly application developers are so dumb and don't know shit about databases", I said "I wonder how clever they could be if I just gave them memcached and simple get/put/by_index".
They taught me a thing or to about how awesome memcache can be (especially with cron-jobs).
Ideally, if you are building the data layer, then all you need to do to enable application developers is get the right complexity class out of the data. If you have ten billion things, then you need to provide the functions that get to a thousand things relevant to what the application developer needs to do. For bigger tasks, computations are best represented with MapReduce, and I feel that MapReduce is way easier to learn for fresh application developers. CouchDB's incremental MapReduce is by far the easiest to learn.
That being said, performance is always going to be an issue. If you enable developers this way, then you need to provide a realistic environment.
- Have a development server with more data than production and with a slower CPU (if you can't do this, then you the ability to connect to production in a read-only mode).
- Force them to profile their code (ab works very well for most situations)
- Work with business people to define how consistency should work
- Train them to do cache invalidation
Related entry: Big Data enables Agile Data.
Sunday, December 12, 2010
Why I gave up on static types
I like programming language theory and how to use typing to do some pretty impressive things, but I'm getting older now and I just don't give a shit about types for day to day stuff. I also gave up on object-orientated code. I also said F-U to relational database theory. Why?
Because people using your product don't give a shit about how it gets done. That's the reality. They don't care if you use assembler or JavaScript. They just don't. The question is: can you make people happy. The more important question is: can you sell? can your team sell? can your sales team make compromises to make the sell?
This last question is the question that I ponder about since it affects my profits. Do I want to put up some academic/aesthetic wall in front of a sale? Or, do I want to enable them to make a sell?
This is where all that rigidity breaks down and I ask a new question. Is this methodology or technology better for sales?
Static typing? No.
Object Orientation? No.
Relational Databases? No.
There is a lot of bull-shit technology out there (especially built on .NET or Java) that is simply a wall to sales. Now, it does depends on what you are doing, but ultimately it comes down to sales.
My issue with static types is that I can't add new members at run-time; nor does it propagate. Everything I do now is basically a giant JavaScript object that I pass around with JSON. I don't care what is in it. From a business point of view, I know that if everything in the system doesn't try to map the JSON into a static class, then I keep all the data; it just propagates. This enables me to change elements at the data store like adding a boolean named "my_sales_team_is_awesome_and_sold_a_feature_that_can_be_added_by_a_bool", then I can sleep knowing that the entire system will just deal with it and pass it along. I don't need to deploy a binary nor compile across an entire system to add a little bool.
My issue with object orientated code is that most of my stuff is non-inherited. I have things that can not be objects. While I do use the JavaScript object a bit, I don't use prototypes. I just treat it like a map and move on with my day. I don't give a shit about binding code to data; this is the worst possible thing you can do. I need all my data in a format that it is (a) obvious what it is and (b) easy to transform by looking from the outside. This is my data model guide line; if any idiot can look at the data and know what it means, then it is a good data model.
My issue with databases is the same as static types. I don't want to plan out how my data is going to look. I don't want to think. I want to be agile and just capture data and throw it into the database. I want to capture as much data as possible then organize it later. I don't want to think about normalizing which I can always break (show me your schema, and I will find a feature that will break it). I just want to put my data somewhere safe and have it replicate. This is why I use CouchDB. It's very relaxing.
Looking back at my life, I realize that I was wasting a bunch of time and energy trying to reach a goal with stupid means. My goal was to enable crazy fast development, and I achieved this goal by simply changing my outlook and aesthetics.
Having said that, I realize that there are reasons these things exist. If you need them, then you should use them. I love static types, but only for raw performance. There are performance patterns that can be implemented as a server that are very flexible, and those are important things to learn as they enable you to deploy safe services. The problem thou is always with specifics.
Oh, it also helps to have mastered grep and write code that enables grep to be useful; this is an amazing productivity boosts for when static types are actually very useful.
I haven't completely given up on types, I just now realize that their place is not where I would have liked it. If you look at my github, then you can probably tell where I've been spending my time in terms of type system.
That's right, I'm a node.js junkie. I just spent a weekend cutting a new version of my platform, and I have to say that I get amazing velocity with it. So much so that I can focus on leveling up my design rather than painting yet another bike shed.
Because people using your product don't give a shit about how it gets done. That's the reality. They don't care if you use assembler or JavaScript. They just don't. The question is: can you make people happy. The more important question is: can you sell? can your team sell? can your sales team make compromises to make the sell?
This last question is the question that I ponder about since it affects my profits. Do I want to put up some academic/aesthetic wall in front of a sale? Or, do I want to enable them to make a sell?
This is where all that rigidity breaks down and I ask a new question. Is this methodology or technology better for sales?
Static typing? No.
Object Orientation? No.
Relational Databases? No.
There is a lot of bull-shit technology out there (especially built on .NET or Java) that is simply a wall to sales. Now, it does depends on what you are doing, but ultimately it comes down to sales.
My issue with static types is that I can't add new members at run-time; nor does it propagate. Everything I do now is basically a giant JavaScript object that I pass around with JSON. I don't care what is in it. From a business point of view, I know that if everything in the system doesn't try to map the JSON into a static class, then I keep all the data; it just propagates. This enables me to change elements at the data store like adding a boolean named "my_sales_team_is_awesome_and_sold_a_feature_that_can_be_added_by_a_bool", then I can sleep knowing that the entire system will just deal with it and pass it along. I don't need to deploy a binary nor compile across an entire system to add a little bool.
My issue with object orientated code is that most of my stuff is non-inherited. I have things that can not be objects. While I do use the JavaScript object a bit, I don't use prototypes. I just treat it like a map and move on with my day. I don't give a shit about binding code to data; this is the worst possible thing you can do. I need all my data in a format that it is (a) obvious what it is and (b) easy to transform by looking from the outside. This is my data model guide line; if any idiot can look at the data and know what it means, then it is a good data model.
My issue with databases is the same as static types. I don't want to plan out how my data is going to look. I don't want to think. I want to be agile and just capture data and throw it into the database. I want to capture as much data as possible then organize it later. I don't want to think about normalizing which I can always break (show me your schema, and I will find a feature that will break it). I just want to put my data somewhere safe and have it replicate. This is why I use CouchDB. It's very relaxing.
Looking back at my life, I realize that I was wasting a bunch of time and energy trying to reach a goal with stupid means. My goal was to enable crazy fast development, and I achieved this goal by simply changing my outlook and aesthetics.
Having said that, I realize that there are reasons these things exist. If you need them, then you should use them. I love static types, but only for raw performance. There are performance patterns that can be implemented as a server that are very flexible, and those are important things to learn as they enable you to deploy safe services. The problem thou is always with specifics.
Oh, it also helps to have mastered grep and write code that enables grep to be useful; this is an amazing productivity boosts for when static types are actually very useful.
I haven't completely given up on types, I just now realize that their place is not where I would have liked it. If you look at my github, then you can probably tell where I've been spending my time in terms of type system.
That's right, I'm a node.js junkie. I just spent a weekend cutting a new version of my platform, and I have to say that I get amazing velocity with it. So much so that I can focus on leveling up my design rather than painting yet another bike shed.
Subscribe to:
Posts (Atom)