Understanding a sea of JSON with Map Reduce

CouchDB stores a lot of data in a sea of JSON, and it isn't exactly easy to get a good grasp on what there is.

For WIN, I force each object to have a name-space field called 'ns'; this enables me to partition the data and enable developers to partition the data. Ideally, this helps in keeping things separate.

A fundamental problem is that I want to have an idea of what it is in the data set and be able (and enable developers) to write appropriate documentation so everyone stays on the same page. I would also like data to adher to some kind of structural quality. However, it would be nice to be able to look for oddities that could become future support issues (it would also be nice if everyone used the same language and kept things consistent; I would rather nip inconsistencies in the bud earlier rather than later).

So, I flatten the structural qualities of each object and count them using this code (for CouchDB's incremental MapReduce).

http://pygments.org/demo/12753/ (alternative http://pastie.org/1384759 )

This enables me to grep the code base and then use blame to work with the developer to resolve oddities. Or, I can turn a blind eye because it isn't in a table that matters that much (i.e. meta data or user controlled data).

I can monitor this for changes daily to determine what is happening on development (where oddities first get introduced).

This mode of thinking enables me to think about unicorns when it comes to the database (oh, and never allowing anyone to delete; everything goes to trash with an trash_goes_out_on field that is set for 60 days in the future when it will be actually deleted).

Database Development Mistakes as NoSQL propaganda

Context

http://stackoverflow.com/questions/621884/database-development-mistakes-made-by-application-developers

Summary

• Not using appropriate indexes
• Not enforcing referential integrity
• Using natural rather than surrogate (technical) primary keys
• Writing queries that require DISTINCT to work
• Favouring aggregation over joins
• Not simplifying complex queries through views
• Not sanitizing input
• Not using prepared statements
• Not normalizing enough
• Normalizing too much
• Using exclusive arcs
• Not doing performance analysis on queries at all
• Over-reliance on UNION ALL and particularly UNION constructs
• Using OR conditions in queries
• Not designing their data model to lend itself to high-performing solutions
• Selfish database design and usage.
• Abusing denormalised data
• Scared of writing SQL
• Dogmatic 'No Stored Procedures' policies.
• Not understanding database design
• Not using version control on the database schema
• Working directly against a live database
• Not reading up and understanding more advanced database concepts (indexes, clustered indexes, constraints, materialized views, etc)
• Failing to test for scalability ... test data of only 3 or 4 rows will never give you the real picture of real live performance
• They only test on toy databases.
• Not using indexes.
• Not communicating with experienced DBAs.
• Poor Performance Caused by Correlated Subqueries
• Forgetting to set up relationships between the tables.
• Not using parameterized queries.
• Favoring "Elegant" code over highly performing code.
• Not doing the correct level of normalization.
• You want to make sure that data is not duplicated
• Using Excel for storing (huge amounts of) data.
• Unnecessarily using a function on a value in a where clause with the result of that index not being used.
• Not adding check constraints to ensure the validity of the data.
• Adding unnormalized columns to tables out of pure laziness or time pressure.
• not so much about the database per se but indeed annoying.
• Not taking advantage of CLUSTERED INDEXES
• Not using a SERIAL (autonumber) datatype as a PRIMARY KEY
• Not UPDATING STATISTICS on a table when many records have been INSERTED or DELETED.

My Thoughts

All of these are consequences of using a one-size fits-all solution for storing your data. Fact is, application developers shouldn't worry about how they use data. They should be able to get their job done without worrying about the long-beard in the back room. I've been in this role, and I can sympathize with it.

Then, I realized something has to change. I took away SQL and built a very simple RESTful layer to the data layer, and then I watched how application developers solved their problems. I was amazed at their cleverness. Instead of saying "oh, these silly application developers are so dumb and don't know shit about databases", I said "I wonder how clever they could be if I just gave them memcached and simple get/put/by_index".

They taught me a thing or to about how awesome memcache can be (especially with cron-jobs).

Ideally, if you are building the data layer, then all you need to do to enable application developers is get the right complexity class out of the data. If you have ten billion things, then you need to provide the functions that get to a thousand things relevant to what the application developer needs to do. For bigger tasks, computations are best represented with MapReduce, and I feel that MapReduce is way easier to learn for fresh application developers. CouchDB's incremental MapReduce is by far the easiest to learn.

That being said, performance is always going to be an issue. If you enable developers this way, then you need to provide a realistic environment.

• Have a development server with more data than production and with a slower CPU (if you can't do this, then you the ability to connect to production in a read-only mode).
  
• Force them to profile their code (ab works very well for most situations)
  
• Work with business people to define how consistency should work
• Train them to do cache invalidation

In my mind, core to the NoSQL movement is the ability to empower more people with persistence rather than knowing edge cases of how a database works. The sooner you empower people, the sooner the business can iterate and the more time you can dedicate to sitting on a beach thinking about unicorns.

Related entry: Big Data enables Agile Data.

Why I gave up on static types

I like programming language theory and how to use typing to do some pretty impressive things, but I'm getting older now and I just don't give a shit about types for day to day stuff. I also gave up on object-orientated code. I also said F-U to relational database theory. Why?

Because people using your product don't give a shit about how it gets done. That's the reality. They don't care if you use assembler or JavaScript. They just don't. The question is: can you make people happy. The more important question is: can you sell? can your team sell? can your sales team make compromises to make the sell?

This last question is the question that I ponder about since it affects my profits. Do I want to put up some academic/aesthetic wall in front of a sale? Or, do I want to enable them to make a sell?

This is where all that rigidity breaks down and I ask a new question. Is this methodology or technology better for sales?

Static typing? No.
Object Orientation? No.
Relational Databases? No.

There is a lot of bull-shit technology out there (especially built on .NET or Java) that is simply a wall to sales. Now, it does depends on what you are doing, but ultimately it comes down to sales.

My issue with static types is that I can't add new members at run-time; nor does it propagate. Everything I do now is basically a giant JavaScript object that I pass around with JSON. I don't care what is in it. From a business point of view, I know that if everything in the system doesn't try to map the JSON into a static class, then I keep all the data; it just propagates. This enables me to change elements at the data store like adding a boolean named "my_sales_team_is_awesome_and_sold_a_feature_that_can_be_added_by_a_bool", then I can sleep knowing that the entire system will just deal with it and pass it along. I don't need to deploy a binary nor compile across an entire system to add a little bool.

My issue with object orientated code is that most of my stuff is non-inherited. I have things that can not be objects. While I do use the JavaScript object a bit, I don't use prototypes. I just treat it like a map and move on with my day. I don't give a shit about binding code to data; this is the worst possible thing you can do. I need all my data in a format that it is (a) obvious what it is and (b) easy to transform by looking from the outside. This is my data model guide line; if any idiot can look at the data and know what it means, then it is a good data model.

My issue with databases is the same as static types. I don't want to plan out how my data is going to look. I don't want to think. I want to be agile and just capture data and throw it into the database. I want to capture as much data as possible then organize it later. I don't want to think about normalizing which I can always break (show me your schema, and I will find a feature that will break it). I just want to put my data somewhere safe and have it replicate. This is why I use CouchDB. It's very relaxing.

Looking back at my life, I realize that I was wasting a bunch of time and energy trying to reach a goal with stupid means. My goal was to enable crazy fast development, and I achieved this goal by simply changing my outlook and aesthetics.

Having said that, I realize that there are reasons these things exist. If you need them, then you should use them. I love static types, but only for raw performance. There are performance patterns that can be implemented as a server that are very flexible, and those are important things to learn as they enable you to deploy safe services. The problem thou is always with specifics.

Oh, it also helps to have mastered grep and write code that enables grep to be useful; this is an amazing productivity boosts for when static types are actually very useful.

I haven't completely given up on types, I just now realize that their place is not where I would have liked it. If you look at my github, then you can probably tell where I've been spending my time in terms of type system.

That's right, I'm a node.js junkie. I just spent a weekend cutting a new version of my platform, and I have to say that I get amazing velocity with it. So much so that I can focus on leveling up my design rather than painting yet another bike shed.

WIN is looking good; good enough to start documenting and testing more hard-core

Well, I put WIN into production. I learned that if you rely on unsupported couchdb code, then strange things happen since there is no debug code. I found a bug in node.js that I need to mock up and send to the node.js team. I learned that I don't like looking at more than 1K code.

I just spent half a day re-factoring and cleaning up win so it makes more sense, and I added crap comments. I also linted to look for stupid issues, so it looks a lot cleaner now.

So, now, I'm going to write the guide ultra-hard-core fashion. I am confident in the patterns that I am going to present, and I'm confident that the system can be hacked to get anything anyone would want.

Why Mustache is for WIN

Mustache is a logic-less templating language. By being lacking in logic, it easily enables cross-language template interpretation. This is important for two reasons.

• It protects work in constructing good DOM. This is true for many template languages, but it makes sure the assets are protected from language change.
• By enabling templates to work in multiple languages, you enable it to work it multiple contexts. For instance, if you have a search feature that you would like ajaxified, then you can work towards producing a JSON object. For SEO, you use the template to send off the HTML. For Ajax, you can just get the JSON object and do the JSON to HTML in the browser. Generally, JSON is more efficient to send over the wire when compared to HTML; ergo, you get a snappier response in addition to faster development time (by only writing one template and not worrying about DOM manipulation).

When I approached Mustache, I was hesitant at first since I do not do much front-end engineering. I wondered what it can't do, and I've looked for counter-examples. Simply put, I can do amazing things with mustache. So can you.

The key is to think of Mustache as just a simple HTML encoder over a giant JSON represention of the module, page, layout, etc. You will put in some silly things in the JSON, but in the end it will enable something very powerful in you architect around getting a giant a JSON object back.

Namely, it is very easy to automate testing on giant JSON files. That is, it is easier to script against JSON than junk HTML. For me and WIN, this is a fairly important question as I would like to be able to crawl my entire projects to look for errors.

Say Yes to Internet Censorship

Why?

Because it will make things worse.

When things are bad, talk begins of revolution.

Viva La Revolución

By the way, this was troll-bait. Just an experiment. Of course, there should be no censorship, but that is obvious to me. Is this not obvious to others???

3 reasons why I don't key off of email anymore.

For some of my clients, I built their stuff such that a user only needed an email and a password. Registration was easy and it was awesome. Now, I have introduced a login name back in. Here is why.

Emails Change

I had clients that lost jobs and they needed to change their email; well, that required writing a change email function. That's not pleasant because emails may already exist due to a prior sign up or a different use case.

People get fired, two employees at a company. One used their personal to sign up to product where as the other used their business. The one who used their personal got fired and the needs to transfer access to the other employee, but its already taken. So, either I have account merger or they manage multiple credentials. Never the less, they have to call in for support if we present an obstacle. Additionally, companies get bought and emails change.

By adding the level of indirection, I'm enabling them to handle these issues themselves rather than supporting it on our end.

Multiple Accounts per Email

If you enable a single email to manage multiple accounts, then you help them out companies that have different billable uses of your product. Otherwise, you require them to be able to setup multiple emails which just sucks.

Multiple Managers/Owners

If you focus on providing a single account, then you can enable your product to be managed by multiple people (or enable collaborative features). It is easy to key off of the account's login name to enable multiple users to access the account.