I use Apache CouchDB in the implementation of OurParents for holding a lot of our big data. One of the things I run into is a concern for the size of any specific object within the dataset. When you have hundreds of thousands of documents each of which with a lot of weight, finding the “big” objects can be a problem. Luckily, and full thanks to Jan Lehnardt, there is this little gem that leverages the underlying SpiderMonkey system within CouchDB to determine the size of a document based on the number of characters in it. Hope it helps you as much as it helps me:
I just wanted to write a short ode to Ajaxian. Ben and Dion both literally created the Ajax community, founding a central watering hole for everyone to come around. Of any tech community I’ve been a part of, the Ajax/Ajaxian community has been the absolute best one. While Jesse James Garrett might have coined the term Ajax, Ben and Dion made it real and gave it heart.
I wouldn’t be where I am in any possible way if it wasn’t for Ajaxian and Ben and Dion. Thanks for all the hard work guys!
As HTML5 grows and slowly eclipses the Ajax community, lets hope we get as awesome a common community space as Ajaxian has been!
// posted by Brad Neuberg @ Wednesday, July 14, 2010
Organizing any event of any magnitude takes exponential effort and cost to the resulting “feel” of that event.
This is something I have come to after years of starting small groups and increasingly large events. JSConf, or any conference as best I can tell, is no exception to this rule. I believe in part this is due to the veil that conference organizers (myself included) put up in order to hide the details in an effort to make the event seem perfect - the truth is very often far from. A conference or any event of substantial size is little more than herding cats with nothing but a spoon and a lot of faith that things will work out, despite any specific time instance. That’s it - complete and total chaos. If you don’t immediately understand what I mean, please continue.
Possibly the most complex and important decision of every conference is the venue. Trust me, the venue is also the most precarious and hazardous of all decisions related to the success of the conference. When an organizer selects a venue they are implicitly selecting the following items:
- Quality, variety, and costing of the food available
- IT capability for the entire conference
- Room cost and availability
- Total number of attendees
- General feel of the conference
- Location of the event
That is a tremendous amount of fundamental concerns that all must be done prior to even the first thought of the conference. Factor in that no hotel seems to be affected by the ongoing economic troubles suffered by everyone else, or at least that is the front they all put up, and before even setting forth, the organizer begins to question their decision. Think about this, at a technology conference, you are selecting the basis for every major component that makes or breaks the conference (internet, room rates, number, feel, food) roughly 6-8 months in advance with little to no validation that the conference will work or who will show up.
Worse still, you are dedicating roughly 55% of your budget to a group that is only contractually required to turn the lights on and unlock the doors. Organizers have little to no protection in venue contracts, but bear all of the risk. If you assume a hotel for “ease of selection”, the organizer has to guarantee at their own personal risk a room block, meaning a set of rooms that must be filled by certain number of attendees otherwise you pay for the rooms to be vacant. You have to make the estimate on the size of the room block long before you really even know the numbers of attendees AND you cannot increase the size without going through another round of negotiation with the hotel. So you are stuck trying to fill rooms in a hotel to (normally) within a 5 room night space considered “acceptable” by the hotel. Worse still, the room rate is “offered” with little bearing to the market and little control over any “considerations”. Most hotels will offer the organizer a “price lower than any other available”, but will sell the rooms on Priceline/Hotwire for at or below the event price. Aside from the organizer or an attendee actually booking a room through such services, the organizer will never find out and therefore can never exercise the contract line item - rendering it completely useless.
The mere selection of the venue is a precarious clusterfuck as well, if you choose a venue in the city which affords excitement and self-directed activities for the attendees, you can look for a room rate of $200-$500/night for your attendees at first negotiation. That pretty much prohibits anyone from attending except for local people who do not help you on the aforementioned room block issue. Also you have to factor in that the cost of the hotel room directly affects the cost of the conference, since you will most likely be providing a space for your invited speakers (yet to be determined) to stay. Consider a two day (16 speakers) single track conference, the cost of hosting speakers so you can ensure that they are ready and “on-call” normally runs roughly $15,000-$30,000 of the budget. This is also why negotiating the venue is absolutely essential, but it is very stressful, frustrating, and can build a tremendous amount of hostility between the organizer and the venue early on, yielding a less than satisfying relationship on going.
If you hold the conference outside the city, the price does go down to anywhere from $100-$250, but the organizer has to balance the likelihood of someone taking a taxi from the nearest airport to get to your conference, which decreases with every mile out from a major airport. Since you aren’t in a major metropolis (possibly in the suburbs) you likely will have to find (or create) things for your attendees to do/see/eat outside of the conference. Or you could just not care - but then the perceived quality of the conference rapidly degrades, because the attendees will just go back to their rooms and complain. So now you are looking at a social budget and the first thing you will consider is the hotel. Hotel parties cost anywhere from $10,000 - $14,000 for a group the size of 150 people, because according to their analysis people will consume $86 worth of food and drink over two hours. Stop for a second a re-read that, you could send everyone to the fanciest of restaurants in the region for that price, but instead you are going to pay that for chicken-on-a-stick and Bud Light. On top of that there is generally a 22%-30% gratuity that must be paid to thank the people cooking and serving the food - regardless of their performance. It is without a doubt, impossible to justify the cost so the organizer will now have to find a new venue for the social events, making it exponentially more complex.
And then there is food and beverages during the day. Most hotels and venues will not let you bring in outside food, feigning rules of “state law” and “labor restrictions” - all of which is complete hogwash barring two states. So they will provide the organizer with the only option for food, which comes with prices that are outrageous. By outrageous, I mean $26 per person for a “continental breakfast” of day-old pastries and $40 per person for deli sandwiches. So for a given attendee, you are looking at spending anywhere from $50 - $75 per person per day. This amount will be even higher if you attempt accommodate your guests throughout the day with coffee and sodas which can easily range from $28 - $36 per person, per day. So tabulating that out, if you run a conference for 150 people spread over 2 days, you are looking at a base price for accommodating your guests of roughly $34,000 on average.
So as a conference organizer, just to get started, you require about $50,000 to just secure the base essentials of speakers and food/beverage. So next time you complain about the price of admission to event X or conference Y consider this, in order to break even for an “ok” event in a reasonable place, the bear minimum the conference organizers should charge without expectation of sponsorship is $189 per day per person. That is just covering your attendance and the presence of the speakers for the event. Mind you that is only a one track conference, as you add more tracks the number of “compensated” speakers grows normally at the exchange of paying guests, which directly changes the cost per person per day. Furthermore that number does not cover the cost of speaker travel, which can range anywhere from $300 to $700 per speaker.
I have been sitting on this post for a while, I do intend on continuing the series, but I wanted to get this out. Seriously, stop bitching about grass roots conferences, if you have an issue with them, then offer to help - it will be gladly accepted. The grass roots, organic conference movement has provided some of the most amazing events in the technology industry, but they are fading because the response is generally one of “OMG I HATE YOU” or “WTF YOU FAILED ON…” instead of looking at what positive things were done. If you think you can do better, I encourage you to go for it. You will have a new appreciation for every other event you attend thereafter.
As you may or may not know, this year we started out with a very risky web site concept. Hailed by some as crazy and others as “elitist BS”, either way it was pretty good filter for speakers. Some found it, others needed a little treasure map. Charles Jolley actually emailed us, provided a fitting description of himself, and asked how he could propose to speak. All we needed to say was “use web inspector” to which we received his one line response: “lol. clever.” At that point, I knew he was the kind of speaker we were looking for. Charles is the creator of the now famous and widely used SproutCore. SproutCore is the Rich Internet Application (RIA) framework at play in such applications as MobileMe from Apple and Bespin from Mozilla (also a JSConf Sponsor 2 years running). Charles currently works for Apple as a senior front end architect, which is without a doubt an impressive title.
Charles plans on demonstrating the HTML5 love that SproutCore 1.1 comes packed to the brim with. Taking RIA and binding it with HTML5 - Brilliant! Charles also has some great things he is planning on demonstrating that will really blow your socks (or boots as the case maybe) off. This is going to be quite the presentation especially for building complex and very rich internet applications. We might even have some buccaneers with contrarian view points strategically placed in the audience to heckle him… You will have to be there to find out more!
Posted on January 29th, 2010 by Kevin Dangoor
The Original Goals
As laid out in my original blog post, we were seeking to create:
- A module system,
- A cross-interpreter standard library,
- A few standard interfaces,
- A package system, and
- A package repository
Note that the idea here is that this group creates specs, which will have multiple implementations.
CommonJS and the ECMAScript Standard
The CommonJS group is a grassroots effort, and not some formal standards body. In some ways, however, it works like a standards body in that the people working on the standard are also implementing the standard-in-progress and using it to build real applications.
We have no control over the ECMAScript language, which is managed by the TC39 working group. However, there are a few people involved in CommonJS who are part of TC39, and I have firsthand knowledge of others who are keeping an eye on how things are going with CommonJS.
The CommonJS standard-in-progress is designed to work on a subset of ECMAScript 5 that can be made to work on today’s ECMAScript 3 interpreters. ECMAScript 3 is the standard that is running in all of the browsers. In other words, in a CommonJS application you can count on Array.prototype.forEach to be implemented. Obviously, applications can do whatever they want (array destructuring? knock yourself out, but your app will only work on SpiderMonkey and Rhino).
One certainty about CommonJS is that inventing new language syntax is out of scope.
The Module System
The CommonJS group has been remarkably good at avoiding bikeshedding. While there is discussion about names of things, there isn’t heated discussion about it. People are far more interested in issues of functionality and ease-of-use. This is a very good thing, and it allowed us to get modules out of the way early on.
Of course, the lack of bikeshedding doesn’t mean that everyone agrees on things. The CommonJS module system has its controversial aspects, but I think it does well given the constraints:
- must work with ES3 syntax (destructuring could actually be useful, but we’re not going to do it)
- modules should have self-contained namespaces and be explicit about data the want to export
- it should be possible to make modules tamper-proof, though this is not a requirement
- using a module should be competitive with using modules in languages like Python and Ruby
Personally, I find this to be reasonably concise with a nice level of explicitness. Some syntax sugar would be good, and I hope we get that in ES-Harmony. But, this syntax works fine today.
Of course, this syntax is not without controversy. The biggest controversy has been that require() is synchronous – the “extramath/silly” module has to be available as the module above is loaded. However, there are a couple of reasonable ways to deal with this problem and I am happily working with CommonJS modules in the browser which is the environment that is least tolerant to synchronous loading.
Controversy or not, this basic module system was ratified last winter and has been implemented on all target environments of CommonJS.
Discussion is ongoing for a tie-in to the module standard: the module transport standard. Without additional help from the browser or some additional scaffolding running in the page, the module syntax above doesn’t work via a
The Standard Library
I had hoped we’d get farther on the standard library than we have, but it can indeed be a long process. system and unit testing are the only ones that have been ratified at this point. We have come a long way on file access.
“What!?!?”, I hear you say, “there’s no standard for accessing files yet?” That’s right. And there are a couple of reasons that it’s been challenging to get there.
I’m hoping to see promises become a part of the standard, because some form of that interface is very convenient to use for asynchronous operations.
Database access has not yet been standardized. It will be interesting to see if we can come up with a good interface that can usefully target SQL and NoSQL databases alike.
We have a recently ratified spec for packages of CommonJS code. I hope this will bring about a collection of good package managers with different focuses and targeting different environments. There have already been a couple of attempts to create package management systems, and the most fleshed out one that I’ve seen is Tusk, which is bundled with Narwhal. Once a few more specs are ratified, Tusk should be able to run on other CommonJS implementations.
Tusk is using GitHub as a package repository, and that is working okay for the time being. A couple weeks back, we got word that the jQuery plugin repository is going to provide a CommonJS package.json file for the ~5000 plugins in their database. This is exciting, because this infrastructure could prove to be very useful to us going forward.
My Personal View
I had hoped to personally have more time to devote to CommonJS in 2009, but I am delighted at how a great collection of people have stepped in and carried the specifications and implementations forward through a lot of hard work and force of will. More than 5,100 messages have gone across the mailing list, and more than 10% of those have been from Kris Kowal. Kris has done a ton of work in ironing out many of the specs and deserves a good deal of credit for where CommonJS is today. He and Tom Robinson even stood in for me when I had to cancel my trip to JSConf.eu (thanks, guys!).
Taking a look at the top posters on the googlegroup, you can see how many people have put so much into CommonJS. More than 10 people have contributed more than 100 messages a piece and, for many of those people, there was a lot of time spent in the email discussions, IRC chats, spec writing and implementation of those specs. Plus, as early adopters, they have the joy of tweaking things as the specs have changed over time. Thanks to all of you for the dedication and the will to get it done.
Comments are closed.