One Degree of Separation

Social networks rely on your primary network - your existing friends and contacts - to introduce you to THEIR friends and contacts. Each of the people in the network are called 'nodes', each with one or more connections to other nodes. Each of those connections is sometimes called a degree of separation; a friend of a friend (FOAF) would then be two degrees of separation. The famous phrase "six degrees of separation" was based on work by psychologist Stanley Milgram who determined that any two Americans, connected in the nation-wide extended network, are separated by an average of five intermediaries, i.e. six steps or degrees.

Despite their connectedness, two people separated by so long a chain are extremely unlikely to ever meet. In fact, we usually only ever meet the friends of our friends: an extremely small fraction of the larger network. Web services like LinkedIn, the business contact network, tracks your chain to three degrees of separation - though I wonder how often the third degrees ever connect. [Friendster tracked the chain even further, and this pursuit has been credited with Friendster's downfall, as tracking long chains is very difficult computationally and has much larger hardware requirements.]

Online Social Networks are not really social, and the network - as degrees of separation - serves mostly to separate. So, if one really wants to 'kill' social nets, one needs to get rid of the 'net' (the multiple degrees of separation that separate people) in order to bring people together. Jyri Engeström argues that social networks should not be based on individual connections between people that can be counted and accumulated, rather people must be connected by shared objects. We agree and take this to the next level by making everything in the virtual community an object, where each object is connected to every other object.

The New Paradigm

As proposed in the last post, what is lacking in the current data islands and the proposed schema solutions is a way of harnessing the true power of the collective to actually reduce information overload and increase discovery. This will require a revolution in content and relationship discovery that can only arise with a completely new kind of information filtration and recommender technology.

"The social web will be powered by recommender systems".
Open Issues in Recommender Systems
John Riedl, Bilbao Recommenders School, 2006

The true power of the collective will be realized with the proper integration of social media, new universal discovery techniques, and associated detailed portable identity and personalization info. The result is a Social Web based on one degree of separation: all people and things are related to each other directly, with each such relationship differing only in type and strength. The following graphic is a representation of such a "one degree" circle of people relationships, but keep in mind that each person is also similarly related to all items, ideas, endeavors, etc. in the system as well.

Critical to this new paradigm are the new universal discovery techniques that I've hinted at previously. Current recommender systems, including collaborative filters, are too primitive and limited to accomplish the task. Instead, we have applied certain bioinformatics concepts to solve the puzzle of simulating the human preference engine without requiring "strong AI". This starts with a quick determination of a person's "core identity", that internal mechanism which is responsible for generating appreciation, and sifting through the chaos and making choices.

Determining that "core identity" is a critical breakthrough as it allows us to quantify the relationship (strength and type) between all people, and between all people and all other things in the system. It also can yield portable data that can be used to quantify such relationships between users and items from multiple data islands, and can even be used in mobile devices and in real-world activity. This discovery system involves no collaborative filtering, psychological testing or interpretation, statistical or stochastic methods, etc.

"But there is no go-to discovery engine - yet. Building a personalized discovery mechanism will mean tapping into all the manners of expression, categorization, and opinions that exist on the Web today. It's no easy feat, but if a company can pull it off and make the formula portable so it works on your mobile phone - well, such a tool could change not just marketing, but all of commerce."
The race to create a 'smart' Google
by Jeffrey M. O'Brien, writing for Fortune Magazine

In addition to the current benefits of the social web, the integration of these universal discovery techniques will allow:

• A brief one-page registration with no need for private information. Qualifies as 'Cold-Start' for people and also items, ideas, endeavors, etc.
• Immediate access to promising relationships of all types, i.e. universal recommendations. These relationships are the predicted interest and affinity between a person and all other people, music, movies, books, recreation, groups, products, services, ads, travel destinations, vocations, jobs, teams, politics, religion, ideas, websites, articles, news items, games, etc.
• Portable data that can be compared and relationships quantified. This portable data can be used between social and data islands, for mobile devices and in real-world activity.
• No language or cultural barriers: no folksonomy or semantic constraints.
• No need for existing relationships. Emphasis is on relationship discovery, though existing friends and contacts are revealing.
• No need to observe history of actions and choices. A one-page registration is enough to provide significantly more information, and better information, than collaborative filters can accumulate.
• The new system will act as a good friend who knows you well and delivers trusted recommendations of all types, both solicited and unsolicited.
• Reduced privacy concerns as personal or demographic data is unnecessary.
• Automatic person-level granularity. Each relationship has a strength and type.
• Universal recommendations allows for highly successful affiliations of all types, direct sales and downloads, and highly targeted advertising as the diverse business model.
• Ratio of discovery to effort is high. No need for constant messages, spam, requests, friend searches, etc.
• Discovery is filtration, so 'information overload' and the 'tyranny of choice' are greatly reduced.
• Enables highly personalized search engine functionality, news aggregation, and many other forms of person-level information filtration.
• Constant excitement of discovery, so no "what's next?" reaction. No limit to novelty and interest, little boredom. No feeling of wasted time.
• Highly useful and usable: the keys to success of any product or service.

Social Standardization and the Death of Social Networks

"...we’re reaching an inflection point where some fundamental conceptions of the web (and social networks) need to change".
from Stop building social networks, by Chris Messina

It seems that everybody is predicting the end of something due to something else, typically calling the later a 'killer app'. Are VOIP and email replacing the phone and fax? Is social media replacing Google search, email, communication in general? Is IM replacing email? Well, who would have predicted that the trusty typewriter would disappear in the span of a few years? It seems many are making another prediction: Social Nets are on their way out, at least in their current configuration. In this post, I'll talk about the problems and proposed solutions.

Social Nets are hugely popular and are obviously doing something right. They were clearly a revolution in online communication and information sharing. Let's first list why people enjoy them. They allow you to:

1. express yourself and try to look cool
2. people-watch / voyeurism / "gawk at strangers"
3. 'collect friends' and compete to see who has more
4. waste time doing semi-fun alone stuff with apps, etc.
5. keep in touch with existing friends (the primary network)
6. make new friends, dates and business contacts (the largely unfulfilled promise of the 'network')
7. manage your personal data
8. exchange knowledge and information
9. re-connect with old friends and colleagues

As for the negatives, here are some of the points mentioned on the blogosphere:

1. 'Friend collecting' is not 'social'. No real communication takes place, and no real friends are made. Checkmarking someone as a friend is not being social. Not much relationship building going on.
2. Information Overload is not reduced, quite the opposite: too many people, messages, spam, etc. There is a limit to our ability to absorb information: our internal filters cannot handle it.

   "There isn’t enough time in the day for any person to find value in what a 1,000 people have to say - our internal filters just won’t allow it. At some point all that information; whether it be valuable or just fluff, becomes nothing more than white noise".
   from Enough with the social crap I think I’m gonna puke, by Steven Hodson

3. "Massive waste of time" / "It takes too much time" / 'Social Net Fatigue'
4. Privacy concerns / 'abuse of trust'. Services track user activity on and off the service, and post some of those activities to the "friends". Combining information from multiple sources may reveal private information.
5. Social nets are 'Walled Gardens'. They are not portable - information is trapped within the bounds of each service. New users must re-enter profile information, must search and re-add network contacts, and must reset notification and privacy preferences for each new social net joined.
6. Social nets are by definition 'network-centric'. Most users are exposed only to friends of friends (i.e. two degrees of separation). This presents an obstacle to discovering true friends and contacts, most of the potential being outside of your network.
7. No Business Model beyond popularity and possibly advertising. Also, because new users on social networks often misrepresent themselves and enter false personal information, demographic data for advertisers is therefore unreliable.
8. The "superficial emptiness"
9. The "what's next?" phenomenon (after exhausting the novelty of the site) / Lack of Innovation
10. Not granular enough - no ability to group friends and contacts in categories, or indicate how close or trustworthy those relationships are.
11. Tired of having to add friends or accept friend requests in all of these networks.
12. Use a given service only because that's where your friends are.

Proposed Solutions:

Many feel that Identity/Info concepts like OpenID, OpenSocial, FOAF, the 'Semantic Web', Microformats, have great potential in solving a few of the above problems.

"a distributed, user-centric identity scheme would destroy almost every "walled garden" social software application on the web".
from Identity Management Will Destroy Social Software, by Brian 'Bex' Huff

The idea is that each internet user would have a single universal and portable profile that would be used and understood by all services, thereby elimiating the need to enter and configure the same information and connections on every new service. Ideally, this would have the effect of removing the walls between services, creating a single large community or 'cloud' where "relationships transcend networks/documents".

The social and data islands that dot the internet can clearly be helped by some kind of standardized profile that can be uploaded to (and modified by) each service. The burden of registration and establishing relationships would be greatly reduced. Such a profile can grow to include all the data that a person might share, including photos and information, music, movie, web site favorites, etc. As long as all services agreed on standardization, this should work pretty well. As an example, browser standardization is largely successful - though differences do exist and can be frustrating for developers and surfers alike.

The Next Revolution:

Schemas, however, will not solve most of the issues mentioned above, and some are made worse (like privacy concerns). Some even argue that standardization and identity aggregation would not be entirely apprieciated. As much as schemas depend on FOAF information, most of the problems with social networks will remain. If one really wants to 'kill' social nets, one needs to get rid of the 'net' part, i.e. the degrees of separation. What is lacking in the current data islands and the proposed schema solutions is a way of harnessing the true power of the collective to actually reduce information overload and increase discovery. The next revolution in content and relationship discovery can only arise with a completely new kind of information filtration and recommender technology.

"The social web will be powered by recommender systems".
Open Issues in Recommender Systems
John Riedl, Bilbao Recommenders School, 2006

The true power of the collective can only be realized with the proper integration of social media, new universal discovery techniques, and associated detailed portable identity and personalization info. The result is a Social Web based on one degree of separation: all people and things are related to each other directly, with each such relationship differing only in type and strength. More on this new paradigm shortly.

Cause vs. Effect of Human Preference

"One crucial unsolved problem for recommender systems is how best to learn about a new user".
Getting to Know You: Learning New User Preferences in Recommender Systems
Rashid, et al, 2002

"Success comes from understanding both data and people"
Open Issues in Recommender Systems
John Riedl, Bilbao Recommenders School, 2006

"The problem with recommendation systems is... it measures and acts upon the effect, not the cause".
Response to “UIEtips Article: Watch and Learn: Recommendation Systems are Redefining the Web”
Adam Smith, 2006

So far, the internet has been all about effect. What other people say they like, you might also like; what you liked in the past suggest what you may like in the future. Google does it with PageRank; Amazon.com and Netflix do it with their recommender systems. They act based on your, or others', past preferences (the effect) rather than the cause of your past preferences. As you interact with the web, applications can record your actions and choices in order to create a filter with which to formulate suggestions that you might appreciate in the future.

But this is not the way the natural social process of recommendation seeking works. If you really want a good recommendation you ask someone who knows you well, as an individual. This is the way good friends do it. We accept recommendations from good friends because they understand our core identity (hopefully) and have no ulterior motives (hopefully). For example, as a single guy, I will never again go on a blind date unless the intermediary is a good friend who understands my taste and my attitudes, values, personality, etc., as well as that of the prospective date. One could make an assumption based on my past dates and relationships, but it would be an assumption based on insufficient (see below) and indirect data: the effect rather than the cause.

What is the cause? Preferences do not appear out of thin air, they are a result of your core identity: some combination of nature and nurture, your genes and your cultural and social influences, the configuration of your brain. This is the direct cause of your preferences: it is your preference engine. Unfortunately, it is a black box that we cannot really open. Possibly in the future there will be a scanning device that can capture and replicate your precise neural configuration. With this copy, and sufficient understanding of the human mind, we might be able to accurately predict your choices. In making a choice, the steps are:

1. Core Identity + Exposure -> Preferences (i.e. Brazilian Supermodels)
2. Preferences + Availability -> Choices      (Damn!)

Current recommender systems, such as collaborative filters, attempt to simulate a filter at the second stage. What we need is a way to accurately simulate your filter at the first: not quite a copy of your brain - but close.

Your preferences are also extremely limited by your limited exposure. Take music as an example. I love music, but I have only heard a tiny fraction of a percent of all music. So how the hell can my current favorites be expected to be entirely descriptive of my true taste or ultimate favorites? I have been exposed to that which is largely popular, better marketed, in English, etc. Music recommender applications suffer from this limitation: they consider only what I have already heard, and so they receive highly skewed data about my true taste. It would be great to have a good friend who is the ultimate "long tail" DJ and can match me to music based on his knowledge of my core identity and detailed knowledge about all music and musical tastes.

"Thus, the task is not so much to see what no one yet has seen, but to think what nobody yet has thought about that which everybody sees".
– Arthur Schopenhauer

It seems obvious that far better recommendations would result from an intimate knowledge of a person's core identity. But identity is mysterious and unapproachable; better left to the fantasies of pipe-smoking psychologists. In reality, it is the chain around the elephant's leg. We all have the tools to break free from the constraints of assumption, but smart people have not previously applied themselves to the task.

Current Recommender Types

There are a number of types of recommender systems currently available. They vary significantly in their mode of action and ultimate user experience. In terms of results, recommender systems are expected to offer sufficient good quality recommendations ('New Favorites'). In addition to this, the quality of the results is also dependent on minimizing false positives ('Trust Busters') and false negatives ('Missed Opportunities'). In other words, users should also not be shown inappropriate results and should not be denied appropriate results.

The quality of the user experience is also influenced by the time and effort required to give the recommender system enough information to make minimally reasonable recommendations. Users are sometimes asked to fill out lengthy questionnaires, or applications require that a user's history of choices or ratings be observed and recorded. It takes time and effort before things start working well. These days, users don't like to wait for anything and expect immediate gratification - delivering instant results upon quick registration is called 'cold start'. However, existing applications that permit a 'cold start' lack anything close to sufficient information, explicit or implicit, required to make accurate, high-quality recommendations.

There are a number of strategies that recommender systems are taking today. These include:

1. Non-personalized: "Web 1.0" technology offering the highest rated or most popular items to all users. No intrinsic personalization, poor quality results, but immediate.
2. Demographic: Require some knowledge about the user in order to group similar users together (i.e. by age, gender, area code, other similar features). Poor quality recommendations, low personalization, though slightly better than the above. May require "private" information, and depending on the length of the questionnaire, registration can take time.
3. Simple answer or ratings matching: Matches users based on explicit matching of answers, selections, ratings, etc. Makes recommendations with extremely limited scope, many missed opportunities, requires answers or observations.
4. Heuristics, probabilistic models (Bayesian, Markov), decision tree, neural net, etc. An application must collect a large amount of user-item preferences, or user/item features before quality recommendations are possble. This approach attempts to identify the underlying logic (or apply certain assumptions, in the case of heuristics) to a user's choices.
5. User-based Collaborative Filtering: similarity of historical choices or actions allows the application to find highly correlated users. The assumption is that users who agreed in the past might tend to agree in the future. Limited immediate results, most items will not be rated/answered (sparsity). Users with non-typical opinions or taste (the 'long tail') may not get good recommendations.
6. Item-based collaborative filtering: Finds items that tend to be preferred together. Limited immediate results, and users with non-typical opinions or taste may not get good recommendations.
7. Content-Based: Find items with similar features (Keywords, author, genre, i.e. DNA) to known preferences of a user. Items must be properly and thoroughly represented as a set of features - this generally requires a large staff. Generally limited to a single domain as there may be few cross-domain features. Limited immediate results.

There are many recommendation engines and recommender applications available on the internet and many more seem to be popping up all the time. Currently they all have severe limitations and offer mediocre to poor quality results when compared to, say, recommendations by a best friend. Examples of current applications include:

• eHarmony requires a very lengthy questionnaire and uses a proprietary empirical heuristic to match people romantically. It's success depends on the quality of the questions and the heuristic, the person's willingness to answer truthfully, and the person's willingness to spend a few hours to register. Mixed results are reported, but there is certainly an advantage over matchmaking sites that allow daters to make their own bad choices.
• Pandora and Last.fm both recommend music though they do so in different ways. Pandora's large staff must determine the separable features ("DNA") of a song and observe a user's choices in order to extract common features of a user's preference. Last.fm seems to work by grouping users of similar taste. Both suffer from reduced choice diversity for slightly different reasons. Both are mildly satisfactory, but also suffer from excessive false negatives and false positives, and require recording your existing preferences. Two roommates using the same account will likely see poor results.
• Amazon.com's recommendations work by observed a user's choices and activity and grouping items (books, CDs, DVDs, etc.) that tend to be chosen or viewed by the same users. After viewing or choosing items, you are presented with: "users who liked X (the currently viewed item) also liked Y (a correlated item). As may be considered a typical pattern, users who buy for multiple people, like for children or friends, will likely see poor results.
• Social DNA sounds like it works similar to Pandora, but the granularity is significantly greater, and unlike eHarmony, there seems to be no heuristic - matching is all or nothing (i.e. explicit ratings and questions). This is expected to lead to extremely high false negatives, relatively few true positives, and, since matches will likely occur with only a tiny fraction of possible DNA (highly limited explicit information yeilds a sparse matrix), considering the complexity of human beings, mostly false positives.

In order to get relatively high quality and accurate recommendations, a large amount of explicit ratings/choices (and/or possibly implicit activity) must be recorded. This is extremely hard to do: users are less likely to maintain interest while the machine learns, and this will be increasingly true in the future. Currently, users must be content with mediocre results, but a trade-off will develop between accuracy/quality and user patience.

Another frequent limitation is that users can act maliciously or inappropriately to skew results. Due to the limitations of current applications, users may feel the need to modify or exaggerate their choices in order to get better results. On the other end, users who want to promote certain items to others may give or encourage false ratings, views or descriptions (called 'Shilling') through manual or automated efforts or attacks. Also, privacy becomes an issue as users may explicitly or implicitly reveal private information about themselves. Details include demographics, personal details, taste, ratings, opinions, etc. Systems administrators (and possibly hackers) will have free access to this data.

Accurate, high quality, robust and broad scope recommendations have been the holy grail for internet futurists for quite some time, though we are still a long way from that goal. The problem is largely technical: recommendations are a really tough problem. Mathematics/statistics, clever algorithms and artificial intelligence are stretching the results to the maximum, given the poor quality data available from users during registration or interaction with the application. The solution is to get high quality data about the user's identity or individuality and match based on that, rather than matching based on a user's history. The problem is that teaching the machine about the core identity of a person is science fiction. Or is it?

Live fuzzy search using n-grams in Rails

Update [July 31, 2014]:

• This post is the basis for a few Rails plugins and gems:
• Fuzzily - a Ruby gem for blazing-fast fuzzy text search
• no_fuzz - Simple Rails-plugin that provides offline fuzzy-search for ActiveRecord [see my fork of this as well]
• act_as_fuzzy_search - add a fuzzy search to model for DataMapper
• rails-fuzzy-search - and more recent fork of the above.
• You can now see these live applications:
• Heebz.com
• Gigcatcher.com

We wanted to give users the ability to perform a search. It is fairly easily set up a small search engine on a site or allow users to perform a search from a major search engine such as Google directly from the site itself. Our site needed something a bit more sophisticated and useful.

First, we wanted to allow users to start typing in the search box, and have the site suggest results from the database as the user types and display them in a drop-down list under the search box. This is sometimes referred to as auto-complete, auto-suggest or "Live Search". Rails accomplishes this typically through its JavaScript macros helpers that wrap the Prototype JavaScript framework and the Scriptaculous JavaScript controls and visual effects library. See also:

• AJAX autocompletion demo
• Create an Auto-Complete Field with script.aculo.us

We needed some extra functionality: we needed a single search feature to search multiple attributes of a model and, upon selection, populate a number of form fields with a number of attributes and associations. It actually gets even more complex than this, and perhaps I'll write more about it in a future article.

Typically, a search looks for exact matches of what is being typed. However, many important results are missed when an exact match is required: people often misspell, and related words that share the same word root will often not be found. So we wanted to allow users to find things that did not match perfectly with what was being typed. Furthermore, we wanted the results to be sorted such that the better the match, the higher a result is located on the drop down list.

One option for Rails developers is to use Ferret, a high-performance text search engine library based on Apache Lucene. Ferret is a reasonable option, and we may eventually go this route, but we wanted something simpler that did not require installing a library - we only needed the fuzzy search capability. Ferret's code is largely in C, rather than Ruby, and its fuzzy search is based on Levenshtein edit distance formula which calculates the "minimum number of operations needed to transform one string into the other, where an operation is an insertion, deletion, or substitution of a single character".

Instead, we decided it should be rather simple to create an n-gram distance algorithm and leverage database indexing to create a fast n-gram distance fuzzy search, purely in Ruby. What is n-gram distance? It is the measure of the similarity of two strings by comparing the number or significance of matching substrings of length n. It is particularly useful when searching a database of strings, for example a directory of names, rather than lengthy text (though that may certainly be possible as well). Our focus will be on a database of strings.

As an example, let's take the most popular name in Europe, and two of it's spellings: Muhammed and Mohammed (there are also many other spellings). Typical live searches will not be able to help someone who has started to type one rather than the other, as the second letter is different. Live searches need at least the first two or three letters of the input to match. First, let's add a word start character (for code, we prepend a space) to the beginning of the words because word boundaries constitute useful information. Next, let's break each word into sliding frames of three letters (trigrams). Muhammed becomes: *mu, muh, uha, ham, amm, mme, med. Mohammed becomes *mo, moh, oha, ham, amm, mme, med. The two names share 4 out of 7 trigrams.

First, migrate a trigrams table with columns: tg (trigram, a string), person_id (the key in the people table, an integer), and score (an integer, though you may not need it). Make an index on the first two columns in that order. Next, generate a trigrams model. Then you must populate the trigrams table with trigrams from the people table. This is easy to do as a separate migration:

class PopulateTrigrams < ActiveRecord::Migration def self.up #require 'enumerator' ? #convert all people into trigram data. use person firstname and lastname data. Person.find(:all).each do |person| # add a space at the beginning as this will emphasize the beginning of the word word = ' ' + person.lastname (0..word.length-3).each do |idx| tg = word[idx,3] # we can score last names differently for search matches Trigram.create(:tg => tg, :person_id => person.id, :score => 2) unless Trigram.find_by_tg_and_person_id(tg,person.id) end word = ' ' + person.firstname (0..word.length-3).each do |idx| tg = word[idx,3] Trigram.create(:tg => tg, :person_id => person.id, :score => 1) unless Trigram.find_by_tg_and_person_id(tg,person.id) end end end def self.down Trigram.delete_all end end

Here, we are are storing trigrams from each person's first and last names, and in this example, we are scoring matches to the last name at twice the score of the first name. This is just an example, you can score it any way you want. If you don't need to distinguish scores, you can eliminate the score column. Anyway, this migration will populate the trigrams table, and since the trigram and person_id are indexed, any search is easy on the database - and fast.

The autocomplete action method in the controller might look like this:

def auto_complete_for_person_full_name limit = 10 #how many results do you want? entry = ' ' + params[:person][:full_name].downcase # space is word boundary, full_name is a virtual attribute trigrams = (0..entry.length-3).collect {|idx| entry[idx,3]} trigram_groups = Trigram.sum(:score, :conditions => [ "tg IN (?)", trigrams], :group => 'person_id') top_ids = trigram_groups.sort_by {|a| -a[1]}.first(limit).map {|a| a[0]} #sorts by score, gets top scorers, returns simple array of ids unsorted_people = Person.find(top_ids) #get unsorted objects @people = top_ids.map{|id| unsorted_people.detect{|person| person.id == id}} #sort objects according to sorted ids render :partial => 'full_names' #AJAX update of dropdown list end

Do you see what's going on here? The entry is split into trigrams, then the scores of all matching trigrams in a person's name are summed. People are sorted based on their total score, and the top matches are sent to the partial. Very simple. The partial might look like this:

<ul class="full_names"><% for person in @people do -%> <li class="person"> <div class="id" style="display:none"><%= person.id %></div><div class="full_name"><%=h person.firstname %> <b><%=h person.lastname %></b></div></li> <% end -%> </ul>

The ID is included, though hidden, so user selection of a person automatically includes the id of the person from the database table for further use.

It works beautifully, and it seems very fast so far - though with a huge number of items, database storage may become an issue. Any further ideas, suggestions, requests?

Adventures in Self-Referential Database Associations

Rails makes object associations easy and has a number of different ways to configure associations and even store extra information about each relationship. Most of the time, those relationships are between business objects that wrap rows in two separate database tables. For example, let's say we wanted to be able to record, read and update grades and attendance of students in a particular school class. We might create a student model and a school class model, and the association between them may contain the grades and attendance information. We use many of these kinds of model object associations, and the procedure is fairly straightforward and written about extensively. See the following links to learn more:

• Associations in Ruby on Rails
• Module: ActiveRecord::Associations::ClassMethods
• Howto on has_many :through in Ruby on Rails

For our project, we also needed a way to store various qualities about relationships between pairs of objects of the same model type, i.e. between rows of the same database table. We would need this for a number of different purposes. Those creating 'social net' applications know this kind of association well; they must document, for example, who are the direct friends of a particular user, and possibly further information about each such relationship. All those users are stored in the same database table, so a separate "join" table is created to store the keys of both users, plus any other information about the relationship.

To accomplish this, we used self-referential has_many :through associations that use a join model and methods to simplify the reading and updating of attributes of the association. To read more about these more specialized associations, here are some sites:

• Self-referential has_many :through associations
• How to model an internal emailing system
• Another has_many :through Self-referential Example

More details on this later.

Why Ruby on Rails?

Ruby is a beautiful, purely object oriented, interpreted programming language influenced by Perl and Smalltalk, among others. Rails is a web application framework built on top of Ruby that is designed to speed and simplify the development of database-driven web applications. Rails is relatively new and not widely used relative to other web programing languages, but nonetheless its growth has been tremendous. Rails has seen so much growth because of its reputation as the fastest and easiest web development framework. Also, to achieve similar functionality Rails typically requires far fewer lines of code than the more popular web development languages.

When we first started out, we were fairly familiar with Perl and PHP - so it was a difficult decision to attempt to bring the team into the Rails community. We expected that the delay required to learn Rails would be later compensated by faster development. So we decided to learn Ruby and Rails and at least build the Builders' Site prototype with Rails.

The Israeli Rails community is rather small, but nonetheless I attempted to try to attract co-founders and developers for a previous related project through the israel.rb group. There are lots of startups in Israel, and I'm not surprised that demand exceeds supply of capable developers. At the time I was not able to interest any qualified Rails hackers to join the Israel team - part of the problem may have been that none of them knew what the project was really about, only that some bombastic American has a stealth startup.

Despite the lack of Rails experts on the team, the learning curve was not excessive and we were quickly creating some fairly complex functionality. The Rails principle of "Convention over Configuration" helps to speed development of more typical functionality. We encountered some difficulty when our functionality quickly exceeded the convention. Luckily there are some real heroes on the internet who publish their experiences with similar issues and try to help those trying to learn and develop along the same lines.

Critics of Ruby and/ or Rails argue that Ruby is slow and web applications developed in Rails do not 'scale'. Most of this criticism appears to be sour grapes. It is true that there are no blockbuster applications written in Rails, only medium sized applications, but reaching "medium size" exceeds the achievements of the vast majority of web startups. Rails defenders argue that scaling Rails applications simply involves purchasing more servers and hardware, and that the greatest limitations are database related - and all web development languages suffer from this. They argue that savings on developer-hours more than makes up for extra costs in hardware.

The debates continue, as does our development. Perhaps I will post some of the obstacles and specific solutions we found. So far we like Rails, and have no plans to go elsewhere. When we are ready to increase our development or scale the application, perhaps we will consider again all our options. The decision will, in part, depend on the availability of capable developers.