Universal Recommendations

What is the Affinax Project?

2009-11-26T01:20:00.005+02:00

The Affinax Project is attempting to solve one of the most complex and interesting puzzles on the internet: how to predict a person's future favorites in any domain of life (social, career, products, services, media, etc.).

We have developed a completely novel technology, a cross-pollination between Bioinformatics and the Semantic Web. If we are right, it will make such predictions cold, rapidly, without tracking a user's behavior, without data mining, without a lengthy registration process, and in any domain. It should do so without many of the drawbacks of existing recommender technologies.

Due to the nature of the technology, it is not possible to perform a simulation or testing using Netflix data. We must build a live proof of concept with real users and real objects to match them to. This will require a few Facebook applications and some efficient algorithms.

We hope to evaluate the technology to academic standards in order to demonstrate that it works. Join us and help us solve this puzzle - any and all relevant skills are welcome.

Changing the World: The Business Model

2008-06-18T01:39:00.014+03:00

What is the business model for the next internet revolution? In this article, I review web monetization issues, especially that of web 2.0. I propose a monetization solution where any site with users, commercial items, and even visitors, can significantly increase its revenue and reduce marketing and advertising expenses. Our affinity targeting system monetizes itself in the process.

Traditionally, business models for web applications, communities, blogs, etc. are an afterthought. Apps and networking sites dream of reaching critical mass and then selling to Google, Microsoft, Yahoo, etc. Thus the revenue model is actually an exit strategy. This dream has been fueled by the observation that the purchase price for such sites is related to reach ("eyeballs", size of the audience). This is reminiscent of Metcalfe's Law. A more thorough analysis of the market value of social networks was recently posted in TechCrunch, by Michael Arrington.

A very few fortunate web startup founders do not need to consider a business model beyond their big exit, even in the current economic climate. The new owners, however, will be forced to monetize their sexy new purchase. For the vast majority of web startup founders the business model will be important and is often considered and tested from the very start.

The default monetization method is advertising, preferred by 58% of web startups (this figure includes affiliate marketing) according to Bizak. Of the strictly advertising sites, Google's AdSense is adopted by 54%. I imagine this number is higher for web 2.0 social sites. Nonetheless, AdSense earnings per visitor (EPV) are the lowest among the various monetization methods. As an example, Tom OKeefe writes about Mahalo's poor Google AdSense earnings, and Allen Stern predicts that affiliate revenue could surpass Google AdSense revenue for Mahalo in the long-term. Decrying AdSense as "worthless", Tom OKeefe asks "What's Next?".

Many of the hugely popular sites are struggling to better monetize. YouTube, for example, is struggling to justify its $1.65 billion purchase price. Also, Facebook faces a rough road ahead, with "only" $150 million in ad sales in 2007 and projections of $265 million in 2008, and Aidan Henry proposes solutions to the "perennial debate surrounding Twitter's revenue model", and the CEO of Mahalo, Jason Calacanis, even chimes in with his own Twitter business model suggestions.

This struggle may no longer be necessary. Our novel Affinity Targeting technology allows a user to be targeted to entities they are most likely to appreciate, in any domain of life. On-line communities and sites with users can increase their earnings by adding both their site and users into our system. Those users are then targeted to entities of interest (products, services, media, jobs, sites, other users, etc.). Targeting, leading to a commercial transaction, will result in affiliate revenue, part of which is shared with the originating site of the purchasing user. According to Bizak.com, affiliate earnings per visitor are 16 times greater than AdSense earnings. An affiliate model with highly specific user targeting should increase such earnings significantly.

The benefits don't end at monetizing eyeballs; sites and sellers can precisely target users to themselves and their items, thereby increasing sales and reducing costs. Communities, groups and fan clubs all seek to attract enthusiastic members. In our system, users will be targeted to the communities they are most likely to appreciate, leading to increased membership and customers. Also, sellers and providers benefit by precise targeting of users to products and services they are most likely to purchase. This will increase sales, and reduce dependence on marketing, advertising, SEO, etc. All sellers and providers are required to do is profile their products, services, jobs, etc. for the system (in the unique way we need the info) and agree to our affiliate model. There are no other costs to them.

The figure above (click to enlarge) depicts a solution to several critical needs: internet sites and sellers must increase their revenue, reduce expenses, and attract the most ideal new users or members. In our solution, sites and sellers add their existing users (no private information is required) and/or items into the system. Users are then targeted (via the targeting engine) to three different kinds of entities (circles): other users (if they are so inclined), groups (sellers, sites, communities, etc.), and items (products, services, media, jobs, etc.). When a user is targeted to a commercial item and makes a purchase, the seller provides an affiliate fee to the system, part of which is shared with the group that brought the user to the system. Also, if a group added an affiliate item into the system that they are not directly selling (for example, an Amazon.com book), part of any affiliate fee earned from that item is shared with that group. Follow the green arrows to see the flow of money. Note that sites and sellers may contribute users and/or items, and users and/or items may be entered independently of a site or seller.

Our plan is to grow the system organically by bootstrapping it on FaceBook and OpenSocial. We will do this in a way where critical mass never becomes a significant issue. At a certain point the affinity matrix of objects will be large enough to attract sites, communities, sellers and providers. At that point, we will offer our own API, customizable web interface, or client software, such that a site and its users can interact with the system the way the site sees fit. In the beginning we will use existing affiliate and payment processors, but eventually this will likely be done with our own systems. Our affinity engine and business model represents the ideal win-win solution for sites, sellers and users: better targeting, discovery, user satisfaction, monetization, reduced expenditures, etc. Ultimately, we see this targeting system attracting a significant fraction of on-line sites, communities and commercial entities.

The Affinity Graph

2008-04-25T21:31:00.004+03:00

Is the Affinity Graph the anticipated Internet Singularity?

Tim Berners-Lee, the father of the World Wide Web, has been talking about this concept of the future "Internet of things." By "things" he means the people and other objects on the internet, and he argues that those things and the connections between them are the key aspects of the web. This, he argues, is the primary evolution of the walled gardens of "Web 2.0" into something far more important. He calls this evolution the Giant Global Graph, while others call it Web 3.0 or the Semantic Web.

The use of the term "Graph" has been met with a bit of consternation by those who argue that we already have the term "network" to describe these connections. Robert Scoble describes the difference in reference to social relationships where your social network is who you know, while your Social Graph describes who you are associated with based on common objects of interest (passions, concerns, politics, religion, work, school, etc.). He says: "The Social Graph is NOT my social network. My Social Network is my friends list. But the Social Graph shows a LOT more than that." A Graph then is not simply the simple connections, but the types and context of connections and the strengths of those connections.

While the Graph will ultimately know what is currently song #3 on your iPod, some metadata about the song, as well as all the other people who have the same song as #3 on their iPods, one must wonder "what's the point"? How does this help me discover that I should be a dolphin trainer, or to find new people that share my way of thinking? Once the monstrous amount of data on the Graph is accessible to robots, many will be applying data mining and filtering algorithms, and massive amounts of CPU, to try to generate usable information about the people and other objects on the web.

Tim Berners-Lee envisioned the ability to create "intelligent agents", sort of like advanced email filters, to perform many of the more tedious tasks, easier and faster. I talked about a similar kind of agent in the post "Your Identity Proxy". Real progress will be achieved when future technology will be able to offer the users a much more personalized and enjoyable experience, and of course better targeting of those users with commercial objects. In practical terms, this will require the storage of as much data as possible about users and their objects so that futuristic computer programs will be able to make sense of the identities of those users and the meanings of those objects, and also to make predictions about the basic affinities between the objects and users. Some even predict that given enough information, "the machine" will begin to transcend the metadata and attain a kind of sentience (or sapience).

This is similar to the ideas of Gary Flake who hypothesized that continued advancements in networked information and other technologies will create a "virtuous cycle" leading to what he terms the "Internet Singularity". As with the Global Graph, we are far from advanced enough technologically to see these concepts realized in the near future.

Let me propose that both the Internet Singularity and the Global Graph are overlapping concepts that are largely achievable today through the Affinity Graph, a major element of this project. As of late 2007, we have had the technology to begin to store the affinity relationships and strengths between users and all other objects on the internet and mobile devices. This is a much simpler abstraction, where we store the most important kind of meaning (affinity) for the typical user. In other words, the most important benefit of the Graph or Singularity, e.g. searching, personalization, and discovery, can be generated, stored and queried in a much more feasible way than is predicted for the Graph, Semantic Web, or Singularity.

With the Affinity Graph, the similarity in meaning of objects, including people, will be known. Universal categorization, classification, hierarchies and affinity matching will all be made fairly trivial. Users will have immediate access to their future favorites in every domain of life; likewise objects (and those that care about them) will know which users are likely to most appreciate those objects (marketers? advertisers? evangelists?). This is the point at which the Utopian dreams of internet visionaries is realized. The Affinity Graph does not make irrelevant other forms of abstractions or metadata upon which computer scientists are free to set loose their strong AI. There are many other kinds of meaning, and those will be explored by computers in time.

The Singularity is here, as is the Global Graph, in ways that are most important to users.

Big World, Short Life

2008-03-19T12:03:00.002+02:00

The world is big and life is short. We've solved this problem.

To restate the problem: in our short lives, we are unlikely to ever find the people and things that we would most enjoy and appreciate. This is unfortunate.

Have you been feeling the pain? Not finding your soulmate? No best buddies? Have the suspicion that the most incredible music is out there, somewhere? Feel like you never found your ideal vocation? Actually there is little chance you could have found the optimal things in life. As I mentioned in a previous post, it would take us thousands of years to meet every other human, listen to every song, read every book, evaluate every vocation, etc.

Many of us have grown to accept our mortality, and the tyranny of time. We've had to accept the limitations in the time we have to explore options and find those optimal things. This acceptance has silenced our normally inquisitive and innovative inclinations to find solutions to problems; it seems an insurmountable problem, and, frankly, dwelling on mortality is not entirely pleasant. Those who haven't accepted mortality will deny the existence of the problem and thus the need for a solution.

I didn't set out to solve the 'short life' problem. Actually, that's not entirely true - I'm a huge health and nutrition nut: I plan to be healthy to at least age 120. But in this post, and in this project, I'm not talking about extending human lifespan. It is the 'big world' problem that we are addressing, and the problem may not be so big after all. The innovation came first, and then it occurred to me that the thousands of years it would take to find your favorites could be compressed significantly.

I'll use the analogy from a previous post. Many of us receive hundreds of emails every day. Without an email filter, it would take us hours to sort through and pick out the emails we prefer. We don't have enough time to perform this task, nor would we want to. The email filter, if it works well, presents to you only those emails that you are most likely to prefer. Reviewing your emails becomes a much quicker and simpler task. Information overload is reduced.

In a similar way, our discovery engine sorts through thousands of years of people, media, opportunities, ideas, causes, products, etc, and presents to you only those things you are most likely to prefer.

So you need no longer fear your own mortality. :-)

Your Identity Proxy

2008-03-12T18:35:00.001+02:00

There seems to be a bit of confusion about the distinction between the terms "identity" and "identification" in popular discussion. The terms are often used interchangeably, and are used differently in different contexts. I thought I would write a bit about these and other related concepts, including a new concept that we introduce.

The terms are infused with the complexity of multiple disciplines (philosophy, psychology, sociology, neurology, religion, etc.), each with their own usage and take on the meanings. To add to the complexity, identity is now an important concept with different meanings for government, commerce, and the internet.

Who are you? Are you different from your neighbor? From your identical twin? Is there something about you that distinguishes you from everybody else? The subjective versions of this are the "self-image" (a person's own model of his identity) and the identity perception of someone by others. Is it "the self" or the the ego of psychology? Is this the "soul" of certain faiths? Is it the mind? The brain? What about the body? Is identity a product of nature, nurture, or both together? Many questions.

Most of us cannot be relied upon to accurately describe our identities, though sometimes best friends can get pretty close (we get closer, see below). This is the reason metadata contributed by users about themselves or their works is not considered accurate. A personal tag cloud is just an ego trip. It is highly subjective. Web page meta keywords are no longer relied upon by search engines or advertisers because they are so inaccurate. This is the source of the delay in the promised "Semantic Web" revolution.

I like to think of Identity as that mental thingy that distinguishes you from every other person. It is the objective, non-corporeal entity that is the sum of all the biology and environmental influences that constitutes what it is to be you, at this moment in your life. Despite the similarities, you have a different identity from your identical twin because your minds and bodies have had different experiences. You also have a different identity than yourself of one year ago because you've had new experiences... and of course your brain has suffered some oxidative degeneration ("vegetable oil", anybody?).

But in the real world, and for the purposes of government, commerce, most things that make the world work, it is the corpus that counts. You are you because you are contained within the body of you. Identity equals body. The body that is recognized as you by facial recognition, and authenticated by fingerprinting, retinal or corneal scanning, etc. Science fiction has enjoyed this mind-body identity confusion with numerous examples in movies and television ("This body is not mine, and I have to be clever to convince my friends of my true identity").

Now, identification is the assertion that you are actually you ("I may look like a fly, but it's really me!"). Having the face of Nancy is an assertion that you are Nancy, i.e. your friends and family will identify you as being the identity they call "Nancy". Identical twins and masks can confuse the identification in opposite ways.

One can authenticate their identification with some available mechanism that provides some level of authentication. Visual similarity to your picture ID card (is ID "identity" or "identification"?) is a common form, voice recognition on the phone is another common one; "Hello, it's Nancy" works only if you sound like Nancy. We can authenticate the body fairly well, but the mind is more difficult ("Nancy doesn't seem like herself today. Maybe she's been taken over by an alien.").

On the internet, there are various uses for the terms identity, identification, authentication and anonymity. Your Facebook profile is a reflection of your identity, or an exhibition of your identity, most probably with identifying elements like your name and photos. In some cases you may have multiple online "identities" representing different facets of your actual identity. Those facets are sometimes identified by usernames and avatars indicative of the identity or sub-identity or idealized identity they represent.

For a new user, identity may initially not be important: an anonymous user is self contained, requiring no identification or authentication. But as other users get to know that user, they will expect that it is consistently backed by the same identity. As it develops a reputation, the identity behind that user identification will want to maintain exclusive ownership of that identification, via some kind of authentication that ensures such exclusivity.

There are many systems for authentication, each attempting to ensure that the user instance is an active reflection of the same identity. Online banking is an example. There are two levels of authentication here. First, the owner of the username is the identity called "Nancy" with these identifying personal details. Second, that the username instance (i.e. the just logged in identity) is also the "Nancy" identity (access management). The first is corpus related: Nancy walks into her bank and gets her login details based on corpus identity. The second is mind identity: does Nancy remember her username and password, or where she scribbled them?

Our project introduces another concept to the scene: the identity proxy. In our case, it is an objective proxy of your identity that makes choices on your behalf, likely the same choices you would make, even when you are not logged in. In a sense, it is like an email filter that follows your instructions and helps you deal with information overload by automating that small bit of your identity that prefers certain emails over others. Ours is much more powerful in reducing information overload because your identity proxy automates the filtration of all available information and options, in every domain of life. Your identity proxy is an accurate and objective reflection of your identity and it understands and automates your decision making processes. There is no greater weapon against the tyranny of choice and information overload.

Without an email filter, it would take us hours per day to delete the spam and read the relevant emails. We would quickly lose patience and only find a fraction of real emails. Likewise, it would take us thousands of years to meet every other human, listen to every song, read every book, evaluate every vocation, etc, in order to find the ones we like. It's a big world, and, sadly, life is short. The identity proxy does not live our lives for us - it makes our lives richer by allowing us to find those things that we wouldn't have found unless we lived for thousands of years.

Also, at it's core, the identity proxy requires no corpus identification, i.e. no personal or demographic details are necessary in the registration process. Nobody can use the registration information to track you down (track down the corpus). Privacy is intact.

Your identity proxy is singular. Having more than one identity proxy is a waste of time because every time you register accurately the system should see you as being identical (or close) to your previous proxy. Registering inaccurately serves no purpose because the proxy will make choices that do not reflect your identity, and the choices will not be as fulfilling for you.

Counsel of the Wise

2008-02-18T01:58:00.002+02:00

One of the consistent recommendations to innovators is to seek the counsel of some very smart people who understand the field of the innovation. One must be open to objective critical feedback in order to determine if the innovation should be developed. Initially, I started with family and friends, but recently I sought the feedback and input of academics and businesspeople with whom I was not formerly acquainted. So far the response has been very positive. In fact, four of those smart people have volunteered to become members of the project's board of advisers.

In addition to all of the great feedback from those meetings and follow-up discussions, the board has another important function - as references. Anyone who would so easily dismiss my claims of a novel disruptive internet paradigm, or the "preference engine" technology, must first reconcile the existence of this volunteer advisory board. My hope is that the added legitimacy of the board will further my contacts in the field, and especially among expert developers and angel investors. Here is the current board that can be contacted for references:

Avi Noy, Ph.D. [Research Page, LinkedIn]
Adjunct Lecturer, Graduate School of Business, University of Haifa
PhD, Information Systems, The University of Haifa
MBA, Technion Institute, Haifa
LLB, The University of Haifa
BSc, Computer Engineering, Cum Laude, Technion Institute
Doron Erblich [LinkedIn]
CEO, Noya Software, Haifa
Center of Computing and Information Systems (Mamram), Israel
BA, Computer Science, Open University, Israel
Yosi Dagan, MBA [LinkedIn]
MBA, Heriot-Watt University, Edinburgh, Scotland
BSc, Electrical Engineering, Ben Gurion University
Yuval Dan-Gur, Ph.D. [Work Page, LinkedIn]
VP Engineering, RADA Electronic Industries
Winner of the Israel National Security Prize
PhD, Information Systems, The University of Haifa
MsC, Industrial and Management Engineering, Technion Institute
BSc, Electrical Engineering, Technion Institute

The Serendipity Revolution

2008-02-12T21:33:00.001+02:00

Traditionally, the success of recommender systems is evaluated by predicting accuracy of recommendations off-line using existing datasets. For example, see the million dollar Netflix prize for a meager 10% improvement of their collaborative filtering algorithm. Netflix provided access to 100 million of its customers’ movie ratings to train new algorithms and test them. In other words, the algorithm is judged more accurate the more it recommends movies the user has already seen. Recommendations based upon this traditional accuracy metric are not the most useful to users.

Researchers know that success of recommendations is better measured by recording user satisfaction - the positive emotional response at having discovered something new that one likes. But that is more difficult to measure - as it requires a community of users and a useful mechanism to compel (or at least strongly encourage) the reporting of satisfaction, it's strength and perhaps type. Satisfaction of recommendations seems to follow in ascending order of the following recommendation types:

Low quality, low accuracy recommendation. Users obviously don't appreciate having their time wasted in evaluating something that the system should have known the user would not be likely to appreciate. These are "trust-busters"; the user will lose trust in the system.
An accurate, but known recommendation. An item the user is already aware of. The user likes the item, but it is not novel. Trust is maintained because at least the system recommended something that the user already likes. Too many of these recommendations imply an excess number of false-negatives or "missed opportunities".
A novel, but obvious recommendation. A novel recommendation is something new and appreciated, but something the user would have discovered on his/her own. For example, a new song from a favorite musician, or a new movie from a favorite director. The user will have a positive, though muted, reaction. Many users will suspect that there were "missed opportunities", given the huge number of unfamiliar items in any domain.
A serendipitous recommendation. A serendipitous recommendation is something new, non-obvious and appreciated that the user would likely not have discovered on his/her own. For example, an unfamiliar song from an unfamiliar musician, or a unfamiliar movie from an unfamiliar director. The user will likely have a very positive reaction, though it has been argued that, in some users, such recommendations may be seen as obscure and not immediately appreciated.

The serendipitous recommendation is obviously the ideal for most users, the problem is that collaborative filters tend to focus on what is commonly known and popular - items that the user has heard about or items that the user would have experienced eventually because of their "blockbuster nature". Many of the most interesting items for the user may be buried in the "long tail", so some collaborative filtering systems have attempted to tweak their algorithms to try to maximize this type of recommendation by reducing the more popular recommendations. Even so, recommendation diversity tends to be reduced in collaborative filtering systems, leading to a large number of false-negatives or "missed opportunities".

Recommendations based on a user's core identity will not focus on the popular, or items from artists or directors the user likes, or that the user's friends like. Instead, the user will be recommended items from the entire item landscape that by definition the user is most likely to appreciate based on that core identity (their "preference engine"). Thus the recommendation diversity (coverage of item space) within a domain (such as music) is as large as the diversity of items within that domain, leading to a large number of serendipitous recommendations - possibly the vast majority. Keep in mind that the number of domains in our community is also unlimited, and the same core identity can be used to recommend anything and everything in life.

Forget Everything You Know

2008-02-04T23:06:00.009+02:00

I'm getting a relatively large amount of email feedback, now that I'm putting a little effort into PR - and much of the feedback reflects existing biases. Recommender and social technologies have largely plateaued, with new innovations being relatively minor and more fine-tuning in nature. The possibility that something is being developed that is completely new, disruptive, and significantly better than existing services is not something many in the field are willing to consider. Admittedly, the secrecy of this project has not helped to convince people of that possibility, but this emotional rejection often takes the form of certain mistaken assumptions.

So I'll make another attempt to preempt the preconceptions and misconceptions with a discussion of some of the mistaken assumptions, both technical and paradigm-related, from the feedback. We can call this the Frequently Made Assumptions (FMA) list:

There are so many social networks, nobody wants another one.: Our system is not a social net, but one effect is that it creates a completely new community paradigm. Immediately after registration, we can quantify a person’s relationship to every other person, item, idea, endeavor, etc. in the system. Those relationships emerge and are presented to the new user as options and suggestions that then leads to further exploration, enjoyment, socializing, purchases and perhaps community involvement. So you don’t NEED to have (or get) your friends or contacts on the system to derive value.
There are so many recommender sites, and the results are lukewarm.: Instead of looking at your past choices and making certain assumptions, we are looking at the CAUSE of your preferences. We call this your "preference engine" which yields results of far higher accuracy and quality than existing recommender systems. Much of the recommendations will be serendipitous and cover the "long tail" as well. We don't care what your friends prefer, or what your past choices were. We know what you will like - from the inside.
"Web 2.0" stuff like this has no business model beyond advertising. I can't see any concrete business or how you want to generate revenue from your invention.: Imagine that every user has immediate access to their future favorites of: people, music, movies, books, recreation, groups, products, services, ads, travel destinations, vocations, jobs, teams, politics, religion, ideas, websites, articles, news items, games, etc. Highly successful affiliations of all types, direct sales and downloads, and highly targeted advertising would be the obvious business model components. Note that even advertisements are objects in our system, and can be targeted with person-level granularity.
No idea is new, no matter how good it is, at least 50 other people have thought of it.: Show me.
The future of the internet is the Semantic Web, haven't you heard?: Yeah, I've been hearing that for a long long time. The semantic web is too primitive and highly problematic. Certain aspects will be adopted by some, but there will be no utopia there.
Are you trying to promote just an idea?: We have the paradigm, a test site, and three patent applications. The test site is the first part of the application, which I am developing in Ruby on Rails. It is operating currently and in the process of being tested by select individuals. I am trying to form a team to develop the second part, which will require some sophisticated social software development and efficient algorithms programming.
LinkedIn (or some other site) is doing the exact same thing!: No, LinkedIn is not doing anything even remotely similar. If it’s not clear what our system does, read on.
Why would visitors want to come to your site or spread the word?: Again, imagine that every user has immediate access to their future favorites of: people, music, movies, books, recreation, groups, products, services, ads, travel destinations, vocations, jobs, teams, politics, religion, ideas, websites, articles, news items, games, etc. The system itself acts as a good friend that knows you best, and understands all. A personal and trusted relationship will develop with each user. Users need not reveal any personal information, so privacy is intact. Add these to the benefits of existing recommender systems and social and professional networks, and I think you will agree that popularity will not be an issue.
Nobody likes unlimited connectivity, you are creating a flood of people/info that will overwhelm users.: This is about reducing information overload and the tyranny of choice – it’s not about creating unlimited connectivity. It’s about giving the user access to only the things he/she is most likely to appreciate: people and every other thing in life. This is the holy grail of internet futurists and marketers, etc. There will be no unwelcome flood – only welcome and surprising options.
People only want to interact with their primary network. They don't want to be bothered by strangers.: First, we are not trying to be another Social Network. Social Networks separate people as nodes or degrees of separation. Separation is not good if it keeps away ideal relationships. But removing those separations, while preventing a flood, requires new kinds of filtration. A user’s potential ultimate friends and romantic partners, ideal business partners, etc. most likely lie far outside of one’s immediate network. Our novel discovery techniques provide the filter, thus there is no flood – only the most ideal are presented as options for the user.
Are you just creating another matchmaking site?: Objects in our system also include people: we expect matchmaking to be a significant part of the interest in our system. But remember that people can also be matched with every other kind of object, so the system is much, much more. Also, user-user matching can also be used to discover friendships, business partnerships, roommates, travel buddies, etc.
Are you just creating a technology to sell to the Amazon.coms of this world so they can make better business?: The discovery engine is not separable from the community. The community must be created (possibly by aggregating or integration of existing communities), but it’s not a social network, nor just a recommendation engine. It’s a holistic and completely new paradigm, and it is destined to change the world.
Are you just creating yet another standalone dotcom, or will this be a feature of other services? Folks are fatigued with so many separate services.: The discovery engine is not separable from the community, however, the community need not be a standalone data island - the format is ideal for integration of multiple services. But even as a standalone entity, the novel paradigm will make it quite popular. Others will want to integrate and take advantage of our universal recommendation system.
You must be constructing a commonality map. This is not new.: There is no “commonality map”, no use of shared things to relate people. This is not a people-centric or shared object-centric social model, as everything within our system is an object. It is beyond the "object-centered" or "people-only" debate. You’ll have to put aside your knowledge of existing methods and community/recommender concepts in order to understand the new disruptive paradigm.
You must be using data mining edge activity to produce (more) accurate node profiles. This is not new.: Our system does not involve data mining. Data mining generates very poor quality information compared to the kind of information we get. As I said, ours is something completely new.
You must be using existing algorithms and filtering methods. Collaborative filtering? Artificial intelligence? Content Based? Horting?: Ours is a completely new paradigm, not horting or any of the other existing algorithms or systems. It is not a collaborative filter of any sort. It does not rely on personal or demographic data and it is not content based, requiring feature analysis. It is something completely novel.
Registration involves Psychological testing and interpretation.: Psychologists will be very interested in what we are doing, but we are not using Psychological testing and interpretation. There are no Rorschach inkblots or squiggles and shapes, no questionnaires, no puzzles, etc.
Registration involves some sort of metaphysical or astrological process.: No pseudoscience, metaphysics or mysticism, I promise. It is a completely logical and quantitative process. Hard to imagine, I know, but I assure you it is quite legitimate. In the future, PhD theses will be based on it.
Registration involves personal and demographic data used to enable audience segmentation.: No personal or demographic information needs to be entered at any time. Audience segmentation is a severe limitation to behavioral targeting and recommendations. Our system provides object-level granularity.
You fill in a sparse matrix to infer missing relationships and preferences.: The matrix is dense from the very start. No missing relationships are inferred and filled in.
Core identity is impossible to determine, and certainly not in a quick registration.: Don't judge what is impossible if you don't understand it. I would be happy to arrange a meeting with you to demonstrate how it works. If you are not willing to listen, then you deserve the box that imprisons you.
How can you derive someone's essential identity from a name and password?: Now THAT would be magic! Of course the answer is that we don't. There is a quick one-page registration and we get their essential identity from this - more than you can possibly imagine. But this is not the essential innovation, only a very cool result.
Why universal? Wouldn't it be easier to stick with a single vertical domain?: Our system prefers diversity, the more object types the better. It can operate on only one domain, but this would be a huge waste. Don’t forget, everything in life can be discovered, separately or simultaneously. The difference in complexity, between one domain and universality, is not great.
How are you going to compile a database of all the "objects" in the world? This is no trivial task!: Our staff will not have to compile anything (as with Pandora), nor are we relying on users to enter feature data. Whatever you're thinking - that ain't it. We are using a completely novel approach.
No matter how you do it, reaching critical mass is gonna be a bitch.: Ours is a slave to the network effect like all the others. However, as long as the system is sufficiently populated with objects of any variety (people OR things), the system will begin to work. In other words, with the right conditions the first user can see benefit. The obvious advantages over social nets, recommender applications, and the multitude of vertical applications should spread the word quickly. Most agree that popularity would not be an issue.
You don't reveal the "magic" on your blog, so the project must be hype and at least appears not to be serious.: First let me apologize for the lack of technical info in this blog. Certainly this will be off-putting to some, but I assure you that it is very serious. This is not an academic project - it is business, and the lack of technical information is simply a necessity for protecting confidential information. In the world of internet start-ups, innovations are rarely disclosed publicly before the first public beta site. Patents offer some protection, but there is no replacement for a healthy maintenance of secrecy at this early stage. I am happy to disclose the technical solutions to appropriate individuals under confidentiality assurances. Contact me at the email address in my profile to arrange a meeting in Israel or the States.
Why be so stealthy - this is not the Manhattan Project!: If this really IS the next internet revolution as we suspect, then we shouldn't treat is as just another cool web app. We’re planning to change the world.
I can't see the purpose of this blog.: Consider the articles in this blog to be teasers for the purpose of attracting interest from those with vision and curiosity. We are looking for investors and hackers (highly competent programmers and developers). Do you know any?
If the system performs as you say, then it will clearly be disruptive. But these claims are a bit extraordinary... why should I take time to arrange a meeting to learn more?: Everyone's gotta gamble with their time - weighing the potential upside, probabilities, and time spent. I know the frustration of wasting my time hearing a stupid idea. But natural curiosity keeps me listening, just in case. Contact me at the email address in my profile to arrange a meeting in Israel or the States and let me change your world. In fact, invite a group of the smartest web-focused people (including angel investors and hackers) you know.

Skepticism vs. Cynicism

2008-01-30T11:53:00.000+02:00

I just received a response email from a professor at a large American university:

"I'm sorry, but your presentation of your idea makes it sound like the equivalent of a perpetual motion machine, as you promise a secret ingredient that will figure out the core of who an individual is without essentially no data about them.

I wish you good luck, but I'm afraid I'm not willing to invest any further time in exploring your ideas. "

Now, I appreciate feedback and certainly don't want to discourage it - but I thought this was a good opportunity to review some of the objections to this project and the limited technical information disclosed. What can we draw from the above email?

A perpetual motion machine would be revolutionary if it were true, and so would this project.
A perpetual motion machine cannot exist because it violates the laws of physics, and so would this project violate our expectations about what is possible.
Anyone proposing such a revolution/violation is not worthy of further consideration, and the ideas are not worthy of further exploration, even a meeting would be a waste of time.

I am a skeptic myself, and so I certainly appreciate skepticism, but our project does not violate any laws of physics. It sounds fantastic, but I assure you the algorithm is sound. The professor claims we have "essentially no data about them [users]" - though obviously we know a huge amount about a registering user. The difference is that we get information that can be used to quantify relationships, and the information can be gathered during a quick one-page registration. Teaching the machine about the "core of who an individual is", i.e. a person's 'preference engine', has never been done - and if this professor can't do it, certainly nobody can. Meetings with three local professors in the last two weeks produced the expected jaw-dropping effect, and because of them I may now have a linear algebra option for the core algorithm, though I'm still not convinced.

Anyway, this emailing professor has chosen not only not to meet, but seems annoyed at having been exposed to such ridiculous suggestions. I also understand a certain amount of frustration that the technical solutions, enabling full evaluation, are not disclosed (though I doubt the emailing professor would have read that far). Readers must also understand that technical solutions cannot be made public, despite patent-pending status. There are too many horror stories about disclosed innovations being stolen by large, well-lawyered corporations. I am willing to meet, traveling considerable distance, in order to satisfy that skepticism. Vision, curiosity and open-mindedness should be sufficient for a meeting, and perhaps this is a useful filter: why would I want to associate with those without these traits?

One Degree of Separation

2008-01-14T12:13:00.001+02:00

Social networks rely on your primary network - your existing friends and contacts - to introduce you to THEIR friends and contacts. Each of the people in the network are called 'nodes', each with one or more connections to other nodes. Each of those connections is sometimes called a degree of separation; a friend of a friend (FOAF) would then be two degrees of separation. The famous phrase "six degrees of separation" was based on work by psychologist Stanley Milgram who determined that any two Americans, connected in the nation-wide extended network, are separated by an average of five intermediaries, i.e. six steps or degrees.

Despite their connectedness, two people separated by so long a chain are extremely unlikely to ever meet. In fact, we usually only ever meet the friends of our friends: an extremely small fraction of the larger network. Web services like LinkedIn, the business contact network, tracks your chain to three degrees of separation - though I wonder how often the third degrees ever connect. [Friendster tracked the chain even further, and this pursuit has been credited with Friendster's downfall, as tracking long chains is very difficult computationally and has much larger hardware requirements.]

Online Social Networks are not really social, and the network - as degrees of separation - serves mostly to separate. So, if one really wants to 'kill' social nets, one needs to get rid of the 'net' (the multiple degrees of separation that separate people) in order to bring people together. Jyri Engeström argues that social networks should not be based on individual connections between people that can be counted and accumulated, rather people must be connected by shared objects. We agree and take this to the next level by making everything in the virtual community an object, where each object is connected to every other object.

The New Paradigm

As proposed in the last post, what is lacking in the current data islands and the proposed schema solutions is a way of harnessing the true power of the collective to actually reduce information overload and increase discovery. This will require a revolution in content and relationship discovery that can only arise with a completely new kind of information filtration and recommender technology.

"The social web will be powered by recommender systems".
Open Issues in Recommender Systems
John Riedl, Bilbao Recommenders School, 2006

The true power of the collective will be realized with the proper integration of social media, new universal discovery techniques, and associated detailed portable identity and personalization info. The result is a Social Web based on one degree of separation: all people and things are related to each other directly, with each such relationship differing only in type and strength. The following graphic is a representation of such a "one degree" circle of people relationships, but keep in mind that each person is also similarly related to all items, ideas, endeavors, etc. in the system as well.

Critical to this new paradigm are the new universal discovery techniques that I've hinted at previously. Current recommender systems, including collaborative filters, are too primitive and limited to accomplish the task. Instead, we have applied certain bioinformatics concepts to solve the puzzle of simulating the human preference engine without requiring "strong AI". This starts with a quick determination of a person's "core identity", that internal mechanism which is responsible for generating appreciation, and sifting through the chaos and making choices.

Determining that "core identity" is a critical breakthrough as it allows us to quantify the relationship (strength and type) between all people, and between all people and all other things in the system. It also can yield portable data that can be used to quantify such relationships between users and items from multiple data islands, and can even be used in mobile devices and in real-world activity. This discovery system involves no collaborative filtering, psychological testing or interpretation, statistical or stochastic methods, etc.

"But there is no go-to discovery engine - yet. Building a personalized discovery mechanism will mean tapping into all the manners of expression, categorization, and opinions that exist on the Web today. It's no easy feat, but if a company can pull it off and make the formula portable so it works on your mobile phone - well, such a tool could change not just marketing, but all of commerce."
The race to create a 'smart' Google
by Jeffrey M. O'Brien, writing for Fortune Magazine

In addition to the current benefits of the social web, the integration of these universal discovery techniques will allow:

A brief one-page registration with no need for private information. Qualifies as 'Cold-Start' for people and also items, ideas, endeavors, etc.
Immediate access to promising relationships of all types, i.e. universal recommendations. These relationships are the predicted interest and affinity between a person and all other people, music, movies, books, recreation, groups, products, services, ads, travel destinations, vocations, jobs, teams, politics, religion, ideas, websites, articles, news items, games, etc.
Portable data that can be compared and relationships quantified. This portable data can be used between social and data islands, for mobile devices and in real-world activity.
No language or cultural barriers: no folksonomy or semantic constraints.
No need for existing relationships. Emphasis is on relationship discovery, though existing friends and contacts are revealing.
No need to observe history of actions and choices. A one-page registration is enough to provide significantly more information, and better information, than collaborative filters can accumulate.
The new system will act as a good friend who knows you well and delivers trusted recommendations of all types, both solicited and unsolicited.
Reduced privacy concerns as personal or demographic data is unnecessary.
Automatic person-level granularity. Each relationship has a strength and type.
Universal recommendations allows for highly successful affiliations of all types, direct sales and downloads, and highly targeted advertising as the diverse business model.
Ratio of discovery to effort is high. No need for constant messages, spam, requests, friend searches, etc.
Discovery is filtration, so 'information overload' and the 'tyranny of choice' are greatly reduced.
Enables highly personalized search engine functionality, news aggregation, and many other forms of person-level information filtration.
Constant excitement of discovery, so no "what's next?" reaction. No limit to novelty and interest, little boredom. No feeling of wasted time.
Highly useful and usable: the keys to success of any product or service.

Social Standardization and the Death of Social Networks

2008-01-08T13:33:00.000+02:00

"...we’re reaching an inflection point where some fundamental conceptions of the web (and social networks) need to change".
from Stop building social networks, by Chris Messina

It seems that everybody is predicting the end of something due to something else, typically calling the later a 'killer app'. Are VOIP and email replacing the phone and fax? Is social media replacing Google search, email, communication in general? Is IM replacing email? Well, who would have predicted that the trusty typewriter would disappear in the span of a few years? It seems many are making another prediction: Social Nets are on their way out, at least in their current configuration. In this post, I'll talk about the problems and proposed solutions.

Social Nets are hugely popular and are obviously doing something right. They were clearly a revolution in online communication and information sharing. Let's first list why people enjoy them. They allow you to:

express yourself and try to look cool
people-watch / voyeurism / "gawk at strangers"
'collect friends' and compete to see who has more
waste time doing semi-fun alone stuff with apps, etc.
keep in touch with existing friends (the primary network)
make new friends, dates and business contacts (the largely unfulfilled promise of the 'network')
manage your personal data
exchange knowledge and information
re-connect with old friends and colleagues

As for the negatives, here are some of the points mentioned on the blogosphere:

'Friend collecting' is not 'social'. No real communication takes place, and no real friends are made. Checkmarking someone as a friend is not being social. Not much relationship building going on.
Information Overload is not reduced, quite the opposite: too many people, messages, spam, etc. There is a limit to our ability to absorb information: our internal filters cannot handle it.
"There isn’t enough time in the day for any person to find value in what a 1,000 people have to say - our internal filters just won’t allow it. At some point all that information; whether it be valuable or just fluff, becomes nothing more than white noise".
from Enough with the social crap I think I’m gonna puke, by Steven Hodson
"Massive waste of time" / "It takes too much time" / 'Social Net Fatigue'
Privacy concerns / 'abuse of trust'. Services track user activity on and off the service, and post some of those activities to the "friends". Combining information from multiple sources may reveal private information.
Social nets are 'Walled Gardens'. They are not portable - information is trapped within the bounds of each service. New users must re-enter profile information, must search and re-add network contacts, and must reset notification and privacy preferences for each new social net joined.
Social nets are by definition 'network-centric'. Most users are exposed only to friends of friends (i.e. two degrees of separation). This presents an obstacle to discovering true friends and contacts, most of the potential being outside of your network.
No Business Model beyond popularity and possibly advertising. Also, because new users on social networks often misrepresent themselves and enter false personal information, demographic data for advertisers is therefore unreliable.
The "superficial emptiness"
The "what's next?" phenomenon (after exhausting the novelty of the site) / Lack of Innovation
Not granular enough - no ability to group friends and contacts in categories, or indicate how close or trustworthy those relationships are.
Tired of having to add friends or accept friend requests in all of these networks.
Use a given service only because that's where your friends are.

Proposed Solutions:

Many feel that Identity/Info concepts like OpenID, OpenSocial, FOAF, the 'Semantic Web', Microformats, have great potential in solving a few of the above problems.

"a distributed, user-centric identity scheme would destroy almost every "walled garden" social software application on the web".
from Identity Management Will Destroy Social Software, by Brian 'Bex' Huff

The idea is that each internet user would have a single universal and portable profile that would be used and understood by all services, thereby elimiating the need to enter and configure the same information and connections on every new service. Ideally, this would have the effect of removing the walls between services, creating a single large community or 'cloud' where "relationships transcend networks/documents".

The social and data islands that dot the internet can clearly be helped by some kind of standardized profile that can be uploaded to (and modified by) each service. The burden of registration and establishing relationships would be greatly reduced. Such a profile can grow to include all the data that a person might share, including photos and information, music, movie, web site favorites, etc. As long as all services agreed on standardization, this should work pretty well. As an example, browser standardization is largely successful - though differences do exist and can be frustrating for developers and surfers alike.

The Next Revolution:

Schemas, however, will not solve most of the issues mentioned above, and some are made worse (like privacy concerns). Some even argue that standardization and identity aggregation would not be entirely apprieciated. As much as schemas depend on FOAF information, most of the problems with social networks will remain. If one really wants to 'kill' social nets, one needs to get rid of the 'net' part, i.e. the degrees of separation. What is lacking in the current data islands and the proposed schema solutions is a way of harnessing the true power of the collective to actually reduce information overload and increase discovery. The next revolution in content and relationship discovery can only arise with a completely new kind of information filtration and recommender technology.

"The social web will be powered by recommender systems".
Open Issues in Recommender Systems
John Riedl, Bilbao Recommenders School, 2006

The true power of the collective can only be realized with the proper integration of social media, new universal discovery techniques, and associated detailed portable identity and personalization info. The result is a Social Web based on one degree of separation: all people and things are related to each other directly, with each such relationship differing only in type and strength. More on this new paradigm shortly.

Cause vs. Effect of Human Preference

2008-01-05T22:13:00.000+02:00

"One crucial unsolved problem for recommender systems is how best to learn about a new user".
Getting to Know You: Learning New User Preferences in Recommender Systems
Rashid, et al, 2002

"Success comes from understanding both data and people"
Open Issues in Recommender Systems
John Riedl, Bilbao Recommenders School, 2006

"The problem with recommendation systems is... it measures and acts upon the effect, not the cause".
Response to “UIEtips Article: Watch and Learn: Recommendation Systems are Redefining the Web”
Adam Smith, 2006

So far, the internet has been all about effect. What other people say they like, you might also like; what you liked in the past suggest what you may like in the future. Google does it with PageRank; Amazon.com and Netflix do it with their recommender systems. They act based on your, or others', past preferences (the effect) rather than the cause of your past preferences. As you interact with the web, applications can record your actions and choices in order to create a filter with which to formulate suggestions that you might appreciate in the future.

But this is not the way the natural social process of recommendation seeking works. If you really want a good recommendation you ask someone who knows you well, as an individual. This is the way good friends do it. We accept recommendations from good friends because they understand our core identity (hopefully) and have no ulterior motives (hopefully). For example, as a single guy, I will never again go on a blind date unless the intermediary is a good friend who understands my taste and my attitudes, values, personality, etc., as well as that of the prospective date. One could make an assumption based on my past dates and relationships, but it would be an assumption based on insufficient (see below) and indirect data: the effect rather than the cause.

What is the cause? Preferences do not appear out of thin air, they are a result of your core identity: some combination of nature and nurture, your genes and your cultural and social influences, the configuration of your brain. This is the direct cause of your preferences: it is your preference engine. Unfortunately, it is a black box that we cannot really open. Possibly in the future there will be a scanning device that can capture and replicate your precise neural configuration. With this copy, and sufficient understanding of the human mind, we might be able to accurately predict your choices. In making a choice, the steps are:

Core Identity + Exposure -> Preferences (i.e. Brazilian Supermodels)
Preferences + Availability -> Choices (Damn!)

Current recommender systems, such as collaborative filters, attempt to simulate a filter at the second stage. What we need is a way to accurately simulate your filter at the first: not quite a copy of your brain - but close.

Your preferences are also extremely limited by your limited exposure. Take music as an example. I love music, but I have only heard a tiny fraction of a percent of all music. So how the hell can my current favorites be expected to be entirely descriptive of my true taste or ultimate favorites? I have been exposed to that which is largely popular, better marketed, in English, etc. Music recommender applications suffer from this limitation: they consider only what I have already heard, and so they receive highly skewed data about my true taste. It would be great to have a good friend who is the ultimate "long tail" DJ and can match me to music based on his knowledge of my core identity and detailed knowledge about all music and musical tastes.

"Thus, the task is not so much to see what no one yet has seen, but to think what nobody yet has thought about that which everybody sees".
– Arthur Schopenhauer

It seems obvious that far better recommendations would result from an intimate knowledge of a person's core identity. But identity is mysterious and unapproachable; better left to the fantasies of pipe-smoking psychologists. In reality, it is the chain around the elephant's leg. We all have the tools to break free from the constraints of assumption, but smart people have not previously applied themselves to the task.

Current Recommender Types

2008-01-02T17:10:00.000+02:00

There are a number of types of recommender systems currently available. They vary significantly in their mode of action and ultimate user experience. In terms of results, recommender systems are expected to offer sufficient good quality recommendations ('New Favorites'). In addition to this, the quality of the results is also dependent on minimizing false positives ('Trust Busters') and false negatives ('Missed Opportunities'). In other words, users should also not be shown inappropriate results and should not be denied appropriate results.

The quality of the user experience is also influenced by the time and effort required to give the recommender system enough information to make minimally reasonable recommendations. Users are sometimes asked to fill out lengthy questionnaires, or applications require that a user's history of choices or ratings be observed and recorded. It takes time and effort before things start working well. These days, users don't like to wait for anything and expect immediate gratification - delivering instant results upon quick registration is called 'cold start'. However, existing applications that permit a 'cold start' lack anything close to sufficient information, explicit or implicit, required to make accurate, high-quality recommendations.

There are a number of strategies that recommender systems are taking today. These include:

Non-personalized: "Web 1.0" technology offering the highest rated or most popular items to all users. No intrinsic personalization, poor quality results, but immediate.
Demographic: Require some knowledge about the user in order to group similar users together (i.e. by age, gender, area code, other similar features). Poor quality recommendations, low personalization, though slightly better than the above. May require "private" information, and depending on the length of the questionnaire, registration can take time.
Simple answer or ratings matching: Matches users based on explicit matching of answers, selections, ratings, etc. Makes recommendations with extremely limited scope, many missed opportunities, requires answers or observations.
Heuristics, probabilistic models (Bayesian, Markov), decision tree, neural net, etc. An application must collect a large amount of user-item preferences, or user/item features before quality recommendations are possble. This approach attempts to identify the underlying logic (or apply certain assumptions, in the case of heuristics) to a user's choices.
User-based Collaborative Filtering: similarity of historical choices or actions allows the application to find highly correlated users. The assumption is that users who agreed in the past might tend to agree in the future. Limited immediate results, most items will not be rated/answered (sparsity). Users with non-typical opinions or taste (the 'long tail') may not get good recommendations.
Item-based collaborative filtering: Finds items that tend to be preferred together. Limited immediate results, and users with non-typical opinions or taste may not get good recommendations.
Content-Based: Find items with similar features (Keywords, author, genre, i.e. DNA) to known preferences of a user. Items must be properly and thoroughly represented as a set of features - this generally requires a large staff. Generally limited to a single domain as there may be few cross-domain features. Limited immediate results.

There are many recommendation engines and recommender applications available on the internet and many more seem to be popping up all the time. Currently they all have severe limitations and offer mediocre to poor quality results when compared to, say, recommendations by a best friend. Examples of current applications include:

eHarmony requires a very lengthy questionnaire and uses a proprietary empirical heuristic to match people romantically. It's success depends on the quality of the questions and the heuristic, the person's willingness to answer truthfully, and the person's willingness to spend a few hours to register. Mixed results are reported, but there is certainly an advantage over matchmaking sites that allow daters to make their own bad choices.
Pandora and Last.fm both recommend music though they do so in different ways. Pandora's large staff must determine the separable features ("DNA") of a song and observe a user's choices in order to extract common features of a user's preference. Last.fm seems to work by grouping users of similar taste. Both suffer from reduced choice diversity for slightly different reasons. Both are mildly satisfactory, but also suffer from excessive false negatives and false positives, and require recording your existing preferences. Two roommates using the same account will likely see poor results.
Amazon.com's recommendations work by observed a user's choices and activity and grouping items (books, CDs, DVDs, etc.) that tend to be chosen or viewed by the same users. After viewing or choosing items, you are presented with: "users who liked X (the currently viewed item) also liked Y (a correlated item). As may be considered a typical pattern, users who buy for multiple people, like for children or friends, will likely see poor results.
Social DNA sounds like it works similar to Pandora, but the granularity is significantly greater, and unlike eHarmony, there seems to be no heuristic - matching is all or nothing (i.e. explicit ratings and questions). This is expected to lead to extremely high false negatives, relatively few true positives, and, since matches will likely occur with only a tiny fraction of possible DNA (highly limited explicit information yeilds a sparse matrix), considering the complexity of human beings, mostly false positives.

In order to get relatively high quality and accurate recommendations, a large amount of explicit ratings/choices (and/or possibly implicit activity) must be recorded. This is extremely hard to do: users are less likely to maintain interest while the machine learns, and this will be increasingly true in the future. Currently, users must be content with mediocre results, but a trade-off will develop between accuracy/quality and user patience.

Another frequent limitation is that users can act maliciously or inappropriately to skew results. Due to the limitations of current applications, users may feel the need to modify or exaggerate their choices in order to get better results. On the other end, users who want to promote certain items to others may give or encourage false ratings, views or descriptions (called 'Shilling') through manual or automated efforts or attacks. Also, privacy becomes an issue as users may explicitly or implicitly reveal private information about themselves. Details include demographics, personal details, taste, ratings, opinions, etc. Systems administrators (and possibly hackers) will have free access to this data.

Accurate, high quality, robust and broad scope recommendations have been the holy grail for internet futurists for quite some time, though we are still a long way from that goal. The problem is largely technical: recommendations are a really tough problem. Mathematics/statistics, clever algorithms and artificial intelligence are stretching the results to the maximum, given the poor quality data available from users during registration or interaction with the application. The solution is to get high quality data about the user's identity or individuality and match based on that, rather than matching based on a user's history. The problem is that teaching the machine about the core identity of a person is science fiction. Or is it?

Live fuzzy search using n-grams in Rails

2007-12-23T14:32:00.017+02:00

Update [December 17, 2009]:

You can now see these live applications:

This post is the basis for a few Rails plugins:

no_fuzz - Simple Rails-plugin that provides offline fuzzy-search for ActiveRecord [see my fork of this as well]
act_as_fuzzy_search - add a fuzzy search to model for DataMapper
rails-fuzzy-search - and more recent fork of the above.

We wanted to give users the ability to perform a search. It is fairly easily set up a small search engine on a site or allow users to perform a search from a major search engine such as Google directly from the site itself. Our site needed something a bit more sophisticated and useful.

First, we wanted to allow users to start typing in the search box, and have the site suggest results from the database as the user types and display them in a drop-down list under the search box. This is sometimes referred to as auto-complete, auto-suggest or "Live Search". Rails accomplishes this typically through its JavaScript macros helpers that wrap the Prototype JavaScript framework and the Scriptaculous JavaScript controls and visual effects library. See also:

We needed some extra functionality: we needed a single search feature to search multiple attributes of a model and, upon selection, populate a number of form fields with a number of attributes and associations. It actually gets even more complex than this, and perhaps I'll write more about it in a future article.

Typically, a search looks for exact matches of what is being typed. However, many important results are missed when an exact match is required: people often misspell, and related words that share the same word root will often not be found. So we wanted to allow users to find things that did not match perfectly with what was being typed. Furthermore, we wanted the results to be sorted such that the better the match, the higher a result is located on the drop down list.

One option for Rails developers is to use Ferret, a high-performance text search engine library based on Apache Lucene. Ferret is a reasonable option, and we may eventually go this route, but we wanted something simpler that did not require installing a library - we only needed the fuzzy search capability. Ferret's code is largely in C, rather than Ruby, and its fuzzy search is based on Levenshtein edit distance formula which calculates the "minimum number of operations needed to transform one string into the other, where an operation is an insertion, deletion, or substitution of a single character".

Instead, we decided it should be rather simple to create an n-gram distance algorithm and leverage database indexing to create a fast n-gram distance fuzzy search, purely in Ruby. What is n-gram distance? It is the measure of the similarity of two strings by comparing the number or significance of matching substrings of length n. It is particularly useful when searching a database of strings, for example a directory of names, rather than lengthy text (though that may certainly be possible as well). Our focus will be on a database of strings.

As an example, let's take the most popular name in Europe, and two of it's spellings: Muhammed and Mohammed (there are also many other spellings). Typical live searches will not be able to help someone who has started to type one rather than the other, as the second letter is different. Live searches need at least the first two or three letters of the input to match. First, let's add a word start character (for code, we prepend a space) to the beginning of the words because word boundaries constitute useful information. Next, let's break each word into sliding frames of three letters (trigrams). Muhammed becomes: *mu, muh, uha, ham, amm, mme, med. Mohammed becomes *mo, moh, oha, ham, amm, mme, med. The two names share 4 out of 7 trigrams.

First, migrate a trigrams table with columns: tg (trigram, a string), person_id (the key in the people table, an integer), and score (an integer, though you may not need it). Make an index on the first two columns in that order. Next, generate a trigrams model. Then you must populate the trigrams table with trigrams from the people table. This is easy to do as a separate migration:

Here, we are are storing trigrams from each person's first and last names, and in this example, we are scoring matches to the last name at twice the score of the first name. This is just an example, you can score it any way you want. If you don't need to distinguish scores, you can eliminate the score column. Anyway, this migration will populate the trigrams table, and since the trigram and person_id are indexed, any search is easy on the database - and fast.

The autocomplete action method in the controller might look like this:

Do you see what's going on here? The entry is split into trigrams, then the scores of all matching trigrams in a person's name are summed. People are sorted based on their total score, and the top matches are sent to the partial. Very simple. The partial might look like this:

The ID is included, though hidden, so user selection of a person automatically includes the id of the person from the database table for further use.

It works beautifully, and it seems very fast so far - though with a huge number of items, database storage may become an issue. Any further ideas, suggestions, requests?

Adventures in Self-Referential Database Associations

2007-12-22T15:47:00.000+02:00

Rails makes object associations easy and has a number of different ways to configure associations and even store extra information about each relationship. Most of the time, those relationships are between business objects that wrap rows in two separate database tables. For example, let's say we wanted to be able to record, read and update grades and attendance of students in a particular school class. We might create a student model and a school class model, and the association between them may contain the grades and attendance information. We use many of these kinds of model object associations, and the procedure is fairly straightforward and written about extensively. See the following links to learn more:

For our project, we also needed a way to store various qualities about relationships between pairs of objects of the same model type, i.e. between rows of the same database table. We would need this for a number of different purposes. Those creating 'social net' applications know this kind of association well; they must document, for example, who are the direct friends of a particular user, and possibly further information about each such relationship. All those users are stored in the same database table, so a separate "join" table is created to store the keys of both users, plus any other information about the relationship.

To accomplish this, we used self-referential has_many :through associations that use a join model and methods to simplify the reading and updating of attributes of the association. To read more about these more specialized associations, here are some sites:

More details on this later.

Why Ruby on Rails?

2007-12-19T19:38:00.001+02:00

Ruby is a beautiful, purely object oriented, interpreted programming language influenced by Perl and Smalltalk, among others. Rails is a web application framework built on top of Ruby that is designed to speed and simplify the development of database-driven web applications. Rails is relatively new and not widely used relative to other web programing languages, but nonetheless its growth has been tremendous. Rails has seen so much growth because of its reputation as the fastest and easiest web development framework. Also, to achieve similar functionality Rails typically requires far fewer lines of code than the more popular web development languages.

When we first started out, we were fairly familiar with Perl and PHP - so it was a difficult decision to attempt to bring the team into the Rails community. We expected that the delay required to learn Rails would be later compensated by faster development. So we decided to learn Ruby and Rails and at least build the Builders' Site prototype with Rails.

The Israeli Rails community is rather small, but nonetheless I attempted to try to attract co-founders and developers for a previous related project through the israel.rb group. There are lots of startups in Israel, and I'm not surprised that demand exceeds supply of capable developers. At the time I was not able to interest any qualified Rails hackers to join the Israel team - part of the problem may have been that none of them knew what the project was really about, only that some bombastic American has a stealth startup.

Despite the lack of Rails experts on the team, the learning curve was not excessive and we were quickly creating some fairly complex functionality. The Rails principle of "Convention over Configuration" helps to speed development of more typical functionality. We encountered some difficulty when our functionality quickly exceeded the convention. Luckily there are some real heroes on the internet who publish their experiences with similar issues and try to help those trying to learn and develop along the same lines.

Critics of Ruby and/ or Rails argue that Ruby is slow and web applications developed in Rails do not 'scale'. Most of this criticism appears to be sour grapes. It is true that there are no blockbuster applications written in Rails, only medium sized applications, but reaching "medium size" exceeds the achievements of the vast majority of web startups. Rails defenders argue that scaling Rails applications simply involves purchasing more servers and hardware, and that the greatest limitations are database related - and all web development languages suffer from this. They argue that savings on developer-hours more than makes up for extra costs in hardware.

The debates continue, as does our development. Perhaps I will post some of the obstacles and specific solutions we found. So far we like Rails, and have no plans to go elsewhere. When we are ready to increase our development or scale the application, perhaps we will consider again all our options. The decision will, in part, depend on the availability of capable developers.

Why do we Patent?

2007-12-19T12:22:00.000+02:00

Now that we've discussed our thoughts on the next internet revolution, and therefore the goals for the project, let's skip ahead and tell you that the algorithms have already been worked out and three patent applications filed. We are patent pending on the algorithms, the various uses of those algorithms, and business methods associated with this project. But there are many who question the ethics and effectiveness of patents, especially patents on software and algorithms. The software industry has demonstrated patents aren't needed for innovation to take place. So why have them?

The first thing to recognize in any discussion of patents is that they have been and are essential to a profit-making business. Innovation can be a hobby, but it is also integral to high-tech business. Patents reward and therefore encourage innovation because the innovator is more likely to see the full benefit from the innovation and hard work and money. Algorithms, like any innovation, can meet the patenting criteria of "unique", "non-obvious" to those skilled in the art, and "useful". They have been successfully defended in the past. There is no real difference between an algorithm, and a design, process or utility. Arguments against patenting in general tend to be ideological in nature, and this is not the forum for debating Marxism.

As a startup project, investment is critical. But investment may be hard to come by unless we have something with which to convince investors that the market won’t be immediately entered by many competitors doing the same thing, especially by those larger and better equipped than we are. Investors want to know that their money will produce gains, not go down the drain due to new entrants copying or reverse engineering the technology whose development they funded. Patents offer investors security. Investment allows startup companies to defend patents.

Also, we are spending significant amounts of our own time, effort and money on development. If we were pursuing a hobby, we would eagerly make our innovations publicly available and open source. There is much to be proud of, show off, and share. Open source is a wonderful thing, it is driving much development today, and can even result in revenue if done properly and developed quickly. This is not an indictment of Open Source, but we decided to try to maintain more control of the development and business of the project. If someone can propose a way to do this without patenting, please let us know.

The Future of the Internet

2007-12-18T13:40:00.000+02:00

I found an interesting analysis of existing applications and ideas about the future of the internet on the CNN website by Jeffrey M. O'Brien, writing for Fortune Magazine. He says:

But there is no go-to discovery engine - yet. Building a personalized discovery mechanism will mean tapping into all the manners of expression, categorization, and opinions that exist on the Web today. It's no easy feat, but if a company can pull it off and make the formula portable so it works on your mobile phone - well, such a tool could change not just marketing, but all of commerce.

"The effect of recommender systems will be one of the most important changes in the next decade," says University of Minnesota computer science professor John Riedl, who built one of the first recommendation engines in the mid-1990s. "The social web is going to be driven by these systems."

I see a growing interest in recommender systems everywhere I look, and I tend to agree with the above that the next big thing likely involves personalized discovery and recommendations. It seems clear that in order to improve the current state of the internet information overload must be reduced as well as the burden of choice - this will require advancements in recommender technology. One can also assume that the internet will continue and expand the benefits of "Web 2.0" and the collaborative internet. These benefits are, like most products and services, increased ease of use and increased usefulness.

Some feel that the "Semantic Web" or similar technologies are the future. Essentially this involves "teaching the machine" about content through the creation of universal, machine readable formats. Those critical of the limited scope of "folksonomies" champion a more standardized approach, thinking that if only people could be compelled to be disciplined in their approach toward organization of content, that some kind of utopia would emerge. There are as many critics of the Semantic Web as there are of folksonomy, and the fact that people have been talking about schemas and such for years, seems to indicate that - short of some radically new approach - there will be no utopia there. Interestingly, it may be a phenomenon called "social network fatigue" that finally forces services to adopt a standardized portable social/data/identity ontology.

The new revolution must also bring people together in much better ways than those that employ the "degrees of separation" format. Online Social Networks are extremely popular because they allow people to keep in virtual contact with their existing friends and contacts, and they can facilitate personal expression and PR. Unfortunately, social nets are not really social, and the network - as degrees of separation - serves mostly to separate. Registering for one of these sites is a rather cold experience unless you already have tons of friends and contacts already registered. Otherwise you have to spam your friends and nag them to register - something that they may not appreciate. The burden of registration and establishing relationships is compounded as more and more social nets are created, each of them requiring the time consuming input of the same personal data.

The next big thing must facilitate discovery of new people based only on similarities of their core identities, rather than focusing only on a user's primary social network. Two identical twins, separated at birth and raised in countries on opposite sides of the planet, speaking different languages, should be able to find each other. Bringing people together must include the separate objectives of: romance, friendship, business relationships, work team formation, roommates, travel and recreation buddies, etc.

In addition to people finding other people of similar core identity, the next big thing will need to do the same for non-human entities, like music, movies, TV, books, news, web pages, articles, games, products, services, vocations, jobs, travel destinations, politics, religion, advertising, or any other activity, enterprise, product, service, endeavor, idea, belief, passion or item.

In order for such matching to be accurate and satisfying for the user, there must be low false positives ("trust busters"), low false negatives ("missed opportunities") and sufficient true positives ("new favorites"). No current recommendation or recommender system comes close to even approaching this goal. Matching cannot be based on identical answers or choices, as there can never be enough questions to encompass all of human individuality. More importantly, people's existing interests in music, books, movies, etc. is not strictly linked to their core identities, but rather often more on prevailing cultural and social influences.

In our short lives, we are unlikely to ever find the people and things that we would most enjoy and appreciate. This is unfortunate.

The new internet revolution must match people to other people and things based solely on that core identity. It must do it with only a brief registration, where, upon registering, you are immediately presented with your ultimate best friends and soulmates, ideal potential business partners, as well as new favorites in every area of life. It should do it without language barriers (i.e. folksonomy or schemas), without the need for a large staff (i.e. Pandora), without a tedious registration (i.e. eHarmony), without having to observe your history (i.e. Amazon.com, Netflix, Last.fm, etc.), without requiring existing friends or contacts (i.e. Facebook, MySpace, LinkedIn, etc.), and it must be completely free for users but able to convert many of those recommendations into revenue.