One of our biggest challenges is determining if a company from one source is the same one as we found in another source. Using our proprietary semantic Identity Resolution engine, we perform advanced matching to know if this company is the same as that company. This enables us to avoid duplicates in the database and to combine multiple information sources to create rich company and executive profiles.  This process is referred to as “Identity Resolution” or “Match & Merge.”

Our matching system is based on advanced semantic & statistical models, natural language processing and machine learning. Using our massive database of web-sourced data the system has created sets of positive and negative rules to decide if two contacts should be matched.

Positive Rules

The inputs are first exposed to a defined rule set made up of certain positive rules that are meant to examine the likelihood that the two inputs refer to the same entity. The names of the companies are examined first and based on the likelihood that they refer to the same company the pair is given a rating between 0 and 1.

For example: Luigi’s Pizzeria and Luigi’s Italian Restaurant will be matched and assigned a score of 1 as it is likely that the two names refer to the same company.

Next, the contact information of each company is examined, in order to ascertain a true connection between the two. This includes the physical addresses of the two entities, the phone numbers, URL’s, employee information and so on. Based on the similarities between all this information, the two entities are combined into a “matched set.”

“Fuzzy Matching” is utilized in the identification of spelling mistakes, and an advanced semantic process analyzes the actual meaning of the content, allowing the system to take into account synonyms and similar-meaning words (such as “restaurant” and diner). In addition, an abbreviation process considers if YMCA is its own name, or just a short hand for Young Men’s Christian Association.

Negative Rules

Every “matched set” is then processed through a defined rule set made up of negative rules that are meant to examine the likelihood that, regardless of the “matched set” status assigned after successfully meeting the standards of the positive rules, these two entities are different companies. The DUNS number, stock symbol and other recognized information is examined for any discrepancies. After this examination, if there is a conflict in information the set is assigned a “problematic” status, re-examined and either accepted as a match or dismissed as opposing companies.

Merge

After two entities are matched, the next step is to merge them correctly into one profile. Sources are assigned priority levels based on the quality, accuracy and recency of the information, so that any conflicting data can be adjusted according to the source with the highest priority. Priority and source quality is assigned independently per attribute and in this way we can refer to the most accurate information provided on one topic regardless of the accuracy of the remaining information provided by the source.

For example: If one source has great company financial data but poor phone records, aside from the overall priority score assigned to the source as a whole, each one of these attributes receives a quality and priority ranking; the source would be considered when dealing with financial information but basically disregarded when handling phone records. In this way the most relevant and accurate information per category is utilized.

Precision and Recall

We need to strike the optimal balance between precision and recall. For our purposes, precision equals the fraction of information that is correctly retrieved while recall equals the fraction of information retrieved relative to what’s available. We have created advanced tools that allow us to dial the precision and recall up or down, to find the right balance. The more “loosely“ we match, the greater the opportunity is to extract more information on the topic, leading to higher recall, the “tighter” the match, the less likely we are to connect the dots incorrectly, improving precision.

Depending on our customers wants and expectations, we can select the appropriate balance between precision and recall to offer the most accurate and rich concentration of knowledge to our customers.

Introducing Event Data into Predictive Models

Event data can easily be introduced as predictor (independent) variables within pre-existing risk and marketing models in two basic modes.  The first adds them as later stage variables, which means the pre-existing model variables are entered on a forced basis (which replicates the current state of the model), and the event variables are subsequently allowed to enter.  This process ensures that the event variables are evaluated for their incremental contribution to the model, and do not displace any pre-existing model variables.  In contrast, the alternative mode starts the model development from scratch, and pre-existing model variables might be replaced by the event variables.  This approach may outmode the current model, but yield a more optimized set of factors.

The event input data should be coded into event types as well as time periods. For example, the number of litigation occurrences in the last 3, 6,12,18,24+ months.  As a simple example, I’ve found very high correlation between the number of lawsuits from different parties and payment delinquency.  Sometimes, source and quantity are desirable, but from a practical perspective they create significant complexity (a single litigation event might now be exploded into many different combinations of source and amount which need to be individually tested).

Treating Events as Triggers

Sometimes, events are hugely significant in their impact on risk and/or reward.  As an obvious example, M&A, which (believe it or not) is a variable ignored in risk models.  The affects of these events cannot be easily quantified in models, and so they are best treated as “triggers” or decisioning input (for subsequent manual review and intervention).

Sentiment Analysis

Sentiment Analysis of companies is one of the more interesting qualitative pieces of data that has recently become available, due to advances in web mining. Briefly stated, sentiment measures the positive or negative “buzz” about a company.  The firms that utilize product sentiment analysis use varying sources and methods to produce “sentiment scores”.  Minimally, sentiment analysis can be used to corroborate certain decisions, and may have predictive ability as well.  Like events, sentiment scores can be used as time-based model variables, or as external triggers.

To conclude, I would like to impress that there should be no doubt that select business events affect the risk and opportunity value of a company.  Event data, and its accompanying sentiment, is available on a near-real time basis on the Internet.  Semantic analysis companies (such as Digital Trowel) have created a process that mines this data and presents it in coded form, which can be made available to scoring and decision models, as well as human monitors.  Virtually any company relying on risk and/or potential models can incorporate this powerful information to enhance its accuracy, by employing them either as internal variables or as external decision factors.

Please contact me with any questions or comments. I can be reached by commenting on the blog, or via email at Steve (at) digitaltrowel.com

Check back soon for more in-depth exploration of the growing text-mining phenomenon.

- Steve

Regardless of whether you’re a lender, supplier, analyst or salesperson, here are some events that negatively impact the growth behavior of a company, and that can be mined from web-data in a very updated manner:

• Litigation: When a company starts to have cash flow problems, one of its first reactions is to delay payment to some suppliers.  At some point, payment delinquency moves from the “tolerant” stage to the litigation stage.  But what happens if you’re modeling the risk of a company and do not have access to their AP/AR data?  How do you recognize litigation without waiting for it to possibly appear in a financial report?  Fortunately, there are fee-based web-based sources that detect and track litigation including LexisNexis, Public Access to Court Electronic Records (PACER), and D&B.  Publicized litigation that has made it into the media can be obtained at little or no direct cost, and recent (2010) examples of major litigation include BP,  Microsoft’s suit against Salesfore.com (patent infringement), Borg-Warner (asbestos product liability), and Chrysler (failure to pay suppliers).  Of course, the above companies are large enough to withstand the litigation payouts to avoid default; but what does this do to their sales & marketing budget and supplier terms?  In our more expanded view of risk, these are important topics!
• Analyst Recommendations: Analyst recommendations often, and quickly, affect a company’s stock price.  Downward recommendations that cause the stock to fall, place pressure on the company to compensate. A typical reaction is to cut expenses in order to boost earnings. Of course, this action does not bode well for the company’s S&M efforts, or their suppliers.
• Partnerships: Partnerships usually indicate positive growth activity, and by logical extension, lower the company risk.
• M&A: M&A logically reduces the target company’s risk.  Although M&A (and even its announcement) should immediately change the risk score of the target company, this is usually not the case, since the scoring models have no way of quickly recognizing the event.
• Key employee movement: When a company hires a heavyweight Senior executive, it is invariably a growth move, which should lower risk (otherwise they would likely not take the new position).
• Insider trading: The purchasing of shares by insiders is often a leading indicator that they expect the stock will go up in the near future (which is itself a leading indicator that the company will expand due to its increased market cap)
• Product introductions:  A new product introduction is typically a leading indicator of growth, hype, success, and similar; these are all leading indicators that reflect a lowering of risk.
• Product recalls (pharma): At a minimum, product recalls offer a negative distraction to sales.  Sometimes, for example in the Pharma sector, recalls can have a devastating affect on sales.  Sometimes, for example in the auto industry, they may have a more temporary affect. But in either case, they diminish the strength of a company.
• Financial announcements:  Financial announcements are excellent leading indicators, on the upside and downside. They appear on the Internet well before they appear in the financial statements that are used to drive typical company risk models.   Competitive tracking:  Significant changes in competitive activity greatly affect market potential models, and could well affect risk models.
• Competitive monitoring becomes increasingly important in economic downturns, since supplier loyalty is overshadowed by the customer need for cost reductions.   Generally speaking, as direct competition grows, it becomes more formidable to deal with, and the competitive events including product, financial, employment, and so on should be quantified and incorporated into both risk and marketing models .Whew! Now that we are all caught up on Business Events, check back for the third and final post of the series that will tie everything together.

Check back in a couple of days for Part 3: Using Web Mined Data to Enhance the Performance of Business Risk and Opportunity Models

Please contact me with any questions or comments. I can be reached by commenting on the blog, or via email at Steve (at) digitaltrowel.com

Cheers!

- Steve

Having been in the business risk and opportunity-modeling arena for many years, I’ve come to the conclusion that the greatest weakness in business data modeling is quite simply the age of the data.  There is no doubt that a downswing in EBIT spells bad news; but by the time that is recognized in a financial report, it’s very late in the game, and no modeling technique can overcome the limitations of old data.  In my search for a better source of leading indicators, I naturally gravitated to the internet.  After all, the Internet offers an unparalleled rich, dynamic source of data in both quantitative (e.g. financial reports) and qualitative (e.g. sentiment) form, and many of these are inherently powerful leading indicators of both risk and opportunity.

Not coincidentally, statistical package developers such as SAS and SPSS have already launched applications that combine text mining and analytics.  However, for many companies, it will be preferable to gather the data as a separate process, and then integrate it into their modeling/decisioning processes. Recently, I’ve found that incorporation of web data can improve the accuracy/timeliness of risk-based decisions by as much as 20%; even larger benefits can be expected in the area of market potential analytics.

Stay tuned for my upcoming musings on this topic. Part 2: Using Web Mined Data to Enhance the Performance of Business Risk and Opportunity Models

Please contact me with any questions or comments. I can be reached by commenting on the blog, or via email at Steve (at) digitaltrowel.com

Looking forward to an active dialogue.

- Steve

Well, in the previous section we mentioned in passing that our technology was based on a synergistic approach, combining syntax, semantics and pragmatics. In this final part of the survey, we’ll explain just how we do this, and why our system yields unparalleled results. In doing so we’ll do our best to abstract away from the underlying mathematics and details of the machine-learning algorithms, and instead present the linguistic principles by which our algorithms work using examples. To wrap things up, we’ll end this review with a snapshot of what Digital Trowel’s Sentiment Analysis looks like in action.

Our technological approach begins with the observation that sentiment is conveyed on three interacting levels of increasing structural complexity. Namely the lexical, phrasal and semantic-event level of structure. We’ll explain.

Lexical sentiment, sometimes referred to as dictionary-based sentiment, is the sentiment attributed to single isolated words. For example:

great, wonderful, terribly, worrisome, helpful, etc…

Though single words clearly carry sentiment, this is the most rudimentary and least reliable sentiment available. To see this consider the following phrases using the above examples:

Great failure
Wonderful fiasco
Terribly surprising comeback
Worrisome transformation for previous skeptics
Helpful in expediting the demise

It should be evident from the above phrases that the initial or “natural” sentiment associated with the isolated words, have all been transformed if not negated. To avoid such “wonderful fiascoes” in deciphering the sentiment, we employ the lexical analysis of sentiment only after the text has undergone syntactic parsing. In simple words syntactic parsing means that sentences are analyzed to determine their grammatical structure and that each word is assigned its corresponding Part Of Speech (POS) tag.

Consider the following example taken from Cisco’s website (where red and green indicate negative and positive sentiment, respectively):

If Cisco does not achieve the desired level of acceptances, the company will withdraw the offer and evaluate alternative ways to expand our activities in the video communications market.

To glean the lexical sentiment, the sentence is first parsed, i.e. grammatically analyzed. For starters, this allows us to determine the subject of the sentence (“Cisco”) as well as any pronominal phrase referring to the subject (“the company”) – both of which have been marked in bold above. Naturally, this is of critical import to us is in determining what company the sentiment is to be associated with. Secondly, once we obtain a phrasal structure of the sentence we are able to determine how a candidate lexical entry interacts with clause-mate entries. In the example above, “desired” is typically associated with positive sentiment, but this sentiment is reversed due to the negation “does not” appearing earlier in the clause. On the other hand in the subsequent clause the verb entries “evaluate” and “expand” maintain and even substantiate their positive sentiment, as there is nothing in the clause to alter their natural interpretation.

Obviously, not all lexical entries are born equal. Entries may vary both in the extent to which they convey a sentiment and their relative intra-clausal effect. For example “excellent” conveys a stronger sentiment than “good”, whereas “great” and “superb” generally indicate the same level of positivity, but “great” is more susceptible to lexical negation (cf. “great mistake” vs. “superb mistake”). Different entries therefore receive different weights, depending on their relative sentimental strength and susceptibility to polarity-transformations. In order to correctly assign weight to these words, DT uses advanced statistical models which are generated using large manually-analyzed text corpora. In addition further factors such as conditional, speculative and contra-factual clause structures are taken into account before the final contribution of specific entries are calculated.

But this is only the first and most rudimentary element of our synergistic approach.  The second more complex element is that associated with the phrasal level of structure. The phrasal level of analysis assigns a sentiment value to full phrases rather than to single words. Consider the following examples:

Cisco Chief Executive John Chambers has said the firm aims to gain market share in a tech recovery.

Boosted by those moves and  …  following last year’s 40 percent decline

Company Struggles in Attempt to Buy Time

In the examples above the lexical level may signal certain entries are positive or negative, but  only a real phrase-level analysis can ascertain the sentiment. It is here that we first allow semantic and pragmatic factors to interact. It is not enough to understand the meaning of each word in isolation, the meaning of the entire phrase must be deciphered, and to so  correctly, context is needed.

Take a look for instance at the third example above. Usually when companies buy something, it’s either a product or another company. Here, however, it is clear that an idiomatic meaning is intended (buying time… stalling).

DT’s SA takes pragmatics to a whole new level. Not only do we use carefully developed word-classes to allow our engine to utilize outside knowledge in interpreting text, but, working with a team of linguists and economists we have developed specialized sets of phrase level interpretive rules, which allow the engine to identify the context of a sentence or phrase.  All of this combined with the simple pragmatic module which is used to identify key companies by resolving anaphora and common nicknames and descriptors and you end up with a context identifier that allows our engine to assign sentiment to even highly complicated, idiomatic or obscure phrases. Believe it or not, allowing our semantic and pragmatic modules to collaborate, our engine is able to pick up on sarcastic, wishy-washy, and even ironic notes in the text.

This brings us to the third level of our Synergistic Sentiment Analysis, which is based on the interpretation of actual events within the text. Transcending both lexical and phrasal levels of interpretation, we have trained our engine to identify key economic events, and together with a team of experienced financial experts, we’ve created a scale of positive and negative weights for these events. Take a look at the following examples:

shares of Cisco Systems (Nasdaq: CSCO) were recently up 47 percent

Cisco expects revenue to grow 1 to 4 percent

Cisco(R) (NASDAQ: CSCO) today announced a revised recommended voluntary cash offer to acquire TANDBERG (OSLO: TAA)

All the above are real examples of events captured by our SA engine and marked as positive. We currently have our engine trained to extract and evaluate dozens of types of events including purchases, stock offerings, workforce changes, legal events, product launches or recalling, hiring and firing of key figures, new facilities, bankruptcy, etc… etc…

The event-level of our SA assigns the highest weights since it combines and epitomizes all of our techniques. Using syntactic, semantic and pragmatic analyses to determine the contribution of the event to the sentiment. In fact, we believe that by identifying and analyzing the key events in the text we are emulating just what an expert would do when attempting to estimate the sentiment associated with a given text excerpt.

Starting from the lexical level, which allows us to pick up on subtle tones in the text , building up to phrases which indicate attitude, and embedding these all within a semantic-pragmatic event extractor and economic-analyzer, we believe we are truly able to capture the sentiment of text very much like a human would, with incredible reliability and consistency. We may not have yet passed the Turintg Test, but we’re surely on the way to improve the ability of machines to “understand” the natural language that humans use!

Well, for now that’s all we can show, without divulging too much

The Space Between the Lines

Welcome back! Sit down and buckle up for a magical tour of the text mining technology focusing on Sentiment Analysis (SA).

Well, first things first. What is Sentiment Analysis anyway? Rephrasing the Wikipedia definition, Sentiment analysis (sometimes called opinion mining) refers to an area of Natural Language Processing (NLP), which aims to determine the attitude of a writer with respect to some topic. This attitude may be their judgment or evaluation, their emotional state when writing or the intended emotional communication the author wishes to convey.

To keep our discussion as concrete as possible we’ll use real life examples to elucidate the different types of attitudes.  Consider the following example:

This year was a setup year for B&N, and 2010 will see its efforts start to pay off [...] In 2010, B&N will rack up significant sales of Nooks and e-books, as some consumers look for an Amazon alternative.

Obviously this excerpt contains an explicit positive evaluation for Barnes and Noble for 2010, but moreover the tone is upbeat, optimistic, and even excited. A good Sentiment Analysis would pick up on this tone and report a highly positive sentiment for B&N and their e-reader Nook, whereas a negative or at least an apprehensive sentiment should be reported for Amazon.

The next example is even more blatant:

Belated Happy New Year and already what a year it’s turning out to be for eReaders! [...] Time’s a fave around here these days, especially considering its December report naming nook one of the Best Travel Gadgets of 2009 as well as rating the device # 2 among the Top Ten Gadgets of the year. While emphasizing nook’s “classy book-lending feature”, the magazine also cited “the powerful, flexible Android operating system that the whole package runs on.”

The exclamation mark, the rhythm, the tone, the profuse use of superlatives and positive adjectives all indicate an extremely positive sentiment for the nook product. It is clear that the author has a favorable opinion of the product, and moreover that he is quite eager to share his enthusiasm with the readers.

Obviously these are not the only attitudes that can be found on the web. Other attitudes may include anticipation, sarcasm, doubt, apprehension, cynicism and even condemnation. It’s our nature to focus on the good, so we’ll spare you examples of the negative attitudes (well, I guess it’s also that we prefer to avoid any unnecessary lawsuits ) but the basic idea of what is meant by an underlying attitude should be clear by now.

It’s important to keep in mind that Sentiment Analysis is not severed from the basic meaning of the sentence. Rather, SA picks up on the basic meaning and further capitalizes on the cadence, the tone, the choice of words, and even the absence thereof, to build a complete picture of the message being conveyed. Note that we’ve implicitly drawn a line between some sort of “basic meaning” of a sentence, and the “ultimate intention” of the  message to be conveyed. Let’s try and be a bit more precise and explicit about this distinction.

Formal linguistic theory usually recognizes 3 levels of abstraction for natural language comprehension: Syntax, Semantics and Pragmatics (we are excluding phonology, phonetics and morphology which are irrelevant here). Simply stated, Syntax is the study of the grammatical structure of sentences, Semantics deals with how words are interpreted and how their interpretation is combined to yield the meaning of the sentence, and Pragmatics is the study of how extra-linguistic, real-world knowledge, so to speak, interacts with the basic meaning of sentences to yield the ultimate message conveyed.

So for example, syntactic theories may attempt to explain why the English sentence “I gave that to you” is fine whereas, “You gave that to I” is ungrammatical. Semantic theories may attempt to explain what the meaning of a word such as “tall” is, and how this meaning can be reconciled with seemingly problematic examples such as “I am tall” vs. “The midget is only 4 feet tall”. Pragmatics, goes one step further and attempts to explain how our knowledge of the world, circumstances, etc. play with and alter the meaning of the conveyed message. So for example, although strictly speaking the sentence: “I have 3 children”, does not formally preclude the possibility that I have more than 3, say 5 children, it would generally be considered wrong, or at least odd, for someone who indeed has 5 children to utter the original sentence: “I have 3 children”. To see how this judgment may change with circumstances, imagine Mr. Jones is being interviewed by the IRS, when he is notified by the interviewer that tax benefits are available to anyone with 3 or more children. Under these circumstance, we would probably no longer consider it odd for Mr. Jones to say “I have 3 children”, even if in fact he had 10 children.

So where does Sentiment Analysis fit in this 3-headed theoretical framework? If you’re guessing the answer lies somewhere between semantics and pragmatics, perhaps with a bit of a syntactic-twist, you’re following this introduction just fine. (If, on the other hand, you thought it was limited to the syntax, you may want to go brew yourself a fresh cup of coffee before you reread the last few paragraphs ).

Mirroring the theoretical image portrayed above, Natural Language Processing algorithms consist of syntactic algorithms (most notably Part Of Speech (POS) parsers and taggers), semantic algorithms (e.g. semantic rulebooks and relation extraction algorithms) and finally pragmatic algorithms (including for example, contextual disambiguating algorithms, and world knowledge look-up algorithms, used in automated translators for instance). At Digital Trowel we’ve honed our Sentiment Analysis algorithms to combine the strengths of these 3 disciplines to produce the most reliable and comprehensive understanding of the message being conveyed, reading not only the text itself, but also between the lines, so to speak.

The mathematical implementation of these algorithms is beyond the scope of this introduction, but this by no means should prevent us from taking advantage of the knowledge we’ve gained thus far to see how Sentiment Analysis techniques may harness the power of the different types of linguistic algorithms in an attempt to achieve their goal. In fact the lion’s share of the third part of this survey aims to do just that. For now, suffice it to say that one of the main reasons we at DT believe that our technology is superior has to do with our synergistic approach of integrating syntactic, semantic and pragmatic algorithms. This is why we call it Synergistic Sentiment Analysis (SSA). BTW, for those of you wondering, synergy is the term used to describe a situation where different entities cooperate advantageously for a final outcome (tx, Wikipedia!). There, you now understand yet another word in the titles above

Before ending this part, let’s focus on the goals of SA, or in other words, what SA is good for. Well, in one sentence, as we already phrased it:

Extracting and discerning the underlying sentiment allows us to transform otherwise inert texts into vibrant business opportunities.

There are many other business opportunities for the SA technology, including some of which we’ve already implemented at DT such as evaluating pharmaceutical forums for client’s sentiment about drugs, as well as sports product satisfaction, but I think this is enough hype for now

The third and final part of this introduction to the field of SA, goes a bit deeper into the SA engine itself, and examines the innovative technology unique to Digital Trowel using real examples… Stay tuned, this is where things get really exciting

A Non-standard Introduction to Sentiment Analysis in 3 Parts

Part 1 – CAPTCHA to Gotcha:

A Brief History of Artificial Intelligence

Alan Turing was a prominent British mathematician and one of the most inspiring pioneers of modern computer science. In 1950, at the age of 38, he published his seminal paper Computing Machinery and Intelligence, which till this day remains probably the single most influential paper in the field of Artificial Intelligence (AI).

Since Digital Trowel’s core technology is based on machine learning, a modern offshoot of AI, it would be conducive (and nice!) to get back to the basics, and learn a bit about the history that continues to shape both the science itself and the challenges we face at DT.

Big words and complications aside, Turing begins his paper with the simple yet perplexing question: “Can machines think?” Nevertheless, realizing that “thinking” is a highly ambiguous term, Turing immediately proposed an alternative question that would be free of obscurities and eschew obfuscations. Instead of dealing with machines’ capacity for thinking, he focused on their capacity to emulate human thought. In simplified terms the question he suggested was:

Could machines be made to simulate human thought well enough so as to fool a person into believing they were actually human?

This question is the essence of what has come to be called the Turing Test. It proceeds as follows: a human judge engages in a natural language conversation with one human and one machine, each of which tries to appear human. All participants are placed in isolated locations. If the judge cannot reliably tell the machine from the human, the machine is said to have passed the test. In order to test the machine’s intelligence rather than its ability to render words into audio, the conversation is limited to a text-only channel such as a computer keyboard and screen.

At the time of its publication, many people viewed the prospect of machines ever reaching the level of human computational power an impossibility, but in his paper, Turing, armed with his visionary intuition and razor-sharp mathematical analysis, set out to invalidate contemporary objections, ending with a speculation of his own, that one day machines would indeed emulate human thought, thereby passing the Turing Test!

Inspired by the challenge, Digital Trowel’s groundbreaking technology has taken several huge steps forward in proving that Turing was right. The technology we’ve developed allows computers to extract not only the facts communicated by the text, but also the underlying sentiment or, if you will, the attitude associated with the message conveyed. In simple words, we’re enabling computers to understand the full meaning not only of the text, but of the subtext – just as a human would. But hold your horses! Before we continue, let’s try to explain why the problem is so difficult, so we can more fully appreciate the profundity of Digital Trowel’s achievement and its extraordinary implications.

Well for one thing, it’s now sixty years later and the question presented by Turing has yet to be settled. In fact, it is far from being resolved. Machines have beaten world chess champions, navigated spacecrafts millions of miles away and even been used to prove mathematical theorems whose intricacy is impenetrable to human beings for their sheer magnitude of computational complexity, yet as of now no computer has been shown to pass the test.

It may be argued that the challenge machines face is simply a matter of raw computing power. It is currently estimated by some experts that the human brain can perform some 38,000 trillion operations per second (that’s 3.8×10¹⁶ operations!) and hold over 3,500 terabytes of memory. In comparison, the world’s most powerful supercomputers (e.g. IBM’s BlueGene) have computational capacity of less than a “mere” 100 trillion operations per second (only 10¹⁴) and less than 10 terabytes of storage. However, if this indeed is the case, and the capacity to “think” lies in raw computation power alone, then according to some versions of Moore’s Law (which predicts the rate at which computation performance evolves with time) machines will ultimately obtain the required criterion by circa 2018. But we may have to wait a bit longer for an answer: recently, futurist Raymond Kurzweil, revised his earlier prediction that Turing test-capable computers would be manufactured by 2020, deferring the predicted date to 2029 (I can’t help but wonder if this prediction has anything to do with the fact that asteroid (35396) 1997 XF₁₁ is anticipated to make a close approach to Earth late in 2028 ).

But how does this all have to do with Digital Trowel’s business? That is, unless we have secret plans in store for purchasing stocks in Scientific American… (which we don’t!). Well to answer that consider first what is sometimes called a Reverse Turing Test. Imagine a modification of the Turing Test wherein the role of the judge has been switched between machines and humans. Now it’s the computer who has to determine whether it is “conversing” with a human or another machine. In fact to some of you this may just ring a (quite annoying) bell. Take a look at the images below:

Ever wonder why every once in a while you’re prompted to try to decipher the jumbled up letters in images such as these? Well, put simply, it’s because you’re taking part in a test that’s not meant for you – you’re serving as a participant in a Reversed Turing Test administered and judged by the security computers of the website with which you are attempting to engage. Humans have (or rather should have!) no problem deciphering the text in the above images, which incidentally are called CAPTCHAs (for Completely Automated Public Turing test to tell Computers and Humans Apart). However, the random distortions in a CAPTCHA make it nearly impossible for computers to decipher the letters. As a result, automated security programs can use these images and the respective responses to make certain it is a human attempting to engage with the website and not some malicious script.

The Demise of the Language Barrier

As autumn briskly descends on New England I find myself admiring nature’s capacity to silently transmute subtle changes into full-fledged displays of majestic beauty. A leaf here, a leaf there… They practically go unnoticed for several weeks – and then suddenly one morning you wake up and look through the window, amazed to see that nearly all the trees on the street are ablaze.

Reflecting back on this past year, I can’t help but draw the analogy to Digital Trowel’s own progress. It seems unbelievable that only one year ago we were a small team of 7 engineers crowded together in 2 small rooms, with not much more than an untested NLP platform and a vision. As the weeks passed by, we added an engineer here, a linguist and a mathematician there, and suddenly we are a mature, full-fledged commercial company, with over 40 developers, selling products and data, ablaze with a proven breakthrough NLP technology in hand.

But we’re still hungry! And our team of scientists is cooking up a storm

We’ve just recently hired another mathematician and a computer scientist, so we now have 5 PhD level scientists on our R&D team. We strongly emphasize the research aspect of the business, because we know that being at the top is a very fragile state. So while we’re very proud to be the first company to produce consistent commercial results with over 90% accuracy and recall across multiple semantic fields, we realize there are still many challenges to be met, and we’re determined to be the first to meet them.

While I’m not yet at liberty to divulge the full extent of our newest developments, I am able to give you a taste of some of the advances we’ve made.

One of the most exciting projects we’re looking into is a multi-lingual text-mining platform codenamed SNUG for Semantically Negotiated Universal Grammar. Admittedly, I don’t think this is exactly what Noam Chomsky had in mind when he first proposed his theory of Universal Grammar, but it may not be too far off. Let me explain:

Linguists use the term Universal Grammar (UG) to denote a theory according to which all humans possess an inherent “hard-wired” capability to acquire a language. It is this linguistic “hardware” we’re all born with that allows children to learn the grammar of a language even when the linguistic data available to them is insufficient.

A quick example is in order. A child born in the U.S.A must learn to form questions out of assertions in English. He may, for example, hear the assertion:

“My sister is pregnant.”

And over time infer that the proper interrogative form of this statement is:

“Is my sister pregnant?”

He may then (subconsciously) form a rule of grammar in his mind, whereby questions in English are formed by moving the first auxiliary verb to the beginning of the sentence. We would then expect that children faced with an assertion such as:

“My sister who is pregnant will be blessed with happiness.”

Would form the incorrect question:

* “Is my sister who pregnant will be blessed with happiness?”

Interestingly enough – they don’t! Only the correct form is acquired by children:

“Will my sister who is pregnant be blessed with happiness?”   (Of course she will!)

The claim (asserted by UG supporters) is that children simply don’t encounter enough sentences as complicated as the one above to make a learned choice. So how do they do it? Well, simply put – they are born knowing. More accurately, they’re born with a set of rules that are triggered and activated in a certain way once they are exposed to sentences in their language.

Of course, things are far more complicated than these simplistic examples, and many questions immediately arise. How, for example, is it that other languages, German and French for instance, form questions by placing the main verb at the beginning of the sentence? For example:

French: Parlez-vous anglais?

German: Sprechen Sie Englisch?

Both mean: Do you speak English?

But a literal translation would be the ungrammatical: * Speak you English?

So, obviously, the theory must be significantly more complex. But don’t worry, that concludes today’s lesson in linguistics 101 . We now return to what got us here in the first place – our new multi-lingual text-mining platform SNUG (We thought it’s kind of cute to build SNUG using CARE ).

So, first things first: No – we’re not out there to decipher the way our mind processes languages, but – Yes we are out there to create a platform that will enable us to process and mine texts in any language.

For us humans, it’s quite hard to fathom learning a new language in say, a week. But for our system that’s actually an achievable feat. In order for us to effectively extract data from free text in a new language, we need to accomplish two things. First, we need to teach our system this new language. Well not exactly the language itself, but more like the statistical distribution of words in this new language. We do this using an automatic training process, by which our system runs through huge text corpora, producing a statistical model of the language, which is further refined every time we train the system.

The second task is translating our semantic rulebooks. Here is where our linguists come into play. Since our rulebooks are basically comprised of highly sophisticated weighted Context-Free Grammars, their translation amounts to a structure-preserving function of semantic rules (a semantic homomorphism). This function can be thought of as taking an English semantic-driven grammar as input and transmuting it into a semantic-driven grammar for a different language. Though languages vary considerably in their actual spoken grammar, they tend to convey events and factual information in a surprisingly similar manner (or rather, the structure of the required semantic rules is quite similar).

Most Central European languages share so many common qualities with English, that many of the rules can be translated verbatim. Some of the challenges arise when translating rulebooks to languages with a word order that significantly differs from the English word order (e.g. German, not to mention Subject-Object-Verb languages such as Japanese, Hindi and Turkish). Languages with a high level of agreement inflection such as French, Italian and Spanish are usually easier to parse as well, though high inflection agreement often incurs the omitting of overt pronouns. So for example “I eat” in Italian will simply be “mangio”. This turns out to be quite problematic for issues of anaphora resolution.

Every new language poses a new and exciting challenge, but crucially, it does not require us to rewrite the code. All we need is a few large text corpora, an expert linguist, a savvy rulebook writer, and a week or two of intensive work, and our system will learn to “speak” a new language. If only it were so simple for people to learn…

Well I guess that’s enough for this post. I hope you’ve enjoyed this quick “intro to linguistics”, and are as excited as I am by the prospect of text-mining – free of language barriers. Stay tuned for more new and interesting features in my next entry.

Meanwhile, wherever you are, I hope you have an autumn as beautiful as the one I am having in Boston. Even more importantly, I hope you take the time to appreciate the beauty of change all around – spoken without any words.

When setting the goals for extraction results, try to find the optimal balance between recall and precision. Remember: Aiming high is good, just make sure there’s an easy way down!

Do not attempt to debug a rulebook for more than 24 hours straight. We’ve been there. We’ve done that. If you think you’re stuck now, you still haven’t seen nada!

Remember the chain of  language processing from last entry? First comes The HTML converter – it hands down the info to CARE, which does all the hard work and in turn passes the parsed relations to the Post Processor so that it can rest on its laurels

Perfect “Anaphora Resolution” is a myth. You can try. For a while, you may even believe you’ve done it. Our prediction: In the end the bubble will burst. Or you’ll go crazy trying to solve all the problems. Or both.

Make sure that all the crucial information that comes in as input, goes out as output

Remember the engineer that didn’t take our advice and corrected extraction rules through the night?

Just like life itself, retrieving the ultimate extractions for a given relation may be an extremely tedious and laborious task. Lighten up, add some spice. Make fun of yourself!

Do everything you do with love and CARE!

But remember that if you’re not enjoying it, you’re probably not doing it right!

Here from Cambridge, MA, wishing you all a wonderful year full of wonderful experiences, CARE-ing, happiness and love!