"Sensors everywhere. Infinite storage. Clouds of processors. Our ability to capture, warehouse, and understand massive amounts of data is changing science, medicine, business, and technology. As our collection of facts and figures grows, so will the opportunity to find answers to fundamental questions. Because in the era of big data, more isn't just more. More is different."
Wired Magazine has a series of articles on next generation analytics and data mining.
Author Tom Clancy's characters frequently lament our intelligence agencies do not have enough human intelligence - HUMINT. Well, how about masses of humans? Crowds, in fact, to produce Open Source Intelligence - OSINT...
Read more at how our intelligence agencies are learning to mine the internet and other public sources in USA Today
I spent a day at the FASTForward conference in Orlando and listen to 4 provocative keynotes. All about Search, as a proxy for web and other analytics. FAST, a Norwegian company in the fast growing enterprise search space, was recently acquired by
Microsoft
a) John Hagel, expounded on the work he has been doing over the last few years around "Pull Platforms" - which leverage a new breed of empowered consumers (with search and other information sources) and talent - as against traditional Push Programs which worked in the past as demand was more predictable and processes could be standardized a lot easier. The biggest threat to most corporations- that these consumers could leverage the talent themselves (or are also the talent themselves) and cut the middleman out.
I talked to him afterwards and he believes the single most important capability for an enterprise is identifying and developing new forms of talent that understand the new consumer. And he is big on networked communities like that of Li and Fung, a Chinese company, which helps apparel designers around the world connect with 10,000 highly specialized providers of production and logistics services. He also cited Cisco Connection On-line with 40,000 partners.
b) Then it was Clare Hart, President of Dow Jones Enterprise Media Group and she showcased "search without the search box" Using her Factiva 2.0 tool kit she demoed two roles at a bank - an investment banker, and a wealth management exec - and the analytics they would see driven by an "event" - say missed earnings at a stock. What's impressive is the range of information (from news feeds, blogs, streaming stock information, and proprietary customer and other data) and data types (docs, graphs, videos etc) all customized by role, linked and triggered by event types which have "anticipatory discovery" done ahead of time. So instead of having users run multiple and inconsistent queries seeking such information, much of it has been thought out and being processed in background. I talk to too many folks who think a better UI to enter transactions or queries is "user empowerment". Giving a user powerful information with little user involvement like Clare showed is a much higher form of user empowerment.
c) Then on to Dave Weinberger, co-author of The Cluetrain Manifesto. His last slide showed a well manicured topiary and next to it a pile of fallen leaves in stunning Fall colors. He loves the disorganized pile and thinks enterprises have tried too hard to organize data into information and made it "soulless" . The web is about links and links should be unpredictable and "messy". Some would call it anarchy, but it was a refreshing perspective on not trying to predefine data models, and instead let communities of internal and external users influence it. They will always bring broader and fresher "tags" than a corporate team can ever think of.
I spent a few minutes with him later and you can see the man's
bright lights - though on his blog he describes himself tongue-in-cheek
as "a Ph.D in philosophy that entitles him to affect an air of smug
obscurity whenever he chooses"
d) Dave was followed by Tom Davenport, who, of course has written many books on analytics. Tom's presentation was as much about disciplined analytical rigor as Dave's was about enjoying the messiness that comes from the web. Tom's presentation focused on companies like Harrah's and Progressive which have for long known the value of mining intelligence from mountain of data. He updated that with newer examples of analytics the web is facilitating from recommendation engines at NetFlix to better customer experiences at Careerbuilder.com as it optimizes across millions of job searches.
"We're bidding adieu to 2007 with a look back at the breaking news, the big events and the must-have gadgets that captivated us this year (give or take a few weeks; we compile this list by early December). To get a glimpse of what's been on our collective consciousness, we mined billions of search queries to discover what sorts of things rose to the top. We encourage you to check out our findings to see if you, too, reflect the zeitgeist — the spirit of the times."
Google Zeitgist has all kinds of factoids like Ron Paul was the most searched Presidential candidate, the most searched for lawsuit was that related to Borat and other useful and useless stuff...
From engineering to gadgets to alternative fuels, Popular Science lists innovations from around the world.
MIT Technology Review on the quant melt-down this past summer - and the crunching computational power they rely on
"Computers also underlie another developing frontier, high-frequency trading, which is a fantastically exaggerated form of day trading. The computer looks for patterns and inefficiencies over minutes or seconds rather than hours or days. An algorithm, for instance, might look for patterns in trading while the Japanese are at lunch, or in the moments before an important announcement. There is a massive amount of such data to crunch. Olsen Financial Technologies, a Zürich-based firm that offers data for sale, says it collects as many as a million price updates per day."
New Scientist (sub required)
"Researchers are working hard to harness the body's inner power - not some mystical life force, but the chemical energy locked up in the body's own food stores - and convert some of that into electricity. The hope is that medical devices can be made to behave like benign parasites, stealing just enough of this energy to run themselves without you, the host, even noticing. We're talking about a matter of tens of microwatts to a few milliwatts for many applications. The same could be done with the waste heat energy that our bodies pour out, or the kinetic energy of our pulsating muscles. Several of these so-called energy-scavenging systems have already reached the prototype stage, and more are under development. It is possible that within a few years many medical devices will be able to do away with batteries entirely. Eventually, your body (or should that be iBody?) might even deliver enough electrical power to run gadgets like your cellphone or MP3 player"
"Using the institute's expertise in pattern-matching technology, he and his colleagues are about to embark on one of the biggest jigsaw puzzles of all time—or, rather, 45m of them. For that is the number of pages which the 600m fragments of paper, stored in more than 16,000 bags that were recovered from Normannenstrasse, are thought to represent."
"WHEN the shredding machines failed and the mob was at the gates, the spooks at East Germany's State Security Service, better known as the Stasi ... resorted to tearing each page up by hand. The fact that many of the resulting shreds are only a few millimetres across is testament to just how much the soon-to-be-ex-members of the intelligence service did not want their work to fall into the public domain."
Mark Nittler and I go back a long way. And every time we meet, it seems like the conversation drifts to some version of REA (Resources, Events, Activities) Accounting. Could be because when we met I was at Price Waterhouse and we had a tool called GENEVA aimed at manipulating large events databases. Could be because he was at Walker Interactive which provided accountants a very flexible coding block to track various non-accounting perspectives of data.
Mark, now at Workday, has introduced a financial module which incorporates the contemporary concept of "tags" (See Dan Farber's perspective here, and Dennis Howlett's here.) Workday also delivers its software as a service - so in many ways, a long cry from the big mainframe, DB2 driven ledgers Walker
delivered.
The question is are conservative accountants ready for these innovations?
"Logical spreadsheets—data management systems that use logic instead of math—allow easier manipulation of data,...
...Currently, (Stanford) administrators manage the scheduling of rooms in the Gates Building using logical spreadsheets. Starting next fall, Genesereth hopes that the Computer Science Department will be run using logical spreadsheets.
...The Department of Defense would like to use the technology for organizing troop deployment and training...
Funding for the team's logical spreadsheet project comes from the Stanford Logic Group and SAP."
Recent Comments