The entire purpose of business software systems has always been to provide business decision makers the KPI’s they need to make informed decisions. That was the goal with mainframes and reports on ribbon paper, and it remains the goal on today’s smartphones. Yet data scientist is a fairly new term; what changed?
It’s not the Algorithms but the Computer Power
While the speed of individual computers has massively grown since I was a computer operator in the 1980s, it’s not that which has created the large jump in computing power in the last decade. Clustered computing, the ability to share the workload among large numbers of computers, is what has exponentially grown the ability to perform far more complex calculations much faster.
The algorithms used have, therefore, become more complex in concert with those advances. Forget the “what took six weeks now takes six minutes” aspect — what wasn’t even reasonable to consider running on a computer is now reasonable.
Most of the algorithms running on computers these days aren’t new, as they weren’t developed for computing. They existed as digital laboratories for mathematical exercises or computing science theory. Now that theory has been coming into practice.
Companies Don’t Create Their Own Accounting Standards
Okay, Enron and other companies showed that, yes, they sometimes do. However, in the U.S. we have GAAP; and other standards exist in other nations. A team of very experienced people sat down and figured out something that everyone can apply without understanding the details that were studied in the creation of the standards.
The same is true with algorithms. A few mathematicians and computer scientists can think up complex new ways to analyze information. A larger but still focused group can implement those algorithms into software systems, and the far larger business world at large can leverage the analysis. However, those algorithms must meet with expected standards, both in business and other arenas. For instance, medical imaging algorithms must pass rigorous FDA approval processes.
The analogy is that the core algorithms, whether they be for accounting or data analysis, can be created by a few, while a far larger group can use those algorithms within their own organizations. Neither usage demands absolute rigidity, interpretations can vary, but the basic tool remains consistent and most people who leverage those tools need not understand the complexities behind the decision to use or the reason why the algorithms are created.