Ruminations-On Reality Vs the Mathematical Model Discussion
I read Joanna Kavena's article and my mind quickly drew parallels between my two passion subjects. Engineering and Sports. It took me weeks to figure out what my thoughts were. Here it is.
I recently read an article by Joanna Kavena in the Institute of Art and Ideas newsletter. https://iai.tv/articles/truth-is-deeper-than-mathematics-auid-3278
The teaser for the article explained the intent of the author:
Mathematics is a powerful tool. But increasingly, mathematic models are being mistaken for reality.
From fitness trackers to large language models, we're increasingly ruled by systems that quantify without understanding.
…. novelist and essayist Joanna Kavenna explores the rise of cyber-Pythagoreanism—the belief that reality can be captured through numbers — and argues that when we mistake data for truth, we risk replacing the real world with algorithmic fiction.
“Reality is ignored in favour of algorithmic fictions, which are mistaken for objective facts,” argues Kavenna.
The article piqued my imagination and I started to abstract the idea to those subjects that I am the most familiar with: engineering and sports. You are invited now to my Chautauqua.
Humans are by nature, wary of uncertainty while reality is by nature infinitely uncertain; because humans cannot change reality, we subconsciously substitute our perception of reality as reality in our minds. We create a mental model of reality from the information that we gain from our senses. Since we are dependent on our senses for our survival, we implicitly believe our perception of reality to be the truth, because we are unable to understand reality otherwise.
Humans are also great story tellers; we create narratives from our perceptions of reality and then we conveniently convince ourselves that these narratives are indeed the reality that is around us. This is also known as the narrative fallacy because it is a narrative emanating from our imagination after having been populated by our perceptions. Even those who are conscious of our proclivity to self-deceive through our perceptions and belief in our narratives will inevitably and subconsciously fall into the trap.
This concept is not new; Plato’s allegory of the Cave addresses the idea directly. (https://en.wikipedia.org/wiki/Allegory_of_the_cave).
In the allegory, Plato describes people who have spent their entire lives chained by their necks and ankles in front of an inner wall with a view of the empty outer wall of the cave. They observe shadows projected onto the outer wall by objects carried behind the inner wall by people who are invisible to the chained “prisoners” and who walk along the inner wall with a fire behind them, creating the shadows on the inner wall in front of the prisoners. The "sign bearers" pronounce the names of the objects, the sounds of which are reflected near the shadows and are understood by the prisoners as if they were coming from the shadows themselves.
Only the shadows and sounds are the prisoners' reality, which are not accurate representations of the real world. The shadows represent distorted and blurred copies of reality we can perceive through our senses, while the objects under the Sun represent the true forms of objects that we can only perceive through reason. Three higher levels exist: natural science; deductive mathematics, geometry, and logic; and the theory of forms.
According to Kavena, the modern incarnation of the Plato’s Cave allegory substitutes the projected shadows with the mathematical models; we have come to believe that the derived mathematical models are reality, rather than a representation of reality.
Why would we opt for creating models, mathematical or otherwise? I thought about those questions in terms of the two subjects I have chosen: the purpose for both is the same: to create models that can predict the long-term future and anticipate the short-term results.
In certain cases, mathematical models are close enough to reality for our purposes, i.e. the difference between the mathematical models and reality is close enough; while in most cases the gaps between mathematical models and reality are so large so that if they are used to predict the long-term future and anticipate the short-term future, these results are consistently misleading because they do not reflect the actual state of reality.
In engineering, the purpose is to create, analyze, and implement processes and products for applications. The purpose of being able to predict the long-term future is to enable the revolutionary advancement of the body of knowledge through using what we already know as the foundation of the new explorations; while the purpose of anticipating the short-term future is to enable the evolutionary and incremental development of new processes and products based on previously implemented and verified processes and products. Mathematical models are relied upon as factual replication of reality, encompassing what we know about reality through our understanding of nature to predict how previously never attempted designs will act and react. Additionally, mathematical models are used in analysis to resolve the difference between the mathematical model and the performance results, whether it is to troubleshoot errant results that deviate from what is anticipated or to correct glitches in the mathematical model to bring the model in line with reality.
In sports, the purpose is to use mathematical models to anticipate short-term human performances under competitive situations so that informed decisions can be made in comparing players, deciding on who should play and who should sit. There is also a desire in sports to predict, in the long term, how contests between teams would turn out. Fantasy leagues and sports book businesses have long been trying to use mathematical models to make long-term predictions. The accuracy of the long-term predictions has been dodgy at best and mythological at worst.
Since the “realities” that are modelled are very different between the two subjects, the varying successes in accomplishing the purposes of predicting and anticipating are expected.
The first question that needs to be asked, and not answered definitively, is: What is reality? That question can and has been explored throughout history in a philosophical framework. Limiting the meaning of reality in the context of the two subjects significantly narrows the focus and limits the discussion options.
What we do know is that reality is uncertain and to make matters more complex, the sources of uncertainty are uncertain and indeterminate for both subjects. Reality is usually experienced in the open loop, i.e., there is a significant amount of unpredictability when we experience reality. Which is exactly why it is impossible for us to fathom the depth and breadth of reality. We can base our conception of reality on what we know and what we perceive, but that is highly dependent on the fidelity of our perception of reality and whether our perceptions apply directly to the specific situation that we are navigating.
Reality can be incredibly simple, or it can be infinitely complex. The problem is that at times reality may seem straightforward when seen through our perceptions, which in our minds, simplicity triggers linear thinking, this could be a detriment in how we create our conception of reality through our perception because our perceptions are not reliable and more often than not, reality turns out to be considerably more complex than our ability to perceive. Indeed, reality can involve many dimensions and independent variables that are subtly coupled and interrelated. These complex interrelationships feeds back upon one another which affect how reality behaves. How much of these interrelationships affect how reality acts and reacts increases the complexity of our task: to be able to predict reality. (Forrester 1968) (Meadows 2008)
Reality in engineering is both dictated and constrained by structures that are superimposed on reality by nature: the physical laws as we understand them. Fortunately, reality in engineering is also associated with a preponderance of detailed legacy knowledge that is derived from prior experience, and substantiation of the laws of nature through those experiences. We have created paradigms within the sciences through a structure that roughly follows Thomas S. Kuhn’s essay (Kuhn 1970): the paradigms resulting from our definition of normal sciences, the appearances of anomalies which challenges the paradigms of normal sciences, and the creation of new paradigms which satisfactorily includes the anomalies in the new candidate paradigm. When sufficient empirical proof of new paradigms warrants the shifting of the old paradigm, the new paradigms are accepted and become the new normal science.
The reality in sports does not have a deterministic foundational structure that is equivalent to the foundational structure that buttresses engineering. The laws of nature do not play a role in the sports reality, outside how nature affects the playing of the sport: gravity, aerodynamics, et. al. The laws of nature do not directly impact on the desired purpose of sports: to anticipate human performance in competitive situations, to make the best decision while under specific situations, and to provide factual basis for anticipating individual human performances for comparison.
What the sports realities involve is the laws of human nature rather than the laws of nature, game actions are predicated upon how each person involved in the game acts and reacts both in general and in specific situations. While it is possible to anticipate human performance in the short-term, this is what coaches and players strive to do in training — to attain some level of consistency. Unfortunately, the individual variations due to human nature can never approach the same level of predictability as the laws of nature.
Even though the purpose of using mathematical models for both engineering and sports to represent reality are identical: having the ability to accurately and precisely anticipate short-term performance; what differentiates the two subjects drastically are the differences in the nature of the realities, those differences also drastically affect how mathematical models are derived and the ability of the mathematical models to meet the purposes.
One way to think of reality and the derived models is to recall the Indian parable of the blind men and the elephant. (https://en.wikipedia.org/wiki/Blind_men_and_an_elephant)
a group of blind men who have never come across an elephant before and who learn and imagine what the elephant is like by touching it. Each blind man feels a different part of the animal's body, but only one part, such as the side or the tusk. They then describe the animal based on their limited experience and their descriptions of the elephant are different from each other. … The moral of the parable is that humans have a tendency to claim absolute truth based on their limited, subjective experience as they ignore other people's limited, subjective experiences which may be equally true.
The lesson? It is impossible to completely comprehend reality through just our senses; we can only draw boundaries around reality and comprehend that partitioned bit of reality to our best ability; and those boundaries define the viability of the models for meeting the purposes.
In engineering, the robustness of the structures — the certainties afforded by the laws of nature, the a priori knowledge and experiences accrued over time, and the testing and verification of the mathematical models enhance the mathematical model’s ability to anticipate in the short-term future because there is less variability present in the mathematical model as compared to reality. This increase in the level of certainty in the mathematical model allows the engineer to create new processes and products using prior designs, which are based on prior experience and empirical evidence. It saves the engineer from having to reinvent the wheel every time they are charged with creating something new. In addition, having a robust mathematical model gives the engineers the tools to resolve many differences between the anticipated performance of proposed new design versus the experimental testing performed on the new designs.
An advantage afforded by the laws of nature is that those laws that dictate the mathematical model are also constraints imposed on the model. The laws of nature thus serve two purposes: it defines both the mathematical model and the boundaries of the mathematical model of reality. There are instances, however, when those constraints tend to over constrain the mathematical model to such a point that it hampers the purpose anticipating the long-term future. It is because the mathematical model is so dependent on the accumulation of prior knowledge that the mathematical model is unable to approximate a reality that has not been experienced and asked to extrapolate beyond the bounds of the prior knowledge. In engineering, the mathematical model had been met with an anomaly and the mathematical model needs to be adjusted to anticipate the long-term future.
In sports, while the purpose is also to anticipate performance, the desired anticipation capability is in a much broader context. The performance to be anticipated in sports is human performance as players are competing with other players’ performances while playing the game. While the mathematical model is also based on experience and data, the lack of the established laws of nature as foundational structure not only hampers the model building but also leads to less accuracy and predictability.
Mathematical models in sports are based on statistics, mostly game statistics. (https://thecuriouspolymath.substack.com/p/stats-for-spikes-termination-scoring) Descriptive statistics are invaluable in many ways: they describe the reality of the sporting contest in inviolate numbers, which gives the user a well-defined representation of what is happening during the competition, even though the accuracy of the representation is dependent on how the data is collected. Statistical data can be considered to be deterministic if the data taking process is unbiased and consistent. Statistics also shed light on the veracity of the perceptions of the participants: coaches, players, and officials. The caveat regarding the deterministic nature of the termination statistical data is that intermediate actions that proceed from dead ball to dead ball sequences are missing. Sports action is a chain of events which are linked by causal relationships, one action leads to another. This is why sports are often modelled using conditional probabilities in the form of Markov chains. (https://polymathtobe.blogspot.com/2021/03/stats-for-spikes-markov-chains.html) If any of the events are not recorded, which often are not, the statistical data set is incomplete, the representation of reality is also incomplete. This is not to say that incomplete data has been a showstopper, it is just that this incompleteness leads to ambiguities and lack of clarity in the meaning of the statistics, which become amplified exponentially when decisions are made in the absence of the data.
Even though engineering models also use statistical analysis to augment the laws of nature, the engineering mathematical model is anchored and constrained on the laws of nature, and the statistical analysis is used for verification purposes, i.e., to account for the uncertainties in the measurement and verification process.
Ever since Michael Lewis wrote Moneyball, the sporting world has built analytics capabilities at every level of competition. The upsurge in analytics tools and personnel in the ranks of sports team personnel has been staggering. The insight gained has been impressive, the increased anticipability of player performance achieved through analytics has shone new light on player abilities, although it has not been as deterministic as the engineering mathematical model. This can be attributable to factors that make engineering mathematical models less susceptible to uncertainty: scientific laws governing modeling and constraining, accuracy and precision of measurement, and experimental verification.
The sports mathematical model is mostly descriptive: what happened in games and what happened in a cluster of games. It gave the sports world a clearer picture of reality at that point and time. The accrual of consistent statistical data also allows limited inferencing ability, but just short-term inferences because the nature of sports statistics may or may not meet the criteria for accurate deployment of statistical inferencing. The amount of data collected is usually not enough to meet the criteria for statistical inference, the statistics being termination point statistics cause the statistically based models to be missing vast amounts of data that cannot be collected, or if they could, tend to be subjective in nature.
Unlike engineering mathematical models, the sports statistically based mathematical model does not automatically include the naturally occurring constraints which come from the laws of nature that is the basis of engineering mathematical models. This is an important omission because the constraints serve to limit the size of the mathematical models by excluding implausible solution spaces.
The question is: why did sports analytics succeed so spectacularly given their apparent shortcomings? My conjecture is that since there had not been any mathematical model to speak of, and a severe lack of confidence and understanding of statistics in general, decisions were made at the critical levels by gut feel and the eye test, neither had been numerically verified. What analytics did was to eliminate the uncertainties associated with heuristics and rules of thumb which also obviated myths and erroneous beliefs; unfortunately, other sources of uncertainties remained within the statistically based mathematical models.
Drawing an analogy with gambling; analytics served the same role as the players counting cards while playing, it leveled the playing field between the house and the player, but the player did not gain an overwhelming advantage over the house, it just seemed that way.
Verification is a mandatory activity before deploying engineering mathematical models. It is concomitant with the creation of the mathematical model. There are two levels of verification: simulation and experimental testing. The computer simulation step does not verify the accuracy of the model with reality; it does verify whether the mathematical model performs as it is derived and whether the mathematical model strictly follows the laws of nature that are built into the mathematical model. As simulation software has evolved to be faster and more accurate, some have made the procedural leap of faith to conclude that simulation can be just as good if not better than experimental verification, as experiments also introduces uncertainties as a part of the data measurement process; it is also not economically viable to constantly test and verify the mathematical model at every step of the process, simulations thus serves the purpose of checking the verity of the mathematical model. What is ignored is that the simulation results are only as accurate as the mathematical model itself. If there exists unmodelled dynamics that is integral to how reality acts and reacts, the simulation results will be inaccurate, reference the parable of the elephant and the blind men; it is probable that the composite descriptions from all the blind men have significant gaps in their perception of the elephant as compared to reality.
There is an apocryphal anecdote from my gradual school days regarding the early development of adaptive pilot systems on fighter jets. The adage that describes the problem was: adapt and learn or crash and burn. Many test pilots had to eject and crash their planes while testing these adaptive systems because the adaptive system models failed to include the unmodelled dynamics that is a part of the airplane dynamics so that even though live pilots can adjust and overcome the missing dynamics, the adaptive autopilot systems were not able to learn in time to solve the unmodelled dynamics problem, they crashed and burned.
After the mathematical model has been verified as being accurate to the laws of nature governing reality as we understand reality, the experimental testing phase is deployed to determine whether the mathematical model had faithfully duplicated reality, within the acceptable limits of uncertainty as determined by statistical analysis. If the disagreement between the computed results from the mathematical model is greater than a computed limit, two sets of action should be initiated: re-examining the simulation results for any loss of accuracy and the lack of accuracy of the mathematical models as compared to reality. This involves the painstaking and prolonged labor of granularly tracking the propagation of errors through the entire experimental testing procedure. The resulting action ranges from duplicating the testing procedure to track down the source of experimental error to challenging the accuracy of the mathematical models as their anticipated information is compared to empirical data, and everything in between.
As stated before, the human element dramatically increased the uncertainty factor in the sports mathematical modelling, the lack of solid guardrails like the laws of nature also increases uncertainties and decreases mathematical constraints that engineering mathematical models do not have to reckon with. The measurement aspect in sports is also subject to uncertainties, much as the measurement aspects increase the uncertainties in the engineering context — even though new statistics have been generated that are more informative, they are still lacking the accuracy and precision that the engineering measurements enjoy.
In essence, there are no such thing as experimental verification of statistically based mathematical models in sports. There is no meaningful and broad means of verifying statistical models to assess the predictability of the mathematical model. Even as more sets of data are generated because more games are being played and reams of data are taken, the uncertainty of the sports mathematical model will always hamper the attempts at predicting the long-term future; it is impossible to integrate finite number of data sets into the mathematical model when the goal is to gain the ability to predict infinitely into the future.
Mathematical models in sports are dependent on probabilities, which account for the large variations and uncertainties in the calculation of the performance metric. The probability functions, while informative about the measured variables, will also propagate the uncertainties and variations onto the mathematical model.
The statistically based sports mathematical model can only reflect what had happened in history, the models are descriptive, not necessarily inferential. This means that the statistical models are not suitable for prediction in the long-term, there are strict requirements placed on the statistical models to qualify them for application as inferential statistics. Many have used statistically based mathematical models inferentially to try to gain short-term anticipability, with varying degrees of success. As the narrative of Moneyball can attest, there are many instances of success in using analytics to uncover underlying trends in performance metrics, many performance metrics have revealed the hidden values of many players that would have been obfuscated because of the gutfeel and eye test methods of evaluating players. I wonder, however, whether the important revelation is the value of analytics or the lack of value of gutfeel and eye test methods?
Parenthetically, Joanna Kavena cited the flood of AI into our landscape as her main motivation to discuss the use of mathematical models to replace reality. Indeed, the advent of the readily available AI engines is forcibly and perhaps inevitably replacing our preferred shadows on the cave walls with replacing human perception-based shadows with AI based shadows on the cave walls.
I have a rudimentary and shallow understanding of how the present-day form of AI operates after having read some general public directed books on the subject (Gary Marcus 2006) (Mitchell 2019), so I am willing to learn from those who are better versed in the technology, but I dip my toe into the speculations about AI.
One thing that stands out is that the AI engines that have been foisted upon us are not Artificial General Intelligence (AGI) engines, they are Large Language Model (LLM) based engines. What does that mean? LLM engines employ massive training databases to achieve their capacity to rapidly and overwhelmingly churn out information that pertains to the user prompts. The speed of the LLM engines and the breadth, and depth of the LLM engine results are impressive, but the LLM does not possess the human ability to accept plausible information and reject implausible information. The LLM engines regurgitates what it has “learned” from the massive databases that was used to train the LLM. The sheer amount of information presented in a short time is truly a time saver, but all that information cannot and should not be accepted without scrutiny as the LLM engines tend to hallucinate and present non-existing information as true information. These hallucinations are embedded in the large set of information as factual representation of reality, camouflaging its fictional nature. It is for that reason that I am weary of its potential for misinformation. Yet, I do not wish to be left behind and be thought of as a Luddite, I have devoted my life to the advancement of technology, so being a Luddite is contrary to my training and indoctrination. Once again, I draw on the blind men and the elephant parable to illustrate the point: if reality is not accurately portrayed in the perception (LLM training data), then the mathematical model does not represent reality.
I view the LLM based engines as yet another law of nature based mathematical model, subject to the same rigorous scrutiny and verification as any other mathematical models. The main difference is that the models are difficult to verify because the output space is so large and the potential of hallucinations being included in the information creates more work to rigorously determine the LLM model’s agreement to reality.
In the engineering context, since the mathematical model is based on the laws of nature and is strictly constrained by what we already know and have verified with testing against reality, the application of LLM models to engineering mathematical models is seemingly less problematic.
In the sports context, since the mathematical model is so fraught with uncertainties and potential errors when used to draw inferences to begin with, the potential use of LLM models is seemingly futile at the present. I can’t speak for the future.
Finally, as the pioneer of Artificial Intelligence Marvin Minsky once observed: Easy things are hard. Which can be interpreted as: things that are seemingly easy to execute in human experiences are hard to implement in AI because of our human blind spot, the amount of common sense, subconscious abilities to apply our experiences, and deep understanding of the heuristic knowledge that we possess which are needed to do the simple things. Much of those capabilities are considered to be so trivial that we fail to recognize them as critical. This is not to say that AI is a dead end, on the contrary, the development of the idea which germinated in Dartmouth in 1956 has come a long way, but at this stage, what is needed is AGI and not LLM.
This is the end of my Chautauqua on my thought on Kavena’s article as it applies in my two playgrounds. As always, I am open to discussions and civilized conversations with knowledgeable people.
References
Forrester, Jay Wright. Principles of Systems. Waltham MA: Pegassus Communications, 1968.
Gary Marcus, Ernest Davis. Rebooting AI Building Artificial Intelligence We Can Trust. New York City: Pantheon Books, 2006.
Kuhn, Thomas S. The Structure of Scientific Revolutions. Chicago: University of Chicago Press, 1970.
Meadows, Donella. Thinking in Systems A Primer. NYC: Chelsea Green, 2008.
Mitchell, Melanie. Artificial Intelligence: A Guide for Thinking Humans. NYC: Farrar, Straus, and Giroux, 2019.