“Extinction of old forms is the almost inevitable consequence of the production of new forms […]”
- Charles Darwin, The Origin of Species
There exists a deleterious belief about the financial world:
That financial markets are somehow too wild and random for machines to understand — they are simply too emotional, too susceptible to the whims of innumerable variables to decode.
Thus humans are indispensable for clever trading, and the careers of traditional speculators are as stable as their starched collars.
Earlier this month, new temporally-aware Cortical Learning AI (CL-AI) yielded 24% non-compounded annual returns trading a portfolio of the S&P e-Mini, Gold & Copper futures over four years. …
Last time we installed ArcGIS with minimum fuss and began exploring their mapmaking tools, including raster pixel tilings and searching for more pre-uploaded GIS items online.
However, there’s a fairly fundamental rule of data science:
The data you want usually isn’t in the place or shape you want it to be.
This is one reason why I believe data science careers are fairly safe from AI replacement. Machine learning is great at re-treading previously seen pathways, but the first 80% of the data process — ETL, including cleaning and migrations — is variable enough to require human input.
It could also be that the people researching AI algorithms aren’t interested in making themselves obsolete, but I doubt it; the drive to automate and simplify your own workflow is generally too high for such far-flung fears. …
In the digital age we never lack for information, but often have trouble finding useful knowledge. The answer is nuance.
Context makes data truly useful.
Add the Pareto Principle: Of all the total information about some topic, 80% of the “useful knowledge” is contained in about 20% of the net information.
Some context matters more than others.
You will often be reminded of this. We don’t mind petting a salamander but steer clear of alligators, even though their shape and function is largely the same.
It’s why Facebook knows what ads to show you, even though it doesn’t have access to all the thoughts in your brain (yet). …
The online dating giant OkCupid runs a blog here on Medium. Every now and then they put out something really fantastic, like this archived article from 2014 that highlighted some fascinating discrepancies on the battlefield of love.
It turned some heads and generated some delightfully disruptive graphs, and has since been deleted — probably because peering at the truth underneath oceans of data tends to be somewhat unpleasant.
You could, of course, argue that OkCupid’s data is biased or this interpretation is skewed. The dating app Hinge put out an analytics article in 2017, which reached quite similar conclusions.
One interesting quote compares ‘likes’ received by users to wealth distribution in…
As we move through the world, walking or flying in physical space, we also move through time. It’s well argued to be the 4th dimension, for good reason: it takes time to do anything.
But then it gets a bit more tricky — we’re probably in something resembling Minkowski Spacetime, which at its simplest means that time is bound together with space rather than being a separate, purely impartial ‘observer’.
This may clue us in as to why time dilation occurs at the edge of a black hole, among other things.
A core data science task is classification: grouping data points into various groups based on certain shared qualities.
In a sense, it’s an exercise as old as life itself: as soon as the first protozoan developed sensory organs, it (accidentally) started to act differently based on various sensory stimulus.
On a higher biological level, it’s a monkey looking at an object hanging from a branch and deciding “food” or “not food”.
On a machine level, it’s your ML model combing through credit transactions and deciding “fraud” or “not fraud”.
The connections between data can often tell us more than the data itself.
Nothing in this world exists in a vacuum — everything is a part of something else, every piece of information is interlinked with other data. Ignore context at your own risk.
But since graphs are everywhere — and there’s no shortage of ways to record them as simple data structures — how do we go about analyzing these graphs?
Well, you could start by feeding them into a neural network, experimenting until something goes horribly wrong, and then trying again with more graphs. …
I’ve written a few articles on Hierarchical Temporal Memory neural networks, which encode data into Sparse Distributed Representations to make noise-resistant predictions that consider multiple time-steps from the past input feed.
The key pioneer of HTM technologies, Numenta, holds weekly research meetings to discuss interesting new theories and advancements in machine learning and neuroscience.
Recently they invited Viviane Clay, a PhD student from the Institute of Cognitive Science in Osnabrück, to discuss her fascinating experiments in embodied reinforcement learning.
Her experimental reasoning can be summarized as such:
This is from John Nelson’s Hexperiment. He uses some wicked cool tools to draw hexagonal filters over NASA photos of the United States, further breaks those hexagons down into tessellating diamonds, and makes the whole thing fairly interactive with Chipotle location density data.
Last week I spent an entire article rambling about the intrinsic power of hexagons: how bees and our brain use the same structure to build maximally efficient grid networks.
While I work on other technically-heavy hexagonal grids, I wanted to share a simple and enjoyable plotting framework. Infuse your graphs with the apiary power of hexbins:
If you’re like me, you’ve always had a strange feeling that bees (much like dolphins) know more than they’re letting on.
They know architecture, and communicate navigational information through dance. Their elaborate caste-based hierarchy and collective survival impetus is striking. There is something mathematical about them, and that is deeply unsettling.
I’ve recently been looking into encoding data to binary representations for use in neuromorphic AI systems. I’ve always been fascinated with mathematical and artistic representations of space, distance and volume.
Searching for the intersection of these ideas has led me to the same destination as bees: the hexagon.
Let me make this quite clear: we are only starting to learn the extent to which honey-gathering insects know more than us. Bees build their hives with tessellating hexagons. What do they know? …