reading patterns in financial waves

source: UMass

There exists a deleterious belief about the financial world:

That financial markets are somehow too wild and random for machines to understand — they are simply too emotional, too susceptible to the whims of innumerable variables to decode.

Thus humans are indispensable for clever trading, and the careers of traditional speculators are as stable as their starched collars.

Nothing is Truly Random

Earlier this month, new temporally-aware Cortical Learning AI (CL-AI) yielded 24% non-compounded annual returns trading a portfolio of the…

geographic formatting & spatial dataframes

mapping roads of the world

Last time we installed ArcGIS with minimum fuss and began exploring their mapmaking tools, including raster pixel tilings and searching for more pre-uploaded GIS items online.

However, there’s a fairly fundamental rule of data science:

The data you want usually isn’t in the place or shape you want it to be.

This is one reason why I believe data science careers are fairly safe from AI replacement. Machine learning is great at re-treading previously seen pathways, but the first 80% of the data process — ETL, including cleaning and migrations — is variable enough to require human input.

It could…

all data has a location if you look hard enough


In the digital age we never lack for information, but often have trouble finding useful knowledge. The answer is nuance.

Add the Pareto Principle: Of all the total information about some topic, 80% of the “useful knowledge” is contained in about 20% of the net information.
Some context matters more than others.

You will often be reminded of this. We don’t mind petting a salamander but steer clear of alligators, even though their shape and function is largely the same.

It’s why Facebook knows what ads to show you, even though it doesn’t have access…

nature is cruel; statistics is worse

source: altmodern / Getty

The online dating giant OkCupid runs a blog here on Medium. Every now and then they put out something really fantastic, like this archived article from 2014 that highlighted some fascinating discrepancies on the battlefield of love.

It turned some heads and generated some delightfully disruptive graphs, and has since been deleted — probably because peering at the truth underneath oceans of data tends to be somewhat unpleasant.

You could, of course, argue that OkCupid’s data is biased or this interpretation is skewed. The dating app Hinge put out an analytics article in 2017, which reached quite similar conclusions.


black holes and temporal dimensions

The Persistence of Memory, Dalí, 1931

As we move through the world, walking or flying in physical space, we also move through time. It’s well argued to be the 4th dimension, for good reason: it takes time to do anything.

But then it gets a bit more tricky — we’re probably in something resembling Minkowski Spacetime, which at its simplest means that time is bound together with space rather than being a separate, purely impartial ‘observer’.

This may clue us in as to why time dilation occurs at the edge of a black hole, among other things.

supraspatial decisionmaking


A core data science task is classification: grouping data points into various groups based on certain shared qualities.

In a sense, it’s an exercise as old as life itself: as soon as the first protozoan developed sensory organs, it (accidentally) started to act differently based on various sensory stimulus.

On a higher biological level, it’s a monkey looking at an object hanging from a branch and deciding “food” or “not food”.
On a machine level, it’s your ML model combing through credit transactions and deciding “fraud” or “not fraud”.

relationships are complicated


The connections between data can often tell us more than the data itself.

Nothing in this world exists in a vacuum — everything is a part of something else, every piece of information is interlinked with other data. Ignore context at your own risk.

But since graphs are everywhere — and there’s no shortage of ways to record them as simple data structures — how do we go about analyzing these graphs?

Well, you could start by feeding them into a neural network, experimenting until something goes horribly wrong, and then trying again with more graphs. …

unpacking Viviene Clay’s findings in reinforcement learning


I’ve written a few articles on Hierarchical Temporal Memory neural networks, which encode data into Sparse Distributed Representations to make noise-resistant predictions that consider multiple time-steps from the past input feed.

The key pioneer of HTM technologies, Numenta, holds weekly research meetings to discuss interesting new theories and advancements in machine learning and neuroscience.
Recently they invited Viviane Clay, a PhD student from the Institute of Cognitive Science in Osnabrück, to discuss her fascinating experiments in embodied reinforcement learning.

Her experimental reasoning can be summarized as such:

  1. Observing nature can lead to clues on how to efficient perform a task

tessellation, resolution and you

John Nelson’s Hexperiment

This is from John Nelson’s Hexperiment. He uses some wicked cool tools to draw hexagonal filters over NASA photos of the United States, further breaks those hexagons down into tessellating diamonds, and makes the whole thing fairly interactive with Chipotle location density data.

Last week I spent an entire article rambling about the intrinsic power of hexagons: how bees and our brain use the same structure to build maximally efficient grid networks.

While I work on other technically-heavy hexagonal grids, I wanted to share a simple and enjoyable plotting framework. Infuse your graphs with the apiary power of hexbins:

bees figured this out ages ago


If you’re like me, you’ve always had a strange feeling that bees (much like dolphins) know more than they’re letting on.

They know architecture, and communicate navigational information through dance. Their elaborate caste-based hierarchy and collective survival impetus is striking. There is something mathematical about them, and that is deeply unsettling.

I’ve recently been looking into encoding data to binary representations for use in neuromorphic AI systems. I’ve always been fascinated with mathematical and artistic representations of space, distance and volume.

Searching for the intersection of these ideas has led me to the same destination as bees: the hexagon.


Mark Cleverley

data scientist, machine learning engineer. passionate about ecology, biotech and AI.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store