Combined Harvester

Let’s change gears, skip the code, and look at a harvester in action. Here’s the url for you to play with and a user-guide / description –

Data Harvesting: Stock Prices
From the S&P500 we choose up to 5 tickers for their daily adjusted-closes. Store those into memory with “Load Prices” against a time-frame of our choosing (as far back as the past decade).

That might take a few seconds, but don’t be too impatient: you just completed your first live harvest. There is no database running in the background, and hitting that button took you foraging into the electronic universe where you automatically catologued and stored data (as we had previously said you could).

A quick word on databases –

This obviously isn’t how we would ordinarily design our site (accessing in-house databases is much quicker) but the idea was to show you harvesting in action and to not get side-tracked.

See below for a quick mention on the GUI.

Randomness & Trend-Following
Now with those prices in-memory, let’s take the view that markets are essentially played at dart-boards (an early reference was “A Random Walk Down Wall Street”, Malkiel) but that price patterns occasionally show up as either trends or mean reversions. One way we might look for those trends is with fractals.

Fractal dimensions are used broadly (river networks, urban growth, human physiology etc.) as well as for market trends. Without getting into Alligator’s teeth(!) and the like, let’s here just assume that we would like to get a handle on those fractal dimensions as part of a trend-following investment strategy.

Counting Boxes & Branches
Box-counting is a known method for analysing complex patterns by breaking datasets, objects, images, etc. into variously-sized box-shaped pieces and asking how observations of detail change with scale. Indeed, fractal dimensions (themselves an older idea, but coined by everyone’s favourite, Mandelbrot, back in the 70s) quantify complexity as a ratio of that change in detail to the change in scale. (If you use Metatrader, or similar, the above will be old-hat to you.)

Analytics & Graphics
With prices in memory, we now perform box-counting at 3 different levels of zoom (2×2, 4×4 and 8×8). Just pick a quarter you want to look at (make sure it is within your date range) and hit “Generate Picture” to see it in action.

For any quarter, we can categorise prices as belonging to either the Low or High bin (y-axis) in timeframe bins measured as either Old or Recent (x-axis), in which case we have a 2 x 2 box. The 4×4 and 8×8 follow similarly.

A white square indicates a date-price bin that never happened; darkest squares the most frequently occurring date-prices. Fewer greys, and more white-space, imply stronger trends.

To capture fractal dimension, we capture how the number of grey squares varies with magnification. That is, the fractal dimension is in fact a gradient, and varies between 1.0 (a straight line) and 2.0 (very waggly line with no white-space).

For context: famously the coastline of Britain is 1.25.

Users don’t tend to want to play with code, so we add an interface to wrap around it. That overlay added a small bit of complexity to our code, but not significantly, and the harvester and analytics still remain its core.

GUI design is a topic of its own, but in this case we have deliberately left the design very light (few pages, logins, styling, choices etc.) as we don’t want to distract excessively versus the engine itself. For the curious, in this case we are using Flask.

Some Sources
If you want to read more, this site is very informative fractalfoundation as well as the usual wikipedia sites (Box_counting, Fractal_dimension) or even Investopedia.

Posted in trendfollowing, visualisation, webscraping.

Leave a Reply