142 14.2.4 Portfolio This section describes a Portfolio object that keeps track of the positions within a portfolio and generates orders of a fixed quantity of stock based on signals. More sophisticated portfolio objects could include risk management and position sizing tools (such as the Kelly Criterion). In fact, in the following chapters we will add such tools to some of our trading strategies to see how they compare to a more "naive" portfolio approach. The portfolio order management system is possibly the most complex component of an eventdriven backtester. Its role is to keep track of all current market positions as well as the market value of the positions (known as the "holdings"). This is simply an estimate of the liquidation value of the position and is derived in part from the data handling facility of the backtester. In addition to the positions and holdings management the portfolio must also be aware of risk factors and position sizing techniques in order to optimise orders that are sent to a brokerage or other form of market access. Unfortunately, Portfolio and Order Management Systems (OMS) can become rather complex! Thus I’ve made a decision here to keep the Portfolio object relatively straightforward, so that you can understand the key ideas and how they are implemented. The nature of an object-oriented design is that it allows, in a natural way, the extension to more complex situations later on. Continuing in the vein of the Event class hierarchy a Portfolio object must be able to handle SignalEvent objects, generate OrderEvent objects and interpret FillEvent objects to update positions. Thus it is no surprise that the Portfolio objects are often the largest component of event-driven systems, in terms of lines of code (LOC). We create a new file portfolio.py and import the necessary libraries. These are the same as most of the other class implementations, with the exception that Portfolio is NOT going to be an abstract base class. Instead it will be normal base class. This means that it can be instantiated and thus is useful as a "first go" Portfolio object when testing out new strategies. Other Portfolios can be derived from it and override sections to add more complexity. For completeness, here is the performance.py file: #!/usr/bin/python # -*- coding: utf-8 -*- # performance.py from __future__ import print_function import numpy as np import pandas as pd def create_sharpe_ratio(returns, periods=252): """ Create the Sharpe ratio for the strategy, based on a benchmark of zero (i.e. no risk-free rate information). Parameters: returns - A pandas Series representing period percentage returns. periods - Daily (252), Hourly (252*6.5), Minutely(252*6.5*60) etc. """ return np.sqrt(periods) * (np.mean(returns)) / np.std(returns) def create_drawdowns(pnl): """ Calculate the largest peak-to-trough drawdown of the PnL curve as well as the duration of the drawdown. Requires that the pnl_returns is a pandas Series.
143 Parameters: pnl - A pandas Series representing period percentage returns. Returns: drawdown, duration - Highest peak-to-trough drawdown and duration. """ # Calculate the cumulative returns curve # and set up the High Water Mark hwm = [0] # Create the drawdown and duration series idx = pnl.index drawdown = pd.Series(index = idx) duration = pd.Series(index = idx) # Loop over the index range for t in range(1, len(idx)): hwm.append(max(hwm[t-1], pnl[t])) drawdown[t]= (hwm[t]-pnl[t]) duration[t]= (0 if drawdown[t] == 0 else duration[t-1]+1) return drawdown, drawdown.max(), duration.max() Here is the import listing for the Portfolio.py file. We need to import the floor function from the math library in order to generate integer-valued order sizes. We also need the FillEvent and OrderEvent objects since the Portfolio handles both. Notice also that we are adding two additional functions, create_sharpe_ratio and create_drawdowns, both from the performance.py file described above. #!/usr/bin/python # -*- coding: utf-8 -*- # portfolio.py from __future__ import print_function import datetime from math import floor try: import Queue as queue except ImportError: import queue import numpy as np import pandas as pd from event import FillEvent, OrderEvent from performance import create_sharpe_ratio, create_drawdowns The initialisation of the Portfolio object requires access to the bars DataHandler, the events Event Queue, a start datetime stamp and an initial capital value (defaulting to 100,000 USD). The Portfolio is designed to handle position sizing and current holdings, but will carry out trading orders in a "dumb" manner by simply sending them directly to the brokerage with a predetermined fixed quantity size, irrespective of cash held. These are all unrealistic assumptions, but they help to outline how a portfolio order management system (OMS) functions in an eventdriven fashion.
144 The portfolio contains the all_positions and current_positions members. The former stores a list of all previous positions recorded at the timestamp of a market data event. A position is simply the quantity of the asset held. Negative positions mean the asset has been shorted. The latter current_positions dictionary stores contains the current positions for the last market bar update, for each symbol. In addition to the positions data the portfolio stores holdings, which describe the current market value of the positions held. "Current market value" in this instance means the closing price obtained from the current market bar, which is clearly an approximation, but is reasonable enough for the time being. all_holdings stores the historical list of all symbol holdings, while current_holdings stores the most up to date dictionary of all symbol holdings values: # portfolio.py class Portfolio(object): """ The Portfolio class handles the positions and market value of all instruments at a resolution of a "bar", i.e. secondly, minutely, 5-min, 30-min, 60 min or EOD. The positions DataFrame stores a time-index of the quantity of positions held. The holdings DataFrame stores the cash and total market holdings value of each symbol for a particular time-index, as well as the percentage change in portfolio total across bars. """ def __init__(self, bars, events, start_date, initial_capital=100000.0): """ Initialises the portfolio with bars and an event queue. Also includes a starting datetime index and initial capital (USD unless otherwise stated). Parameters: bars - The DataHandler object with current market data. events - The Event Queue object. start_date - The start date (bar) of the portfolio. initial_capital - The starting capital in USD. """ self.bars = bars self.events = events self.symbol_list = self.bars.symbol_list self.start_date = start_date self.initial_capital = initial_capital self.all_positions = self.construct_all_positions() self.current_positions = dict( (k,v) for k, v in \ [(s, 0) for s in self.symbol_list] ) self.all_holdings = self.construct_all_holdings() self.current_holdings = self.construct_current_holdings() The following method, construct_all_positions, simply creates a dictionary for each symbol, sets the value to zero for each and then adds a datetime key, finally adding it to a list. It uses a dictionary comprehension, which is similar in spirit to a list comprehension: # portfolio.py
145 def construct_all_positions(self): """ Constructs the positions list using the start_date to determine when the time index will begin. """ d = dict( (k,v) for k, v in [(s, 0) for s in self.symbol_list] ) d[’datetime’] = self.start_date return [d] The construct_all_holdings method is similar to the above but adds extra keys for cash, commission and total, which respectively represent the spare cash in the account after any purchases, the cumulative commission accrued and the total account equity including cash and any open positions. Short positions are treated as negative. The starting cash and total account equity are both set to the initial capital. In this manner there are separate "accounts" for each symbol, the "cash on hand", the "commission" paid (Interactive Broker fees) and a "total" portfolio value. Clearly this does not take into account margin requirements or shorting constraints, but is sufficient to give you a flavour of how such an OMS is created: # portfolio.py def construct_all_holdings(self): """ Constructs the holdings list using the start_date to determine when the time index will begin. """ d = dict( (k,v) for k, v in [(s, 0.0) for s in self.symbol_list] ) d[’datetime’] = self.start_date d[’cash’] = self.initial_capital d[’commission’] = 0.0 d[’total’] = self.initial_capital return [d] The following method, construct_current_holdings is almost identical to the method above except that it doesn’t wrap the dictionary in a list, because it is only creating a single entry: # portfolio.py def construct_current_holdings(self): """ This constructs the dictionary which will hold the instantaneous value of the portfolio across all symbols. """ d = dict( (k,v) for k, v in [(s, 0.0) for s in self.symbol_list] ) d[’cash’] = self.initial_capital d[’commission’] = 0.0 d[’total’] = self.initial_capital return d On every heartbeat, that is every time new market data is requested from the DataHandler object, the portfolio must update the current market value of all the positions held. In a live trading scenario this information can be downloaded and parsed directly from the brokerage, but for a backtesting implementation it is necessary to calculate these values manually from the bars DataHandler. Unfortunately there is no such as thing as the "current market value" due to bid/ask spreads and liquidity issues. Thus it is necessary to estimate it by multiplying the quantity of the asset held by a particular approximate "price". The approach I have taken here is to use the closing
146 price of the last bar received. For an intraday strategy this is relatively realistic. For a daily strategy this is less realistic as the opening price can differ substantially from the closing price. The method update_timeindex handles the new holdings tracking. It firstly obtains the latest prices from the market data handler and creates a new dictionary of symbols to represent the current positions, by setting the "new" positions equal to the "current" positions. The current positions are only modified when a FillEvent is obtained, which is handled later on in the portfolio code. The method then appends this set of current positions to the all_positions list. The holdings are then updated in a similar manner, with the exception that the market value is recalculated by multiplying the current positions count with the closing price of the latest bar. Finally the new holdings are appended to all_holdings: # portfolio.py def update_timeindex(self, event): """ Adds a new record to the positions matrix for the current market data bar. This reflects the PREVIOUS bar, i.e. all current market data at this stage is known (OHLCV). Makes use of a MarketEvent from the events queue. """ latest_datetime = self.bars.get_latest_bar_datetime( self.symbol_list[0] ) # Update positions # ================ dp = dict( (k,v) for k, v in [(s, 0) for s in self.symbol_list] ) dp[’datetime’] = latest_datetime for s in self.symbol_list: dp[s] = self.current_positions[s] # Append the current positions self.all_positions.append(dp) # Update holdings # =============== dh = dict( (k,v) for k, v in [(s, 0) for s in self.symbol_list] ) dh[’datetime’] = latest_datetime dh[’cash’] = self.current_holdings[’cash’] dh[’commission’] = self.current_holdings[’commission’] dh[’total’] = self.current_holdings[’cash’] for s in self.symbol_list: # Approximation to the real value market_value = self.current_positions[s] * \ self.bars.get_latest_bar_value(s, "adj_close") dh[s] = market_value dh[’total’] += market_value # Append the current holdings self.all_holdings.append(dh) The method update_positions_from_fill determines whether a FillEvent is a Buy or a Sell and then updates the current_positions dictionary accordingly by adding/subtracting
147 the correct quantity of shares: # portfolio.py def update_positions_from_fill(self, fill): """ Takes a Fill object and updates the position matrix to reflect the new position. Parameters: fill - The Fill object to update the positions with. """ # Check whether the fill is a buy or sell fill_dir = 0 if fill.direction == ’BUY’: fill_dir = 1 if fill.direction == ’SELL’: fill_dir = -1 # Update positions list with new quantities self.current_positions[fill.symbol] += fill_dir*fill.quantity The corresponding update_holdings_from_fill is similar to the above method but updates the holdings values instead. In order to simulate the cost of a fill, the following method does not use the cost associated from the FillEvent. Why is this? Simply put, in a backtesting environment the fill cost is actually unknown (the market impact and the depth of book are unknown) and thus is must be estimated. Thus the fill cost is set to the the "current market price", which is the closing price of the last bar. The holdings for a particular symbol are then set to be equal to the fill cost multiplied by the transacted quantity. For most lower frequency trading strategies in liquid markets this is a reasonable approximation, but at high frequency these issues will need to be considered in a production backtest and live trading engine. Once the fill cost is known the current holdings, cash and total values can all be updated. The cumulative commission is also updated: # portfolio.py def update_holdings_from_fill(self, fill): """ Takes a Fill object and updates the holdings matrix to reflect the holdings value. Parameters: fill - The Fill object to update the holdings with. """ # Check whether the fill is a buy or sell fill_dir = 0 if fill.direction == ’BUY’: fill_dir = 1 if fill.direction == ’SELL’: fill_dir = -1 # Update holdings list with new quantities fill_cost = self.bars.get_latest_bar_value(fill.symbol, "adj_close") cost = fill_dir * fill_cost * fill.quantity self.current_holdings[fill.symbol] += cost self.current_holdings[’commission’] += fill.commission self.current_holdings[’cash’] -= (cost + fill.commission)
148 self.current_holdings[’total’] -= (cost + fill.commission) The pure virtual update_fill method from the Portfolio class is implemented here. It simply executes the two preceding methods, update_positions_from_fill and update_holdings_from_fill, upon receipt of a fill event: # portfolio.py def update_fill(self, event): """ Updates the portfolio current positions and holdings from a FillEvent. """ if event.type == ’FILL’: self.update_positions_from_fill(event) self.update_holdings_from_fill(event) While the Portfolio object must handle FillEvents, it must also take care of generating OrderEvents upon the receipt of one or more SignalEvents. The generate_naive_order method simply takes a signal to go long or short an asset, sending an order to do so for 100 shares of such an asset. Clearly 100 is an arbitrary value, and will clearly depend upon the portfolio total equity in a production simulation. In a realistic implementation this value will be determined by a risk management or position sizing overlay. However, this is a simplistic Portfolio and so it "naively" sends all orders directly from the signals, without a risk system. The method handles longing, shorting and exiting of a position, based on the current quantity and particular symbol. Corresponding OrderEvent objects are then generated: # portfolio.py def generate_naive_order(self, signal): """ Simply files an Order object as a constant quantity sizing of the signal object, without risk management or position sizing considerations. Parameters: signal - The tuple containing Signal information. """ order = None symbol = signal.symbol direction = signal.signal_type strength = signal.strength mkt_quantity = 100 cur_quantity = self.current_positions[symbol] order_type = ’MKT’ if direction == ’LONG’ and cur_quantity == 0: order = OrderEvent(symbol, order_type, mkt_quantity, ’BUY’) if direction == ’SHORT’ and cur_quantity == 0: order = OrderEvent(symbol, order_type, mkt_quantity, ’SELL’) if direction == ’EXIT’ and cur_quantity > 0: order = OrderEvent(symbol, order_type, abs(cur_quantity), ’SELL’) if direction == ’EXIT’ and cur_quantity < 0: order = OrderEvent(symbol, order_type, abs(cur_quantity), ’BUY’)
149 return order The update_signal method simply calls the above method and adds the generated order to the events queue: # portfolio.py def update_signal(self, event): """ Acts on a SignalEvent to generate new orders based on the portfolio logic. """ if event.type == ’SIGNAL’: order_event = self.generate_naive_order(event) self.events.put(order_event) The penultimate method in the Portfolio is the generation of an equity curve. This simply creates a returns stream, useful for performance calculations, and then normalises the equity curve to be percentage based. Thus the account initial size is equal to 1.0, as opposed to the absolute dollar amount: # portfolio.py def create_equity_curve_dataframe(self): """ Creates a pandas DataFrame from the all_holdings list of dictionaries. """ curve = pd.DataFrame(self.all_holdings) curve.set_index(’datetime’, inplace=True) curve[’returns’] = curve[’total’].pct_change() curve[’equity_curve’] = (1.0+curve[’returns’]).cumprod() self.equity_curve = curve The final method in the Portfolio is the output of the equity curve and various performance statistics related to the strategy. The final line outputs a file, equity.csv, to the same directory as the code, which can loaded into a Matplotlib Python script (or a spreadsheet such as MS Excel or LibreOffice Calc) for subsequent analysis. Note that the Drawdown Duration is given in terms of the absolute number of "bars" that the drawdown carried on for, as opposed to a particular timeframe. def output_summary_stats(self): """ Creates a list of summary statistics for the portfolio. """ total_return = self.equity_curve[’equity_curve’][-1] returns = self.equity_curve[’returns’] pnl = self.equity_curve[’equity_curve’] sharpe_ratio = create_sharpe_ratio(returns, periods=252*60*6.5) drawdown, max_dd, dd_duration = create_drawdowns(pnl) self.equity_curve[’drawdown’] = drawdown stats = [("Total Return", "%0.2f%%" % \ ((total_return - 1.0) * 100.0)), ("Sharpe Ratio", "%0.2f" % sharpe_ratio), ("Max Drawdown", "%0.2f%%" % (max_dd * 100.0)), ("Drawdown Duration", "%d" % dd_duration)]
150 self.equity_curve.to_csv(’equity.csv’) return stats The Portfolio object is the most complex aspect of the entire event-driven backtest system. The implementation here, while intricate, is relatively elementary in its handling of positions. 14.2.5 Execution Handler In this section we will study the execution of trade orders by creating a class hierarchy that will represent a simulated order handling mechanism and ultimately tie into a brokerage or other means of market connectivity. The ExecutionHandler described here is exceedingly simple, since it fills all orders at the current market price. This is highly unrealistic, but serves as a good baseline for improvement. As with the previous abstract base class hierarchies, we must import the necessary properties and decorators from the abc library. In addition we need to import the FillEvent and OrderEvent: #!/usr/bin/python # -*- coding: utf-8 -*- # execution.py from __future__ import print_function from abc import ABCMeta, abstractmethod import datetime try: import Queue as queue except ImportError: import queue from event import FillEvent, OrderEvent The ExecutionHandler is similar to previous abstract base classes and simply has one pure virtual method, execute_order: # execution.py class ExecutionHandler(object): """ The ExecutionHandler abstract class handles the interaction between a set of order objects generated by a Portfolio and the ultimate set of Fill objects that actually occur in the market. The handlers can be used to subclass simulated brokerages or live brokerages, with identical interfaces. This allows strategies to be backtested in a very similar manner to the live trading engine. """ __metaclass__ = ABCMeta @abstractmethod def execute_order(self, event): """ Takes an Order event and executes it, producing a Fill event that gets placed onto the Events queue.
151 Parameters: event - Contains an Event object with order information. """ raise NotImplementedError("Should implement execute_order()") In order to backtest strategies we need to simulate how a trade will be transacted. The simplest possible implementation is to assume all orders are filled at the current market price for all quantities. This is clearly extremely unrealistic and a big part of improving backtest realism will come from designing more sophisticated models of slippage and market impact. Note that the FillEvent is given a value of None for the fill_cost (see the penultimate line in execute_order) as we have already taken care of the cost of fill in the Portfolio object described above. In a more realistic implementation we would make use of the "current" market data value to obtain a realistic fill cost. I have simply utilised ARCA as the exchange although for backtesting purposes this is purely a string placeholder. In a live execution environment this venue dependence would be far more important: # execution.py class SimulatedExecutionHandler(ExecutionHandler): """ The simulated execution handler simply converts all order objects into their equivalent fill objects automatically without latency, slippage or fill-ratio issues. This allows a straightforward "first go" test of any strategy, before implementation with a more sophisticated execution handler. """ def __init__(self, events): """ Initialises the handler, setting the event queues up internally. Parameters: events - The Queue of Event objects. """ self.events = events def execute_order(self, event): """ Simply converts Order objects into Fill objects naively, i.e. without any latency, slippage or fill ratio problems. Parameters: event - Contains an Event object with order information. """ if event.type == ’ORDER’: fill_event = FillEvent( datetime.datetime.utcnow(), event.symbol, ’ARCA’, event.quantity, event.direction, None ) self.events.put(fill_event)
152 14.2.6 Backtest We are now in a position to create the Backtest class hierarchy. The Backtest object encapsulates the event-handling logic and essentially ties together all of the other classes that we have discussed above. The Backtest object is designed to carry out a nested while-loop event-driven system in order to handle the events placed on the Event Queue object. The outer while-loop is known as the "heartbeat loop" and decides the temporal resolution of the backtesting system. In a live environment this value will be a positive number, such as 600 seconds (every ten minutes). Thus the market data and positions will only be updated on this timeframe. For the backtester described here the "heartbeat" can be set to zero, irrespective of the strategy frequency, since the data is already available by virtue of the fact it is historical! We can run the backtest at whatever speed we like, since the event-driven system is agnostic to when the data became available, so long as it has an associated timestamp. Hence I’ve only included it to demonstrate how a live trading engine would function. The outer loop thus ends once the DataHandler lets the Backtest object know, by using a boolean continue_backtest attribute. The inner while-loop actually processes the signals and sends them to the correct component depending upon the event type. Thus the Event Queue is continually being populated and depopulated with events. This is what it means for a system to be event-driven. The first task is to import the necessary libraries. We import pprint ("pretty-print"), because we want to display the stats in an output-friendly manner: #!/usr/bin/python # -*- coding: utf-8 -*- # backtest.py from __future__ import print_function import datetime import pprint try: import Queue as queue except ImportError: import queue import time The initialisation of the Backtest object requires the CSV directory, the full symbol list of traded symbols, the initial capital, the heartbeat time in milliseconds, the start datetime stamp of the backtest as well as the DataHandler, ExecutionHandler, Portfolio and Strategy objects. A Queue is used to hold the events. The signals, orders and fills are counted: # backtest.py class Backtest(object): """ Enscapsulates the settings and components for carrying out an event-driven backtest. """ def __init__( self, csv_dir, symbol_list, initial_capital, heartbeat, start_date, data_handler, execution_handler, portfolio, strategy ): """ Initialises the backtest.
153 Parameters: csv_dir - The hard root to the CSV data directory. symbol_list - The list of symbol strings. intial_capital - The starting capital for the portfolio. heartbeat - Backtest "heartbeat" in seconds start_date - The start datetime of the strategy. data_handler - (Class) Handles the market data feed. execution_handler - (Class) Handles the orders/fills for trades. portfolio - (Class) Keeps track of portfolio current and prior positions. strategy - (Class) Generates signals based on market data. """ self.csv_dir = csv_dir self.symbol_list = symbol_list self.initial_capital = initial_capital self.heartbeat = heartbeat self.start_date = start_date self.data_handler_cls = data_handler self.execution_handler_cls = execution_handler self.portfolio_cls = portfolio self.strategy_cls = strategy self.events = queue.Queue() self.signals = 0 self.orders = 0 self.fills = 0 self.num_strats = 1 self._generate_trading_instances() The first method, _generate_trading_instances, attaches all of the trading objects (DataHandler, Strategy, Portfolio and ExecutionHandler) to various internal members: # backtest.py def _generate_trading_instances(self): """ Generates the trading instance objects from their class types. """ print( "Creating DataHandler, Strategy, Portfolio and ExecutionHandler" ) self.data_handler = self.data_handler_cls(self.events, self.csv_dir, self.symbol_list) self.strategy = self.strategy_cls(self.data_handler, self.events) self.portfolio = self.portfolio_cls(self.data_handler, self.events, self.start_date, self.initial_capital) self.execution_handler = self.execution_handler_cls(self.events) The _run_backtest method is where the signal handling of the Backtest engine is carried out. As described above there are two while loops, one nested within another. The outer keeps track of the heartbeat of the system, while the inner checks if there is an event in the Queue object, and acts on it by calling the appropriate method on the necessary object.
154 For a MarketEvent, the Strategy object is told to recalculate new signals, while the Portfolio object is told to reindex the time. If a SignalEvent object is received the Portfolio is told to handle the new signal and convert it into a set of OrderEvents, if appropriate. If an OrderEvent is received the ExecutionHandler is sent the order to be transmitted to the broker (if in a real trading setting). Finally, if a FillEvent is received, the Portfolio will update itself to be aware of the new positions: # backtest.py def _run_backtest(self): """ Executes the backtest. """ i = 0 while True: i += 1 print i # Update the market bars if self.data_handler.continue_backtest == True: self.data_handler.update_bars() else: break # Handle the events while True: try: event = self.events.get(False) except queue.Empty: break else: if event is not None: if event.type == ’MARKET’: self.strategy.calculate_signals(event) self.portfolio.update_timeindex(event) elif event.type == ’SIGNAL’: self.signals += 1 self.portfolio.update_signal(event) elif event.type == ’ORDER’: self.orders += 1 self.execution_handler.execute_order(event) elif event.type == ’FILL’: self.fills += 1 self.portfolio.update_fill(event) time.sleep(self.heartbeat) Once the backtest simulation is complete the performance of the strategy can be displayed to the terminal/console. The equity curve pandas DataFrame is created and the summary statistics are displayed, as well as the count of Signals, Orders and Fills: # backtest.py def _output_performance(self): """ Outputs the strategy performance from the backtest.
155 """ self.portfolio.create_equity_curve_dataframe() print("Creating summary stats...") stats = self.portfolio.output_summary_stats() print("Creating equity curve...") print(self.portfolio.equity_curve.tail(10)) pprint.pprint(stats) print("Signals: %s" % self.signals) print("Orders: %s" % self.orders) print("Fills: %s" % self.fills) The last method to be implemented is simulate_trading. It simply calls the two previously described methods, in order: # backtest.py def simulate_trading(self): """ Simulates the backtest and outputs portfolio performance. """ self._run_backtest() self._output_performance() This concludes the event-driven backtester operational objects. 14.3 Event-Driven Execution Above we described a basic ExecutionHandler class that simply created a corresponding FillEvent instance for every OrderEvent. This is precisely what we need for a "first pass" backtest, but when we wish to actually hook up the system to a brokerage, we need more sophisticated handling. In this section we define the IBExecutionHandler, a class that allows us to talk to the popular Interactive Brokers API and thus automate our execution. The essential idea of the IBExecutionHandler class is to receive OrderEvent instances from the events queue and then to execute them directly against the Interactive Brokers order API using the open source IbPy library. The class will also handle the "Server Response" messages sent back via the API. At this stage, the only action taken will be to create corresponding FillEvent instances that will then be sent back to the events queue. The class itself could feasibly become rather complex, with execution optimisation logic as well as sophisticated error handling. However, I have opted to keep it relatively simple here so that you can see the main ideas and extend it in the direction that suits your particular trading style. As always, the first task is to create the Python file and import the necessary libraries. The file is called ib_execution.py and lives in the same directory as the other event-driven files. We import the necessary date/time handling libraries, the IbPy objects and the specific Event objects that are handled by IBExecutionHandler: #!/usr/bin/python # -*- coding: utf-8 -*- # ib_execution.py from __future__ import print_function import datetime import time
156 from ib.ext.Contract import Contract from ib.ext.Order import Order from ib.opt import ibConnection, message from event import FillEvent, OrderEvent from execution import ExecutionHandler We now define the IBExecutionHandler class. The __init__ constructor firstly requires knowledge of the events queue. It also requires specification of order_routing, which I’ve defaulted to "SMART". If you have specific exchange requirements, you can specify them here. The default currency has also been set to US Dollars. Within the method we create a fill_dict dictionary, needed later for usage in generating FillEvent instances. We also create a tws_conn connection object to store our connection information to the Interactive Brokers API. We also have to create an initial default order_id, which keeps track of all subsequent orders to avoid duplicates. Finally we register the message handlers (which we’ll define in more detail below): # ib_execution.py class IBExecutionHandler(ExecutionHandler): """ Handles order execution via the Interactive Brokers API, for use against accounts when trading live directly. """ def __init__( self, events, order_routing="SMART", currency="USD" ): """ Initialises the IBExecutionHandler instance. """ self.events = events self.order_routing = order_routing self.currency = currency self.fill_dict = {} self.tws_conn = self.create_tws_connection() self.order_id = self.create_initial_order_id() self.register_handlers() The IB API utilises a message-based event system that allows our class to respond in particular ways to certain messages, in a similar manner to the event-driven backtester itself. I’ve not included any real error handling (for the purposes of brevity), beyond output to the terminal, via the _error_handler method. The _reply_handler method, on the other hand, is used to determine if a FillEvent instance needs to be created. The method asks if an "openOrder" message has been received and checks whether an entry in our fill_dict for this particular orderId has already been set. If not then one is created. If it sees an "orderStatus" message and that particular message states than an order has been filled, then it calls create_fill to create a FillEvent. It also outputs the message to the terminal for logging/debug purposes: # ib_execution.py def _error_handler(self, msg): """
157 Handles the capturing of error messages """ # Currently no error handling. print("Server Error: %s" % msg) def _reply_handler(self, msg): """ Handles of server replies """ # Handle open order orderId processing if msg.typeName == "openOrder" and \ msg.orderId == self.order_id and \ not self.fill_dict.has_key(msg.orderId): self.create_fill_dict_entry(msg) # Handle Fills if msg.typeName == "orderStatus" and \ msg.status == "Filled" and \ self.fill_dict[msg.orderId]["filled"] == False: self.create_fill(msg) print("Server Response: %s, %s\n" % (msg.typeName, msg)) The following method, create_tws_connection, creates a connection to the IB API using the IbPy ibConnection object. It uses a default port of 7496 and a default clientId of 10. Once the object is created, the connect method is called to perform the connection: # ib_execution.py def create_tws_connection(self): """ Connect to the Trader Workstation (TWS) running on the usual port of 7496, with a clientId of 10. The clientId is chosen by us and we will need separate IDs for both the execution connection and market data connection, if the latter is used elsewhere. """ tws_conn = ibConnection() tws_conn.connect() return tws_conn To keep track of separate orders (for the purposes of tracking fills) the following method create_initial_order_id is used. I’ve defaulted it to "1", but a more sophisticated approach would be th query IB for the latest available ID and use that. You can always reset the current API order ID via the Trader Workstation > Global Configuration > API Settings panel: # ib_execution.py def create_initial_order_id(self): """ Creates the initial order ID used for Interactive Brokers to keep track of submitted orders. """ # There is scope for more logic here, but we # will use "1" as the default for now. return 1 The following method, register_handlers, simply registers the error and reply handler methods defined above with the TWS connection: # ib_execution.py
158 def register_handlers(self): """ Register the error and server reply message handling functions. """ # Assign the error handling function defined above # to the TWS connection self.tws_conn.register(self._error_handler, ’Error’) # Assign all of the server reply messages to the # reply_handler function defined above self.tws_conn.registerAll(self._reply_handler) In order to actually transact a trade it is necessary to create an IbPy Contract instance and then pair it with an IbPy Order instance, which will be sent to the IB API. The following method, create_contract, generates the first component of this pair. It expects a ticker symbol, a security type (e.g. stock or future), an exchange/primary exchange and a currency. It returns the Contract instance: # ib_execution.py def create_contract(self, symbol, sec_type, exch, prim_exch, curr): """ Create a Contract object defining what will be purchased, at which exchange and in which currency. symbol - The ticker symbol for the contract sec_type - The security type for the contract (’STK’ is ’stock’) exch - The exchange to carry out the contract on prim_exch - The primary exchange to carry out the contract on curr - The currency in which to purchase the contract """ contract = Contract() contract.m_symbol = symbol contract.m_secType = sec_type contract.m_exchange = exch contract.m_primaryExch = prim_exch contract.m_currency = curr return contract The following method, create_order, generates the second component of the pair, namely the Order instance. It expects an order type (e.g. market or limit), a quantity of the asset to trade and an "action" (buy or sell). It returns the Order instance: # ib_execution.py def create_order(self, order_type, quantity, action): """ Create an Order object (Market/Limit) to go long/short. order_type - ’MKT’, ’LMT’ for Market or Limit orders quantity - Integral number of assets to order action - ’BUY’ or ’SELL’ """ order = Order() order.m_orderType = order_type order.m_totalQuantity = quantity
159 order.m_action = action return order In order to avoid duplicating FillEvent instances for a particular order ID, we utilise a dictionary called the fill_dict to store keys that match particular order IDs. When a fill has been generated the "filled" key of an entry for a particular order ID is set to True. If a subsequent "Server Response" message is received from IB stating that an order has been filled (and is a duplicate message) it will not lead to a new fill. The following method create_fill_dict_entry carries this out: # ib_execution.py def create_fill_dict_entry(self, msg): """ Creates an entry in the Fill Dictionary that lists orderIds and provides security information. This is needed for the event-driven behaviour of the IB server message behaviour. """ self.fill_dict[msg.orderId] = { "symbol": msg.contract.m_symbol, "exchange": msg.contract.m_exchange, "direction": msg.order.m_action, "filled": False } The following method, create_fill, actually creates the FillEvent instance and places it onto the events queue: # ib_execution.py def create_fill(self, msg): """ Handles the creation of the FillEvent that will be placed onto the events queue subsequent to an order being filled. """ fd = self.fill_dict[msg.orderId] # Prepare the fill data symbol = fd["symbol"] exchange = fd["exchange"] filled = msg.filled direction = fd["direction"] fill_cost = msg.avgFillPrice # Create a fill event object fill = FillEvent( datetime.datetime.utcnow(), symbol, exchange, filled, direction, fill_cost ) # Make sure that multiple messages don’t create # additional fills. self.fill_dict[msg.orderId]["filled"] = True # Place the fill event onto the event queue self.events.put(fill_event)
160 Now that all of the preceeding methods having been implemented it remains to override the execute_order method from the ExecutionHandler abstract base class. This method actually carries out the order placement with the IB API. We first check that the event being received to this method is actually an OrderEvent and then prepare the Contract and Order objects with their respective parameters. Once both are created the IbPy method placeOrder of the connection object is called with an associated order_id. It is extremely important to call the time.sleep(1) method to ensure the order actually goes through to IB. Removal of this line leads to inconsistent behaviour of the API, at least on my system! Finally, we increment the order ID to ensure we don’t duplicate orders: # ib_execution.py def execute_order(self, event): """ Creates the necessary InteractiveBrokers order object and submits it to IB via their API. The results are then queried in order to generate a corresponding Fill object, which is placed back on the event queue. Parameters: event - Contains an Event object with order information. """ if event.type == ’ORDER’: # Prepare the parameters for the asset order asset = event.symbol asset_type = "STK" order_type = event.order_type quantity = event.quantity direction = event.direction # Create the Interactive Brokers contract via the # passed Order event ib_contract = self.create_contract( asset, asset_type, self.order_routing, self.order_routing, self.currency ) # Create the Interactive Brokers order via the # passed Order event ib_order = self.create_order( order_type, quantity, direction ) # Use the connection to the send the order to IB self.tws_conn.placeOrder( self.order_id, ib_contract, ib_order ) # NOTE: This following line is crucial. # It ensures the order goes through! time.sleep(1) # Increment the order ID for this session
161 self.order_id += 1 This class forms the basis of an Interactive Brokers execution handler and can be used in place of the simulated execution handler, which is only suitable for backtesting. Before the IB handler can be utilised, however, it is necessary to create a live market feed handler to replace the historical data feed handler of the backtester system. In this way we are reusing as much as possible from the backtest and live systems to ensure that code "swap out" is minimised and thus behaviour across both is similar, if not identical.
162
Chapter 15 Trading Strategy Implementation In this chapter we are going to consider the full implementation of trading strategies using the aforementioned event-driven backtesting system. In particular we will generate equity curves for all trading strategies using notional portfolio amounts, thus simulating the concepts of margin/leverage, which is a far more realistic approach compared to vectorised/returns based approaches. The first set of strategies are able to be carried out with freely available data, either from Yahoo Finance, Google Finance or Quandl. These strategies are suitable for long-term algorithmic traders who may wish to only study the trade signal generation aspect of the strategy or even the full end-to-end system. Such strategies often possess smaller Sharpe ratios, but are far easier to implement and execute. The latter strategy is carried out using intraday equities data. This data is often not freely available and a commercial data vendor is usually necessary to provide sufficient quality and quantity of data. I myself use DTN IQFeed for intraday bars. Such strategies often possess much larger Sharpe ratios, but require more sophisticated implementation as the high frequency requires extensive automation. We will see that our first two attempts at creating a trading strategy on interday data are not altogether successful. It can be challenging to come up with a profitable trading strategy on interday data once transaction costs have been taken into account. The latter is something that many texts on algorithmic trading tend to leave out. However, it is my belief that as many factors as possible must be added to the backtest in order to minimises surprises going forward. In addition, this book is primarily about how to effectively create a realistic interday or intraday backtesting system (as well as a live execution platform) and less about particular individual strategies. It is far harder to create a realistic robust backtester than it is to find trading strategies on the internet! While the first two strategies presented are not particularly attractive, the latter strategy (on intraday data) performs well and gives us confidence in using higher frequency data. 15.1 Moving Average Crossover Strategy I’m quite fond of the Moving Average Crossover technical system because it is the first nontrivial strategy that is extremely handy for testing a new backtesting implementation. On a daily timeframe, over a number of years, with long lookback periods, few signals are generated on a single stock and thus it is easy to manually verify that the system is behaving as would be expected. In order to actually generate such a simulation based on the prior backtesting code we need to subclass the Strategy object as described in the previous chapter to create the MovingAverageCrossStrategy object, which will contain the logic of the simple moving averages and the generation of trading signals. In addition we need to create the __main__ function that will load the Backtest object and actually encapsulate the execution of the program. The following file, mac.py, contains both of these objects. 163
164 The first task, as always, is to correctly import the necessary components. We are importing nearly all of the objects that have been described in the previous chapter: #!/usr/bin/python # -*- coding: utf-8 -*- # mac.py from __future__ import print_function import datetime import numpy as np import pandas as pd import statsmodels.api as sm from strategy import Strategy from event import SignalEvent from backtest import Backtest from data import HistoricCSVDataHandler from execution import SimulatedExecutionHandler from portfolio import Portfolio Now we turn to the creation of the MovingAverageCrossStrategy. The strategy requires both the bars DataHandler, the events Event Queue and the lookback periods for the simple moving averages that are going to be employed within the strategy. I’ve chosen 100 and 400 as the "short" and "long" lookback periods for this strategy. The final attribute, bought, is used to tell the Strategy when the backtest is actually "in the market". Entry signals are only generated if this is "OUT" and exit signals are only ever generated if this is "LONG" or "SHORT": # mac.py class MovingAverageCrossStrategy(Strategy): """ Carries out a basic Moving Average Crossover strategy with a short/long simple weighted moving average. Default short/long windows are 100/400 periods respectively. """ def __init__( self, bars, events, short_window=100, long_window=400 ): """ Initialises the Moving Average Cross Strategy. Parameters: bars - The DataHandler object that provides bar information events - The Event Queue object. short_window - The short moving average lookback. long_window - The long moving average lookback. """ self.bars = bars self.symbol_list = self.bars.symbol_list self.events = events self.short_window = short_window self.long_window = long_window
165 # Set to True if a symbol is in the market self.bought = self._calculate_initial_bought() Since the strategy begins out of the market we set the initial "bought" value to be "OUT", for each symbol: # mac.py def _calculate_initial_bought(self): """ Adds keys to the bought dictionary for all symbols and sets them to ’OUT’. """ bought = {} for s in self.symbol_list: bought[s] = ’OUT’ return bought The core of the strategy is the calculate_signals method. It reacts to a MarketEvent object and for each symbol traded obtains the latest N bar closing prices, where N is equal to the largest lookback period. It then calculates both the short and long period simple moving averages. The rule of the strategy is to enter the market (go long a stock) when the short moving average value exceeds the long moving average value. Conversely, if the long moving average value exceeds the short moving average value the strategy is told to exit the market. This logic is handled by placing a SignalEvent object on the events Event Queue in each of the respective situations and then updating the "bought" attribute (per symbol) to be "LONG" or "OUT", respectively. Since this is a long-only strategy, we won’t be considering "SHORT" positions: # mac.py def calculate_signals(self, event): """ Generates a new set of signals based on the MAC SMA with the short window crossing the long window meaning a long entry and vice versa for a short entry. Parameters event - A MarketEvent object. """ if event.type == ’MARKET’: for s in self.symbol_list: bars = self.bars.get_latest_bars_values( s, "adj_close", N=self.long_window ) bar_date = self.bars.get_latest_bar_datetime(s) if bars is not None and bars != []: short_sma = np.mean(bars[-self.short_window:]) long_sma = np.mean(bars[-self.long_window:]) symbol = s dt = datetime.datetime.utcnow() sig_dir = "" if short_sma > long_sma and self.bought[s] == "OUT": print("LONG: %s" % bar_date) sig_dir = ’LONG’
166 signal = SignalEvent(1, symbol, dt, sig_dir, 1.0) self.events.put(signal) self.bought[s] = ’LONG’ elif short_sma < long_sma and self.bought[s] == "LONG": print("SHORT: %s" % bar_date) sig_dir = ’EXIT’ signal = SignalEvent(1, symbol, dt, sig_dir, 1.0) self.events.put(signal) self.bought[s] = ’OUT’ That concludes the MovingAverageCrossStrategy object implementation. The final task of the entire backtesting system is populate a __main__ method in mac.py to actually execute the backtest. Firstly, make sure to change the value of csv_dir to the absolute path of your CSV file directory for the financial data. You will also need to download the CSV file of the AAPL stock (from Yahoo Finance), which is given by the following link (for Jan 1st 1990 to Jan 1st 2002), since this is the stock we will be testing the strategy on: http://ichart.finance.yahoo.com/table.csv?s=AAPL&a=00&b=1&c=1990&d=00&e=1 &f=2002&g=d&ignore=.csv Make sure to place this file in the path pointed to from the main function in csv_dir. The __main__ function simply instantiates a new backtest object and then calls the simulate_trading method on it to execute it: # mac.py if __name__ == "__main__": csv_dir = ’/path/to/your/csv/file’ # CHANGE THIS! symbol_list = [’AAPL’] initial_capital = 100000.0 heartbeat = 0.0 start_date = datetime.datetime(1990, 1, 1, 0, 0, 0) backtest = Backtest( csv_dir, symbol_list, initial_capital, heartbeat, start_date, HistoricCSVDataHandler, SimulatedExecutionHandler, Portfolio, MovingAverageCrossStrategy ) backtest.simulate_trading() To run the code, make sure you have already set up a Python environment (as described in the previous chapters) and then navigate the directory where your code is stored. You should simply be able to run: python mac.py You will see the following listing (truncated due to the bar count printout!): .. .. 3029 3030 Creating summary stats... Creating equity curve... AAPL cash commission total returns equity_curve drawdown datetime 2001-12-18 0 99211 13 99211 0 0.99211 0.025383 2001-12-19 0 99211 13 99211 0 0.99211 0.025383 2001-12-20 0 99211 13 99211 0 0.99211 0.025383 2001-12-21 0 99211 13 99211 0 0.99211 0.025383
167 2001-12-24 0 99211 13 99211 0 0.99211 0.025383 2001-12-26 0 99211 13 99211 0 0.99211 0.025383 2001-12-27 0 99211 13 99211 0 0.99211 0.025383 2001-12-28 0 99211 13 99211 0 0.99211 0.025383 2001-12-31 0 99211 13 99211 0 0.99211 0.025383 2001-12-31 0 99211 13 99211 0 0.99211 0.025383 [(’Total Return’, ’-0.79%’), (’Sharpe Ratio’, ’-0.09’), (’Max Drawdown’, ’2.56%’), (’Drawdown Duration’, ’2312’)] Signals: 10 Orders: 10 Fills: 10 The performance of this strategy can be seen in Fig 15.1: Figure 15.1: Equity Curve, Daily Returns and Drawdowns for the Moving Average Crossover strategy Evidently the returns and Sharpe Ratio are not stellar for AAPL stock on this particular set of technical indicators! Clearly we have some work to do in the next set of strategies to find a system that can generate positive performance.
168 15.2 S&P500 Forecasting Trade In this sectiuon we will consider a trading strategy built around the forecasting engine discussed in prior chapters. We will attempt to trade off the predictions made by a stock market forecaster. We are going to attempt to forecast SPY, which is the ETF that tracks the value of the S&P500. Ultimately we want to answer the question as to whether a basic forecasting algorithm using lagged price data, with slight predictive performance, provides us with any benefit over a buy-and-hold strategy. The rules for this strategy are as follows: 1. Fit a forecasting model to a subset of S&P500 data. This could be Logistic Regression, a Discriminant Analyser (Linear or Quadratic), a Support Vector Machine or a Random Forest. The procedure to do this was outlined in the Forecasting chapter. 2. Use two prior lags of adjusted closing returns data as a predictor for tomorrow’s returns. If the returns are predicted as positive then go long. If the returns are predicted as negative then exit. We’re not going to consider short selling for this particular strategy. Implementation For this strategy we are going to create the snp_forecast.py file and import the following necessary libraries: #!/usr/bin/python # -*- coding: utf-8 -*- # snp_forecast.py from __future__ import print_function import datetime import pandas as pd from sklearn.qda import QDA from strategy import Strategy from event import SignalEvent from backtest import Backtest from data import HistoricCSVDataHandler from execution import SimulatedExecutionHandler from portfolio import Portfolio from create_lagged_series import create_lagged_series We have imported Pandas and Scikit-Learn in order to carry out the fitting procedure for the supervised classifier model. We have also imported the necessary classes from the event-driven backtester. Finally, we have imported the create_lagged_series function, which we used in the Forecasting chapter. The next step is to create the SPYDailyForecastStrategy as a subclass of the Strategy abstract base class. Since we will "hardcode" the parameters of the strategy directly into the class, for simplicity, the only parameters necessary for the __init__ constructor are the bars data handler and the events queue. We set the self.model_*** start/end/test dates as datetime objects and then tell the class that we are out of the market (self.long_market = False). Finally, we set self.model to be the trained model from the create_symbol_forecast_model below: # snp_forecast.py class SPYDailyForecastStrategy(Strategy): """
169 S&P500 forecast strategy. It uses a Quadratic Discriminant Analyser to predict the returns for a subsequent time period and then generated long/exit signals based on the prediction. """ def __init__(self, bars, events): self.bars = bars self.symbol_list = self.bars.symbol_list self.events = events self.datetime_now = datetime.datetime.utcnow() self.model_start_date = datetime.datetime(2001,1,10) self.model_end_date = datetime.datetime(2005,12,31) self.model_start_test_date = datetime.datetime(2005,1,1) self.long_market = False self.short_market = False self.bar_index = 0 self.model = self.create_symbol_forecast_model() Here we define the create_symbol_forecast_model. It essentially calls the create_lagged_series function, which produces a Pandas DataFrame with five daily returns lags for each current predictor. We then consider only the two most recent of these lags. This is because we are making the modelling decision that the predictive power of earlier lags is likely to be minimal. At this stage we create the training and test data, the latter of which can be used to test our model if we wish. I have opted to not output testing data, since we have already trained the model before in the Forecasting chapter. Finally we fit the training data to the Quadratic Discriminant Analyser and then return the model. Note that we could easily replace the model with a Random Forest, Support Vector Machine or Logistic Regression, for instance. All we need to do is import the correct library from ScikitLearn and simply replace the model = QDA() line: # snp_forecast.py def create_symbol_forecast_model(self): # Create a lagged series of the S&P500 US stock market index snpret = create_lagged_series( self.symbol_list[0], self.model_start_date, self.model_end_date, lags=5 ) # Use the prior two days of returns as predictor # values, with direction as the response X = snpret[["Lag1","Lag2"]] y = snpret["Direction"] # Create training and test sets start_test = self.model_start_test_date X_train = X[X.index < start_test] X_test = X[X.index >= start_test] y_train = y[y.index < start_test] y_test = y[y.index >= start_test] model = QDA() model.fit(X_train, y_train) return model
170 At this stage we are ready to override the calculate_signals method of the Strategy base class. We firstly calculate some convenience parameters that enter our SignalEvent object and then only generate a set of signals if we have received a MarketEvent object (a basic sanity check). We wait for five bars to have elapsed (i.e. five days in this strategy!) and then obtain the lagged returns values. We then wrap these values in a Pandas Series so that the predict method of the model will function correctly. We then calculate a prediction, which manifests itself as a +1 or -1. If the prediction is a +1 and we are not already long the market, we create a SignalEvent to go long and let the class know we are now in the market. If the prediction is -1 and we are long the market, then we simply exit the market: # snp_forecast.py def calculate_signals(self, event): """ Calculate the SignalEvents based on market data. """ sym = self.symbol_list[0] dt = self.datetime_now if event.type == ’MARKET’: self.bar_index += 1 if self.bar_index > 5: lags = self.bars.get_latest_bars_values( self.symbol_list[0], "returns", N=3 ) pred_series = pd.Series( { ’Lag1’: lags[1]*100.0, ’Lag2’: lags[2]*100.0 } ) pred = self.model.predict(pred_series) if pred > 0 and not self.long_market: self.long_market = True signal = SignalEvent(1, sym, dt, ’LONG’, 1.0) self.events.put(signal) if pred < 0 and self.long_market: self.long_market = False signal = SignalEvent(1, sym, dt, ’EXIT’, 1.0) self.events.put(signal) In order to run the strategy you will need to download a CSV file from Yahoo Finance for SPY and place it in a suitable directory (note that you will need to change your path below!). We then wrap the backtest up via the Backtest class and carry out the test by calling simulate_trading: # snp_forecast.py if __name__ == "__main__": csv_dir = ’/path/to/your/csv/file’ # CHANGE THIS! symbol_list = [’SPY’] initial_capital = 100000.0 heartbeat = 0.0 start_date = datetime.datetime(2006,1,3) backtest = Backtest(
171 csv_dir, symbol_list, initial_capital, heartbeat, start_date, HistoricCSVDataHandler, SimulatedExecutionHandler, Portfolio, SPYDailyForecastStrategy ) backtest.simulate_trading() The output of the strategy is as follows and is net of transaction costs: .. .. 2209 2210 Creating summary stats... Creating equity curve... SPY cash commission total returns equity_curve \ datetime 2014-09-29 19754 90563.3 349.7 110317.3 -0.000326 1.103173 2014-09-30 19702 90563.3 349.7 110265.3 -0.000471 1.102653 2014-10-01 19435 90563.3 349.7 109998.3 -0.002421 1.099983 2014-10-02 19438 90563.3 349.7 110001.3 0.000027 1.100013 2014-10-03 19652 90563.3 349.7 110215.3 0.001945 1.102153 2014-10-06 19629 90563.3 349.7 110192.3 -0.000209 1.101923 2014-10-07 19326 90563.3 349.7 109889.3 -0.002750 1.098893 2014-10-08 19664 90563.3 349.7 110227.3 0.003076 1.102273 2014-10-09 19274 90563.3 349.7 109837.3 -0.003538 1.098373 2014-10-09 0 109836.0 351.0 109836.0 -0.000012 1.098360 drawdown datetime 2014-09-29 0.003340 2014-09-30 0.003860 2014-10-01 0.006530 2014-10-02 0.006500 2014-10-03 0.004360 2014-10-06 0.004590 2014-10-07 0.007620 2014-10-08 0.004240 2014-10-09 0.008140 2014-10-09 0.008153 [(’Total Return’, ’9.84%’), (’Sharpe Ratio’, ’0.54’), (’Max Drawdown’, ’5.99%’), (’Drawdown Duration’, ’811’)] Signals: 270 Orders: 270 Fills: 270 The following visualisation in Fig 15.2 shows the Equity Curve, the Daily Returns and the Drawdown of the strategy as a function of time: Note immediately that the performance is not great! We have a Sharpe Ratio < 1 but a reasonable drawdown of just under 6%. It turns out that if we had simply bought and held SPY in this time period we would have performed similarly, if slightly worse. Hence we have not actually gained very much from our predictive strategy once transaction costs are included. I specifically wanted to include this example because it uses an "end to end" realistic implementation of such a strategy that takes into account conservative, realistic transaction costs. As can be seen it is not easy to make a predictive forecaster on daily data that produces good performance!
172 Figure 15.2: Equity Curve, Daily Returns and Drawdowns for the SPY forecast strategy Our final strategy will make use of other time series and a higher frequency. We will see that performance can be improved dramatically after modifying certain aspects of the system. 15.3 Mean-Reverting Equity Pairs Trade In order to seek higher Sharpe ratios for our trading, we need to consider higher-frequency intraday strategies. The first major issue is that obtaining data is significantly less straightforward because high quality intraday data is usually not free. As stated above I use DTN IQFeed for intraday minutely bars and thus you will need your own DTN account to obtain the data required for this strategy. The second issue is that backtesting simulations take substantially longer, especially with the event-driven model that we have constructed here. Once we begin considering a backtest of a diversified portfolio of minutely data spanning years, and then performing any parameter optimisation, we rapidly realise that simulations can take hours or even days to calculate on a modern desktop PC. This will need to be factored in to your research process. The third issue is that live execution will now need to be fully automated since we are edging into higher-frequency trading. This means that such execution environments and code must be highly reliable and bug-free, otherwise the potential for significant losses can occur. This strategy expands on the previous interday strategy above to make use of intraday data. In particular we are going to use minutely OHLCV bars, as opposed to daily OHLCV. The rules for the strategy are straightforward: 1. Identify a pair of equities that possess a residuals time series which has been statistically identified as mean-reverting. In this case, I have found two energy sector US equities with tickers AREX and WLL.
173 2. Create the residuals time series of the pair by performing a rolling linear regression, for a particular lookback window, via the ordinary least squares (OLS) algorithm. This lookback period is a parameter to be optimised. 3. Create a rolling z-score of the residuals time series of the same lookback period and use this to determine entry/exit thresholds for trading signals. 4. If the upper threshold is exceeded when not in the market then enter the market (long or short depending on direction of threshold excess). If the lower threshold is exceeded when in the market, exit the market. Once again, the upper and lower thresholds are parameters to be optimised. Indeed we could have used the Cointegrated Augmented Dickey-Fuller (CADF) test to identify an even more accurate hedging parameter. This would make an interesting extension of the strategy. Implementation The first step, as always, is to import the necessary libraries. We require pandas for the rolling_apply method, which is used to apply the z-score calculation with a lookback window on a rolling basis. We import statsmodels because it provides a means of calculating the ordinary least squares (OLS) algorithm for the linear regression, necessary to obtain the hedging ratio for the construction of the residuals. We also require a slightly modified DataHandler and Portfolio in order to carry out minutely bars trading on DTN IQFeed data. In order to create these files you can simply copy all of the code in portfolio.py and data.py into the new files hft_portfolio.py and hft_data.py respectively and then modify the necessary sections, which I will outline below. Here is the import listing for intraday_mr.py: #!/usr/bin/python # -*- coding: utf-8 -*- # intraday_mr.py from __future__ import print_function import datetime import numpy as np import pandas as pd import statsmodels.api as sm from strategy import Strategy from event import SignalEvent from backtest import Backtest from hft_data import HistoricCSVDataHandlerHFT from hft_portfolio import PortfolioHFT from execution import SimulatedExecutionHandler In the following snippet we create the IntradayOLSMRStrategy class derived from the Strategy abstract base class. The constructor __init__ method requires access to the bars historical data provider, the events queue, a zscore_low threshold and a zscore_high threshold, used to determine when the residual series between the two pairs is mean-reverting. In addition, we specify the OLS lookback window (set to 100 here), which is a parameter that is subject to potential optimisation. At the start of the simulation we are neither long or short the market, so we set both self.long_market and self.short_market equal to False: # intraday_mr.py
174 class IntradayOLSMRStrategy(Strategy): """ Uses ordinary least squares (OLS) to perform a rolling linear regression to determine the hedge ratio between a pair of equities. The z-score of the residuals time series is then calculated in a rolling fashion and if it exceeds an interval of thresholds (defaulting to [0.5, 3.0]) then a long/short signal pair are generated (for the high threshold) or an exit signal pair are generated (for the low threshold). """ def __init__( self, bars, events, ols_window=100, zscore_low=0.5, zscore_high=3.0 ): """ Initialises the stat arb strategy. Parameters: bars - The DataHandler object that provides bar information events - The Event Queue object. """ self.bars = bars self.symbol_list = self.bars.symbol_list self.events = events self.ols_window = ols_window self.zscore_low = zscore_low self.zscore_high = zscore_high self.pair = (’AREX’, ’WLL’) self.datetime = datetime.datetime.utcnow() self.long_market = False self.short_market = False The following method, calculate_xy_signals, takes the current zscore (from the rolling calculation performed below) and determines whether new trading signals need to be generated. These signals are then returned. There are four potential states that we may be interested in. They are: 1. Long the market and below the negative zscore higher threshold 2. Long the market and between the absolute value of the zscore lower threshold 3. Short the market and above the positive zscore higher threshold 4. Short the market and between the absolute value of the zscore lower threshold In either case it is necessary to generate two signals, one for the first component of the pair (AREX) and one for the second component of the pair (WLL). If none of these conditions are reached, then a pair of None values are returned: # intraday_mr.py def calculate_xy_signals(self, zscore_last): """ Calculates the actual x, y signal pairings to be sent to the signal generator.
175 Parameters zscore_last - The current zscore to test against """ y_signal = None x_signal = None p0 = self.pair[0] p1 = self.pair[1] dt = self.datetime hr = abs(self.hedge_ratio) # If we’re long the market and below the # negative of the high zscore threshold if zscore_last <= -self.zscore_high and not self.long_market: self.long_market = True y_signal = SignalEvent(1, p0, dt, ’LONG’, 1.0) x_signal = SignalEvent(1, p1, dt, ’SHORT’, hr) # If we’re long the market and between the # absolute value of the low zscore threshold if abs(zscore_last) <= self.zscore_low and self.long_market: self.long_market = False y_signal = SignalEvent(1, p0, dt, ’EXIT’, 1.0) x_signal = SignalEvent(1, p1, dt, ’EXIT’, 1.0) # If we’re short the market and above # the high zscore threshold if zscore_last >= self.zscore_high and not self.short_market: self.short_market = True y_signal = SignalEvent(1, p0, dt, ’SHORT’, 1.0) x_signal = SignalEvent(1, p1, dt, ’LONG’, hr) # If we’re short the market and between the # absolute value of the low zscore threshold if abs(zscore_last) <= self.zscore_low and self.short_market: self.short_market = False y_signal = SignalEvent(1, p0, dt, ’EXIT’, 1.0) x_signal = SignalEvent(1, p1, dt, ’EXIT’, 1.0) return y_signal, x_signal The following method, calculate_signals_for_pairs obtains the latest set of bars for each component of the pair (in this case 100 bars) and uses them to construct an ordinary least squares based linear regression. This allows identification of the hedge ratio, necessary for the construction of the residuals time series. Once the hedge ratio is constructed, a spread series of residuals is constructed. The next step is to calculate the latest zscore from the residual series by subtracting its mean and dividing by its standard deviation over the lookback period. Finally, the y_signal and x_signal are calculated on the basis of this zscore. If the signals are not both None then the SignalEvent instances are sent back to the events queue: # intraday_mr.py def calculate_signals_for_pairs(self): """ Generates a new set of signals based on the mean reversion strategy.
176 Calculates the hedge ratio between the pair of tickers. We use OLS for this, althought we should ideall use CADF. """ # Obtain the latest window of values for each # component of the pair of tickers y = self.bars.get_latest_bars_values( self.pair[0], "close", N=self.ols_window ) x = self.bars.get_latest_bars_values( self.pair[1], "close", N=self.ols_window ) if y is not None and x is not None: # Check that all window periods are available if len(y) >= self.ols_window and len(x) >= self.ols_window: # Calculate the current hedge ratio using OLS self.hedge_ratio = sm.OLS(y, x).fit().params[0] # Calculate the current z-score of the residuals spread = y - self.hedge_ratio * x zscore_last = ((spread - spread.mean())/spread.std())[-1] # Calculate signals and add to events queue y_signal, x_signal = self.calculate_xy_signals(zscore_last) if y_signal is not None and x_signal is not None: self.events.put(y_signal) self.events.put(x_signal) The final method, calculate_signals is overidden from the base class and is used to check whether a received event from the queue is actually a MarketEvent, in which case the calculation of the new signals is carried out: # intraday_mr.py def calculate_signals(self, event): """ Calculate the SignalEvents based on market data. """ if event.type == ’MARKET’: self.calculate_signals_for_pairs() The __main__ section ties together the components to produce a backtest for the strategy. We tell the simulation where the ticker minutely data is stored. I’m using DTN IQFeed format. I truncated both files so that they began and ended on the same respective minute. For this particular pair of AREX and WLL, the common start date is 8th November 2007 at 10:41:00AM. Finally, we build the backtest object and begin simulating the trading: # intraday_mr.py if __name__ == "__main__": csv_dir = ’/path/to/your/csv/file’ # CHANGE THIS! symbol_list = [’AREX’, ’WLL’] initial_capital = 100000.0 heartbeat = 0.0 start_date = datetime.datetime(2007, 11, 8, 10, 41, 0) backtest = Backtest( csv_dir, symbol_list, initial_capital, heartbeat,
177 start_date, HistoricCSVDataHandlerHFT, SimulatedExecutionHandler, PortfolioHFT, IntradayOLSMRStrategy ) backtest.simulate_trading() However, before we can execute this file we need to make some modifications to the data handler and portfolio objects. In particular, it is necessary to create new files hft_data.py and hft_portfolio.py which are copies of data.py and portfolio.py respectively. In hft_data.py we need to rename HistoricCSVDataHandler to HistoricCSVDataHandlerHFT and replace the names list in the _open_convert_csv_files method. The old line is: names=[ ’datetime’, ’open’, ’high’, ’low’, ’close’, ’volume’, ’adj_close’ ] This must be replaced with: names=[ ’datetime’, ’open’, ’low’, ’high’, ’close’, ’volume’, ’oi’ ] This is to ensure that the new format for DTN IQFeed works with the backtester. The other change is to rename Portfolio to PortfolioHFT in hft_portfolio.py. We must then modify a few lines in order to account for the minutely frequency of the DTN data. In particular, within the update_timeindex method, we must change the following code: for s in self.symbol_list: # Approximation to the real value market_value = self.current_positions[s] * \ self.bars.get_latest_bar_value(s, "adj_close") dh[s] = market_value dh[’total’] += market_value To: for s in self.symbol_list: # Approximation to the real value market_value = self.current_positions[s] * \ self.bars.get_latest_bar_value(s, "close") dh[s] = market_value dh[’total’] += market_value This ensures we obtain the close price, rather than the adj_close price. The latter is for Yahoo Finance, whereas the former is for DTN IQFeed. We must also make a similar adjustment in update_holdings_from_fill. We need to change the following code: # Update holdings list with new quantities fill_cost = self.bars.get_latest_bar_value( fill.symbol, "adj_close" ) To: # Update holdings list with new quantities fill_cost = self.bars.get_latest_bar_value( fill.symbol, "close" )
178 The final change is occurs in the output_summary_stats method at the bottom of the file. We need to modify how the Sharpe Ratio is calculated to take into account minutely trading. The following line: sharpe_ratio = create_sharpe_ratio(returns) Must be changed to: sharpe_ratio = create_sharpe_ratio(returns, periods=252*6.5*60) This completes the necessary changes. Upon execution of intraday_mr.py we get the following (truncated) output from the backtest simulation: .. .. 375072 375073 Creating summary stats... Creating equity curve... AREX WLL cash commission total returns \ datetime 2014-03-11 15:53:00 2098 -6802 120604.3 9721.4 115900.3 -0.000052 2014-03-11 15:54:00 2101 -6799 120604.3 9721.4 115906.3 0.000052 2014-03-11 15:55:00 2100 -6802 120604.3 9721.4 115902.3 -0.000035 2014-03-11 15:56:00 2097 -6810 120604.3 9721.4 115891.3 -0.000095 2014-03-11 15:57:00 2098 -6801 120604.3 9721.4 115901.3 0.000086 2014-03-11 15:58:00 2098 -6800 120604.3 9721.4 115902.3 0.000009 2014-03-11 15:59:00 2099 -6800 120604.3 9721.4 115903.3 0.000009 2014-03-11 16:00:00 2100 -6801 120604.3 9721.4 115903.3 0.000000 2014-03-11 16:01:00 2100 -6801 120604.3 9721.4 115903.3 0.000000 2014-03-11 16:01:00 2100 -6801 120604.3 9721.4 115903.3 0.000000 equity_curve drawdown datetime 2014-03-11 15:53:00 1.159003 0.003933 2014-03-11 15:54:00 1.159063 0.003873 2014-03-11 15:55:00 1.159023 0.003913 2014-03-11 15:56:00 1.158913 0.004023 2014-03-11 15:57:00 1.159013 0.003923 2014-03-11 15:58:00 1.159023 0.003913 2014-03-11 15:59:00 1.159033 0.003903 2014-03-11 16:00:00 1.159033 0.003903 2014-03-11 16:01:00 1.159033 0.003903 2014-03-11 16:01:00 1.159033 0.003903 [(’Total Return’, ’15.90%’), (’Sharpe Ratio’, ’1.89’), (’Max Drawdown’, ’3.03%’), (’Drawdown Duration’, ’120718’)] Signals: 7594 Orders: 7478 Fills: 7478 You can see that the strategy performs adequately well during this period. It has a total return of just under 16%. The Sharpe ratio is reasonable (when compared to a typical daily strategy), but given the high-frequency nature of the strategy we should be expecting more. The major attraction of this strategy is that the maximum drawdown is low (approximately 3%). This suggests we could apply more leverage to gain more return. The performance of this strategy can be seen in Fig 15.3: Note that these figures are based on trading a total of 100 shares. You can adjust the leverage by simply adjusting the generate_naive_order method of the Portfolio class. Look for the
179 Figure 15.3: Equity Curve, Daily Returns and Drawdowns for intraday mean-reversion strategy attribute known as mkt_quantity. It will be set to 100. Changing this to 2000, for instance, provides these results: .. .. [(’Total Return’, ’392.85%’), (’Sharpe Ratio’, ’2.29’), (’Max Drawdown’, ’45.69%’), (’Drawdown Duration’, ’102150’)] .. .. Clearly the Sharpe Ratio and Total Return are much more attractive, but we have to endure a 45% maximum drawdown over this period as well! 15.4 Plotting Performance The three Figures displayed above are all created using the plot_performance.py script. For completeness I’ve included the code so that you can use it as a base to create your own performance charts. It is necessary to run this in the same directory as the output file from the backtest, namely where equity.csv resides. The listing is as follows: #!/usr/bin/python # -*- coding: utf-8 -*- # plot_performance.py import os.path
180 import numpy as np import matplotlib.pyplot as plt import pandas as pd if __name__ == "__main__": data = pd.io.parsers.read_csv( "equity.csv", header=0, parse_dates=True, index_col=0 ).sort() # Plot three charts: Equity curve, # period returns, drawdowns fig = plt.figure() # Set the outer colour to white fig.patch.set_facecolor(’white’) # Plot the equity curve ax1 = fig.add_subplot(311, ylabel=’Portfolio value, %’) data[’equity_curve’].plot(ax=ax1, color="blue", lw=2.) plt.grid(True) # Plot the returns ax2 = fig.add_subplot(312, ylabel=’Period returns, %’) data[’returns’].plot(ax=ax2, color="black", lw=2.) plt.grid(True) # Plot the returns ax3 = fig.add_subplot(313, ylabel=’Drawdowns, %’) data[’drawdown’].plot(ax=ax3, color="red", lw=2.) plt.grid(True) # Plot the figure plt.show()
Chapter 16 Strategy Optimisation In prior chapters we have considered how to create both an underlying predictive model (such as with the Suppor Vector Machine and Random Forest Classifier) as well as a trading strategy based upon it. Along the way we have seen that there are many parameters to such models. In the case of an SVM we have the "tuning" parameters γ and C. In a Moving Average Crossover trading strategy we have the parameters for the two lookback windows of the moving average filters. In this chapter we are going to describe optimisation methods to improve the performance of our trading strategies by tuning the parameters in a systematic fashion. For this we will use mechanisms from the statistical field of Model Selection, such as cross-validation and grid search. The literature on model selection and parameter optimisation is vast and most of the methods are somewhat beyond the scope of this book. I want to introduce the subject here so that you can explore more sophisticated techniques at your own pace. 16.1 Parameter Optimisation At this stage nearly all of the trading strategies and underlying statistical models have required one or more parameters in order to be utilised. In momentum strategies using technical indicators, such as with moving averages (simple or exponential), there is a need to specify a lookback window. The same is true of many mean-reverting strategies, which require a (rolling) lookback window in order to calculate a regression between two time series. Particular statistical machine learning models such as a logistic regression, SVM or Random Forest also require parameters in order to be calculated. The biggest danger when considering parameter optimisation is that of overfitting a model or trading strategy. This problem occurs when a model is trained on an in sample retained slice of training data and is optimised to perform well (by the appropriate performance measure), but performance degrades substantially when applied to out of sample data. For instance, a trading strategy could perform extremely well in the backtest (the in sample data) but when deployed for live trading can be completely unprofitable. An additional concern of parameter optimisation is that it can become very computationally expensive. With modern computational systems this is less of an issue than it once was, due to parallelisation and fast CPUs. However, multiple parameter optimisation can increase computational complexity by orders of magnitudes. One must be aware of this as part of the research and development process. 16.1.1 Which Parameters to Optimise? A statistical-based algorithmic trading model will often have many parameters and different measures of performance. An underlying statistical learning algorithm will have its own set of parameters. In the case of a multiple linear or logistic regression these would be the βi coefficients. In the case of a random forest one such parameter would be the number of underlying decision trees to use in the ensemble. Once applied to a trading model other parameters might be entry 181
182 and exit thresholds, such as a z-score of a particular time series. The z-score itself might have an implicit rolling lookback window. As can be seen the number of parameters can be quite extensive. In addition to parameters there are numerous means of evaluating the performance of a statistical model and the trading strategy based upon it. We have defined concepts such as the hit rate and the confusion matrix. In addition there are more statistical measures such as the Mean Squared Error (MSE). These are performance measures that would be optimised at the statistical model level, via parameters relevant to their domain. The actual trading strategy is evaluated on different criteria, such as compound annual growth rate (CAGR) and maximum drawdown. We would need to vary entry and exit criteria, as well as other thresholds that are not directly related to the statistical model. Hence this motivates the question as to which set of parameters to optimise and when. In the following sections we are going to optimise both the statistical model parameters, at the early research and development stage, as well as the parameters associated with a trading strategy using an underlying optimised statistical model, on each of their respective performance measures. 16.1.2 Optimisation is Expensive With multiple real-valued parameters, optimisation can rapidly become extremely expensive, as each new parameter adds an additional spatial dimension. If we consider the example of a grid search (to be discussed in full below), and have a single parameter α, then we might wish to vary α within the set {0.1, 0.2, 0.3, 0.4, 0.5}. This requires 5 simulations. If we now consider an additional parameter β, which may vary in the range {0.2, 0.4, 0.6, 0.8, 1.0}, then we will have to consider 5 2 = 25 simulations. Another parameter, γ, with 5 variations brings this to 5 3 = 125 simulations. If each paramater had 10 separate values to be tested, this would be equal to 103 = 1000 simulations. As can be seen the parameter search space can rapidly make such simulations extremely expensive. It is clear that a trade-off exists between conducting an exhaustive parameter search and maintaining a reasonable total simulation time. While parallelism, including many-core CPUs and graphics processing units (GPUs), have mitigated the issue somewhat, we still need to be careful when introducing parameters. This notion of reducing parameters is also an issue of model effectiveness, as we shall see below. 16.1.3 Overfitting Overfitting is the process of optimising a parameter, or set of parameters, against a particular data set such that an appropriate performance measure (or error measure) is found to be maximised (or minimised), but when applied to an unseen data set, such a performance measure degrades substantially. The concept is closely related to the idea of the bias-variance dilemma. The bias-variance dilemma concerns the situation where a statistical model has a trade-off between being a low-bias model or a low-variance model, or a compromise between the two. Bias refers to the difference between the model’s estimation of a parameter and the true "population" value of the parameter, or erroneous assumptions in the statistical model. Variance refers to the error from the sensitivity of the model to small fluctuations in the training set (in sample data). In all statistical models one is simultaneously trying to minimise both the bias error and the variance error in order to improve model accuracy. Such a situation can lead to overfitting in models, as the training error can be substantially reduced by introducing models with more flexibility (variation). However, such models can perform extremely poorly on new (out of sample) data since they were essentially "fit" to the in sample data. A common example of a high-bias, low-variance model is that of linear regression applied to a non-linear data set. Additions of new points do not affect the regression slope dramatically (assuming they are not too far from the remaining data), but since the problem is inherently non-linear, there is a systematic bias in the results by using a linear model. A common example of a low-bias, high-variance model is that of a polynomial spline fit applied to a non-linear data set. The parameter of the model (the degree of the polynomial)
183 could be adjusted to fit such a model very precisely (i.e. low-bias on the training data), but additions of new points would almost certainly lead to the model having to modify its degree of polynomial to fit the new data. This would make it a very high-variance model on the in sample data. Such a model would likely have very poor predictability or inferential capability on out of sample data. Overfitting can also manifest itself on the trading strategy and not just the statistical model. For instance, we could optimise the Sharpe ratio by varying entry and exit threshold parameters. While this may improve profitability in the backtest (or minimise risk substantially), it would likely not be behaviour that is replicated when the strategy was deployed live, as we might have been fitting such optimisations to noise in the historical data. We will discuss techniques below to minimise overfitting, as much as possible. However one has to be aware that it is an ever-present danger in both algorithmic trading and statistical analysis in general. 16.2 Model Selection In this section we are going to consider how to optimise the statistical model that will underly a trading strategy. In the field of statistics and machine learning this is known as Model Selection. While I won’t present an exhaustive discussion on the various model selection techniques, I will describe some of the basic mechanisms such as Cross Validation and Grid Search that work well for trading strategies. 16.2.1 Cross Validation Cross Validation is a technique used to assess how a statistical model will generalise to new data that it has not been exposed to before. Such a technique is usually used on predictive models, such as the aforementioned supervised classifiers used to predict the sign of the following daily returns of an asset price series. Fundamentally, the goal of cross validation is to minimise error on out of sample data without leading to an overfit model. In this section we will describe the training/test split and k-fold cross validation, as well as use techniques within Scikit-Learn to automatically carry out these procedures on statistical models we have already developed. Train/Test Split The simplest example of cross validation is known as a training/test split, or a 2-fold cross validation. Once a prior historical data set is assembled (such as a daily time series of asset prices), it is split into two components. The ratio of the split is usually varied between 0.5 and 0.8. In the latter case this means 80% of the data is used for training and 20% is used for testing. All of the statistics of interest, such as the hit rate, confusion matrix or mean-squared error are calculated on the test set, which has not been used within the training process. To carry out this process in Python with Scikit-Learn we can use the sklearn cross_validation train_test_split method. We will continue with our model as discussed in the chapter on Forecasting. In particular, we are going to modify forecast.py and create a new file called train_test_split.py. We will need to add the new import to the list of imports: #!/usr/bin/python # -*- coding: utf-8 -*- # train_test_split.py from __future__ import print_function import datetime import sklearn
184 from sklearn.cross_validation import train_test_split from sklearn.ensemble import RandomForestClassifier from sklearn.linear_model import LogisticRegression from sklearn.lda import LDA from sklearn.metrics import confusion_matrix from sklearn.qda import QDA from sklearn.svm import LinearSVC, SVC from create_lagged_series import create_lagged_series In forecast.py we originally split the data based on a particular date within the time series: # forecast.py .. # The test data is split into two parts: Before and after 1st Jan 2005. start_test = datetime.datetime(2005,1,1) # Create training and test sets X_train = X[X.index < start_test] X_test = X[X.index >= start_test] y_train = y[y.index < start_test] y_test = y[y.index >= start_test] .. This can be replaced with the method train_test_split from Scikit-Learn in the train_test_split.py file. For completeness, the full __main__ method is provided below: # train_test_split.py if __name__ == "__main__": # Create a lagged series of the S&P500 US stock market index snpret = create_lagged_series( "^GSPC", datetime.datetime(2001,1,10), datetime.datetime(2005,12,31), lags=5 ) # Use the prior two days of returns as predictor # values, with direction as the response X = snpret[["Lag1","Lag2"]] y = snpret["Direction"] # Train/test split X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.8, random_state=42 ) # Create the (parametrised) models print("Hit Rates/Confusion Matrices:\n") models = [("LR", LogisticRegression()), ("LDA", LDA()), ("QDA", QDA()), ("LSVC", LinearSVC()), ("RSVM", SVC( C=1000000.0, cache_size=200, class_weight=None, coef0=0.0, degree=3, gamma=0.0001, kernel=’rbf’, max_iter=-1, probability=False, random_state=None,
185 shrinking=True, tol=0.001, verbose=False) ), ("RF", RandomForestClassifier( n_estimators=1000, criterion=’gini’, max_depth=None, min_samples_split=2, min_samples_leaf=1, max_features=’auto’, bootstrap=True, oob_score=False, n_jobs=1, random_state=None, verbose=0) )] # Iterate through the models for m in models: # Train each of the models on the training set m[1].fit(X_train, y_train) # Make an array of predictions on the test set pred = m[1].predict(X_test) # Output the hit-rate and the confusion matrix for each model print("%s:\n%0.3f" % (m[0], m[1].score(X_test, y_test))) print("%s\n" % confusion_matrix(pred, y_test)) Notice that we have picked the ratio of the training set to be 80% of the data, leaving the testing data with only 20%. In addition we have specified a random_state to randomise the sampling within the selection of data. This means that the data is not sequentially divided chronologically, but rather is sampled randomly. The results of the cross-validation on the model are as follows (yous will likely appear slightly different due to the nature of the fitting procedure): Hit Rates/Confusion Matrices: LR: 0.511 [[ 70 70] [419 441]] LDA: 0.513 [[ 69 67] [420 444]] QDA: 0.503 [[ 83 91] [406 420]] LSVC: 0.513 [[ 69 67] [420 444]] RSVM: 0.506 [[ 8 13] [481 498]]
186 RF: 0.490 [[200 221] [289 290]] It can be seen that the hit rates are substantially lower than those found in the aforementioned forecasting chapter. Consequently we can likely conclude that the particular choice of training/test split lead to an over-optimistic view of the predictive capability of the classifier. The next step is to increase the number of times a cross-validation is performed in order to minimise any potential overfitting. For this we will use k-fold cross validation. K-Fold Cross Validation Rather than partitioning the set into a single training and test set, we can use k-fold cross validation to randomly partition the the set into k equally sized subsamples. For each iteration (of which there are k), one of the k subsamples is retained as a test set, while the remaining k − 1 subsamples together form a training set. A statistical model is then trained on each of the k folds and its performance evaluated on its specific k-th test set. The purpose of this is to combine the results of each model into an emsemble by means of averaging the results of the prediction (or otherwise) to produce a single prediction. The main benefit of using k-fold cross validation is that the every predictor within the original data set is used both for training and testing only once. This motivates a question as to how to choose k, which is now another parameter! Generally, k = 10 is used but one can also perform another analysis to choose an optimal value of k. We will now make use of the cross_validation module of Scikit-Learn to obtain the KFold k-fold cross validation object. We create a new file called k_fold_cross_val.py, which is a copy of train_test_split.py and modify the imports by adding the following line: #!/usr/bin/python # -*- coding: utf-8 -*- # k_fold_cross_val.py from __future__ import print_function import datetime import pandas as pd import sklearn from sklearn import cross_validation from sklearn.metrics import confusion_matrix from sklearn.svm import SVC from create_lagged_series import create_lagged_series We then need to make changes the __main__ function by removing the train_test_split method and replacing it with an instance of KFold. It takes five parameters. The first isthe length of the dataset, which in this case 1250 days. The second value is K representing the number of folds, which in this case is 10. The third value is indices, which I have set to False. This means that the actual index values are used for the arrays returned by the iterator. The fourth and fifth are used to randomise the order of the samples. As before in forecast.py and train_test_split.py we obtain the lagged series of the S&P500. We then create a set of vectors of predictors (X) and responses (y). We then utilise the KFold object and iterate over it. During each iteration we create the training and testing sets for each of the X and y vectors. These are then fed into a radial support vector machine with identical parameters to the aforementioned files and the model is fit. Finally the hit rate and confusion matrix for each instance of the SVM is output.
187 # k_fold_cross_val.py if __name__ == "__main__": # Create a lagged series of the S&P500 US stock market index snpret = create_lagged_series( "^GSPC", datetime.datetime(2001,1,10), datetime.datetime(2005,12,31), lags=5 ) # Use the prior two days of returns as predictor # values, with direction as the response X = snpret[["Lag1","Lag2"]] y = snpret["Direction"] # Create a k-fold cross validation object kf = cross_validation.KFold( len(snpret), n_folds=10, indices=False, shuffle=True, random_state=42 ) # Use the kf object to create index arrays that # state which elements have been retained for training # and which elements have beenr retained for testing # for each k-element iteration for train_index, test_index in kf: X_train = X.ix[X.index[train_index]] X_test = X.ix[X.index[test_index]] y_train = y.ix[y.index[train_index]] y_test = y.ix[y.index[test_index]] # In this instance only use the # Radial Support Vector Machine (SVM) print("Hit Rate/Confusion Matrix:") model = SVC( C=1000000.0, cache_size=200, class_weight=None, coef0=0.0, degree=3, gamma=0.0001, kernel=’rbf’, max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False ) # Train the model on the retained training data model.fit(X_train, y_train) # Make an array of predictions on the test set pred = model.predict(X_test) # Output the hit-rate and the confusion matrix for each model print("%0.3f" % model.score(X_test, y_test)) print("%s\n" % confusion_matrix(pred, y_test)) The output of the code is as follows: Hit Rate/Confusion Matrix: 0.528 [[11 10] [49 55]]
188 Hit Rate/Confusion Matrix: 0.400 [[ 2 5] [70 48]] Hit Rate/Confusion Matrix: 0.528 [[ 8 8] [51 58]] Hit Rate/Confusion Matrix: 0.536 [[ 6 3] [55 61]] Hit Rate/Confusion Matrix: 0.512 [[ 7 5] [56 57]] Hit Rate/Confusion Matrix: 0.480 [[11 11] [54 49]] Hit Rate/Confusion Matrix: 0.608 [[12 13] [36 64]] Hit Rate/Confusion Matrix: 0.440 [[ 8 17] [53 47]] Hit Rate/Confusion Matrix: 0.560 [[10 9] [46 60]] Hit Rate/Confusion Matrix: 0.528 [[ 9 11] [48 57]] It is the clear that the hit rate and confusion matrices vary dramatically across the various folds. This is indicative that the model is prone to overfitting, on this particular dataset. A remedy for this is to use significantly more data, either at a higher frequency or over a longer duration. In order to utilise this model in a trading strrategy it would be necessary to combine each of these individually trained classifiers (i.e. each of the K objects) into an ensemble average and then use that combined model for classification within the strategy. Note that technically it is not appropriate to use simple cross-validation techniques on temporally ordered data (i.e. time-series). There are more sophisticated mechanisms for coping with autocorrelation in this fashion, but I wanted to highlight the approach so we have used time series data for simplicity.
189 16.2.2 Grid Search We have so far seen that k-fold cross validation helps us to avoid overfitting in the data by performing validation on every element of the sample. We now turn our attention to optimising the hyper-parameters of a particular statistical model. Such parameters are those not directly learnt by the model estimation procedure. For instance, C and γ for a support vector machine. In essence they are the parameters that we need to specify when calling the initialisation of each statistical model. For this procedure we will use a process known as a grid search. The basic idea is to take a range of parameters and assess the performance of the statistical model on each parameter element within the range. To achieve this in Scikit-Learn we can create a ParameterGrid. Such an object will produce a list of Python dictionaries that each contain a parameter combination to be fed into a statistical model. An example code snippet that produces a parameter grid, for parameters related to a support vector machine, is given below: >>> from sklearn.grid_search import ParameterGrid >>> param_grid = {’C’: [1, 10, 100, 1000], ’gamma’: [0.001, 0.0001]} >>> list(ParameterGrid(param_grid)) [{’C’: 1, ’gamma’: 0.001}, {’C’: 1, ’gamma’: 0.0001}, {’C’: 10, ’gamma’: 0.001}, {’C’: 10, ’gamma’: 0.0001}, {’C’: 100, ’gamma’: 0.001}, {’C’: 100, ’gamma’: 0.0001}, {’C’: 1000, ’gamma’: 0.001}, {’C’: 1000, ’gamma’: 0.0001}] Now that we have a suitable means of generating a ParameterGrid we need to feed this into a statistical model iteratively to search for an optimal performance score. In this case we are going to seek to maximise the hit rate of the classifier. The GridSearchCV mechanism from Scikit-Learn allows us to perform the actual grid search. In fact, it allows us to perform not only a standard grid search but also a cross validation scheme at the same time. We are now going to create a new file, grid_search.py, that once again uses create_lagged_series.py and a support vector machine to perform a cross-validated hyperparameter grid search. To this end we must import the correct libraries: #!/usr/bin/python # -*- coding: utf-8 -*- # grid_search.py from __future__ import print_function import datetime import sklearn from sklearn import cross_validation from sklearn.cross_validation import train_test_split from sklearn.grid_search import GridSearchCV from sklearn.metrics import classification_report from sklearn.svm import SVC from create_lagged_series import create_lagged_series As before with k_fold_cross_val.py we create a lagged series and then use the previous two days of returns as predictors. We initially create a training/test split such that 50% of the
190 data can be used for training and cross validation while the remaining data can be "held out" for evaluation. Subsequently we create the tuned_parameters list, which contains a single dictionary denoting the parameters we wish to test over. This will create a cartesian product of all parameter lists, i.e. a list of pairs of every possible parameter combination. Once the parameter list is created we pass it to the GridSearchCV class, along with the type of classifier that we’re interested in (namely a radial support vector machine), with a k-fold cross-validation k-value of 10. Finally, we train the model and output the best estimator and its associated hit rate scores. In this way we have not only optimised the model parameters via cross validation but we have also optimised the hyperparameters of the model via a parametrised grid search, all in one class! Such succinctness of the code allows significant experimentation without being bogged down by excessive "data wrangling". if __name__ == "__main__": # Create a lagged series of the S&P500 US stock market index snpret = create_lagged_series( "^GSPC", datetime.datetime(2001,1,10), datetime.datetime(2005,12,31), lags=5 ) # Use the prior two days of returns as predictor # values, with direction as the response X = snpret[["Lag1","Lag2"]] y = snpret["Direction"] # Train/test split X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.5, random_state=42 ) # Set the parameters by cross-validation tuned_parameters = [ {’kernel’: [’rbf’], ’gamma’: [1e-3, 1e-4], ’C’: [1, 10, 100, 1000]} ] # Perform the grid search on the tuned parameters model = GridSearchCV(SVC(C=1), tuned_parameters, cv=10) model.fit(X_train, y_train) print("Optimised parameters found on training set:") print(model.best_estimator_, "\n") print("Grid scores calculated on training set:") for params, mean_score, scores in model.grid_scores_: print("%0.3f for %r" % (mean_score, params)) The output of the grid search cross validation procedure is as follows: Optimised parameters found on training set: SVC(C=1, cache_size=200, class_weight=None, coef0=0.0, degree=3, gamma=0.001, kernel=’rbf’, max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False) Grid scores calculated on training set: 0.541 for {’kernel’: ’rbf’, ’C’: 1, ’gamma’: 0.001} 0.541 for {’kernel’: ’rbf’, ’C’: 1, ’gamma’: 0.0001} 0.541 for {’kernel’: ’rbf’, ’C’: 10, ’gamma’: 0.001}
191 0.541 for {’kernel’: ’rbf’, ’C’: 10, ’gamma’: 0.0001} 0.541 for {’kernel’: ’rbf’, ’C’: 100, ’gamma’: 0.001} 0.541 for {’kernel’: ’rbf’, ’C’: 100, ’gamma’: 0.0001} 0.538 for {’kernel’: ’rbf’, ’C’: 1000, ’gamma’: 0.001} 0.541 for {’kernel’: ’rbf’, ’C’: 1000, ’gamma’: 0.0001} As we can see γ = 0.001 and C = 1 provides the best hit rate, on the validation set, for this particular radial kernel support vector machine. This model could now form the basis of a forecasting-based trading strategy, as we have previously demonstrated in the prior chapter. 16.3 Optimising Strategies Up until this point we have concentrated on model selection and optimising the underlying statistical model that (might) form the basis of a trading strategy. However, a predictive model and a functioning, profitable algorithmic strategy are two different entities. We now turn our attention to optimising parameters that have a direct effect on profitability and risk metrics. To achieve this we are going to make use of the event-driven backtesting software that was described in a previous chapter. We will consider a particular strategy that has three parameters associated with it and search through the space formed by the cartesian product of parameters, using a grid search mechanism. We will then attempt to maximise particular metrics such as the Sharpe Ratio or minimise others such as the maximum drawdown. 16.3.1 Intraday Mean Reverting Pairs The strategy of interest to us in this chapter is the "Intraday Mean Reverting Equity Pairs Trade" using the energy equities AREX and WLL. It contains three parameters that we are capable of optimising: The linear regression lookback period, the residuals z-score entry threshold and the residuals z-score exit threshold. We will consider a range of values for each parameter and then calculate a backtest for the strategy across each of these ranges, outputting the total return, Sharpe ratio and drawdown characteristics of each simulation, to a CSV file for each parameter set. This will allow us to ascertain an optimised Sharpe or minimised max drawdown for our trading strategy. 16.3.2 Parameter Adjustment Since the event-driven backtesting software is quite CPU-intensive, we will restrict the parameter range to three values per parameter. This will give us a total of 3 3 = 27 separate simulations to carry out. The parameter ranges are listed below: • OLS Lookback Window - wl ∈ {50, 100, 200} • Z-Score Entry Threshold - zh ∈ {2.0, 3.0, 4.0} • Z-Score Exit Threshold - zl ∈ {0.5, 1.0, 1.5} To carry out the set of simulations a cartesian product of all three ranges will be calculated and then the simulation will be carried out for each combination of parameters. The first task is to modify the intraday_mr.py file to include the product method from the itertools library: # intraday_mr.py .. from itertools import product .. We can then modify the __main__ method to include the generation of a parameter list for all three of the parameters discussed above.