Add OOS Reconciliation page to Meta Analysis (#2300)

04 Research Environment/10 Meta Analysis/04 Live Analysis/03 Reconciliation.html → 04 Research Environment/10 Meta Analysis/05 Live Reconciliation/01 Introduction.html

@@ -1,11 +1,12 @@
+<p>This page shows you how to generate the out-of-sample (OOS) backtest reconciliation curve from a live deployment in the Research Environment so you can quantitatively and visually compare live versus backtest performance. You read the live deployment's launch datetime and starting equity, run a backtest with matching parameters, and overlay the two "Strategy Equity" curves along with every order fill on a single chart per security.</p>
+
 <p>
-Reconciliation is a way to <a href = "https://www.quantconnect.com/forum/discussion/7606/a-new-reconciliation-metric/p1">quantify</a> the difference between an algorithm's live performance and its out-of-sample (OOS) performance (a backtest run over the live deployment period).
+Reconciliation is a way to <a href="https://www.quantconnect.com/forum/discussion/7606/a-new-reconciliation-metric/p1">quantify</a> the difference between an algorithm's live performance and its out-of-sample (OOS) performance (a backtest run over the live deployment period).
 </p>
 <p>
 Seeing the difference between live performance and OOS performance gives you a way to determine if the algorithm is making unrealistic assumptions, exploiting data differences, or merely exhibiting behavior that is impractical or impossible in live trading.
 </p>
-<p>A perfectly reconciled algorithm has an exact overlap between its live equity and OOS backtest curves. Any deviation means that the performance of the algorithm has differed for some reason. Several factors can contribute to this, often stemming from the algorithm design.</p>
-<p>
+<p>A perfectly reconciled algorithm has an exact overlap between its live equity and OOS backtest curves. Any deviation means that the performance of the algorithm has differed for some reason. Several factors can contribute to this, often stemming from the algorithm design. For a catalogue of common deviation causes (data, modeling, brokerage, third-party indicators, and real-time scheduled events), see <a href="/docs/v2/writing-algorithms/live-trading/reconciliation">Reconciliation</a> in the Writing Algorithms documentation.</p>
 
 <img class="docs-image" src="https://cdn.quantconnect.com/i/tu/reconciliation-4.png" alt="Live Deployment Reconciliation" width="100%">
 
@@ -20,7 +21,7 @@
 Reconciliation is scored using two metrics: returns correlation and dynamic time warping (DTW) distance.
 </p>
 <h4>What is DTW Distance?</h4>
-<p>Dynamic Time Warp (DTW) Distance quantifies the difference between two time-series. It is an algorithm that measures the shortest path between the points of two time-series. It uses Euclidean distance as a measurement of <b>point-to-point distance</b> and returns an overall measurement of the distance on the scale of the initial time-series values. We apply DTW to the returns curve of the live and OOS performance, so the DTW distance measurement is on the scale of percent returns. </p>
+<p>Dynamic Time Warp (DTW) Distance quantifies the difference between two time-series. It is an algorithm that measures the shortest path between the points of two time-series. It uses Euclidean distance as a measurement of <b>point-to-point distance</b> and returns an overall measurement of the distance on the scale of the initial time-series values. We apply DTW to the returns curve of the live and OOS performance, so the DTW distance measurement is on the scale of percent returns.</p>
 
 $$\begin{equation}
 DTW(X,Y) = min\bigg\{\sum_{l=1}^{L}\left(x_{m_l} - y_{n_l}\right)^{2}\in P^{N\times M}\bigg\}

04 Research Environment/10 Meta Analysis/05 Live Reconciliation/02 Get Live Deployment Parameters.php

@@ -0,0 +1,67 @@
+<p>Follow these steps to read the live deployment's start datetime, starting equity, and end datetime — the three parameters the OOS backtest must match. All three values come from the live "Strategy Equity" chart, so you only need the project Id.</p>
+
+<ol>
+    <li>Define the project Id.</li>
+    <div class="section-example-container">
+        <pre class="python">project_id = 23034953</pre>
+    </div>
+
+    <p>The following table provides links to documentation that explains how to get the project Id, depending on the platform you use:</p>
+
+    <table class="qc-table table">
+        <thead>
+            <tr>
+                <th style='width: 50%'>Platform</th>
+                <th>Project Id</th>
+            </tr>
+        </thead>
+        <tbody>
+            <tr>
+                <td>Cloud Platform</td>
+                <td><a href='/docs/v2/cloud-platform/projects/getting-started#13-Get-Project-Id'>Get Project Id</a></td>
+            </tr>
+            <tr>
+                <td>Local Platform</td>
+                <td><a href='/docs/v2/local-platform/projects/getting-started#14-Get-Project-Id'>Get Project Id</a></td>
+            </tr>
+            <tr>
+                <td>CLI</td>
+                <td><a href='/docs/v2/lean-cli/projects/project-management#07-Get-Project-Id'>Get Project Id</a></td>
+            </tr>
+        </tbody>
+    </table>
+
+    <li>Read the live "Strategy Equity" chart with the <code class="csharp">ReadLiveChart</code><code class="python">read_live_chart</code> method. The first and last <code>Equity</code> points give you the start datetime, starting equity, and end datetime.</li>
+    <div class="section-example-container">
+        <pre class="python">from datetime import datetime
+from time import sleep, time
+
+def read_chart(project_id, chart_name, start=0, end=int(time()), count=500):
+    # Retry up to 10 times until the chart data finishes loading.
+    for attempt in range(10):
+        result = api.read_live_chart(project_id, chart_name, start, end, count)
+        if result.success:
+            return result.chart
+        print(f"Chart data is loading... (attempt {attempt + 1}/10)")
+        sleep(10)
+    raise RuntimeError(f"Failed to read {chart_name} chart after 10 attempts")
+
+strategy_equity = read_chart(project_id, 'Strategy Equity')
+# The first few points in the series can have a None close, so keep only
+# the points with a valid close value before extracting start/end.
+valid_values = [v for v in strategy_equity.series['Equity'].values if v.close is not None]
+
+# Start datetime and starting equity: first valid point.
+start_datetime = valid_values[0].time
+starting_cash = valid_values[0].close
+# End datetime: last valid timestamp of the live Strategy Equity series.
+# Uncomment the next line instead to reconcile up to "now" and see what
+# would have happened had you not stopped the live algorithm:
+# end_datetime = datetime.utcnow()
+end_datetime = valid_values[-1].time
+
+print(f"Start (UTC): {start_datetime}")
+print(f"Starting equity: ${starting_cash:,.2f}")
+print(f"End (UTC): {end_datetime}")</pre>
+    </div>
+</ol>

04 Research Environment/10 Meta Analysis/05 Live Reconciliation/03 Run OOS Backtest.php

@@ -0,0 +1,41 @@
+<p>Follow these steps to run an out-of-sample backtest that mirrors the live deployment.</p>
+
+<ol>
+    <li>In the project's main algorithm file, set the start date and starting cash to match the values you read in the previous step. Use <code class="csharp">SetStartDate</code><code class="python">set_start_date</code> and <code class="csharp">SetCash</code><code class="python">set_cash</code> so the backtest begins at the same moment and with the same equity as the live deployment. You can either hard-code the values or expose them as <a href='/docs/v2/writing-algorithms/parameter-and-optimization/parameters'>parameters</a>.</li>
+
+    <li>Compile the project by calling the <code class="csharp">CreateCompile</code><code class="python">create_compile</code> method, then poll <code class="csharp">ReadCompile</code><code class="python">read_compile</code> until the compile state is <code>BuildSuccess</code>.</li>
+    <div class="section-example-container">
+        <pre class="python">from time import sleep
+
+compilation = api.create_compile(project_id)
+compile_id = compilation.compile_id
+
+# Poll until the build succeeds.
+for attempt in range(10):
+    result = api.read_compile(project_id, compile_id)
+    if result.state == 'BuildSuccess':
+        break
+    if result.state == 'BuildError':
+        raise Exception(f"Compilation failed: {result.logs}")
+    print(f"Compile in queue... (attempt {attempt + 1}/10)")
+    sleep(5)</pre>
+    </div>
+
+    <li>Create the OOS backtest with the <code class="csharp">CreateBacktest</code><code class="python">create_backtest</code> method.</li>
+    <div class="section-example-container">
+        <pre class="python">backtest = api.create_backtest(project_id, compile_id, 'OOS Reconciliation')
+backtest_id = backtest.backtest_id
+print(f"Backtest Id: {backtest_id}")</pre>
+    </div>
+
+    <li>Poll the <code class="csharp">ReadBacktest</code><code class="python">read_backtest</code> method until the <code>completed</code> flag is <code>True</code>. Log the <code>progress</code> attribute on each poll so you can watch the backtest advance.</li>
+    <div class="section-example-container">
+        <pre class="python">completed = False
+while not completed:
+    result = api.read_backtest(project_id, backtest_id)
+    completed = result.completed
+    print(f"Backtest running... {result.progress:.2%}")
+    sleep(10)
+print("Backtest completed.")</pre>
+    </div>
+</ol>

04 Research Environment/10 Meta Analysis/05 Live Reconciliation/04 Plot Equity Curves.php

@@ -0,0 +1,80 @@
+<p>Follow these steps to plot the live and OOS backtest equity curves on the same axes.</p>
+
+<ol>
+    <li>Read the live "Strategy Equity" chart using the retry helper from the previous step.</li>
+    <div class="section-example-container">
+        <pre class="python">live_equity_chart = read_chart(project_id, 'Strategy Equity')</pre>
+    </div>
+
+    <li>Read the backtest "Strategy Equity" chart by calling the <code class="csharp">ReadBacktestChart</code><code class="python">read_backtest_chart</code> method with the same retry pattern.</li>
+    <div class="section-example-container">
+        <pre class="python">def read_backtest_chart(project_id, backtest_id, chart_name, start=0, end=int(time()), count=500):
+    for attempt in range(10):
+        result = api.read_backtest_chart(project_id, chart_name, start, end, count, backtest_id)
+        if result.success:
+            return result.chart
+        print(f"Chart data is loading... (attempt {attempt + 1}/10)")
+        sleep(10)
+    raise RuntimeError(f"Failed to read backtest {chart_name} chart after 10 attempts")
+
+backtest_equity_chart = read_backtest_chart(project_id, backtest_id, 'Strategy Equity')</pre>
+    </div>
+
+    <li>Extract the <code>Equity</code> series from each chart into a <code>pandas.Series</code> indexed by timestamp. Preserve every timestamp from both the live and backtest series and strip any timezone info so <code>pandas</code> and <code>plotly</code> can align and render the curves cleanly.</li>
+    <div class="section-example-container">
+        <pre class="python">import pandas as pd
+
+def to_naive(t):
+    ts = pd.Timestamp(t)
+    return ts.tz_convert('UTC').tz_localize(None) if ts.tzinfo else ts
+
+def to_series(chart, series_name='Equity'):
+    values = [v for v in chart.series[series_name].values if v.close is not None]
+    return pd.Series(
+        [v.close for v in values],
+        index=pd.DatetimeIndex([to_naive(v.time) for v in values])
+    )
+
+live_series = to_series(live_equity_chart)
+backtest_series = to_series(backtest_equity_chart)
+
+# Keep every timestamp from both sources; align and forward-fill on the union.
+df = pd.concat([live_series.rename('Live'), backtest_series.rename('OOS Backtest')], axis=1).sort_index().ffill()</pre>
+    </div>
+
+    <li>Plot both curves on a single <code>matplotlib</code> axis.</li>
+    <div class="section-example-container">
+        <pre class="python">import matplotlib.pyplot as plt
+
+fig, ax = plt.subplots(figsize=(12, 6))
+ax.plot(df.index, df['Live'], label='Live')
+ax.plot(df.index, df['OOS Backtest'], label='OOS Backtest')
+ax.set_title('Live vs OOS Backtest Equity')
+ax.set_xlabel('Time')
+ax.set_ylabel('Portfolio Value ($)')
+ax.legend()
+plt.show()</pre>
+    </div>
+    <img class='docs-image' src="https://cdn.quantconnect.com/i/tu/reconciliation-4.png" alt="Live vs OOS backtest equity curves">
+
+    <li>Score the reconciliation with the annualized returns DTW distance and the Pearson correlation of daily returns. Use <code>tslearn</code>'s <code>dtw</code> with a Sakoe-Chiba band so the algorithm runs in linear time.</li>
+    <div class="section-example-container">
+        <pre class="python">from tslearn.metrics import dtw as DynamicTimeWarping
+
+returns = df.pct_change().dropna()
+
+# Pearson correlation between live and OOS backtest daily returns (closer to 1 is better).
+returns_correlation = returns.corr().iloc[0, 1]
+
+# Raw DTW distance on the returns curves.
+raw_dtw = DynamicTimeWarping(
+    returns['Live'], returns['OOS Backtest'],
+    global_constraint='sakoe_chiba', sakoe_chiba_radius=3
+)
+# Annualize so the distance is on the scale of yearly percent returns (closer to 0 is better).
+annualized_dtw = abs(((1 + (raw_dtw / returns.shape[0])) ** 252) - 1)
+
+print(f"Returns correlation: {returns_correlation:.3f}")
+print(f"Annualized returns DTW: {annualized_dtw:.3f}")</pre>
+    </div>
+</ol>

04 Research Environment/10 Meta Analysis/05 Live Reconciliation/05 Plot Order Fills.php

@@ -0,0 +1,91 @@
+<p>Follow these steps to overlay live and OOS backtest order fills on a single marker-only chart per symbol. The chart deliberately omits candlesticks and any price history so the comparison between live and backtest executions is not drowned out by other series.</p>
+
+<ol>
+    <li>Read the live and backtest orders. Both endpoints can take a few seconds to load the first time, so retry until the response reports success.</li>
+    <div class="section-example-container">
+        <pre class="python">from time import sleep
+
+def read_orders(fetch):
+    for attempt in range(10):
+        result = fetch()
+        if result:
+            return result
+        print(f"Orders loading... (attempt {attempt + 1}/10)")
+        sleep(10)
+    raise RuntimeError("Failed to read orders after 10 attempts")
+
+live_orders = read_orders(lambda: api.read_live_orders(project_id, 0, 100))
+backtest_orders = read_orders(lambda: api.read_backtest_orders(project_id, backtest_id, 0, 100))</pre>
+    </div>
+    <p>By default, you receive the orders with an Id between 0 and 100. To read more, call the method repeatedly in windows of up to 100 Ids. For more on the order objects returned, see <a href='/docs/v2/research-environment/meta-analysis/live-analysis#03-Plot-Order-Fills'>Plot Order Fills</a> in the Live Analysis documentation.</p>
+
+    <li>Organize the trade times and prices for each security into a dictionary for both the live and backtest fills.</li>
+    <div class="section-example-container">
+        <pre class="python">import pandas as pd
+
+def to_naive(t):
+    # Strip tzinfo so plotly can serialize the fill times.
+    ts = pd.Timestamp(t)
+    return ts.tz_convert('UTC').tz_localize(None) if ts.tzinfo else ts
+
+class OrderData:
+    def __init__(self):
+        self.buy_fill_times = []
+        self.buy_fill_prices = []
+        self.sell_fill_times = []
+        self.sell_fill_prices = []
+
+def group_by_symbol(orders):
+    data_by_symbol = {}
+    for order in [x.order for x in orders]:
+        if order.symbol not in data_by_symbol:
+            data_by_symbol[order.symbol] = OrderData()
+        data = data_by_symbol[order.symbol]
+        is_buy = order.quantity &gt; 0
+        (data.buy_fill_times if is_buy else data.sell_fill_times).append(to_naive(order.last_fill_time))
+        (data.buy_fill_prices if is_buy else data.sell_fill_prices).append(order.price)
+    return data_by_symbol
+
+live_by_symbol = group_by_symbol(live_orders)
+backtest_by_symbol = group_by_symbol(backtest_orders)</pre>
+    </div>
+
+    <li>Plot one figure per symbol with four marker traces: live buys, live sells, backtest buys, backtest sells. Distinct markers keep live versus backtest executions visually separable.</li>
+    <div class="section-example-container">
+        <pre class="python">import plotly.graph_objects as go
+
+symbols = set(live_by_symbol.keys()) | set(backtest_by_symbol.keys())
+
+for symbol in symbols:
+    live = live_by_symbol.get(symbol, OrderData())
+    bt = backtest_by_symbol.get(symbol, OrderData())
+
+    fig = go.Figure(layout=go.Layout(
+        title=go.layout.Title(text=f'{symbol.value} Live vs OOS Backtest Fills'),
+        xaxis_title='Fill Time',
+        yaxis_title='Fill Price',
+        height=600
+    ))
+
+    fig.add_trace(go.Scatter(
+        x=live.buy_fill_times, y=live.buy_fill_prices, mode='markers', name='Live Buys',
+        marker=go.scatter.Marker(color='aqua', symbol='triangle-up', size=12)
+    ))
+    fig.add_trace(go.Scatter(
+        x=live.sell_fill_times, y=live.sell_fill_prices, mode='markers', name='Live Sells',
+        marker=go.scatter.Marker(color='indigo', symbol='triangle-down', size=12)
+    ))
+    fig.add_trace(go.Scatter(
+        x=bt.buy_fill_times, y=bt.buy_fill_prices, mode='markers', name='OOS Backtest Buys',
+        marker=go.scatter.Marker(color='aqua', symbol='triangle-up-open', size=12, line=dict(width=2))
+    ))
+    fig.add_trace(go.Scatter(
+        x=bt.sell_fill_times, y=bt.sell_fill_prices, mode='markers', name='OOS Backtest Sells',
+        marker=go.scatter.Marker(color='indigo', symbol='triangle-down-open', size=12, line=dict(width=2))
+    ))
+
+    fig.show()</pre>
+    </div>
+</ol>
+
+<p>Note: the preceding plots only show the last fill of each trade. If your trade has partial fills, the plots only display the last fill.</p>