Transaction Cost Modeling Beyond Flat Assumptions

Every backtest makes assumptions about what it costs to trade. Most of those assumptions are wrong — not because the researcher is careless, but because the true cost of executing a trade depends on variables that a backtest cannot fully observe.

The simplest approach is to assign a flat cost per trade: ten basis points, twenty basis points, whatever feels conservative enough. This is common. It is quick. And it is often misleading.

In our own research workflow, transaction cost modeling was one of the last components we formalized — and one of the first we wished we had formalized sooner. The difference between a flat assumption and a structured cost model changed the ranking of several strategies we were evaluating. That is not a theoretical risk. It is something we observed directly.

This article describes why flat cost assumptions break down, what a more realistic model looks like, and where the practical boundary sits between useful precision and false exactness.

A flat cost assumption treats every trade as equally expensive. Buy $500 of a large-cap at the open, or sell $50,000 of a micro-cap into a thin close — same cost.

That is not how markets work.

Real transaction costs vary along at least four dimensions: the security's liquidity, the size of the order relative to typical volume, the urgency of execution, and the prevailing volatility at the time of the trade. A flat assumption collapses all of these into a single number, which means it simultaneously overstates the cost of easy trades and understates the cost of hard ones.

The distortion is not symmetric. Strategies that trade liquid, high-volume names are penalized too heavily. Strategies that trade illiquid or small-cap names are not penalized enough. If you are comparing two strategies using the same flat cost, the comparison is biased toward the one that happens to trade harder-to-execute securities.

We encountered this directly when comparing a large-cap momentum strategy against a small-cap mean-reversion strategy under a uniform 15-basis-point assumption. The small-cap strategy appeared to outperform. Once we replaced the flat cost with a liquidity-adjusted model, the ranking reversed.

Transaction costs are not a single number. They are a composite of several distinct components, each driven by different market mechanics.

Commission and fees are the most visible and the least interesting. For most institutional and semi-institutional accounts, these are small and predictable. They still need to be included, but they are rarely the source of modeling error.

The bid-ask spread is where the cost becomes less trivial. Every market order crosses the spread — buying at the ask and selling at the bid. The spread itself varies by security, by time of day, and by market conditions. During periods of elevated volatility, spreads widen. For less liquid securities, spreads can be substantial even in calm markets.

Market impact is the hardest component to estimate and often the largest. When an order is large relative to the security's typical volume, the act of trading itself moves the price. This is not a failure of execution. It is a structural feature of how order books work. A backtest that ignores market impact is implicitly assuming infinite liquidity — that any size can be executed at the quoted price without moving it.

Slippage captures the residual gap between the price assumed by the model and the price actually received. It includes timing delays, partial fills, and the difference between a theoretical signal price and the achievable execution price.

Opportunity cost is subtler. When a trade is not executed — because the price moved away before the order could fill, or because the position was scaled down to limit impact — there is a cost to the missed trade that never appears in the P&L but still affects realized performance.

A useful cost model does not need to be perfect. It needs to be less wrong than a flat assumption in a way that matters for strategy comparison and capital allocation decisions.

The most common practical approach is a spread-plus-impact model. The base cost is estimated as half the prevailing bid-ask spread, which captures the cost of crossing from one side to the other. On top of that, an impact term is added as a function of order size relative to average daily volume.

In our pipeline, we estimate the spread component using the median quoted spread over a trailing window — typically 20 to 60 trading days. This is imperfect because spreads vary intraday and across conditions, but it is directionally correct and avoids the worst distortions of a flat number.

The impact component follows a standard square-root model: the expected impact scales with the square root of the participation rate. If you are trading 1% of daily volume, the impact is proportional to the square root of 0.01. If you are trading 5%, it is proportional to the square root of 0.05. The coefficient is calibrated to the asset class — typically 10 to 30 basis points for equities.

This is not an exact science. The square-root model is well-supported empirically, but the specific calibration depends on market conditions, execution venue, and order handling. In practice, we treat the cost estimate as a range rather than a point — running backtests at the estimated cost, then again at 1.5 times and 2 times the estimate, to see whether the strategy's edge survives a reasonable stress test.

The first thing that changes is turnover sensitivity. Strategies with high turnover — frequent rebalancing, tight holding periods — are disproportionately affected by realistic cost modeling. A strategy that trades weekly and shows 12% annualized return under flat costs may show 8% under a spread-plus-impact model. That is not a small difference.

The second change is capacity estimation. A flat cost model cannot tell you when a strategy runs out of room. A size-dependent cost model can. If impact costs rise as position size grows, there is a natural ceiling on how much capital the strategy can absorb before the edge is consumed by its own execution costs. That ceiling is invisible under flat assumptions.

The third and often most surprising change is strategy ranking. In our work, we have seen multiple instances where the best strategy under flat costs was not the best strategy under realistic costs. This happens because flat costs systematically favor strategies that trade less liquid instruments — the flat number understates their true cost. Once you correct for that, the ordering can shift.

This is the core reason to care about cost modeling. Not because the exact number matters to the decimal point. It does not. But because the relative cost differences between strategies are large enough to change portfolio-level decisions.

The most common mistake is the one we have already described: using a flat cost and treating it as conservative. Flat costs are only conservative for the easiest trades. For the hardest trades, they are aggressive.

The second mistake is modeling costs as of today and applying them historically. Spreads, commissions, and market microstructure have changed dramatically over the past two decades. A security that trades with a 2-basis-point spread today may have traded with a 15-basis-point spread in 2005. A backtest that applies today's costs to 2005 data understates historical transaction friction.

The third mistake is ignoring the relationship between strategy turnover and cost. A strategy that rebalances daily faces not just more trades, but potentially worse fills — because repeated buying and selling of the same names creates predictable order flow that the market can front-run or anticipate. In practice, cost per trade often increases with rebalancing frequency, but most models treat it as constant.

A subtler mistake is neglecting the cost of not trading. If a strategy's cost model is too aggressive, the optimizer will avoid trades that are genuinely profitable net of realistic costs. Overly conservative cost modeling is not free — it has its own distortion.

Because no cost model is exact, the sensible approach is to test sensitivity rather than chase precision.

In our workflow, we run every strategy evaluation at three cost levels: the base estimate, 1.5 times the estimate, and 2 times the estimate. If a strategy is profitable at all three levels, we have more confidence that the edge is real and not a narrow artifact of favorable cost assumptions. If the strategy breaks at 1.5 times, we know the edge is fragile and highly sensitive to execution quality.

This stress-testing approach matters more than getting the base estimate exactly right. A strategy with a moderate but stress-resilient edge is almost always preferable to one with a higher nominal edge that collapses under slightly worse execution.

The calibration itself should be reviewed periodically. Market microstructure evolves. Spreads compress as electronic market-making improves. New venues change execution dynamics. A cost model calibrated in 2020 may not be appropriate for 2026 data without adjustment.

There is a point beyond which additional precision in cost modeling does not improve research decisions. That point arrives sooner than most researchers expect.

The goal is not to predict the exact cost of a future trade. That depends on real-time order book conditions, execution algorithms, and market state at the moment of the trade — none of which a backtest can know in advance.

The goal is to make cost assumptions realistic enough that strategy comparisons, capital allocation decisions, and capacity estimates are not materially distorted. That requires moving beyond flat assumptions and into a liquidity-aware model. It does not require building a full microstructure simulator.

In practice, the spread-plus-impact model described above — calibrated with historical spread data and stressed at multiple levels — captures the majority of the useful information. Further refinement yields diminishing returns for most research workflows.

The honest answer is that you will never know your true transaction costs until you trade live. What a good cost model does is reduce the gap between the backtest's assumption and reality enough that the transition to live performance is not a shock.

Before trusting a backtest's performance numbers, ask:

Transaction cost modeling is one of those areas where doing something simple but structured is dramatically better than doing nothing — and where doing something elaborate is only marginally better than doing something simple.

The flat-cost assumption is the nothing. A spread-plus-impact model with stress testing is the simple-but-structured. A full microstructure simulation is the elaborate.

For most systematic research, the middle path is the right one. It catches the errors that matter. It reveals the strategies that are fragile. And it does so without requiring infrastructure that only the largest quantitative firms can justify.

The transition from flat costs to structured costs is one of the highest-return improvements a research process can make. Not because the numbers change dramatically, but because the decisions change.