Sina Bolouri
Back to Science articles

Introduction to Time Series

ARMA Models as Dynamic Filters

How autoregressive feedback and moving-average innovations create memory, persistence, oscillation, and finite shock effects.

Article 3Time-Series Analysisintermediate20-25 minutespartially-verified
time seriesforecastingdata sciencearmaautoregressionmoving averagecausalityinvertibilitybackshift operatorcharacteristic roots

Introduction: The Problem Before the Method

A correlogram can show that a series remembers its past, but it does not yet explain how that memory is generated. One process may carry forward a fraction of its previous state. Another may combine a small number of recent shocks. A third may do both.

Autoregressive-moving-average models provide a compact language for these mechanisms. The autoregressive part describes feedback from past observations. The moving-average part describes how current and recent innovations enter the observation. Their combination can reproduce persistent decay, damped cycles, short-lived shock effects, and many mixtures of those patterns.

The term moving average is potentially confusing. In ARMA modeling it does not mean a rolling descriptive average. It means a finite linear combination of unobserved innovations.

The Core Idea

A zero-mean ARMA((p,q)) process satisfies

ϕ(B)Xt=θ(B)Wt,\phi(B)X_t=\theta(B)W_t,

where (B) is the backshift operator and

ϕ(B)=1−ϕ1B−⋯−ϕpBp,\phi(B)=1-\phi_1B-\cdots-\phi_pB^p, θ(B)=1+θ1B+⋯+θqBq.\theta(B)=1+\theta_1B+\cdots+\theta_qB^q.

Here:

  • (X_t) is the observed process,
  • (W_t) is white noise with variance (\sigma^2),
  • (p) is the autoregressive order,
  • (q) is the moving-average order.

Expanded, the model is

Xt=ϕ1Xt−1+⋯+ϕpXt−p+Wt+θ1Wt−1+⋯+θqWt−q.X_t = \phi_1X_{t-1}+\cdots+\phi_pX_{t-p} + W_t+\theta_1W_{t-1}+\cdots+\theta_qW_{t-q}.

The AR side transmits past state. The MA side distributes innovation effects over a finite window. The observed memory can nevertheless be infinite because feedback repeatedly propagates earlier shocks.

Why the Concept Exists

A purely autoregressive model may need many lags to reproduce a short, irregular shock pattern. A purely moving-average model may need many innovation terms to reproduce gradual persistence. ARMA models combine both structures and can achieve a compact representation.

Their value is not merely curve fitting. They provide:

  • a generative equation,
  • a theoretical autocovariance structure,
  • forecast recursions,
  • innovation estimates,
  • and a basis for uncertainty intervals.

That usefulness depends on two structural properties: causality and invertibility.

Background and Prerequisites

Autoregression

An AR((p)) model is

ϕ(B)Xt=Wt.\phi(B)X_t=W_t.

For AR(1),

Xt=ϕXt−1+Wt.X_t=\phi X_{t-1}+W_t.

When (|\phi|<1), repeated substitution gives

Xt=Wt+ϕWt−1+ϕ2Wt−2+⋯ .X_t = W_t+\phi W_{t-1}+\phi^2W_{t-2}+\cdots.

The present is therefore a weighted sum of all past innovations.

Moving average

An MA((q)) model is

Xt=θ(B)Wt.X_t=\theta(B)W_t.

For MA(1),

Xt=Wt+θWt−1.X_t=W_t+\theta W_{t-1}.

Only two innovations enter each observation, and the theoretical ACF is zero after lag one.

Mixed structure

For ARMA(1,1),

Xt−ϕXt−1=Wt+θWt−1.X_t-\phi X_{t-1}=W_t+\theta W_{t-1}.

The MA term shapes the immediate shock response; the AR term propagates it forward.

Assumptions

Weak stationarity

The model is intended for a process whose mean and covariance structure do not change with time. A nonstationary series generally requires transformation or ARIMA structure.

White-noise innovations

The innovations should have zero mean, constant variance, and no serial correlation. Gaussianity is useful for exact likelihood and interval construction but is not required for the basic second-order ARMA definition.

Fixed coefficients

The parameters are assumed stable over the modeled period. Regime changes or time-varying dynamics violate this assumption.

No unresolved common factors

If (\phi(z)) and (\theta(z)) share a root, the model can be algebraically reduced. Retaining common factors makes order and parameter interpretation redundant.

How It Works

Causality

An ARMA model is causal when (X_t) can be represented using current and past innovations:

Xt=ψ(B)Wt=∑j=0∞ψjWt−j,∑j=0∞∣ψj∣<∞.X_t = \psi(B)W_t = \sum_{j=0}^{\infty}\psi_jW_{t-j}, \qquad \sum_{j=0}^{\infty}|\psi_j|<\infty.

For the usual polynomial convention, causality holds when every root of

ϕ(z)=0\phi(z)=0

lies outside the unit circle: (|z|>1) (Brockwell & Davis, 2016).

Causality matters because it rules out a representation in which the present depends on future shocks. It also ensures stable decay of impulse effects.

Invertibility

A model is invertible when innovations can be recovered from present and past observations:

Wt=π(B)Xt=∑j=0∞πjXt−j.W_t = \pi(B)X_t = \sum_{j=0}^{\infty}\pi_jX_{t-j}.

Invertibility holds when every root of

θ(z)=0\theta(z)=0

lies outside the unit circle.

This restriction chooses a unique, stable innovation representation. Without it, different MA parameter values can generate the same autocovariance structure.

Dynamic-filter interpretation

The transfer function is formally

ψ(B)=θ(B)ϕ(B).\psi(B)=\frac{\theta(B)}{\phi(B)}.

It maps innovation input into observed output. The coefficients (\psi_j) form the impulse-response sequence. A one-unit innovation at time (t) contributes (\psi_j) to (X_{t+j}).

The Practical Procedure

1. Write the model with an explicit sign convention

Software and textbooks differ in whether MA terms appear with plus or minus signs. Always map the reported coefficients back to the equation being used.

2. Remove common polynomial factors

Factor (\phi(z)) and (\theta(z)) or compare roots numerically. A common factor indicates that the nominal orders are larger than necessary.

3. Check AR roots for causality

Every root of the AR polynomial must lie outside the unit circle for a causal stationary representation.

4. Check MA roots for invertibility

Every root of the MA polynomial must lie outside the unit circle for a standard identifiable representation.

5. Inspect the implied impulse response

Compute or plot (\psi_j). Slow decay means persistent shock effects; alternating or sinusoidal decay indicates oscillatory dynamics.

6. Compare theoretical and sample dependence

The fitted model's theoretical ACF should plausibly match the observed stationary series, while the residual ACF should not retain systematic structure.

Mathematical or Technical Foundation

AR(1): geometric memory

For

Xt=ϕXt−1+Wt,∣ϕ∣<1,X_t=\phi X_{t-1}+W_t, \qquad |\phi|<1,

the variance is

γ(0)=σ21−ϕ2,\gamma(0)=\frac{\sigma^2}{1-\phi^2},

and

ρ(h)=ϕ∣h∣.\rho(h)=\phi^{|h|}.

As (\phi) approaches one, shock effects decay more slowly. If (\phi<0), signs alternate.

MA(1): finite innovation overlap

For

Xt=Wt+θWt−1,X_t=W_t+\theta W_{t-1}, γ(0)=σ2(1+θ2),γ(1)=θσ2,\gamma(0)=\sigma^2(1+\theta^2), \qquad \gamma(1)=\theta\sigma^2,

and (\gamma(h)=0) for (|h|>1).

The MA(1) ACF cannot exceed (1/2) in absolute value because

∣θ1+θ2∣≤12.\left|\frac{\theta}{1+\theta^2}\right|\leq\frac12.

The parameter (\theta) and its reciprocal (1/\theta) produce the same lag-one autocorrelation. Invertibility selects the representation with the root outside the unit circle.

AR(2): decay and oscillation

For

Xt=ϕ1Xt−1+ϕ2Xt−2+Wt,X_t=\phi_1X_{t-1}+\phi_2X_{t-2}+W_t,

multiplying by (X_{t-h}), taking expectations, and using innovation orthogonality gives, for (h\geq2),

γ(h)=ϕ1γ(h−1)+ϕ2γ(h−2).\gamma(h)=\phi_1\gamma(h-1)+\phi_2\gamma(h-2).

The autocorrelation satisfies the same recursion:

ρ(h)=ϕ1ρ(h−1)+ϕ2ρ(h−2).\rho(h)=\phi_1\rho(h-1)+\phi_2\rho(h-2).

The characteristic equation is

r2−ϕ1r−ϕ2=0.r^2-\phi_1r-\phi_2=0.

Real roots produce combinations of exponential decay. Complex-conjugate roots produce damped oscillation. The oscillation is not added manually; it emerges from the feedback geometry.

ARMA autocovariance recursion

For ARMA((p,q)), autocovariances beyond the MA range satisfy the homogeneous AR recursion:

γ(h)=ϕ1γ(h−1)+⋯+ϕpγ(h−p),h>q.\gamma(h) = \phi_1\gamma(h-1)+\cdots+\phi_p\gamma(h-p), \qquad h>q.

The MA terms determine the initial autocovariances; the AR polynomial governs the long-lag tail.

| Model | Theoretical ACF | Theoretical PACF | |---|---|---| | AR((p)) | tails off | cuts off after (p) | | MA((q)) | cuts off after (q) | tails off | | ARMA((p,q)) | tails off | tails off |

Finite samples blur these ideal patterns, so the table should be treated as an identification guide rather than a deterministic rule.

Worked Example

Source-derived example: AR(2) autocorrelation

Consider

Xt=0.2Xt−1+0.08Xt−2+Wt.X_t=0.2X_{t-1}+0.08X_{t-2}+W_t.

For a stationary AR(2),

ρ(1)=ϕ11−ϕ2=0.20.92≈0.2174.\rho(1)=\frac{\phi_1}{1-\phi_2} = \frac{0.2}{0.92} \approx0.2174.

Then

ρ(2)=ϕ1ρ(1)+ϕ2≈0.2(0.2174)+0.08≈0.1235,\rho(2) = \phi_1\rho(1)+\phi_2 \approx 0.2(0.2174)+0.08 \approx0.1235,

and

ρ(3)=ϕ1ρ(2)+ϕ2ρ(1)≈0.0421.\rho(3) = \phi_1\rho(2)+\phi_2\rho(1) \approx0.0421.

The exact values depend on the coefficient convention, but the method is general: obtain the initial equations and propagate the recursion.

A simulation in R can compare sample and theoretical behavior:

set.seed(42)
x <- arima.sim(
  model = list(ar = c(0.2, 0.08)),
  n = 1000
)

acf(x, lag.max = 20)

Original explanatory example: vibration recovery after impact

Suppose a drive-unit vibration feature jumps after a mechanical impact and then oscillates while returning toward baseline.

An AR(2) model with complex-conjugate roots can represent the damped recovery. An MA term can represent a short sensor or preprocessing echo. The model should not be interpreted as a mechanical law without physical validation, but its impulse response can summarize how disturbances persist in the measured feature.

Interpreting the Results

Causality says that the modeled present can be generated from past innovations. Invertibility says that those innovations can be reconstructed from observed history. Neither property proves that the model is scientifically correct.

A root close to the unit circle implies slow decay and numerical sensitivity. Forecasts may remain influenced by the current state for many steps, and parameter uncertainty can become important.

A damped theoretical ACF supports AR-type feedback. A finite cutoff supports MA-type innovation overlap. Mixed tail behavior supports ARMA candidates, but fitted residuals and out-of-sample performance must decide whether the structure is adequate.

Real-World Applications

ARMA models are useful for stationary deviations around a controlled operating level, short-term demand fluctuations after trend removal, forecast errors from larger structural models, and residual dynamics in sensor or software telemetry.

In control and signal-processing language, the ARMA model is a rational linear filter. In econometrics, it often models transitory dynamics after integration or regression effects are addressed. In anomaly detection, standardized innovations can provide a model-relative surprise measure.

Common Misunderstandings

“The MA term averages observations”

It combines innovations, not observed values. A rolling mean is a data transformation; an MA((q)) is a stochastic model.

“Stationary and causal are identical”

For finite-order AR models under the usual formulation they are closely linked through root conditions, but causality specifically concerns a representation using present and past innovations.

“Any MA parameter is identifiable”

Noninvertible and invertible MA representations can share the same autocovariance function. The invertibility restriction chooses the standard representation.

“A root barely outside the unit circle is unproblematic”

It is technically admissible but can imply extreme persistence, unstable estimates, and poor finite-sample separation from a unit root.

“The ACF table identifies the true model”

The cutoff and tail-off patterns are asymptotic theoretical signatures. Sampling error, preprocessing, and mixed dynamics can obscure them.

Limitations and Failure Modes

ARMA models encode linear, time-invariant dependence. They can fail under structural breaks, changing variance, nonlinear thresholds, intermittent demand, long memory, or time-varying coefficients. Gaussian innovations may understate tail risk. A high-order model can mimic many patterns while remaining difficult to interpret and estimate.

Near-canceling AR and MA factors create weak identifiability. Even without exact common roots, nearly common factors can produce large parameter uncertainty while the net process appears simple.

Alternatives and Trade-Offs

ARIMA extends ARMA through differencing. Seasonal ARIMA adds repeated-lag structure. State-space models allow time-varying hidden level, trend, and seasonality. Exponential smoothing is often preferable when evolving components matter more than stationary lag dynamics. Vector autoregressions model interactions among multiple series. Nonlinear or regime-switching models are useful when dynamics change with state.

ARMA is most attractive when a transformed univariate series is stationary and its dependence is adequately linear and short-memory.

Connection to Broader Topics

The causal MA((\infty)) representation supplies forecast weights and forecast-error variances. Invertibility supplies innovation estimates used in likelihood and diagnostics. Polynomial roots return in ARIMA unit-root analysis and seasonal modeling. The same models can also be written in state-space form, connecting classical time series with Kalman filtering.

Connection to Portfolio or Learning

For predictive-maintenance features, ARMA can serve as a baseline model for stationary deviations after operating schedules and trend are removed. A portfolio implementation could expose:

  • fitted coefficients,
  • root locations,
  • impulse responses,
  • theoretical versus sample ACF,
  • and residual diagnostics.

Such a tool should label the model as a statistical description rather than a verified physical mechanism.

Key Takeaways

  1. AR terms propagate past state; MA terms distribute the effects of current and recent innovations.
  2. The polynomial form (\phi(B)X_t=\theta(B)W_t) makes model order, roots, filtering, and cancellation visible.
  3. Causality requires AR roots outside the unit circle and supports a stable past-innovation representation.
  4. Invertibility requires MA roots outside the unit circle and supports stable recovery of innovations from observations.
  5. MA models have finite theoretical ACF cutoffs, while AR models have recursively decaying or oscillating ACFs.
  6. Common or nearly common AR and MA factors make nominal model order misleading.
  7. Root conditions are necessary structural checks, not evidence that the fitted model captures the real system.

Review Questions

  1. What practical interpretation distinguishes an AR coefficient from an MA coefficient?
  2. Why does an AR(1) model have an infinite innovation representation?
  3. What ambiguity does invertibility resolve for an MA(1) process?
  4. How do complex AR roots appear in the autocorrelation function?
  5. Why should common AR and MA factors be removed?
  6. What would a root close to the unit circle imply for shock persistence and estimation?

Further-Learning Path

Autocovariance and ACF
→ reveal the dependence patterns to be explained.

ARMA filters and root conditions
→ provide a stable generative mechanism.

PACF and Yule–Walker estimation
→ connect theoretical structure to model identification and parameter estimates.

Likelihood, AIC, and residual diagnostics
→ compare candidates and test adequacy.

ARIMA and state-space models
→ extend the framework to nonstationary and component-based series.

Suggested Related Monograph Articles

  1. Understanding Dependence in Time Series — prerequisite. Defines the ACF patterns ARMA models explain.
  2. From Correlation Patterns to a Fitted Model — continuation. Covers PACF, prediction, and Yule–Walker estimation.
  3. Diagnosing and Selecting Time-Series Models — continuation. Evaluates whether a fitted ARMA model is adequate.
  4. Impulse Responses for Industrial Sensor Dynamics — practical application. Interprets shock propagation in condition-monitoring features.
  5. State-Space Representations of ARMA Models — deeper theory. Connects ARMA filters with recursive estimation.

References

Author not identified. (n.d.). Introduction to time series [Course notes, Modules 3-4]. Full citation details could not be confirmed.

Box, G. E. P., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015). Time series analysis: Forecasting and control (5th ed.). Wiley.

Brockwell, P. J., & Davis, R. A. (2016). Introduction to time series and forecasting (3rd ed.). Springer. https://doi.org/10.1007/978-3-319-29854-2

Hyndman, R. J., & Athanasopoulos, G. (2021). Forecasting: Principles and practice (3rd ed.). OTexts. https://otexts.com/fpp3/

R Core Team. (n.d.). R documentation. R Foundation for Statistical Computing. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/00Index.html

Shumway, R. H., & Stoffer, D. S. (2025). Time series analysis and its applications: With R examples (5th ed.). Springer. https://doi.org/10.1007/978-3-031-70584-7

Research and Verification Notes

  • Uploaded material used: M3L1 for ARMA definitions, polynomial factoring, simulation, causality, and invertibility; M3L2 and M4L1 for AR(2), AR((p)), and ARMA autocovariance recursions.
  • Authoritative verification: Root conditions and filter representations were checked against Brockwell and Davis (2016), Box et al. (2015), and Shumway and Stoffer (2025).
  • Terminology clarification: “Moving average” was explicitly distinguished from a rolling average. The plus-sign MA convention used here must be checked against any target software.
  • Original material: The vibration-recovery example and portfolio implementation suggestions are explanatory additions.
  • Code status: The R simulation was not executed in the current environment.
  • Potential review issue: Exact numerical examples should be recalculated if the source or website adopts a different AR/MA sign convention.
  • Missing metadata: Course-note authorship and institutional details remain unavailable.