The 80/20 core: everything below reduces to one move — write the unknown dynamics as (something) × (unknown parameters), then solve least squares. Master that and you have 80% of system ID.
Let's do the simplest, most-used case: a linear discrete-time model
xk+1=Axk+Buk.
Step 1 — Stack unknowns.Why?A,B are what we want; group them:
Θ=[AB],zk=[xkuk].
Then the model is simply xk+1=Θzk. This is the key trick: dynamics = matrix × features.
Step 2 — Collect data into matrices.Why? One equation per time step; stack them so we can solve all at once.
X+=[x1x2⋯xN],Z=[z0z1⋯zN−1].
The model over all data reads X+≈ΘZ.
Step 3 — Define the cost.Why? Data is noisy, so we can't hit it exactly; minimize squared mismatch (Gaussian-noise MLE):
J(Θ)=∥X+−ΘZ∥F2=tr[(X+−ΘZ)(X+−ΘZ)⊤].
Step 4 — Set derivative to zero.Why? Minimum of a convex quadratic.
∂Θ∂J=−2(X+−ΘZ)Z⊤=0.
Rearrange: X+Z⊤=ΘZZ⊤.
That Z† is the whole subject in one symbol: it's the least-squares "divide by the data."
Real dynamics aren't linear. But we can still keep the linear-in-parameters form:
x˙=Θϕ(x,u),ϕ=[1,x,u,x2,xu,sinx,…]⊤.
Build a big matrix Φ of candidate functions evaluated on data, then solve the same least-squares — optionally with sparsity (drop tiny coefficients) so the recovered model has few, interpretable terms. That's SINDy (Sparse Identification of Nonlinear Dynamics).
Why stack? So one matrix equation covers both rows:
X+=[21.8],Z=[1120].
Why pseudoinverse?Z is square & invertible here, so it's exact.
\Theta = X_+Z^{-1}=[0.9\;\;1.1].$$
So $a=0.9,\ b=1.1$. **Check:** row 0: $0.9(1)+1.1(1)=2.0$ ✓; row 1: $0.9(2)+1.1(0)=1.8$ ✓.
## Worked Example 2 — why excitation matters
Say every $u_k=0$ during the flight. Then the $u$-row of $Z$ is all zeros, so $ZZ^\top$ is singular → $b$ is **unidentifiable**. *Why this step matters:* you literally never poked the input channel, so no data can tell you $b$. **Fix:** inject a rich signal (chirp / multisine / PRBS) — *persistent excitation*.
## Worked Example 3 — nonlinear via a library
Pendulum-like data suspected: $\dot\theta_2 = -c\sin\theta_1$. Use $\phi=[1,\ \theta_1,\ \sin\theta_1]$. Least squares returns coefficients $\approx[0,\ 0,\ -c]$; sparsity zeroes the first two. **Why the library form?** It keeps the fit *linear in unknowns* even though the model is nonlinear in state — so the same $\Theta=\dot X\,\Phi^\dagger$ formula works.
---
> [!recall]- Feynman: explain to a 12-year-old
> Imagine you have a toy car and you don't know the rules of how it drives. You film it lots of times: "when I push the joystick *this* much, it ends up *there*." Then you make a little math machine that guesses the next spot from the current spot and your push. You tweak the machine until its guesses match your films as closely as possible. Now you can predict where the car will go *before* pushing — that's system identification! The one rule: you have to drive the car in *lots of different ways*, or you'll never learn what the buttons you never pressed actually do.
---
## Common mistakes (steel-manned)
> [!mistake] "More candidate features → better model."
> **Feels right:** more functions = more flexibility. **Reality:** you overfit noise and $ZZ^\top$ becomes ill-conditioned. **Fix:** sparsity/regularization ($L_1$, or ridge $\Theta = X_{+}Z^\top(ZZ^\top+\lambda I)^{-1}$) and validate on *held-out* data.
> [!mistake] "Low training error = good dynamics model."
> **Feels right:** it matched the data. **Fix:** always test **multi-step rollout** — feed predictions back in. Small one-step errors compound over a trajectory; a good ID model must stay stable when *simulated forward*.
> [!mistake] "I can differentiate raw sensor data to get $\dot{x}$."
> **Feels right:** derivative is just a finite difference. **Fix:** finite differences amplify high-frequency noise; smooth first (Savitzky–Golay, TV-regularized differentiation).
---
## Forecast-then-Verify
> [!example] Predict before you compute
> **Q:** In Example 1, if I had a *third* data row that's an exact linear combo of the first two, does $\Theta$ change? **Forecast:** No — it adds no new information; least squares already fits the span. **Verify:** $ZZ^\top$ rank unchanged, solution identical. ✓
---
## #flashcards/coding
What is the goal of system identification? ::: Recover a predictive dynamics model $x_{k+1}=f_\theta(x_k,u_k)$ (or $\dot x=f_\theta$) from measured state/input data by minimizing prediction error.
State the least-squares solution for a linear model $X_+ = \Theta Z$. ::: $\Theta = X_+Z^\top(ZZ^\top)^{-1} = X_+Z^\dagger$ (Moore–Penrose pseudoinverse).
Why can $ZZ^\top$ become singular, and what fixes it? ::: When inputs/states don't excite all directions (poor excitation). Fix: use rich inputs (persistent excitation: chirp, PRBS, multisine).
What key property lets nonlinear SINDy still use linear least squares? ::: The model is made **linear in the parameters**: $\dot x = \Theta\,\phi(x,u)$ with a fixed feature/candidate library $\phi$.
Why prefer discrete-time ID over continuous-time in practice? ::: Continuous needs $\dot x$ from noisy data (numerical differentiation amplifies noise); discrete-time avoids differentiation.
Why is one-step training error a misleading metric? ::: Errors compound in multi-step rollout; must validate simulated trajectories, not just next-step predictions.
What does dividing by $ZZ^\top$ physically represent? ::: De-weighting by input/state excitation "energy" (a covariance); directions barely explored are down-weighted.
State the ridge-regularized dynamics fit. ::: $\Theta = X_{+}Z^\top(ZZ^\top+\lambda I)^{-1}$ with $\lambda>0$ (also $L_1$/thresholding for sparsity, as in SINDy).
---
## Connections
- [[Least Squares and the Pseudoinverse]]
- [[Kalman Filter — State Estimation]]
- [[Model Predictive Control (MPC)]]
- [[SINDy — Sparse Identification of Nonlinear Dynamics]]
- [[Persistent Excitation and Experiment Design]]
- [[Regularization — Ridge and Lasso]]
- [[Digital Twins in Aerospace]]
- [[Numerical Differentiation of Noisy Signals]]
## 🖼️ Concept Map
```mermaid
flowchart TD
HIDDEN[Hidden EOM xdot = f x u] -->|recovered by| SYSID[System Identification]
DATA[Flight-test data x u] -->|feeds| SYSID
SYSID -->|minimizes| PRED[Prediction error]
SYSID -->|two flavours| DISC[Discrete-time model]
SYSID -->|two flavours| CONT[Continuous-time model]
CONT -->|needs| DERIV[Estimate xdot from noisy x]
DERIV -->|amplifies| NOISE[Noise problem]
DISC -->|core move| LLS[Linear least squares]
LLS -->|key trick| FEAT[Dynamics = matrix x features]
FEAT -->|stack unknowns| THETA[Theta = A B]
SYSID -->|model feeds| APPS[MPC / Kalman / Digital twin]
```
## 🔊 Hinglish (regional understanding)
> [!intuition]- Hinglish mein samjho
> System identification ka matlab hai: humein aircraft ki asli equation of motion nahi pata (unknown aerodynamic coefficients, actuator lag waghaira), toh hum **data se seekhte hain** ki system agle step mein kaha jaayega. Basically hum flight ka data lete hain — har time pe state $x_k$ aur control input $u_k$ — aur ek model fit karte hain jo bole $x_{k+1}=f_\theta(x_k,u_k)$. Parameters $\theta$ ko aise tune karte hain ki model ka prediction real data se match kare.
>
> Sabse important insight (80/20): **dynamics ko likho as matrix $\times$ features**. Linear case mein $x_{k+1}=[A\;B]\,[x_k;u_k]$, aur saara data stack karke ek hi least-squares solve: $\Theta = X_+ Z^\dagger$. Yeh pseudoinverse hi poora khel hai — yeh "data se divide karna" hai. Non-linear dynamics ke liye bhi trick same rehti hai: features ki ek library banao ($x^2$, $\sin x$, $xu$...) aur wahi least squares chalao — isko SINDy kehte hain.
>
> Do cheezein yaad rakho. Pehli: **persistent excitation** — agar tumne input channel ko poke hi nahi kiya (u=0 hamesha), toh $b$ kabhi nahi seekh paoge, kyunki $ZZ^\top$ singular ho jayega. Isliye chirp/PRBS jaise rich signals use karo. Doosri: sirf one-step error dekhke khush mat ho — **multi-step rollout** pe test karo, kyunki chhoti errors trajectory pe compound hoti hain aur model unstable dikh sakta hai.
>
> Yeh topic aerospace mein critical hai: identified model seedha MPC controller, Kalman filter, ya digital twin mein feed hota hai. Matlab ek accha model se aap flight simulate, predict aur control — sab kuch kar sakte ho bina exact physics likhe.
![[audio/5.6.16-System-identification-—-learning-dynamics-from-data.mp3]]
Test yourself — Machine Learning (Aerospace Applications)
