Application to GNC — learned guidance laws

What exactly is being learned?

Recipe 1 — Imitation Learning (Behavioral Cloning)

WHAT: treat guidance as plain supervised regression. Generate many optimal trajectories offline with a trusted optimizer; each timestep gives a $(s_i, a_i^\star)$ pair. Fit a net to reproduce $a^\star$.

WHY it works: the optimizer already knows the right answer; we only need a fast approximator of the map $s\mapsto a^\star$. We are compressing a lookup-into-solver into weights.

HOW — derive the loss from first principles. We want $\pi_\theta(s)\approx a^\star(s)$. "Approximate" means minimize expected squared error over the state distribution $\rho$ we'll actually see:

$J(\theta)=\mathbb E_{s\sim\rho}\big[\,\lVert \pi_\theta(s)-a^\star(s)\rVert^2\,\big].$

Why squared error? Under a Gaussian noise model $a^\star = \pi_\theta(s)+\varepsilon,\ \varepsilon\sim\mathcal N(0,\sigma^2 I)$, maximizing the log-likelihood of the data gives exactly $-\tfrac1{2\sigma^2}\sum\lVert a_i^\star-\pi_\theta(s_i)\rVert^2 + \text{const}$. Dropping constants, MLE is least squares. With $N$ samples the empirical loss is

$\hat J(\theta)=\frac1N\sum_{i=1}^N \lVert \pi_\theta(s_i)-a_i^\star\rVert^2 .$

Gradient step: $\theta \leftarrow \theta - \eta\,\nabla_\theta \hat J$.

Recipe 2 — Reinforcement Learning

WHAT: define a reward $r(s,a)$ (e.g. $-$miss-distance, $-$fuel), and learn $\pi_\theta$ that maximizes expected discounted return

$G=\mathbb E_{\pi_\theta}\!\Big[\sum_{t=0}^{T}\gamma^t r(s_t,a_t)\Big],\qquad 0<\gamma\le1.$

WHY: no expert optimizer required — useful when the optimal law is unknown or the dynamics are messy/uncertain.

HOW — the policy gradient, derived. We want $\nabla_\theta J$ where $J=\mathbb E_{\tau\sim p_\theta}[R(\tau)]$, $\tau$ a trajectory, $R$ its return.

$\nabla_\theta J=\nabla_\theta\!\int p_\theta(\tau)R(\tau)\,d\tau=\int R(\tau)\,\nabla_\theta p_\theta(\tau)\,d\tau.$

Use the log-derivative trick $\nabla_\theta p_\theta = p_\theta\,\nabla_\theta\log p_\theta$:

$\nabla_\theta J=\int p_\theta(\tau)R(\tau)\nabla_\theta\log p_\theta(\tau)\,d\tau=\mathbb E_\tau\big[R(\tau)\,\nabla_\theta\log p_\theta(\tau)\big].$

Since $\log p_\theta(\tau)=\log p(s_0)+\sum_t[\log P(s_{t+1}|s_t,a_t)+\log\pi_\theta(a_t|s_t)]$, only the policy term depends on $\theta$ (dynamics don't!), so

$\boxed{\;\nabla_\theta J=\mathbb E\Big[\sum_t \nabla_\theta\log\pi_\theta(a_t|s_t)\;R(\tau)\Big]\;}$

This is REINFORCE. The dynamics dropping out is why RL works even when the plant model is unknown.

Sanity check against a classical law

Proportional Navigation commands acceleration $a_c=N V_c \dot\lambda$ (navigation gain $N$, closing speed $V_c$, LOS rate $\dot\lambda$). A well-trained net should recover PN-like behavior on nominal engagements — a great Forecast-then-Verify test: predict "the net's command scales linearly with $\dot\lambda$," then plot net output vs $\dot\lambda$ and check the slope $\approx N V_c$.

Worked examples

Active-recall flashcards

Connections

• Proportional Navigation — classical guidance law the net often mimics.
• Reinforcement Learning · Policy Gradient Methods · REINFORCE
• Behavioral Cloning · DAgger · Distribution Shift
• Neural Networks as Function Approximators
• Optimal Control / Model Predictive Control — the "expert" being cloned.
• GNC — Navigation · GNC — Control — the loop this guidance feeds.
• Feature Engineering · Maximum Likelihood Estimation

Concept Map

Hinglish (regional understanding)