Application to GNC — learned guidance laws

5.6.19 · Coding › Machine Learning (Aerospace Applications)

Exactly kya seekha ja raha hai?

Recipe 1 — Imitation Learning (Behavioral Cloning)

KYA: guidance ko plain supervised regression treat karo. Ek trusted optimizer se offline kaafi saari optimal trajectories generate karo; har timestep ek $(s_i,a_i^{\star })$ pair deta hai. Net ko $a^{\star }$ reproduce karne ke liye fit karo.

YE KYUN kaam karta hai: optimizer already sahi jawaab jaanta hai; humein sirf map $s\mapsto a^{\star }$ ka ek fast approximator chahiye. Hum ek solver-mein-lookup ko weights mein compress kar rahe hain.

KAISE — loss ko first principles se derive karo. Hum chahte hain ${\pi }_{\theta }(s)\approx a^{\star }(s)$. "Approximate" ka matlab hai state distribution $\rho$ ke upar expected squared error minimize karo jo hum actually dekhenge:

$J(\theta )={\mathbb{E}}_{s\sim \rho }[\Vert {\pi }_{\theta }(s)-a^{\star }(s){\Vert }^2].$

Squared error kyun? Gaussian noise model $a^{\star }={\pi }_{\theta }(s)+\epsilon ,\epsilon \sim \mathcal{N}(0,{\sigma }^{2}I)$ ke under, data ki log-likelihood maximize karne se exactly $-\frac{1}{2{\sigma }^2}\sum \Vert a_i^{\star }-{\pi }_{\theta }(s_i){\Vert }^2+\text{const}$ milta hai. Constants drop karne par, MLE hai hi least squares. $N$ samples ke saath empirical loss hai

$\hat{J}(\theta )=\frac{1}{N}{\sum }_{i=1}^N\Vert {\pi }_{\theta }(s_i)-a_i^{\star }{\Vert }^2.$

Gradient step: $\theta \leftarrow \theta -\eta {\nabla }_{\theta }\hat{J}$.

Recipe 2 — Reinforcement Learning

KYA: ek reward $r(s,a)$ define karo (jaise $-$miss-distance, $-$fuel), aur ${\pi }_{\theta }$ seekho jo expected discounted return maximize kare

$G={\mathbb{E}}_{{\pi }_{\theta }}[{\sum }_{t=0}^T{\gamma }^{t}r(s_t,a_t)],0<\gamma \le 1.$

KYUN: koi expert optimizer ki zaroorat nahi — useful jab optimal law unknown ho ya dynamics messy/uncertain ho.

KAISE — policy gradient, derive kiya. Hum ${\nabla }_{\theta }J$ chahte hain jahan $J={\mathbb{E}}_{\tau \sim p_{\theta }}[R(\tau )]$, $\tau$ ek trajectory hai, $R$ uska return hai.

${\nabla }_{\theta }J={\nabla }_{\theta }\int p_{\theta }(\tau )R(\tau )d\tau =\int R(\tau ){\nabla }_{\theta }{p}_{\theta }(\tau )d\tau .$

Log-derivative trick ${\nabla }_{\theta }{p}_{\theta }=p_{\theta }{\nabla }_{\theta }\log p_{\theta }$ use karo:

${\nabla }_{\theta }J=\int p_{\theta }(\tau )R(\tau ){\nabla }_{\theta }\log p_{\theta }(\tau )d\tau ={\mathbb{E}}_{\tau }[R(\tau ){\nabla }_{\theta }\log p_{\theta }(\tau )].$

Kyunki $\log p_{\theta }(\tau )=\log p(s_0)+{\sum }_t[\log P(s_{t+1}|s_t,a_t)+\log {\pi }_{\theta }(a_t|s_t)]$, sirf policy term $\theta$ pe depend karta hai (dynamics nahi!), isliye

$\boxed{{\nabla }_{\theta }J=\mathbb{E}[\sum _t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)R(\tau )]}$

Yahi REINFORCE hai. Dynamics ka drop out hona yahi reason hai kyun RL tab bhi kaam karta hai jab plant model unknown ho.

Classical law ke against sanity check

Proportional Navigation acceleration command karta hai $a_c=N{V}_c\dot{\lambda }$ (navigation gain $N$, closing speed $V_c$, LOS rate $\dot{\lambda }$). Ek well-trained net ko nominal engagements pe PN-jaisa behavior recover karna chahiye — yeh ek zabardast Forecast-then-Verify test hai: predict karo "net ka command $\dot{\lambda }$ ke saath linearly scale karta hai," phir net output vs $\dot{\lambda }$ plot karo aur slope $\approx N{V}_c$ check karo.

Worked examples

Active-recall flashcards

Connections

• Proportional Navigation — classical guidance law jise net aksar mimic karta hai.
• Reinforcement Learning · Policy Gradient Methods · REINFORCE
• Behavioral Cloning · DAgger · Distribution Shift
• Neural Networks as Function Approximators
• Optimal Control / Model Predictive Control — woh "expert" jise clone kiya ja raha hai.
• GNC — Navigation · GNC — Control — woh loop jise yeh guidance feed karta hai.
• Feature Engineering · Maximum Likelihood Estimation

Concept Map