Discount factor and returns — Hinglish notes

5.1.6 · AI-ML › Reinforcement Learning Foundations

HUM KYA define kar rahe hain?

$γ = 0$ → myopic agent: sirf immediate reward $R_{t + 1}$ matter karta hai.
$γ \to 1$ → far-sighted agent: door ke rewards bhi almost utna hi count karte hain jitna paas ke.

HUM DISCOUNT KARTE HI KYUN HAIN?

Teen independent reasons, sab ek hi direction mein point karte hain:

Mathematical convergence. Un tasks mein jo kabhi khatam nahi hote (continuing tasks), $\sum R$ infinite ho sakta hai. $γ < 1$ se discounting sum ko finite rakhti hai (neeche prove kiya gaya hai).
Future ke baare mein uncertainty. $γ$ ko ek "survival probability" ki tarah socho — har step par ek chance hai ki process ruk jaaye. Woh reward jo tumhe shayad milega hi nahi, usse kam count karna chahiye.
Preference / economics. Interest rates ki tarah: paisa (reward) abhi reinvest ho sakta hai, isliye genuinely worth zyada hai baad mein same amount se.

KAISE: bounded-return result scratch se derive karo

Claim: Agar har reward bounded hai, $∣ R ∣ \leq R_{m a x}$ , aur $0 \leq γ < 1$ , toh $G_{t}$ finite hai.

Step 1 — Har term ko bound karo. $∣ G_{t} ∣ = \sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1} \leq \sum_{k = 0}^{\infty} γ^{k} ∣ R_{t + k + 1} ∣ \leq \sum_{k = 0}^{\infty} γ^{k} R_{m a x}$ Yeh step kyun? Triangle inequality absolute value ko andar le jaati hai, phir hum har reward ko uske worst case $R_{m a x}$ se replace kar dete hain.

Step 2 — Geometric series sum karo. Maano $S = \sum_{k = 0}^{\infty} γ^{k} = 1 + γ + γ^{2} + \dots$ . $γ$ se multiply karo: $γ S = γ + γ^{2} + \dots ⟹ S - γ S = 1 ⟹ S = \frac{1}{1 - γ}$ Yeh step kyun? Ek shifted copy subtract karne se beech ke saare terms telescope ho jaate hain — classic geometric-series trick, valid hai kyunki $γ < 1$ se $γ^{k} \to 0$ ho jaata hai.

Step 3 — Combine karo. $∣ G_{t} ∣ \leq \frac{R _{m a x}}{1 - γ}$ Toh return finite hai. Dhyan do: jaise $γ \to 1$ , yeh bound blow up ho jaata hai — long horizons mein care chahiye.

Effective horizon ( $γ$ ka 80/20)

Weights $γ^{k}$ geometrically shrink hote hain. Do related "horizon" numbers saamne aate hain — inhe confuse mat karo:

1. $\frac{1}{1 - γ}$ scale. Yeh $\sum_{k = 0}^{\infty} γ^{k}$ (total weight) ki value hai aur woh characteristic scale hai jiske upar weights $1/ e$ -ish tak decay hote hain. Yahi number usually rough planning horizon ke roop mein quote kiya jaata hai.

2. True mean step jisko tum dekh rahe ho. Weights ko ek probability distribution $P (k) = (1 - γ) γ^{k}$ mein normalize karo (inki sum 1 hai). Iska mean index hai: $E [k] = \sum_{k = 0}^{\infty} k (1 - γ) γ^{k} = \frac{γ}{1 - γ} .$ Yeh step kyun? $\sum_{k} k γ^{k} = \frac{γ}{( 1 - γ ) ^{2}}$ use karke aur normalizer $(1 - γ)$ se multiply karne par $\frac{γ}{1 - γ}$ milta hai — dhyan do numerator mein extra $γ$ hai, isliye mean $\frac{1}{1 - γ}$ se ek kam hai.

$H_{scale} = \frac{1}{1 - γ} (weight scale), E [k] = \frac{γ}{1 - γ} (mean step)$

$γ$	$\frac{1}{1 - γ}$ (scale)	$\frac{γ}{1 - γ}$ (mean step)	roughly matlab
$0.9$	$10$	$9$	"agle ~10 steps"
$0.99$	$100$	$99$	"agle ~100 steps"
$0.5$	$2$	$1$	"bahut short-sighted"

Worked examples

Common mistakes (Steel-manned)

Flashcards

RL mein return

G_{t}

kya hota hai?

Step

t

se total discounted future reward:

G_{t} = \sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1}

G_{t}

kaun se reward index se start hota hai, aur kyun?

R_{t + 1}

se — reward time

t

par action ke baad milta hai.

Discount factor

γ

kya hai aur iska range kya hai?

[0, 1]

mein ek weight jo scale karta hai ki future rewards kitna count karein; har future step ek extra

γ

se multiply hota hai.

γ = 0

kya produce karta hai?

Ek myopic agent jo sirf immediate reward

R_{t + 1}

ki care karta hai.

γ \to 1

kya produce karta hai?

Ek far-sighted agent jo distant rewards ko almost utna hi weight karta hai jitna near ones ko.

Return ki recursive form kya hai?

G_{t} = R_{t + 1} + γ G_{t + 1}

Episodic task mein return recursion ko kaun si terminal condition chahiye?

Final reward ke baad

G_{T} = 0

(termination ke baad kuch nahi aata).

Bounded reward ke saath

∣ G_{t} ∣

ka upper bound kya hai?

\frac{R _{m a x}}{1 - γ}

Geometric series

\sum_{k = 0}^{\infty} γ^{k}

ka sum kya hai?

\frac{1}{1 - γ}

jab

∣ γ ∣ < 1

γ = 0.9

ke saath hamesha constant reward

+ 1

ka return kya hai?

1/ (1 - 0.9) = 10

Kisi diye gaye

γ

ke liye decay-scale horizon kya hai?

\frac{1}{1 - γ}

(total weight / characteristic scale).

Normalized discount weights ke under mean step index kya hai?

\frac{γ}{1 - γ}

(decay scale se ek kam;

γ \to 1

par agree karte hain).

Continuing tasks mein discount kyun karte hain?

Infinite reward sum ko finite (convergent) rakhne ke liye.

Kya bada

γ

hamesha return badhata hai?

Nahi — yeh future ko amplify karta hai, isliye net-negative future rewards ke saath return ghatta hai.

Recall Feynman: 12-saal ke bachche ko explain karo

Socho tum candies collect kar rahe ho. Jo candy tumhe abhi milti hai usse tum pakka enjoy kar sakte ho. Jo candy koi agle hafte dene ka promise karta hai woh thodi kam worth hai — shayad tum shift kar jao, shayad woh bhool jaaye. Toh tum decide karte ho "agle hafte ki candy aaj ki candy ka 90% worth hai." Uske agli hafte, 90% ka 90%, aur aise chalte jaata hai. Return bas yahi hai — apni saari future candies ko add karo, door waali ko shrink karne ke baad. $γ$ (jaise $0.9$ ) tumhara "kitna future par trust karta hoon" number hai.

Connections

Markov Decision Process — returns MDP trajectories par define hote hain.
Value functions and Bellman equations — $v (s) = E [G_{t} ∣ S_{t} = s]$ directly $G_{t} = R_{t + 1} + γ G_{t + 1}$ par build hota hai.
Episodic vs continuing tasks — decide karta hai ki $γ = 1$ allowed hai ya nahi.
Reward hypothesis — $R_{t}$ 's kya represent karte hain.
Geometric series — woh math jo convergence aur horizon ko power karta hai.
Temporal Difference learning — return ko bootstrap karne ke liye recursive form use karta hai.