Question bank — Multi-agent reinforcement learning

True. With one agent the joint action is just $a^1$ and the transition/reward lose all dependence on other agents, recovering the MDP tuple exactly.

False. NE only means no one gains by unilateral deviation; a jointly better outcome may exist (e.g. Prisoner's Dilemma), but no agent can reach it alone — see Game Theory & Nash Equilibrium.

False. Shared reward aligns goals, but each agent still sees opponents' changing policies inside ${\tilde{P}}^i$, so the environment still moves under its feet.

True. Coordination games have several (e.g. "both pick A" and "both pick B"), which is exactly why decentralized agents may fail to agree on which one to converge to.

True by construction of the rewards — one agent's gain is exactly the other's loss at every step, so the discounted returns are negatives of each other.

False. The proof assumes a stationary transition; with learning opponents ${\tilde{P}}^i$ is non-stationary, so the guarantee is void — this is the Independent-Q-Learning (IQL) trap.

True. If every agent's action is a named input, nothing about the value is "hidden" behind a changing opponent policy, so the critic's target is stationary.

True. If $\partial Q_{tot}/\partial Q^i\ge 0$ for all $i$, raising any local $Q^i$ can only raise $Q_{tot}$, so local argmaxes assemble the global argmax — that's the IGM property.

False. The opponent copy is also improving, so the effective opponent policy keeps changing — self-play is a source of non-stationarity, not a cure for it.

Old transitions were generated by stale opponent policies; they describe a game that no longer exists, so they teach the wrong lesson. Replay + non-stationarity is the classic IQL failure.

Wrong index. The chain rule links my params to my action, so it must be ${\nabla }_{a^i}{Q}^i$ evaluated at $a^i={\mu }^i(o^i)$ — you only push through your own action.

No pure strategy is NE here: any fixed choice is beaten by a best response (Paper beats Rock), so the opponent has a profitable deviation. The unique NE is uniform mixing $(1/3,1/3,1/3)$.

A product is not monotone when a factor is negative — raising a positive $Q^i$ can lower $Q_{tot}$ if the other factor is negative, breaking IGM. That's why QMIX forces nonnegative mixing weights.

The whole point of CTDE is decentralized execution: at test time each agent uses only ${\pi }^i(a^i\mid o^i)$ from its local observation. The centralized critic is a training-only scaffold.

NE is a best response to the others, not a global optimum. Change the opponents and your NE policy may become terrible; it is a fixed point, not an absolute maximum.

There is no single objective — each agent has its own value $V^i$ with possibly conflicting incentives, so "optimal" is replaced by NE, a mutual best-response fixed point.

The critic only needs to give an accurate, stationary learning signal during training; the actor must run in the real world where agents can't see each other, so it may depend only on $o^i$.

Replay reuses many past transitions; those were sampled under old opponent policies, so as opponents drift the buffer fills with data describing environments that no longer exist.

The learning dynamics and even what "success" means depend entirely on whether interests align — cooperation seeks a shared max, zero-sum seeks a minimax saddle, and mixed needs equilibrium reasoning.

Monotonicity is exactly enough to make local greedy action selection agree with global greedy selection (IGM), which is what decentralized execution requires — a non-monotone mixer could make agents disagree.

With $\mathbf{a}$ as an explicit input, the value no longer depends on an unknown, changing opponent policy — the opponents' actions are named arguments, so the mapping is fixed. See Actor-Critic Methods.

Discounting vanishes, returns become undiscounted sums that may diverge in infinite-horizon games; that's why the definition restricts $\gamma \in [0,1)$ to keep values finite.

No — shared reward gives a common $Q_{tot}$, but you still need a mixing structure (e.g. monotone) so that per-agent argmaxes reconstruct the joint argmax; alignment of goals doesn't guarantee IGM.

Yes it can still be an NE: neither agent can improve by switching alone (switching to B alone breaks coordination and pays less), so "both A" is a locally stable fixed point — a suboptimal one.

Yes, when a saddle point exists in pure strategies (a game with a "value" achieved by pure minimax choices). Mixing is only forced when no pure saddle exists, as in Rock–Paper–Scissors.

It grows as $k^N$ (exponential in $N$), so a critic taking the full joint action becomes intractable for many agents — motivating factorization methods like QMIX and value decomposition.