Agent Conversation: google/gemini-3.1-pro-preview
Model: google/gemini-3.1-pro-preview
Total steps: 7
Messages: 15
UserStep 0
Task: rl-value-atari
Online RL: Value-Based Methods for Visual Control (Atari)
Objective
Design and implement a value-based RL algorithm for visual/Atari environments using CNN feature extraction. Your code goes in custom_value_atari.py. Three reference implementations (DQN, DoubleDQN, C51) are provided as read-only.
Background
Atari games require learning from raw pixel observations (84x84 grayscale, 4 stacked frames). Value-based methods must learn effective visual representations alongside Q-value estimation. Key challenges include high-dimensional observations, sparse rewards, and memory-efficient experience replay. Different approaches address these through distributional value functions, frame stacking, or architecture innovations.
Constraints
- Network architecture dimensions are FIXED and cannot be modified
- Total parameter count is enforced at runtime
- Focus on algorithmic innovation: new loss functions, update rules, exploration strategies, etc.
- Do NOT simply copy a reference implementation with minor changes
Evaluation
Trained and evaluated on Breakout, Pong, BeamRider. Additional held-out environments (not shown during intermediate testing) are used to assess generalization. Metric: mean episodic return over 10 evaluation episodes (higher is better).
cleanrl/cleanrl/custom_value_atari.py [EDITABLE — lines 186–249 only]
1: # Custom value-based RL algorithm for Atari -- MLS-Bench
2: #
3: # EDITABLE section: QNetwork head and ValueAlgorithm classes.
4: # FIXED sections: everything else (config, env, buffer, encoder, eval, training loop).
5: import os
6: import random
7: import time
8: from dataclasses import dataclass
9:
10: import gymnasium as gym
11: import numpy as np
12: import torch
13: import torch.nn as nn
14: import torch.nn.functional as F
15: import torch.optim as optim
16: import tyro
17:
18: from cleanrl_utils.atari_wrappers import (
19: ClipRewardEnv,
20: EpisodicLifeEnv,
21: FireResetEnv,
22: MaxAndSkipEnv,
23: NoopResetEnv,
24: )
25: from cleanrl_utils.buffers import ReplayBuffer
26:
27:
28: # =====================================================================
29: # FIXED: Configuration
30: # =====================================================================
31: @dataclass
32: class Args:
33: exp_name: str = os.path.basename(__file__)[: -len(".py")]
34: """the name of this experiment"""
35: seed: int = 1
36: """seed of the experiment"""
37: torch_deterministic: bool = True
38: """if toggled, `torch.backends.cudnn.deterministic=False`"""
39: cuda: bool = True
40: """if toggled, cuda will be enabled by default"""
41:
42: # Algorithm specific arguments
43: env_id: str = "BreakoutNoFrameskip-v4"
44: """the id of the environment""AssistantStep 1edit
Tool Input
1{
2 "content": "import math\n\nclass QNetwork(nn.Module):\n \"\"\"Dueling Implicit Quantile Network (IQN) head.\"\"\"\n def __init__(self, envs):\n super().__init__()\n self.encoder = NatureDQNEncoder()\n \n self.n_actions = envs.single_action_space.n\n self.n_cos = 64\n self.phi = nn.Linear(self.n_cos, ENCODER_FEATURE_DIM)\n \n # Dueling streams\n self.fc_adv = nn.Linear(ENCODER_FEATURE_DIM, 512)\n self.fc_val = nn.Linear(ENCODER_FEATURE_DIM, 512)\n \n self.adv_out = nn.Linear(512, self.n_actions)\n self.val_out = nn.Linear(512, 1)\n\n def forward(self, x, num_quantiles=32):\n quantiles, _ = self.get_quantiles(x, num_quantiles)\n return quantiles.mean(dim=1)\n\n def get_quantiles(self, x, num_quantiles=32):\n batch_size = x.shape[0]\n features = self.encoder(x)\n \n tau = torch.rand(batch_size, num_quantiles, 1, device=x.device)\n \n pi_mtx = math.pi * torch.arange(1, self.n_cos + 1, device=x.device, dtype=torch.float32)\n pi_mtx = pi_mtx.view(1, 1, self.n_cos)\n cos_tau = torch.cos(tau * pi_mtx)\n \n phi_tau = F.relu(self.phi(cos_tau))\n \n features = features.unsqueeze(1)\n merged = features * phi_tau\n \n adv = F.relu(self.fc_adv(merged))\n val = F.relu(self.fc_val(merged))\n \n adv = self.adv_out(adv)\n val = self.val_out(val)\n \n quantiles = val + adv - adv.mean(dim=2, keepdim=True)\n return quantiles, tau\n\n\nclass ValueAlgorithm:\n \"\"\"Dueling IQN - Implicit Quantile Networks with Dueling Architecture.\"\"\"\n def __init__(self, envs, device, args):\n self.device = device\n self.gamma = args.gamma\n self.tau_update = args.tau\n self.target_network_frequency = args.target_network_frequency\n \n self.q_network = QNetwork(envs).to(device)\n self.target_network = QNetwork(envs).to(device)\n self.target_network.load_state_dict(self.q_network.state_dict())\n self.optimizer = optim.Adam(self.q_network.parameters(), lr=args.learning_rate, eps=0.01 / args.batch_size)\n\n def select_action(self, obs, epsilon):\n with torch.no_grad():\n q_values = self.q_network(torch.Tensor(obs).to(self.device), num_quantiles=32)\n return torch.argmax(q_values, dim=1).cpu().numpy()\n\n def update(self, batch, global_step):\n with torch.no_grad():\n next_q = self.q_network(batch.next_observations, num_quantiles=32)\n next_actions = next_q.argmax(dim=1)\n \n next_quantiles_target, _ = self.target_network.get_quantiles(batch.next_observations, num_quantiles=64)\n next_actions_expanded = next_actions.unsqueeze(1).unsqueeze(2).expand(-1, 64, -1)\n next_quantiles_best = next_quantiles_target.gather(2, next_actions_expanded).squeeze(2)\n \n target_quantiles = batch.rewards + self.gamma * next_quantiles_best * (1 - batch.dones)\n \n current_quantiles_all, tau = self.q_network.get_quantiles(batch.observations, num_quantiles=64)\n actions_expanded = batch.actions.unsqueeze(1).expand(-1, 64, -1)\n current_quantiles = current_quantiles_all.gather(2, actions_expanded).squeeze(2)\n \n td_errors = target_quantiles.unsqueeze(1) - current_quantiles.unsqueeze(2)\n abs_errors = torch.abs(td_errors)\n huber_loss = torch.where(abs_errors <= 1.0, 0.5 * abs_errors.pow(2), 1.0 * (abs_errors - 0.5))\n \n tau_expanded = tau.expand(-1, -1, 64)\n indicator = (td_errors.detach() < 0).float()\n quantile_loss = torch.abs(tau_expanded - indicator) * huber_loss\n loss = quantile_loss.mean(dim=2).sum(dim=1).mean()\n \n self.optimizer.zero_grad()\n loss.backward()\n self.optimizer.step()\n \n if global_step % self.target_network_frequency == 0:\n for target_param, q_param in zip(self.target_network.parameters(), self.q_network.parameters()):\n target_param.data.copy_(\n self.tau_update * q_param.data + (1.0 - self.tau_update) * target_param.data\n )\n \n return {\"td_loss\": loss.item(), \"q_values\": next_q.mean().item()}",
3 "op": "replace",
4 "end_line": 249,
5 "start_line": 186,
6 "filename": "cleanrl/cleanrl/custom_value_atari.py"
7}Tool ResultStep 1
OK: Replaced lines 186..249 with 101 line(s) in cleanrl/cleanrl/custom_value_atari.py. Editable range: 186–286. [Current file: cleanrl/cleanrl/custom_value_atari.py | editable: 186–286 | total: 396 lines] 183| # ===================================================================== 184| # EDITABLE: QNetwork head and ValueAlgorithm 185| # ===================================================================== 186 import math 187 188 class QNetwork(nn.Module): 189 """Dueling Implicit Quantile Network (IQN) head.""" 190 def __init__(self, envs): 191 super().__init__() 192 self.encoder = NatureDQNEncoder() 193 194 self.n_actions = envs.single_action_space.n 195 self.n_cos = 64 196 self.phi = nn.Linear(self.n_cos, ENCODER_FEATURE_DIM) 197 198 # Dueling streams 199 self.fc_adv = nn.Linear(ENCODER_FEATURE_DIM, 512) 200 self.fc_val = nn.L
AssistantStep 2test
Tool ResultStep 2
[Test #1] (2 tests remaining; call submit(n=1) to submit this result)
## Seed 42
### breakout-v4 (scripts/breakout.sh)
/usr/bin/bash: warning: setlocale: LC_ALL: cannot change locale (zh_CN.UTF-8)
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.
Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.
See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.
/usr/local/lib/python3.8/dist-packages/pygame/pkgdata.py:25: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
from pkg_resources import resource_stream, resource_exists
Traceback (most recent call last):
File "cleanrl/custom_value_atari.py", line 334, in <module>
_check_param_bud
AssistantStep 3edit
Tool Input
1{
2 "start_line": 186,
3 "filename": "cleanrl/cleanrl/custom_value_atari.py",
4 "end_line": 249,
5 "op": "replace",
6 "content": "import math\n\nclass QNetwork(nn.Module):\n \"\"\"Implicit Quantile Network (IQN) head.\"\"\"\n def __init__(self, envs):\n super().__init__()\n self.encoder = NatureDQNEncoder()\n \n self.n_actions = envs.single_action_space.n\n self.n_cos = 64\n self.phi = nn.Linear(self.n_cos, ENCODER_FEATURE_DIM)\n \n self.head = nn.Linear(ENCODER_FEATURE_DIM, self.n_actions)\n\n def forward(self, x, num_quantiles=32):\n quantiles, _ = self.get_quantiles(x, num_quantiles)\n return quantiles.mean(dim=1)\n\n def get_quantiles(self, x, num_quantiles=32):\n batch_size = x.shape[0]\n features = self.encoder(x)\n \n # [batch_size, num_quantiles, 1]\n tau = torch.rand(batch_size, num_quantiles, 1, device=x.device)\n \n # [1, 1, n_cos]\n pi_mtx = math.pi * torch.arange(1, self.n_cos + 1, device=x.device, dtype=torch.float32)\n pi_mtx = pi_mtx.view(1, 1, self.n_cos)\n cos_tau = torch.cos(tau * pi_mtx)\n \n # [batch_size, num_quantiles, ENCODER_FEATURE_DIM]\n phi_tau = F.relu(self.phi(cos_tau))\n \n features = features.unsqueeze(1)\n merged = features * phi_tau\n \n # [batch_size, num_quantiles, n_actions]\n quantiles = self.head(merged)\n return quantiles, tau\n\n\nclass ValueAlgorithm:\n \"\"\"IQN - Implicit Quantile Networks.\"\"\"\n def __init__(self, envs, device, args):\n self.device = device\n self.gamma = args.gamma\n self.tau_update = args.tau\n self.target_network_frequency = args.target_network_frequency\n \n self.N = 64\n self.N_prime = 64\n self.K = 32\n \n self.q_network = QNetwork(envs).to(device)\n self.target_network = QNetwork(envs).to(device)\n self.target_network.load_state_dict(self.q_network.state_dict())\n self.optimizer = optim.Adam(self.q_network.parameters(), lr=args.learning_rate, eps=0.01 / args.batch_size)\n\n def select_action(self, obs, epsilon):\n with torch.no_grad():\n q_values = self.q_network(torch.Tensor(obs).to(self.device), num_quantiles=self.K)\n return torch.argmax(q_values, dim=1).cpu().numpy()\n\n def update(self, batch, global_step):\n with torch.no_grad():\n # Action selection using online network\n next_q = self.q_network(batch.next_observations, num_quantiles=self.K)\n next_actions = next_q.argmax(dim=1)\n \n # Evaluate using target network\n next_quantiles_target, _ = self.target_network.get_quantiles(batch.next_observations, num_quantiles=self.N_prime)\n next_actions_expanded = next_actions.unsqueeze(1).unsqueeze(2).expand(-1, self.N_prime, -1)\n next_quantiles_best = next_quantiles_target.gather(2, next_actions_expanded).squeeze(2)\n \n # [batch, N_prime]\n target_quantiles = batch.rewards + self.gamma * next_quantiles_best * (1 - batch.dones)\n \n current_quantiles_all, tau = self.q_network.get_quantiles(batch.observations, num_quantiles=self.N)\n actions_expanded = batch.actions.unsqueeze(1).expand(-1, self.N, -1)\n # [batch, N]\n current_quantiles = current_quantiles_all.gather(2, actions_expanded).squeeze(2)\n \n # [batch, N, N_prime]\n td_errors = target_quantiles.unsqueeze(1) - current_quantiles.unsqueeze(2)\n abs_errors = torch.abs(td_errors)\n huber_loss = torch.where(abs_errors <= 1.0, 0.5 * abs_errors.pow(2), 1.0 * (abs_errors - 0.5))\n \n # [batch, N, 1] -> [batch, N, N_prime]\n tau_expanded = tau.expand(-1, -1, self.N_prime)\n indicator = (td_errors.detach() < 0).float()\n quantile_loss = torch.abs(tau_expanded - indicator) * huber_loss\n \n # Expected sum over N and mean over N_prime\n loss = quantile_loss.mean(dim=2).sum(dim=1).mean()\n \n self.optimizer.zero_grad()\n loss.backward()\n self.optimizer.step()\n \n # Target network update\n if global_step % self.target_network_frequency == 0:\n for target_param, q_param in zip(self.target_network.parameters(), self.q_network.parameters()):\n target_param.data.copy_(\n self.tau_update * q_param.data + (1.0 - self.tau_update) * target_param.data\n )\n \n return {\"td_loss\": loss.item(), \"q_values\": current_quantiles.mean().item()}"
7}Tool ResultStep 3
OK: Replaced lines 186..249 with 107 line(s) in cleanrl/cleanrl/custom_value_atari.py. Editable range: 186–329. [Current file: cleanrl/cleanrl/custom_value_atari.py | editable: 186–329 | total: 439 lines] 183| # ===================================================================== 184| # EDITABLE: QNetwork head and ValueAlgorithm 185| # ===================================================================== 186 import math 187 188 class QNetwork(nn.Module): 189 """Implicit Quantile Network (IQN) head.""" 190 def __init__(self, envs): 191 super().__init__() 192 self.encoder = NatureDQNEncoder() 193 194 self.n_actions = envs.single_action_space.n 195 self.n_cos = 64 196 self.phi = nn.Linear(self.n_cos, ENCODER_FEATURE_DIM) 197 198 self.head = nn.Linear(ENCODER_FEATURE_DIM, self.n_actions) 199 200 def forward(self, x, num_quantiles=32): 201
AssistantStep 4edit
Tool Input
1{
2 "filename": "cleanrl/cleanrl/custom_value_atari.py",
3 "op": "replace",
4 "start_line": 293,
5 "end_line": 329,
6 "content": ""
7}Tool ResultStep 4
OK: Replaced lines 293..329 with 0 line(s) in cleanrl/cleanrl/custom_value_atari.py. Editable range: 186–292. [Current file: cleanrl/cleanrl/custom_value_atari.py | editable: 186–292 | total: 402 lines] 183| # ===================================================================== 184| # EDITABLE: QNetwork head and ValueAlgorithm 185| # ===================================================================== 186 import math 187 188 class QNetwork(nn.Module): 189 """Implicit Quantile Network (IQN) head.""" 190 def __init__(self, envs): 191 super().__init__() 192 self.encoder = NatureDQNEncoder() 193 194 self.n_actions = envs.single_action_space.n 195 self.n_cos = 64 196 self.phi = nn.Linear(self.n_cos, ENCODER_FEATURE_DIM) 197 198 self.head = nn.Linear(ENCODER_FEATURE_DIM, self.n_actions) 199 200 def forward(self, x, num_quantiles=32): 201
AssistantStep 5test
Tool ResultStep 5
[Test #2] (1 test remaining; call submit(n=2) to submit this result) ### breakout-v4 (scripts/breakout.sh) Training metrics (last steps): TRAIN_METRICS step=1226000 td_loss=0.2596 q_values=2.1374 TRAIN_METRICS step=1227000 td_loss=0.3087 q_values=2.3986 TRAIN_METRICS step=1228000 td_loss=0.6806 q_values=2.2894 TRAIN_METRICS step=1229000 td_loss=0.7080 q_values=2.2376 TRAIN_METRICS step=1230000 td_loss=0.2956 q_values=2.4530 Evaluation (breakout-v4): Eval episodic_return: 19.90 Eval episodic_return: 65.50 Eval episodic_return: 125.50 Final eval return: 125.50 ### seaquest-v4 (scripts/seaquest.sh) Training metrics (last steps): TRAIN_METRICS step=2052000 td_loss=0.5072 q_values=4.1472 TRAIN_METRICS step=2053000 td_loss=0.3457 q_values=4.0003 TRAIN_METRICS step=2054000 td_loss=0.2918 q_values=4.1691 TRAIN_METRICS step=2055000 td_loss=0.3635 q_values=3.8411 TRAIN_METRICS step=2056000 td_loss=0.3188 q_values=4.1004 Evaluation (seaquest-v4): Eval episodic_return: 960.00 Eval episodic_retur
AssistantStep 6edit
Tool Input
1{
2 "content": "class QNetwork(nn.Module):\n \"\"\"QR-DQN quantile Q-network.\"\"\"\n def __init__(self, envs, n_quantiles=200):\n super().__init__()\n self.n_quantiles = n_quantiles\n self.n = envs.single_action_space.n\n self.encoder = NatureDQNEncoder()\n self.head = nn.Linear(ENCODER_FEATURE_DIM, self.n * n_quantiles)\n\n def forward(self, x):\n features = self.encoder(x)\n quantiles = self.head(features).view(len(x), self.n, self.n_quantiles)\n return quantiles.mean(dim=2)\n\n def get_quantiles(self, x):\n features = self.encoder(x)\n return self.head(features).view(len(x), self.n, self.n_quantiles)\n\n\nclass ValueAlgorithm:\n \"\"\"Munchausen Double QR-DQN.\"\"\"\n def __init__(self, envs, device, args):\n self.device = device\n self.gamma = args.gamma\n self.target_network_frequency = args.target_network_frequency\n self.n_quantiles = 200\n\n # Munchausen hyperparameters\n self.tau_entropy = 0.03\n self.alpha_m = 0.9\n self.clip_log_pi = -1.0 \n\n self.q_network = QNetwork(envs, n_quantiles=self.n_quantiles).to(device)\n self.target_network = QNetwork(envs, n_quantiles=self.n_quantiles).to(device)\n self.target_network.load_state_dict(self.q_network.state_dict())\n self.optimizer = optim.Adam(self.q_network.parameters(), lr=args.learning_rate, eps=0.01 / args.batch_size)\n\n self.tau = torch.arange(1, self.n_quantiles + 1, dtype=torch.float32, device=device)\n self.tau = (2 * self.tau - 1) / (2 * self.n_quantiles)\n\n def select_action(self, obs, epsilon):\n q_values = self.q_network(torch.Tensor(obs).to(self.device))\n return torch.argmax(q_values, dim=1).cpu().numpy()\n\n def update(self, batch, global_step):\n with torch.no_grad():\n # Online network for action selection\n next_quantiles_online = self.q_network.get_quantiles(batch.next_observations)\n next_q_online = next_quantiles_online.mean(dim=2)\n \n # Compute policy for next state\n v_next = next_q_online / self.tau_entropy\n max_v_next = v_next.max(dim=1, keepdim=True)[0]\n exp_v_next = torch.exp(v_next - max_v_next)\n exp_v_next.clamp_(min=1e-8)\n log_pi_next = (v_next - max_v_next) - torch.log(exp_v_next.sum(dim=1, keepdim=True))\n\n next_actions = next_q_online.argmax(dim=1)\n \n # Target quantiles for next state best actions\n next_quantiles_target = self.target_network.get_quantiles(batch.next_observations)\n next_quantiles_best = next_quantiles_target[torch.arange(len(batch.next_observations)), next_actions]\n next_log_pi_a = log_pi_next[torch.arange(len(batch.next_observations)), next_actions]\n\n # Compute Munchausen reward for current state\n curr_quantiles_online = self.q_network.get_quantiles(batch.observations)\n curr_q_online = curr_quantiles_online.mean(dim=2)\n v_curr = curr_q_online / self.tau_entropy\n max_v_curr = v_curr.max(dim=1, keepdim=True)[0]\n exp_v_curr = torch.exp(v_curr - max_v_curr)\n exp_v_curr.clamp_(min=1e-8)\n log_pi_curr = (v_curr - max_v_curr) - torch.log(exp_v_curr.sum(dim=1, keepdim=True))\n \n log_pi_curr_a = log_pi_curr[torch.arange(len(batch.observations)), batch.actions.flatten()]\n log_pi_curr_a = log_pi_curr_a.clamp(min=self.clip_log_pi)\n \n m_reward = batch.rewards.flatten() + self.alpha_m * self.tau_entropy * log_pi_curr_a\n m_target_penalty = - self.tau_entropy * next_log_pi_a\n \n target_quantiles = m_reward.unsqueeze(-1) + self.gamma * (next_quantiles_best + m_target_penalty.unsqueeze(-1)) * (1 - batch.dones)\n \n current_quantiles_all = self.q_network.get_quantiles(batch.observations)\n current_quantiles = current_quantiles_all[torch.arange(len(batch.observations)), batch.actions.flatten()]\n\n td_errors = target_quantiles.unsqueeze(1) - current_quantiles.unsqueeze(2) #[batch, N, N]\n \n # huber loss\n huber_loss = F.huber_loss(\n current_quantiles.unsqueeze(2), \n target_quantiles.unsqueeze(1).expand(-1, self.n_quantiles, -1), \n reduction=\"none\", delta=1.0\n )\n \n quantiles = self.tau.view(1, -1, 1) # [1, N(pred), 1]\n quantile_loss = abs(quantiles - (td_errors.detach() < 0).float()) * huber_loss\n loss = quantile_loss.sum(dim=1).mean(dim=1).mean()\n\n self.optimizer.zero_grad()\n loss.backward()\n self.optimizer.step()\n\n # Hard target update\n if global_step % self.target_network_frequency == 0:\n self.target_network.load_state_dict(self.q_network.state_dict())\n\n return {\"td_loss\": loss.item(), \"q_values\": current_quantiles.mean().item()}",
3 "op": "replace",
4 "start_line": 188,
5 "filename": "cleanrl/cleanrl/custom_value_atari.py",
6 "end_line": 329
7}Tool ResultStep 6
ERROR: Lines 188..329 exceed the editable range. You may only edit lines 186–292. [Current file: cleanrl/cleanrl/custom_value_atari.py | editable: 186–292 | total: 402 lines] 183| # ===================================================================== 184| # EDITABLE: QNetwork head and ValueAlgorithm 185| # ===================================================================== 186 import math 187 188 class QNetwork(nn.Module): 189 """Implicit Quantile Network (IQN) head.""" 190 def __init__(self, envs): 191 super().__init__() 192 self.encoder = NatureDQNEncoder() 193 194 self.n_actions = envs.single_action_space.n 195 self.n_cos = 64 196 self.phi = nn.Linear(self.n_cos, ENCODER_FEATURE_DIM) 197 198 self.head = nn.Linear(ENCODER_FEATURE_DIM, self.n_actions) 199 200 def forward(self, x, num_quantiles=32): 201 quantiles, _ = self.
AssistantStep 7test
Tool ResultStep 7
[Test #3] (0 tests remaining) ## Seed 42 ### breakout-v4 (scripts/breakout.sh) Training metrics (last steps): TRAIN_METRICS step=595000 td_loss=0.3878 q_values=1.6901 TRAIN_METRICS step=596000 td_loss=0.8974 q_values=1.6855 TRAIN_METRICS step=597000 td_loss=0.3214 q_values=1.9017 TRAIN_METRICS step=598000 td_loss=0.3228 q_values=1.7292 TRAIN_METRICS step=599000 td_loss=0.7023 q_values=1.4924 Evaluation (breakout-v4): Eval episodic_return: 7.10 Eval episodic_return: 15.60 Eval episodic_return: 14.70 Final eval return: 14.70 ### seaquest-v4 (scripts/seaquest.sh) Training metrics (last steps): TRAIN_METRICS step=895000 td_loss=0.5321 q_values=2.1462 TRAIN_METRICS step=896000 td_loss=0.3722 q_values=2.0393 TRAIN_METRICS step=897000 td_loss=0.6574 q_values=2.1741 TRAIN_METRICS step=898000 td_loss=0.3680 q_values=2.2312 TRAIN_METRICS step=899000 td_loss=0.4294 q_values=2.3324 Evaluation (seaquest-v4): Eval episodic_return: 426.00 Eval episodic_return: 248.00 Eval episodic_return: 582.00