Agent Conversation: gpt-5.4-pro
Model: gpt-5.4-pro
Total steps: 7
Messages: 15
UserStep 0
Task: quant-graph-stock
Graph-Based Multi-Stock Prediction on CSI300
Objective
Design and implement a graph-based stock prediction model that leverages inter-stock relationships through a stock-concept graph. Your code goes in custom_model.py. Three reference implementations (HIST, GATs, LightGBM) are provided as read-only.
Evaluation
Signal quality: IC, ICIR, Rank IC. Portfolio (TopkDropout, top 50, drop 5): Annualized Return, Max Drawdown, Information Ratio. Automatic via qlib's workflow.
Workflow Configuration
workflow_config.yaml lines 14-26 and 32-45 are editable. This covers the model plus dataset adapter/preprocessor configuration. Instruments, date ranges, train/valid/test splits, and evaluation settings are fixed.
qlib/custom_model.py [EDITABLE — lines 58–156 only]
1: # Custom graph-based stock prediction model for MLS-Bench
2: #
3: # EDITABLE section: CustomModel class with fit() and predict() methods.
4: # FIXED sections: imports and stock-concept graph loading below.
5: import os
6: import numpy as np
7: import pandas as pd
8: import torch
9: import torch.nn as nn
10: import torch.nn.functional as F
11: from qlib.model.base import Model
12: from qlib.data.dataset import DatasetH
13: from qlib.data.dataset.handler import DataHandlerLP
14:
15: DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
16:
17: # =====================================================================
18: # FIXED: Stock-concept graph data loading utilities
19: # =====================================================================
20: # Paths to pre-downloaded graph data
21: STOCK2CONCEPT_PATH = os.path.expanduser("~/.qlib/qlib_data/qlib_csi300_stock2concept.npy")
22: STOCK_INDEX_PATH = os.path.expanduser("~/.qlib/qlib_data/qlib_csi300_stock_index.npy")
23:
24: # Load the stock-concept mapping matrix and stock index
25: # stock2concept_matrix: shape (num_stocks, num_concepts), binary membership
26: # stock_index_dict: dict mapping instrument name -> integer index
27: _stock2concept_matrix = np.load(STOCK2CONCEPT_PATH)
28: _stock_index_dict = np.load(STOCK_INDEX_PATH, allow_pickle=True).item()
29:
30:
31: def get_stock_index(instruments, default_index=733):
32: """Map instrument names to integer indices for stock2concept lookup.
33:
34: Args:
35: instruments: array-like of instrument name strings
36: default_index: fallback index for unknown instruments (733 = padding)
37:
38: Returns:
39: np.ndarray of integer indices
40: """
41: indices = np.array([_stock_index_dict.get(inst, default_index)
42: for inst in instruments])
43: return indices.astype(int)
44:
45:
46: def get_concept_matrix(stock_indices):
47: """Get the concept membership matrix for giAssistantStep 1edit
Tool Input
1{
2 "op": "replace",
3 "filename": "qlib/workflow_config.yaml",
4 "content": " infer_processors: []",
5 "after_line": 31,
6 "start_line": 32,
7 "end_line": 39
8}Tool ResultStep 1
OK: Replaced lines 32..39 with 1 line(s) in qlib/workflow_config.yaml. Editable range: 14–26, 32–38.
[Current file: qlib/workflow_config.yaml | editable: 14–26 | total: 76 lines]
11| - "." # So custom_model.py is importable via module_path
12|
13| task:
14 model:
15 class: CustomModel
16 module_path: custom_model
17 kwargs: {}
18
19 dataset:
20 class: DatasetH
21 module_path: qlib.data.dataset
22 kwargs:
23 handler:
24 class: Alpha360
25 module_path: qlib.contrib.data.handler
26 kwargs:
27| start_time: "2008-01-01"
28| end_time: "2020-08-01"
29| fit_start_time: "2008-01-01"AssistantStep 2edit
Tool Input
1{
2 "op": "replace",
3 "filename": "qlib/workflow_config.yaml",
4 "content": " infer_processors: []\n learn_processors:\n - class: DropnaLabel\n - class: CSRankNorm\n kwargs:\n fields_group: label\n label: [\"Ref($close, -2) / Ref($close, -1) - 1\"]",
5 "after_line": 31,
6 "start_line": 32,
7 "end_line": 45
8}Tool ResultStep 2
ERROR: Lines 32..45 exceed the editable range. You may only edit lines 14–26, 32–38.
[Current file: qlib/workflow_config.yaml | editable: 14–26 | total: 76 lines]
11| - "." # So custom_model.py is importable via module_path
12|
13| task:
14 model:
15 class: CustomModel
16 module_path: custom_model
17 kwargs: {}
18
19 dataset:
20 class: DatasetH
21 module_path: qlib.data.dataset
22 kwargs:
23 handler:
24 class: Alpha360
25 module_path: qlib.contrib.data.handler
26 kwargs:
27| start_time: "2008-01-01"
28| end_time: "2020-08-01"
29| fit_start_time: "2008-01-01"AssistantStep 3edit
Tool Input
1{
2 "op": "replace",
3 "filename": "qlib/custom_model.py",
4 "content": "class CustomModel(Model):\n \"\"\"Graph-Residual Boosting with stock-specific leave-one-out concept adaptation.\"\"\"\n\n def __init__(self):\n super().__init__()\n self.params = {\n \"objective\": \"mse\",\n \"metric\": \"l2\",\n \"learning_rate\": 0.04,\n \"subsample\": 0.88,\n \"colsample_bytree\": 0.88,\n \"lambda_l1\": 180.0,\n \"lambda_l2\": 520.0,\n \"max_depth\": 8,\n \"num_leaves\": 210,\n \"num_threads\": 20,\n \"verbosity\": -1,\n }\n self.model = None\n self.global_gamma = 0.0\n self.gamma = np.zeros(_stock2concept_matrix.shape[0], dtype=np.float32)\n self.shrink = 64.0\n\n def _cs(self, values, index):\n s = pd.Series(values, index=index)\n g = s.groupby(level=0)\n z = (s - g.transform(\"mean\")) / (g.transform(\"std\") + 1e-6)\n return z.fillna(0.0).values.astype(np.float32)\n\n def _peer_scores(self, preds, index):\n inst = index.get_level_values(\"instrument\")\n sid = get_stock_index(inst)\n counts = pd.Series(1, index=index).groupby(level=0).size().values\n starts = np.roll(np.cumsum(counts), 1)\n starts[0] = 0\n out = np.zeros(len(preds), dtype=np.float32)\n for st, n in zip(starts, counts):\n ed = st + n\n ids = sid[st:ed]\n mat = get_concept_matrix(ids)\n x = preds[st:ed].astype(np.float32)\n deg_s = mat.sum(1)\n deg_c = mat.sum(0)\n csum = mat.T @ x\n peer = (mat * ((csum[None, :] - x[:, None] * mat) / np.maximum(deg_c[None, :] - mat, 1.0))).sum(1)\n peer = peer / np.maximum(deg_s, 1.0)\n peer[deg_s == 0] = x[deg_s == 0]\n out[st:ed] = peer\n return out, sid\n\n def fit(self, dataset: DatasetH):\n import lightgbm as lgb\n\n df_train, df_valid = dataset.prepare(\n [\"train\", \"valid\"], col_set=[\"feature\", \"label\"], data_key=DataHandlerLP.DK_L\n )\n xtr, ytr = df_train[\"feature\"], np.squeeze(df_train[\"label\"].values)\n xva, yva = df_valid[\"feature\"], np.squeeze(df_valid[\"label\"].values)\n dtr = lgb.Dataset(xtr.values, label=ytr, free_raw_data=False)\n dva = lgb.Dataset(xva.values, label=yva, free_raw_data=False)\n self.model = lgb.train(\n self.params,\n dtr,\n num_boost_round=800,\n valid_sets=[dtr, dva],\n valid_names=[\"train\", \"valid\"],\n callbacks=[lgb.early_stopping(60), lgb.log_evaluation(period=100)],\n )\n\n p = self._cs(self.model.predict(xva.values), xva.index)\n q, sid = self._peer_scores(p, xva.index)\n q = self._cs(q, xva.index)\n y = self._cs(yva, xva.index)\n z, r = q - p, y - p\n self.global_gamma = float((z * r).sum() / ((z * z).sum() + 1e-6))\n uniq, inv = np.unique(sid, return_inverse=True)\n cnt = np.bincount(inv).astype(np.float32)\n num = np.bincount(inv, weights=z * r)\n den = np.bincount(inv, weights=z * z) + 1e-6\n raw = num / den\n w = cnt / (cnt + self.shrink)\n self.gamma = np.full(_stock2concept_matrix.shape[0], self.global_gamma, dtype=np.float32)\n self.gamma[uniq] = np.clip(w * raw + (1.0 - w) * self.global_gamma, -2.0, 2.0)\n\n def predict(self, dataset: DatasetH, segment=\"test\"):\n if self.model is None:\n raise ValueError(\"model is not fitted yet!\")\n x = dataset.prepare(segment, col_set=\"feature\", data_key=DataHandlerLP.DK_I)\n p = self._cs(self.model.predict(x.values), x.index)\n q, sid = self._peer_scores(p, x.index)\n q = self._cs(q, x.index)\n pred = p + self.gamma[np.clip(sid, 0, len(self.gamma) - 1)] * (q - p)\n return pd.Series(pred, index=x.index, name=\"score\")",
5 "after_line": 60,
6 "start_line": 61,
7 "end_line": 156
8}Tool ResultStep 3
OK: Replaced lines 61..156 with 93 line(s) in qlib/custom_model.py. Editable range: 58–153.
[Current file: qlib/custom_model.py | editable: 58–153 | total: 153 lines]
55| return _stock2concept_matrix[stock_indices].astype(np.float32)
56|
57|
58 # =====================================================================
59 # EDITABLE: CustomModel — implement your stock prediction model here
60 # =====================================================================
61 class CustomModel(Model):
62 """Graph-Residual Boosting with stock-specific leave-one-out concept adaptation."""
63
64 def __init__(self):
65 super().__init__()
66 self.params = {
67 "objective": "mse",
68 "metric": "l2",
69 "learning_rate": 0.04,
70 "subsample": 0.88,
71 "colsample_bytree": 0.88,
72 "lambda_l1": 180.0,
73
AssistantStep 4test
Tool ResultStep 4
[Test #1] (2 tests remaining; call submit(n=1) to submit this result)
### csi300 (scripts/csi300.sh)
Signal analysis (csi300):
SIGNAL_METRIC IC=0.030702
SIGNAL_METRIC ICIR=0.228542
SIGNAL_METRIC Rank_IC=0.040747
SIGNAL_METRIC Rank_ICIR=0.302767
Portfolio analysis (csi300):
PORTFOLIO_METRIC annualized_return=0.007753
PORTFOLIO_METRIC max_drawdown=-0.116040
PORTFOLIO_METRIC information_ratio=0.121168
### csi100 (scripts/csi100.sh)
Signal analysis (csi100):
SIGNAL_METRIC IC=0.026503
SIGNAL_METRIC ICIR=0.149326
SIGNAL_METRIC Rank_IC=0.039147
SIGNAL_METRIC Rank_ICIR=0.203550
Portfolio analysis (csi100):
PORTFOLIO_METRIC annualized_return=-0.048121
PORTFOLIO_METRIC max_drawdown=-0.199534
PORTFOLIO_METRIC information_ratio=-0.864424
[Leaderboard] Results saved: {'ic_csi300': 0.030702, 'icir_csi300': 0.228542, 'rank_ic_csi300': 0.040747, 'rank_icir_csi300': 0.302767, 'annualized_return_csi300': 0.007753, 'max_drawdown_csi300': -0.11604, 'information_ratio_csi300': 0.121168, 'elapsed_csi300'
AssistantStep 5edit
Tool Input
1{
2 "op": "replace",
3 "filename": "qlib/custom_model.py",
4 "content": "class CustomModel(Model):\n \"\"\"Tail-aware concept residual boosting on top of LightGBM.\"\"\"\n\n def __init__(self):\n super().__init__()\n self.params = {\n \"objective\": \"mse\",\n \"metric\": \"l2\",\n \"colsample_bytree\": 0.8879,\n \"learning_rate\": 0.0421,\n \"subsample\": 0.8789,\n \"lambda_l1\": 205.6999,\n \"lambda_l2\": 580.9768,\n \"max_depth\": 8,\n \"num_leaves\": 210,\n \"num_threads\": 20,\n \"verbosity\": -1,\n }\n self.model = None\n self.ridge = 12.0\n self.mix = 0.0\n self.coef = np.zeros(_stock2concept_matrix.shape[1], dtype=np.float32)\n\n def _cs(self, values, index):\n s = pd.Series(np.asarray(values).reshape(-1), index=index)\n g = s.groupby(level=0)\n z = (s - g.transform(\"mean\")) / (g.transform(\"std\") + 1e-6)\n return z.fillna(0.0).values.astype(np.float32)\n\n def _graph_block(self, base, stock_idx):\n mat = get_concept_matrix(stock_idx)\n deg_s = np.maximum(mat.sum(1, keepdims=True), 1.0)\n deg_c = mat.sum(0, keepdims=True)\n peer_sum = (mat.T @ base).reshape(1, -1)\n loo = (peer_sum - base[:, None] * mat) / np.maximum(deg_c - mat, 1.0)\n rel = np.clip((deg_c - 1.0) / (deg_c + 2.0), 0.0, 1.0)\n return (mat * rel * (loo - base[:, None]) / deg_s).astype(np.float32)\n\n def fit(self, dataset: DatasetH):\n import lightgbm as lgb\n\n df_train, df_valid = dataset.prepare(\n [\"train\", \"valid\"], col_set=[\"feature\", \"label\"], data_key=DataHandlerLP.DK_L\n )\n xtr, ytr = df_train[\"feature\"], np.squeeze(df_train[\"label\"].values)\n xva, yva = df_valid[\"feature\"], np.squeeze(df_valid[\"label\"].values)\n wtr = 1.0 + 2.0 * np.abs(ytr)\n wva = 1.0 + 2.0 * np.abs(yva)\n self.model = lgb.train(\n self.params,\n lgb.Dataset(xtr.values, label=ytr, weight=wtr, free_raw_data=False),\n num_boost_round=1000,\n valid_sets=[\n lgb.Dataset(xtr.values, label=ytr, weight=wtr, free_raw_data=False),\n lgb.Dataset(xva.values, label=yva, weight=wva, free_raw_data=False),\n ],\n valid_names=[\"train\", \"valid\"],\n callbacks=[lgb.early_stopping(50), lgb.log_evaluation(period=100)],\n )\n\n p = self._cs(self.model.predict(xva.values), xva.index)\n y = self._cs(yva, xva.index)\n r = y - p\n sw = (1.0 + 2.0 * np.abs(y)).astype(np.float32)\n sid = get_stock_index(xva.index.get_level_values(\"instrument\"))\n counts = pd.Series(1, index=xva.index).groupby(level=0).size().values\n starts = np.roll(np.cumsum(counts), 1)\n starts[0] = 0\n dim = _stock2concept_matrix.shape[1]\n gram = np.eye(dim, dtype=np.float64) * self.ridge\n rhs = np.zeros(dim, dtype=np.float64)\n for st, n in zip(starts, counts):\n ed = st + n\n feat = self._graph_block(p[st:ed], sid[st:ed])\n w = sw[st:ed, None]\n gram += feat.T @ (feat * w)\n rhs += feat.T @ (r[st:ed] * sw[st:ed])\n self.coef = np.clip(np.linalg.solve(gram, rhs), -3.0, 3.0).astype(np.float32)\n\n corr = np.zeros_like(p)\n for st, n in zip(starts, counts):\n ed = st + n\n corr[st:ed] = self._graph_block(p[st:ed], sid[st:ed]) @ self.coef\n self.mix = float(np.clip((sw * corr * r).sum() / ((sw * corr * corr).sum() + 1e-6), 0.0, 1.5))\n\n def predict(self, dataset: DatasetH, segment=\"test\"):\n if self.model is None:\n raise ValueError(\"model is not fitted yet!\")\n x = dataset.prepare(segment, col_set=\"feature\", data_key=DataHandlerLP.DK_I)\n p = self._cs(self.model.predict(x.values), x.index)\n sid = get_stock_index(x.index.get_level_values(\"instrument\"))\n counts = pd.Series(1, index=x.index).groupby(level=0).size().values\n starts = np.roll(np.cumsum(counts), 1)\n starts[0] = 0\n corr = np.zeros_like(p)\n for st, n in zip(starts, counts):\n ed = st + n\n corr[st:ed] = self._graph_block(p[st:ed], sid[st:ed]) @ self.coef\n pred = self._cs(p + self.mix * corr, x.index)\n return pd.Series(pred, index=x.index, name=\"score\")",
5 "after_line": 60,
6 "start_line": 61,
7 "end_line": 153
8}Tool ResultStep 5
OK: Replaced lines 61..153 with 100 line(s) in qlib/custom_model.py. Editable range: 58–160.
[Current file: qlib/custom_model.py | editable: 58–160 | total: 160 lines]
55| return _stock2concept_matrix[stock_indices].astype(np.float32)
56|
57|
58 # =====================================================================
59 # EDITABLE: CustomModel — implement your stock prediction model here
60 # =====================================================================
61 class CustomModel(Model):
62 """Tail-aware concept residual boosting on top of LightGBM."""
63
64 def __init__(self):
65 super().__init__()
66 self.params = {
67 "objective": "mse",
68 "metric": "l2",
69 "colsample_bytree": 0.8879,
70 "learning_rate": 0.0421,
71 "subsample": 0.8789,
72 "lambda_l1": 205.6999,
73 "lambda_l2"
AssistantStep 6test
Tool ResultStep 6
[Test #2] (1 test remaining; call submit(n=2) to submit this result)
### csi300 (scripts/csi300.sh)
Signal analysis (csi300):
SIGNAL_METRIC IC=0.034166
SIGNAL_METRIC ICIR=0.266014
SIGNAL_METRIC Rank_IC=0.039496
SIGNAL_METRIC Rank_ICIR=0.324934
Portfolio analysis (csi300):
PORTFOLIO_METRIC annualized_return=0.020915
PORTFOLIO_METRIC max_drawdown=-0.145865
PORTFOLIO_METRIC information_ratio=0.261379
### csi100 (scripts/csi100.sh)
Signal analysis (csi100):
SIGNAL_METRIC IC=0.036874
SIGNAL_METRIC ICIR=0.198693
SIGNAL_METRIC Rank_IC=0.038826
SIGNAL_METRIC Rank_ICIR=0.217251
Portfolio analysis (csi100):
PORTFOLIO_METRIC annualized_return=-0.015515
PORTFOLIO_METRIC max_drawdown=-0.161950
PORTFOLIO_METRIC information_ratio=-0.289346
[Leaderboard] Results saved: {'ic_csi300': 0.034166, 'icir_csi300': 0.266014, 'rank_ic_csi300': 0.039496, 'rank_icir_csi300': 0.324934, 'annualized_return_csi300': 0.020915, 'max_drawdown_csi300': -0.145865, 'information_ratio_csi300': 0.261379, 'elapsed_csi300'
AssistantStep 7submit
Tool Input
1{
2 "n": 2
3}Tool ResultStep 7
[submit] Submitting result from test #2 as final. ### csi300 (scripts/csi300.sh) Signal analysis (csi300): SIGNAL_METRIC IC=0.034166 SIGNAL_METRIC ICIR=0.266014 SIGNAL_METRIC Rank_IC=0.039496 SIGNAL_METRIC Rank_ICIR=0.324934 Portfolio analysis (csi300): PORTFOLIO_METRIC annualized_return=0.020915 PORTFOLIO_METRIC max_drawdown=-0.145865 PORTFOLIO_METRIC information_ratio=0.261379 ### csi100 (scripts/csi100.sh) Signal analysis (csi100): SIGNAL_METRIC IC=0.036874 SIGNAL_METRIC ICIR=0.198693 SIGNAL_METRIC Rank_IC=0.038826 SIGNAL_METRIC Rank_ICIR=0.217251 Portfolio analysis (csi100): PORTFOLIO_METRIC annualized_return=-0.015515 PORTFOLIO_METRIC max_drawdown=-0.161950 PORTFOLIO_METRIC information_ratio=-0.289346 ### csi300_recent (scripts/csi300_recent.sh) Signal analysis (csi300_recent): SIGNAL_METRIC IC=0.018364 SIGNAL_METRIC ICIR=0.128976 SIGNAL_METRIC Rank_IC=0.035098 SIGNAL_METRIC Rank_ICIR=0.256845 Portfolio analysis (csi300_recent): PORTFOLIO_METRIC annualized_return=0.040932 PORTF