自定义PyEnvironment的 time_step与 time_step_spec不匹配
我正在TensorFlow Agents中创建一个自定义的PyEnvironment，用以模拟田径十项全能。到目前为止，我已经在一定意义上实现了一个可运行的环境，可以使用 _step 和 _reset，但在实现基于Reverb的回放缓冲区时，出现了以下问题：
Traceback (most recent call last):
File "/home/perry/Documents/Programming/decathlon_training_simulation_tf/train_network.py", line 308, in <module>
).run(env_train_py.reset())
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/perry/Documents/Programming/decathlon_training_simulation_tf/.venv/lib/python3.11/site-packages/tf_agents/drivers/py_driver.py", line 119, in run
action_step = self.policy.action(time_step, policy_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/perry/Documents/Programming/decathlon_training_simulation_tf/.venv/lib/python3.11/site-packages/tf_agents/policies/py_policy.py", line 169, in action
return self._action(time_step, policy_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/perry/Documents/Programming/decathlon_training_simulation_tf/.venv/lib/python3.11/site-packages/tf_agents/policies/py_tf_eager_policy.py", line 107, in _action
policy_step = self._policy_action_fn(time_step, policy_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/perry/Documents/Programming/decathlon_training_simulation_tf/.venv/lib/python3.11/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/perry/Documents/Programming/decathlon_training_simulation_tf/.venv/lib/python3.11/site-packages/tf_agents/policies/tf_policy.py", line 317, in action
nest_utils.assert_same_structure(
File "/home/perry/Documents/Programming/decathlon_training_simulation_tf/.venv/lib/python3.11/site-packages/tf_agents/utils/nest_utils.py", line 131, in assert_same_structure
raise exception(
ValueError: time_step and time_step_spec structures do not match:
TimeStep(
{'step_type': .,
'reward': .,
'discount': .,
'observation': {'dev_strength': {'r': ., 'best': .},
'dev_speed': {'r': ., 'best': .},
'dev_jumping': {'r': ., 'best': .},
'dev_endurance': {'r': ., 'best': .},
'dev_100_tech': {'r': ., 'best': .},
'dev_lj_tech': {'r': ., 'best': .},
'dev_sp_tech': {'r': ., 'best': .},
'dev_hj_tech': {'r': ., 'best': .},
'dev_400_tech': {'r': ., 'best': .},
'dev_110h_tech': {'r': ., 'best': .},
'dev_dt_tech': {'r': ., 'best': .},
'dev_pv_tech': {'r': ., 'best': .},
'dev_jt_tech': {'r': ., 'best': .},
'dev_1500_tech': {'r': ., 'best': .},
'dev_100': {'start': ., 'best': ., 'shift': ., 'k': .},
'dev_lj': {'start': ., 'best': ., 'shift': ., 'k': .},
'dev_sp': {'start': ., 'best': ., 'shift': ., 'k': .},
'dev_hj': {'start': ., 'best': ., 'shift': ., 'k': .},
'dev_400': {'start': ., 'best': ., 'shift': ., 'k': .},
'dev_110h': {'start': ., 'best': ., 'shift': ., 'k': .},
'dev_dt': {'start': ., 'best': ., 'shift': ., 'k': .},
'dev_pv': {'start': ., 'best': ., 'shift': ., 'k': .},
'dev_jt': {'start': ., 'best': ., 'shift': ., 'k': .},
'dev_1500': {'start': ., 'best': ., 'shift': ., 'k': .},
'100_i': .,
'lj_i': .,
'sp_i': .,
'hj_i': .,
'400_i': .,
'110h_i': .,
'dt_i': .,
'pv_i': .,
'jt_i': .,
'1500_i': .,
'100_o': .,
'lj_o': .,
'sp_o': .,
'hj_o': .,
'400_o': .,
'110h_o': .,
'dt_o': .,
'pv_o': .,
'jt_o': .,
'1500_o': .,
'inv_speed': .,
'inv_strength': .,
'inv_jumping': .,
'inv_endurance': .,
'inv_100_tech': .,
'inv_lj_tech': .,
'inv_sp_tech': .,
'inv_hj_tech': .,
'inv_400_tech': .,
'inv_110h_tech': .,
'inv_dt_tech': .,
'inv_pv_tech': .,
'inv_jt_tech': .,
'inv_1500_tech': .,
't': .}})
vs.
TimeStep(
{'step_type': ., 'reward': ., 'discount': ., 'observation': .})
Values:
TimeStep(
{'step_type': <tf.Tensor 'time_step:0' shape=(1,) dtype=int32>,
'reward': <tf.Tensor 'time_step_1:0' shape=(1,) dtype=float32>,
'discount': <tf.Tensor 'time_step_2:0' shape=(1,) dtype=float32>,
'observation': {'dev_strength': {'r': <tf.Tensor 'time_step_3:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_4:0' shape=(1,) dtype=int64>},
'dev_speed': {'r': <tf.Tensor 'time_step_5:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_6:0' shape=(1,) dtype=int64>},
'dev_jumping': {'r': <tf.Tensor 'time_step_7:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_8:0' shape=(1,) dtype=int64>},
'dev_endurance': {'r': <tf.Tensor 'time_step_9:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_10:0' shape=(1,) dtype=int64>},
'dev_100_tech': {'r': <tf.Tensor 'time_step_11:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_12:0' shape=(1,) dtype=int64>},
'dev_lj_tech': {'r': <tf.Tensor 'time_step_13:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_14:0' shape=(1,) dtype=int64>},
'dev_sp_tech': {'r': <tf.Tensor 'time_step_15:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_16:0' shape=(1,) dtype=int64>},
'dev_hj_tech': {'r': <tf.Tensor 'time_step_17:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_18:0' shape=(1,) dtype=int64>},
'dev_400_tech': {'r': <tf.Tensor 'time_step_19:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_20:0' shape=(1,) dtype=int64>},
'dev_110h_tech': {'r': <tf.Tensor 'time_step_21:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_22:0' shape=(1,) dtype=int64>},
'dev_dt_tech': {'r': <tf.Tensor 'time_step_23:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_24:0' shape=(1,) dtype=int64>},
'dev_pv_tech': {'r': <tf.Tensor 'time_step_25:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_26:0' shape=(1,) dtype=int64>},
'dev_jt_tech': {'r': <tf.Tensor 'time_step_27:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_28:0' shape=(1,) dtype=int64>},
'dev_1500_tech': {'r': <tf.Tensor 'time_step_29:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_30:0' shape=(1,) dtype=int64>},
'dev_100': {'start': <tf.Tensor 'time_step_31:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_32:0' shape=(1,) dtype=float64>,
'shift': <tf.Tensor 'time_step_33:0' shape=(1,) dtype=int64>,
'k': <tf.Tensor 'time_step_34:0' shape=(1,) dtype=float64>},
'dev_lj': {'start': <tf.Tensor 'time_step_35:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_36:0' shape=(1,) dtype=float64>,
'shift': <tf.Tensor 'time_step_37:0' shape=(1,) dtype=int64>,
'k': <tf.Tensor 'time_step_38:0' shape=(1,) dtype=float64>},
'dev_sp': {'start': <tf.Tensor 'time_step_39:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_40:0' shape=(1,) dtype=float64>,
'shift': <tf.Tensor 'time_step_41:0' shape=(1,) dtype=int64>,
'k': <tf.Tensor 'time_step_42:0' shape=(1,) dtype=float64>},
'dev_hj': {'start': <tf.Tensor 'time_step_43:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_44:0' shape=(1,) dtype=float64>,
'shift': <tf.Tensor 'time_step_45:0' shape=(1,) dtype=int64>,
'k': <tf.Tensor 'time_step_46:0' shape=(1,) dtype=float64>},
'dev_400': {'start': <tf.Tensor 'time_step_47:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_48:0' shape=(1,) dtype=float64>,
'shift': <tf.Tensor 'time_step_49:0' shape=(1,) dtype=int64>,
'k': <tf.Tensor 'time_step_50:0' shape=(1,) dtype=float64>},
'dev_110h': {'start': <tf.Tensor 'time_step_51:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_52:0' shape=(1,) dtype=float64>,
'shift': <tf.Tensor 'time_step_53:0' shape=(1,) dtype=int64>,
'k': <tf.Tensor 'time_step_54:0' shape=(1,) dtype=float64>},
'dev_dt': {'start': <tf.Tensor 'time_step_55:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_56:0' shape=(1,) dtype=float64>,
'shift': <tf.Tensor 'time_step_57:0' shape=(1,) dtype=int64>,
'k': <tf.Tensor 'time_step_58:0' shape=(1,) dtype=float64>},
'dev_pv': {'start': <tf.Tensor 'time_step_59:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_60:0' shape=(1,) dtype=float64>,
'shift': <tf.Tensor 'time_step_61:0' shape=(1,) dtype=int64>,
'k': <tf.Tensor 'time_step_62:0' shape=(1,) dtype=float64>},
'dev_jt': {'start': <tf.Tensor 'time_step_63:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_64:0' shape=(1,) dtype=float64>,
'shift': <tf.Tensor 'time_step_65:0' shape=(1,) dtype=int64>,
'k': <tf.Tensor 'time_step_66:0' shape=(1,) dtype=float64>},
'dev_1500': {'start': <tf.Tensor 'time_step_67:0' shape=(1,) dtype=float64>,
'best': <tf.Tensor 'time_step_68:0' shape=(1,) dtype=float64>,
'shift': <tf.Tensor 'time_step_69:0' shape=(1,) dtype=int64>,
'k': <tf.Tensor 'time_step_70:0' shape=(1,) dtype=float64>},
'100_i': <tf.Tensor 'time_step_71:0' shape=(1,) dtype=float64>,
'lj_i': <tf.Tensor 'time_step_72:0' shape=(1,) dtype=float64>,
'sp_i': <tf.Tensor 'time_step_73:0' shape=(1,) dtype=float64>,
'hj_i': <tf.Tensor 'time_step_74:0' shape=(1,) dtype=float64>,
'400_i': <tf.Tensor 'time_step_75:0' shape=(1,) dtype=float64>,
'110h_i': <tf.Tensor 'time_step_76:0' shape=(1,) dtype=float64>,
'dt_i': <tf.Tensor 'time_step_77:0' shape=(1,) dtype=float64>,
'pv_i': <tf.Tensor 'time_step_78:0' shape=(1,) dtype=float64>,
'jt_i': <tf.Tensor 'time_step_79:0' shape=(1,) dtype=float64>,
'1500_i': <tf.Tensor 'time_step_80:0' shape=(1,) dtype=float64>,
'100_o': <tf.Tensor 'time_step_81:0' shape=(1,) dtype=float64>,
'lj_o': <tf.Tensor 'time_step_82:0' shape=(1,) dtype=float64>,
'sp_o': <tf.Tensor 'time_step_83:0' shape=(1,) dtype=float64>,
'hj_o': <tf.Tensor 'time_step_84:0' shape=(1,) dtype=float64>,
'400_o': <tf.Tensor 'time_step_85:0' shape=(1,) dtype=float64>,
'110h_o': <tf.Tensor 'time_step_86:0' shape=(1,) dtype=float64>,
'dt_o': <tf.Tensor 'time_step_87:0' shape=(1,) dtype=float64>,
'pv_o': <tf.Tensor 'time_step_88:0' shape=(1,) dtype=float64>,
'jt_o': <tf.Tensor 'time_step_89:0' shape=(1,) dtype=float64>,
'1500_o': <tf.Tensor 'time_step_90:0' shape=(1,) dtype=float64>,
'inv_speed': <tf.Tensor 'time_step_91:0' shape=(1,) dtype=int64>,
'inv_strength': <tf.Tensor 'time_step_92:0' shape=(1,) dtype=int64>,
'inv_jumping': <tf.Tensor 'time_step_93:0' shape=(1,) dtype=int64>,
'inv_endurance': <tf.Tensor 'time_step_94:0' shape=(1,) dtype=int64>,
'inv_100_tech': <tf.Tensor 'time_step_95:0' shape=(1,) dtype=int64>,
'inv_lj_tech': <tf.Tensor 'time_step_96:0' shape=(1,) dtype=int64>,
'inv_sp_tech': <tf.Tensor 'time_step_97:0' shape=(1,) dtype=int64>,
'inv_hj_tech': <tf.Tensor 'time_step_98:0' shape=(1,) dtype=int64>,
'inv_400_tech': <tf.Tensor 'time_step_99:0' shape=(1,) dtype=int64>,
'inv_110h_tech': <tf.Tensor 'time_step_100:0' shape=(1,) dtype=int64>,
'inv_dt_tech': <tf.Tensor 'time_step_101:0' shape=(1,) dtype=int64>,
'inv_pv_tech': <tf.Tensor 'time_step_102:0' shape=(1,) dtype=int64>,
'inv_jt_tech': <tf.Tensor 'time_step_103:0' shape=(1,) dtype=int64>,
'inv_1500_tech': <tf.Tensor 'time_step_104:0' shape=(1,) dtype=int64>,
't': <tf.Tensor 'time_step_105:0' shape=(1,) dtype=int64>}})
vs.
TimeStep(
{'step_type': TensorSpec(shape=(), dtype=tf.int32, name='step_type'),
'reward': TensorSpec(shape=(), dtype=tf.float32, name='reward'),
'discount': BoundedTensorSpec(shape=(), dtype=tf.float32, name='discount', minimum=array(0., dtype=float32), maximum=array(1., dtype=float32)),
'observation': BoundedTensorSpec(shape=(), dtype=tf.float32, name='observation_spec', minimum=array(0., dtype=float32), maximum=array(1000., dtype=float32))}).
以下是我的规格：
# MDP specs.
self._action_spec = array_spec.BoundedArraySpec(
shape=(), dtype=np.int32,
minimum=0, maximum=len(self.latent_abilities) + len(self.events) - 1,
name='action'
)
self._observation_spec = array_spec.BoundedArraySpec(
shape=(), dtype=np.float32,
minimum=0, maximum=1000,
name='observation_spec'
)
# self._observation_spec = array_spec.BoundedArraySpec(
# shape=(, len(self.latent_abilities) + len(self.events)*4 + len(self.latent_abilities) + len(self.events) + 1), dtype=np.float32,
# minimum=0, maximum=1000,
# name='observation_spec',
# )
# self._observation_spec = {
# # Latent ability development.
# **{
# f'dev_{ability}': {
# 'r': array_spec.BoundedArraySpec(
# shape=(), dtype=np.float32,
# minimum=0, maximum=1000,
# ),
# 'best': array_spec.BoundedArraySpec(
# shape=(), dtype=np.float32,
# minimum=0, maximum=1000,
# )
# }
# for ability in self.latent_abilities
# },
# # Event technical development.
# **{
# f'dev_{event}_tech': {
# 'r': array_spec.BoundedArraySpec(
# shape=(), dtype=np.float32,
# minimum=0, maximum=1000,
# ),
# 'best': array_spec.BoundedArraySpec(
# shape=(), dtype=np.float32,
# minimum=0, maximum=1000,
# )
# }
# for event in self.events
# },
# # Event development.
# **{
# f'dev_{event}': {
# 'r': array_spec.BoundedArraySpec(
# shape=(), dtype=np.float32,
# minimum=0, maximum=1000,
# ),
# 'best': array_spec.BoundedArraySpec(
# shape=(), dtype=np.float32,
# minimum=0, maximum=1000,
# )
# }
# for event in self.events
# },
# # Initial marks.
# **{
# f'{event}_i': array_spec.BoundedArraySpec(
# shape=(), dtype=np.float32,
# minimum=0, maximum=1000,
# )
# for event in self.events
# },
# # Optimal marks.
# **{
# f'{event}_o': array_spec.BoundedArraySpec(
# shape=(), dtype=np.float32,
# minimum=0, maximum=1000,
# )
# for event in self.events
# },
# # Investment in latent abilities.
# **{
# f'inv_{ability}': array_spec.BoundedArraySpec(
# shape=(), dtype=np.int32,
# minimum=0, maximum=1000,
# )
# for ability in self.latent_abilities
# },
# # Investment in technical abilities.
# **{
# f'inv_{event}_tech': array_spec.BoundedArraySpec(
# shape=(), dtype=np.int32,
# minimum=0, maximum=1000,
# )
# for event in self.events
# },
# # Time.
# 't': array_spec.BoundedArraySpec(
# shape=(),
# dtype=np.float32,
# minimum=0,
# maximum=self.time_bounds[1],
# )
# }
这是我的训练脚本：
# Training environment.
from training_env import CombinedEventEnvironment
# Utilities.
from util import (
preprocess_marks,
observation_from_state,
construct_state,
athlete_profile,
score_performances,
calc_score_combined_event,
simulated_competition,
)
# Reverb.
import reverb
# TensorFlow.
import tensorflow as tf
# TensorFlow Agents.
from tf_agents.agents.dqn import dqn_agent
from tf_agents.drivers import py_driver
from tf_agents.environments import tf_py_environment
from tf_agents.eval import metric_utils
from tf_agents.networks import sequential
from tf_agents.policies import (
py_tf_eager_policy,
random_tf_policy,
)
from tf_agents.replay_buffers import (
reverb_replay_buffer,
reverb_utils,
)
from tf_agents.trajectories import trajectory
from tf_agents.specs import tensor_spec
from tf_agents.utils import common
# Data processing.
import numpy as np
import pandas as pd
# General utilities.
import argparse
import json
# Arguments.
parser = argparse.ArgumentParser()
parser.add_argument('--events', '-e', default='assets/events_dec.json')
parser.add_argument(
'--event_improvement_directions',
'-eid',
default='assets/events_improvement_directions_dec.json'
)
parser.add_argument(
'--latent_abilities',
'-l',
default='assets/latent_abilities_dec.json'
)
parser.add_argument('--factor_loadings', '-fl', default='assets/factor_loadings_dec.json')
parser.add_argument('--coef_scoring', '-cs', default='assets/coef_scoring_dec.json')
parser.add_argument(
'--coef_dev_latent',
'-cdl',
default='assets/coef_dev_latent_dec.json'
)
parser.add_argument('--coef_dev_tech', '-cdt', default='assets/coef_dev_tech_dec.json')
parser.add_argument('--coef_dev_event', '-cde', default='assets/coef_dev_event_dec.json')
parser.add_argument('--marks', '-m', default='data/DecData.csv')
parser.add_argument('--profiles', '-prof', default=None)
args = parser.parse_args()
# Load assets.
with open(args.events) as _: events = json.load(_)
with open(args.event_improvement_directions) as _:
event_improvement_directions = json.load(_)
with open(args.latent_abilities) as _: latent_abilities = json.load(_)
with open(args.factor_loadings) as _: factor_loadings = json.load(_)
with open(args.coef_scoring) as _: coef_scoring = json.load(_)
with open(args.coef_dev_latent) as _: coef_dev_latent = json.load(_)
with open(args.coef_dev_tech) as _: coef_dev_tech = json.load(_)
with open(args.coef_dev_event) as _: coef_dev_event = json.load(_)
# Learning parameters.
num_iterations = 20_000
initial_collect_steps = 100
collect_steps_per_iteration = 1
replay_buffer_max_length = 1000 # 100_000
batch_size = 64
learning_rate = 1e-3
log_interval = 200
num_eval_episodes = 10
eval_interval = 1000
# Load data.
df = pd.read_csv(args.marks)
df_ath = pd.read_csv(args.profiles) if args.profiles is not None else None
df = preprocess_marks(df, events=events)
# Build environment.
env = CombinedEventEnvironment(
# Events.
events=events,
event_improvement_directions=event_improvement_directions,
# Factor structure.
latent_abilities=latent_abilities,
factor_loadings=factor_loadings,
# Scoring coefficients.
coef_scoring=coef_scoring,
# Development curve parameters.
coef_dev_latent=coef_dev_latent,
coef_dev_tech=coef_dev_tech,
coef_dev_event=coef_dev_event,
# Time bounds.
time_bounds=(5, 100),
# Discount factor.
gamma_=0.99,
# Data.
df=df,
df_ath=df_ath,
save_athlete_profiles=True,
)
env_train_py = CombinedEventEnvironment(
# Events.
events=events,
event_improvement_directions=event_improvement_directions,
# Factor structure.
latent_abilities=latent_abilities,
factor_loadings=factor_loadings,
# Scoring coefficients.
coef_scoring=coef_scoring,
# Development curve parameters.
coef_dev_latent=coef_dev_latent,
coef_dev_tech=coef_dev_tech,
coef_dev_event=coef_dev_event,
# Time bounds.
time_bounds=(5, 100),
# Discount factor.
gamma_=0.99,
# Data.
df=df,
df_ath=df_ath,
save_athlete_profiles=True,
)
env_eval_py = CombinedEventEnvironment(
# Events.
events=events,
event_improvement_directions=event_improvement_directions,
# Factor structure.
latent_abilities=latent_abilities,
factor_loadings=factor_loadings,
# Scoring coefficients.
coef_scoring=coef_scoring,
# Development curve parameters.
coef_dev_latent=coef_dev_latent,
coef_dev_tech=coef_dev_tech,
coef_dev_event=coef_dev_event,
# Time bounds.
time_bounds=(5, 100),
# Discount factor.
gamma_=0.99,
# Data.
df=df,
df_ath=df_ath,
save_athlete_profiles=True,
)
print(env_train_py.reset())
print(env_train_py.step(action=2))
train_env = tf_py_environment.TFPyEnvironment(env_train_py)
eval_env = tf_py_environment.TFPyEnvironment(env_eval_py)
# Build network.
fc_layer_params = (100, 50)
action_tensor_spec = tensor_spec.from_spec(env_train_py.action_spec())
num_actions = action_tensor_spec.maximum - action_tensor_spec.minimum + 1
print("Specs:")
print(action_tensor_spec)
print(env_train_py.time_step_spec())
def dense_layer(num_units):
return tf.keras.layers.Dense(
num_units,
activation=tf.keras.activations.relu,
kernel_initializer=tf.keras.initializers.VarianceScaling(
scale=2.0, mode='fan_in', distribution='truncated_normal',
)
)
dense_layers = [dense_layer(num_units) for num_units in fc_layer_params]
q_values_layer = tf.keras.layers.Dense(
num_actions,
activation=None,
kernel_initializer=tf.keras.initializers.RandomUniform(
minval=-0.03, maxval=0.03,
),
bias_initializer=tf.keras.initializers.Constant(-0.2)
)
q_net = sequential.Sequential(
layers=[
*dense_layers,
q_values_layer
],
input_spec=tf.TensorSpec(shape=(1,), dtype=tf.int32)
)
# Optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
train_step_counter = tf.Variable(0)
# Instantiate DQN agent.
agent = dqn_agent.DqnAgent(
train_env.time_step_spec(),
train_env.action_spec(),
q_network=q_net,
optimizer=optimizer,
td_errors_loss_fn=common.element_wise_squared_loss,
train_step_counter=train_step_counter,
)
agent.initialize()
# Set up policies for data collection and learning.
policy_collect = agent.collect_policy
policy_eval = agent.policy
policy_random = random_tf_policy.RandomTFPolicy(
train_env.time_step_spec(),
train_env.action_spec()
)
time_step = train_env.reset()
print("Random Policy Action:", policy_random.action(time_step))
# Average return.
def compute_average_return(env, policy, n_episodes=10):
"""
Computes the average return across all episodes.
"""
def episode():
time_step = env.reset()
episode_return = 0.0
while not time_step.is_last():
action_step = policy.action(time_step)
time_step = env.step(action_step.action)
episode_return += time_step.reward
return episode_return
total_return = sum([
episode()
for i in range(n_episodes)
])
return total_return/n_episodes
# Test.
print("Avg. return test:", compute_average_return(eval_env, policy_random, num_eval_episodes))
# Replay buffer.
table_name = 'uniform_table'
replay_buffer_signature = tensor_spec.from_spec(
agent.collect_data_spec
)
replay_buffer_signature = tensor_spec.add_outer_dim(
replay_buffer_signature
)
table = reverb.Table(
table_name,
max_size=replay_buffer_max_length,
sampler=reverb.selectors.Uniform(),
remover=reverb.selectors.Fifo(),
rate_limiter=reverb.rate_limiters.MinSize(1),
signature=replay_buffer_signature
)
# Create a server to collect data in our table.
reverb_server = reverb.Server([table])
# Create a replay buffer that lives on our server.
replay_buffer = reverb_replay_buffer.ReverbReplayBuffer(
agent.collect_data_spec,
table_name=table_name,
sequence_length=2,
local_server=reverb_server,
)
# Create an observer that drops data into that replay buffer.
rb_observer = reverb_utils.ReverbAddTrajectoryObserver(
replay_buffer.py_client,
table_name,
sequence_length=2,
)
# Collect data for training by exploring using the random policy.
py_driver.PyDriver(
env,
py_tf_eager_policy.PyTFEagerPolicy(
policy_random, use_tf_function=True
),
[rb_observer],
max_steps=initial_collect_steps
).run(env_train_py.reset())
# Out of curiosity:
print(iter(replay_buffer.as_dataset()).next())
我知道状态各组成部分的字段类型应该是什么，只是不确定如何正确嵌套它们，以便每个字段的结构和数据类型都匹配。我猜这与 _step() 或 _reset() 的返回值有关，因为它们提供下一个时间步。
任何帮助将不胜感激！
解决方案
我解决了这个问题。需要做的是：
首先，我意识到在观测返回类型的规范上犯了一个错误。我写了一个函数，从 _state 中提取相关信息来构造一个观测值，该观测值被表示为一个NumPy数组：
def observation_from_state(_state, latent_abilities, events) -> np.array:
"""
Converts the components of the state to observation format
for the TensorFlow Agents framework.
"""
# Initial performances.
i = [
_state[f'dev_{event}']['start']
for event in events
]
# Optima.
o = [
_state[f'dev_{event}']['best']
for event in events
]
# k parameters from event development curves.
k = [
_state[f'dev_{event}']['k']
for event in events
]
# Latent ability development.
inv_latent = [
_state[f'inv_{ability}']
for ability in latent_abilities
]
# Technical development.
inv_tech = [
_state[f'inv_{event}_tech']
for event in events
]
# Time budget.
t = _state['t']
# Combine these into a single array.
obs = np.array([
*i,
*o,
*k,
*inv_latent,
*inv_tech,
t
], dtype=np.float32)
return obs
非常重要的是，必须指定这个NumPy数组的 dtype。如果数据类型不匹配，无论你的 time_step_spec 与TimeStep在形状、界限等方面的匹配程度有多接近，观测值都必须具备合适的数据类型。
然后，我需要在 _reset() 和 _step() 的末尾返回那个观测值。例如，
obs_t = tf.convert_to_tensor(obs, dtype=tf.float32)
print("Tensor Spec from Tensor:", tf.TensorSpec.from_tensor(obs_t))
# Transition to next step.
if self._state['t'] == 0:
self._episode_ended = True
# return ts.termination(np.array([self._state], dtype=np.int32), reward)
return ts.termination(
# observation=self._state,
observation=obs,
reward=reward,
)
else:
# return ts.transition(np.array([self._state], dtype=np.int32), reward=0.0, discount=1.0)
return ts.transition(
# observation=self._state,
observation=obs,
reward=reward,
discount=self.gamma_
)
接着，我需要修改观测的规范，使其反映出合适的形状。
samp_obs = self.func_obs_from_state(
self._state,
latent_abilities=self.latent_abilities,
events=self.events,
)
# MDP specs.
self._action_spec = array_spec.BoundedArraySpec(
shape=(), dtype=np.int32,
minimum=np.array(0, dtype=np.int32),
maximum=np.array(
len(self.latent_abilities) + len(self.events) - 1, dtype=np.int32
),
name='action'
)
self._observation_spec = array_spec.BoundedArraySpec(
# shape=(), dtype=np.float32,
shape=(samp_obs.shape[0],), dtype=np.float32,
minimum=np.array(0, dtype=np.float32),
name='observation'
)
请注意，观测规范是通过对取样观测值计算观测的形状来实现的。这使得观测形式的变化能够自动改变观测规范。对于我正在做的这个项目很重要，但对许多使用场景来说可能并非如此。
站内所有文章版权归属LeftHeroAI导航站，无授权禁止任何主体转载、抄袭、复制内容，亦不得私自架设镜像站点。一经侵权，本站将通过法律途径追责。
相关文章
汇集全球数千款 AI 工具，每日持续更新，配套 AI 开发学习资源，轻松用 AI 提升工作效率。快捷键 Ctrl+D/⌘+D 一键收藏本站。