commit 20dff1e593de7799b06b1b6dd2299404664d8810
parent dac6a95df8de38079ff93afc0f63d6f7513c5ca1
Author: William Lindholm <85635561+LindholmLabs@users.noreply.github.com>
Date: Wed, 11 Dec 2024 23:29:10 +0000
Skapades med Colab
Diffstat:
1 file changed, 11 insertions(+), 63 deletions(-)
diff --git a/CS4287_Prj2_24273759_24293059_24284335.ipynb b/CS4287_Prj2_24273759_24293059_24284335.ipynb
@@ -37,7 +37,7 @@
"\n",
" Inspiration taken from: Lecture P: RL – DQN and the Cartpole\n",
"\n",
- " understaning of Cartpole problem: https://www.gymlibrary.dev/environments/classic_control/cart_pole/"
+ " understanding of Cartpole problem: https://www.gymlibrary.dev/environments/classic_control/cart_pole/"
],
"metadata": {
"id": "zHY7sojS4vsX"
@@ -805,9 +805,7 @@
"\n",
"The training performance indicates that the agent starts with poor rewards around -200, showing no success in the early episodes as it explores the environment. Around episode 100, the agent begins to show improvement, with rewards fluctuating and occasionally exceeding -150. Around episode 200, some progress is made, and the agent achieves higher rewards (closer to -100) in some episodes, but overall it is still quite inconsistent. This means that the agent is learning and improving but has not fully converged on an optimal policy. Additional training, fine-tuning of hyperparameters, or adjustments to the exploration strategy could help stabilize and improve its performance further.\n",
"\n",
- "**Reflections**\n",
- "\n",
- "In short, we are quite disappointed with our results. We underestimated the time required for training, not leaving us enough time to fine tune our model, or accurately display results. We did however see indications of learning, and managed to improve it from our starting point.\n"
+ "\n"
],
"metadata": {
"id": "R9XZ5zqr1sW1"
@@ -823,73 +821,23 @@
}
},
{
- "cell_type": "code",
+ "cell_type": "markdown",
"source": [
- "# Reset the environment and render it to see the initial state\n",
- "state = env.reset()\n",
- "env.render()\n",
- "total_reward = 0\n",
+ "The agent is learning, as indicated by our plot, and we managed to reduce the amount of catastrophic forgetting by a lot.\n",
"\n",
- "for step in range(max_steps):\n",
- " # Choose action using the greedy policy (no exploration, only exploitation)\n",
- " action = epsilon_greedy_policy(state, epsilon=0)\n",
+ "**Reflections**\n",
"\n",
- " # Take the action in the environment and observe the result\n",
- " state, reward, done, _ = env.step(action)\n",
+ "But, we are quite disappointed with our results. We underestimated the time required for training, not leaving us enough time to fine tune our hyper parameters, or accurately display results. We did however see indications of learning, and managed to improve it from our starting point.\n",
"\n",
- " # Render the environment to visualize the agent's actions\n",
- " env.render()\n",
- " total_reward += reward\n",
+ "**Possible Improvemens**\n",
"\n",
- " # End the evaluation if the episode is done\n",
- " if done:\n",
- " break\n",
+ "We think its possible to improve the model by furher experimenting with the hyper parameters, or maybe implementing double Q-learning to maybe get a more stable learning process (Lecture Q). As we also did not see the model fully converge, we think letting it run for more episodes would also produce a better agent.\n",
"\n",
- "print(f\"Total Reward During Evaluation: {total_reward}\")\n",
- "env.close()"
+ "To further test the agent, producing more plots, such as a clearly showing the epsilon value over episodes, how the learning rate changes would be benefitial. Perhaps also showing some screenshots from the game to give the reader a better understanding could be helpful, but here we ran into some trouble with not finding correct documentation in time for the older version of gym."
],
"metadata": {
- "id": "MP0_RNylvX5Q",
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "outputId": "ee516172-f099-416a-88a8-1d94f01db426"
- },
- "execution_count": null,
- "outputs": [
- {
- "output_type": "stream",
- "name": "stderr",
- "text": [
- "/usr/local/lib/python3.10/dist-packages/gym/core.py:49: DeprecationWarning: \u001b[33mWARN: You are calling render method, but you didn't specified the argument render_mode at environment initialization. To maintain backward compatibility, the environment will render in human mode.\n",
- "If you want to render in human mode, initialize the environment in this way: gym.make('EnvName', render_mode='human') and don't call the render method.\n",
- "See here for more information: https://www.gymlibrary.ml/content/api/\u001b[0m\n",
- " deprecation(\n",
- "/usr/local/lib/python3.10/dist-packages/pygame/pkgdata.py:25: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html\n",
- " from pkg_resources import resource_stream, resource_exists\n",
- "/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py:3154: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.\n",
- "Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages\n",
- " declare_namespace(pkg)\n",
- "/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py:3154: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.cloud')`.\n",
- "Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages\n",
- " declare_namespace(pkg)\n",
- "/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py:3154: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.\n",
- "Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages\n",
- " declare_namespace(pkg)\n",
- "/usr/local/lib/python3.10/dist-packages/gym/core.py:49: DeprecationWarning: \u001b[33mWARN: You are calling render method, but you didn't specified the argument render_mode at environment initialization. To maintain backward compatibility, the environment will render in human mode.\n",
- "If you want to render in human mode, initialize the environment in this way: gym.make('EnvName', render_mode='human') and don't call the render method.\n",
- "See here for more information: https://www.gymlibrary.ml/content/api/\u001b[0m\n",
- " deprecation(\n"
- ]
- },
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "Total Reward During Evaluation: -135.0\n"
- ]
- }
- ]
+ "id": "7fCDUpb5-8KN"
+ }
}
],
"metadata": {
