MineRL 0.2.0 [rebased] (#136)

* Add ability to observe placement of non-agent placeable blocks * Fix loud broken pipe error on data loading framework * Add action_space and observation_space to data pipeline * Made obsevation space a member of data class * Add spaces to package * Add other type to handlers * test handlers * Add debug flags to data pipeline to reduce typical printing * Enable debug flag in pipeline * use logger log levels * Expose logger * Unexpose logger * Add queue size parameter * Add debug statements * debug for days * debug for days * Add ability to observe placement of non-agent placeable blocks * Fix loud broken pipe error on data loading framework * Add action_space and observation_space to data pipeline * Made obsevation space a member of data class * Add other type to handlers * test handlers * Add debug flags to data pipeline to reduce typical printing * use logger log levels * Expose logger * Unexpose logger * Add queue size parameter * Reveering changes. * Addding data versioning. * Updating Malmo for gamma fix. * Update load_data to use current environment * Update load_data function * Align data based on actual number of frames in recording * A workable stream viewer! * Making stream viewer a module in minerl. * Adding a new streamviewier script. * viewier. * Windows requirement.txt * Updating. * Updating one more time. * Importing on windows. * need more color * Updating your cool applicaton.xtxt * yeah dude! * remove debug * Added version support to download script * Updating the head to support the new viewer. * Updating viewer to support reward plots. * Updating the new data viewer. * Add ability to return metadata in seq_iter * Fix metadata * Fix metadata * Pushing. code. * Update load data * Add metadat if not there * Fix * Deprecating seq_iter. * Increase timeout in download script closes #131 * Remove teleport observation for final state * Allowing for random streams. * Fix minerl data download version * Fix minerl data download version * Update documentation for data pipeline * Update setup.py * Delete mc_1.log * Update documentation for data itter * Updating the documentation. * Updating Malmo version * Updating the viewer for the new version of python3.5. Why oh why matplotlib :( * Adding a dependency for pySmartDL. * Adding new updates. * Yes.
minerllabs · Jul 12, 2019 · 4f05f0c · 4f05f0c
1 parent 913dd02
commit 4f05f0c
Show file tree

Hide file tree

Showing 18 changed files with 922 additions and 96 deletions.
diff --git a/.gitmodules b/.gitmodules
@@ -1,3 +1,6 @@
 [submodule "minerl/env/Malmo"]
 	path = minerl/env/Malmo
 	url = https://github.com/cmu-rl/Malmo.git
+[submodule "minerl/dependencies/pySmartDL"]
+	path = minerl/dependencies/pySmartDL
+	url = https://github.com/minerllabs/pySmartDL.git
diff --git a/docs/source/assets/cropped_viewer.gif b/docs/source/assets/cropped_viewer.gif
diff --git a/docs/source/assets/minerl_viewer.gif b/docs/source/assets/minerl_viewer.gif
diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -76,7 +76,7 @@ components:
    :caption: Tutorials and Guides
    :maxdepth: 2
 
-   tutorials/getting_started
+   tutorials/index
    tutorials/first_agent
    tutorials/data_sampling
 

diff --git a/docs/source/tutorials/data_sampling.rst b/docs/source/tutorials/data_sampling.rst
@@ -17,25 +17,94 @@ Now we can build the datast for :code:`MineRLObtainDiamond-v0`
 
 .. code-block:: python
 
-    data = minerl.data.make('MineRLObtainDiamond-v0')
+    data = minerl.data.make(
+        'MineRLObtainDiamond-v0', 
+        data_dir='/your/local/path')
     
     # Iterate through a single epoch gathering sequences of at most 32 steps
-    for obs, rew, done, act in data.seq_iter(num_epochs=1, max_sequence_len=32):
-        print("Number of diffrent actions:", len(act))
-        for action in act:
-            print(act)
-        print("Number of diffrent observations:", len(obs), obs)
-        for observation in obs:
-            print(obs)
-        print("Rewards:", rew)
-        print("Dones:", done)
+    for current_state, action, reward, next_state, done \
+        in data.sarsd_iter(
+            num_epochs=1, max_sequence_len=32):
 
+            # Print the POV @ the first step of the sequence
+            print(current_state['pov'][0])
 
+            # Print the final reward pf the sequence!
+            print(reward[-1])
 
+            # Check if final (next_state) is terminal.
+            print(done[-1])
 
+            # ... do something with the data.
+            print("At the end of trajectories the length"
+                  "can be < max_sequence_len", len(reward))
 
-.. note:: 
+
+.. warning:: 
     The :code:`minerl` package uses environment variables to locate the data directory.
     For portability, plese define :code:`MINERL_DATA_ROOT` as 
-    :code:`/your/local/path/data_texture_0_low_res` in your system environment variables.
+    :code:`/your/local/path/` in your system environment variables.
+
+
+
+=============================================================
+Visualizing The Data :code:`minerl.viewer`
+=============================================================
+
+To help you get familiar with the MineRL dataset,
+the :code:`minerl` python package also provides a data trajectory viewer called
+:code:`minerl.viewer`:
+
+
+.. image:: ../assets/cropped_viewer.gif
+  :width: 90 %
+  :alt: 
+  :align: center
+
+
+The :code:`minerl.viewer` program lets you step through individual
+trajectories, 
+showing the observation seen the player, the action
+they took (including camera, movement, and any action described by an MineRL
+environment's action space), and the reward they received.
+
+.. exec::
+
+    import minerl
+    import minerl.viewer
+
+    help_str = minerl.viewer.parser.format_help()
+
+    print(".. code-block:: bash\n") 
+    for line  in help_str.split("\n"):
+        print("\t{}".format(line))
 
+
+**Try it out on a random trajectory by running:** 
+
+.. code-block:: bash
+
+    # Make sure your MINERL_DATA_ROOT is set!
+    export MINERL_DATA_ROOT='/your/local/path'
+
+    # Visualizes a random trajectory of MineRLObtainDiamondDense-v0
+    python3 -m minerl.viewer MineRLObtainDiamondDense-v0 
+
+
+
+**Try it out on a specific trajectory by running:**
+
+.. exec::
+
+    import minerl
+    import minerl.viewer
+
+    traj_name = minerl.viewer._DOC_TRAJ_NAME
+
+    print(".. code-block:: bash\n")
+
+    print('\t# Make sure your MINERL_DATA_ROOT is set!')
+    print("\texport MINERL_DATA_ROOT='/your/local/path'")
+    print("\t# Visualizes a specific trajectory. {}...".format(traj_name[:17]))
+    print("\tpython3 -m minerl.viewer MineRLTreechop-v0 \\")
+    print("\t\t{}".format(traj_name))
diff --git a/docs/source/tutorials/getting_started.rst → docs/source/tutorials/index.rst b/docs/source/tutorials/getting_started.rst → docs/source/tutorials/index.rst
diff --git a/minerl/__init__.py b/minerl/__init__.py
@@ -1,2 +1,5 @@
+
+import minerl.dependencies 
 import minerl.data
-import minerl.env
+import minerl.env
+import minerl.env.spaces as spaces
diff --git a/minerl/data/__init__.py b/minerl/data/__init__.py
@@ -2,6 +2,9 @@
 from minerl.data.download import download
 import os
 
+from minerl.data.version import DATA_VERSION, FILE_PREFIX, VERSION_FILE_NAME
+
+import minerl.data.version
 
 def make(environment=None , data_dir=None, num_workers=4, worker_batch_size=32, minimum_size_to_dequeue=32, force_download=False):
     """
@@ -35,6 +38,9 @@ def make(environment=None , data_dir=None, num_workers=4, worker_batch_size=32,
             raise ValueError("No data_dir provided and $MINERL_DATA_ROOT undefined."
                              "Specify force_download=True to download default dataset")
 
+
+    minerl.data.version.assert_version(data_dir)
+
     d = DataPipeline(
         os.path.join(data_dir, environment),
         environment,