Task Execution with Semantic Segmentation Environment (TESSE) CORE

TESSE is an open-source Unity plugin that enables users to develop robotic, computer vision and reinforcement learning technologies. The large amount of data often required for these applications makes real-world data collection difficult; as an alternative, this simulation framework allows researchers and practitioners to generate large datasets in varied environments to develop these technologies.

The core features of TESSE include the following:

• A robot agent is core to the TESSE plugin. We currently provide a pill robot, with more robots to come.
• There is access to multiple agent cameras: red-green-blue (RGB), depth, and semantic segmentation.
• There is access to physics information for the agent (e.g., position, velocity, collisions).
• A user can configure various aspects of the agent, such as camera locations, image sizes, and toggling collisions.
• There are various control paradigms for the agent.
• There are various time paradigms, as well.

Network interfaces are available to send data back and forth between Unity. The tesse-interface provides a Python package that uses the network interface. This package is compatible with multiple versions of Python and has minimal dependencies.

The capabilities of TESSE encompass many aspects of similar systems used for robotics and reinforcement learning in a single framework. However, existing systems often require multiple packages and a large amount of software integration to deliver the same set of functionality. With TESSE, we strive to have few software dependencies and provide a flexible interface so that we support a wide range of use cases without enforcing a stringent workflow.