Google Summer of Code'23 at Ivy (unify.ai)
on
"Multi-backend framework support of GradSLAM in Ivy by implementing missing PyTorch APIs"


Contributor: Dhruv Rajendra Patel
Mentors: Aarsh Chaubey, Anna Tzatzopoulou, Rishabh Kumar

Introduction

GradSLAM -> an end-to-end differentiable SLAM library written in PyTorch

To develop multi-framework software support for GradSLAM in Ivy.

  • Implement missing PyTorch APIs in Ivy to develop multi-framework support.
  • Create a tutorial for running GradSLAM using Ivy.
  • Showcase better runtime performance of GradSLAM using JAX as the backend framework.

    • Frameworks (to be supported) - PyTorch, Tensorflow, JAX, NumPy

    Installing and Setting up your environment

    Setting up and Installing your environment - Ivy and GradSLAM

    About GradSLAM

    As shown above, the GradSLAM library consists of different modules. We show that only framework-specific or logic containing modules (geometry, structures, odometry, SLAM) require transpilation.

    Timeline and Completed Work

    A detailed documentation of the work done across the entire coding duration can be found here.

    GradSLAM modules

    Doing Basic API Testing
    An example of transpiling a geometry API is shown below:

    Here, param to in ivy.transpile can be set to any of ["torch", "tensorflow", "jax", "numpy"]

    1.) gradslam.geometry: [Jupyter Notebook]
  • Developed transpilation tests for all geometry APIs and all of them (except 2 as below) successfully pass for all frameworks.
  • [so3_hat, se3_hat] APIs from geometry.se3utils submodule fail due to torch.zeros transpilation issue in Ivy's graph compiler (more details in Spreadsheet). This will be available soon once fixed on Ivy's official repo.

    • 2.) gradslam.odometry: [Jupyter Notebook]
  • Developed Transpilation tests for both - high-level logic of ICP in form of classes and low-level util functions (gauss_newton_solve, solve_linear_system, downsampling, etc.).
  • APIs like solve_linear_system have been successfully tested.
  • Transpilation tests for the remaining APIs are also developed, but require some additional core dev and fixes (torch.inv JAX transpilation, Class Transpilation support, and key error issue) in Ivy's official repo.

    • 3.) gradslam.structures: [Jupyter Notebook]
  • Developed transpilation tests for high-level datastructures (PointClouds, RGBDImages) as well as utils.
  • Low-level APIs like list<->pad conversions are successfully tested, but awaiting some core fixes (class transpilation, OS APIs) for testing the remaining ones.

    • 4.) gradslam.slam: [Jupyter Notebook]
  • Similar to gradslam.odometry.
  • Developed transpilation tests for all APIs, but awaiting some core fixes from Ivy's official repo for testing.

    • Ivy Transpilation: API-level Documentation

    Ivy GradSLAM Documentation: Contains an extensive documentation describing Timeline, Completed Work, issues/errors, and more.
    Spreadsheet - Documentation of GradSLAM APIs testing for all frameworks. Comprises failures, PRs for resolving them and checklist of all the APIs.

    GradSLAM using Ivy

    A Jupyter Notebook for testing GradSLAM with different frameworks using Ivy.
    NOTE: The above notebook still has some required core dependencies on Ivy side, and is being worked upon by the official Ivy team. I will update the same here once the required Ivy dependencies/features are available on their repo and then this notebook can be used directly.

    Conclusion

  • Transpilation tests for all GradSLAM APIs are developed.
  • Most Low-level APIs are already tested successfully, and the remaining tests for high-level class logic are already developed and will be tested once we have class transpilation and other fixes like torch.zeros, torch.inv APIs in Ivy's graph compiler.
  • Final Inference time analysis of using GradSLAM with different frameworks will be performed soon.
    • Robotics Deep Learning SLAM PyTorch NumPy Tensorflow JAX