Asynchronous Inference Execution with TFLite Model

Hi I’m currently using the TFLite benchmark tool to measure the performance of my model. I’m interested in executing some of the graph’s operations asynchronously. I noticed that TensorFlow Lite seems to support asynchronous operations under tensorflow/tensorflow/lite/core/async/, but I’m unsure how to utilize this in my benchmarking process.

Could anyone provide guidance or examples on how to implement asynchronous inference execution using TFLite? Any help or pointers would be greatly appreciated. Thank you!

Hi @rita19991020 ,

TFLite asynchronous execution is right now under experimental phase. Once it is fully functional we will get back to you or track the link.

However, for the time being, for your specific usecase if you would like to proceed with Asynchronous execution, modify benchmarking tool’s code by introducing delays between certain operations in the graph.

Thank You