New heterogeneous computing paradigms on dedicated hardware with increased parallelization offer exciting solutions with large potential gains. The growing applications of machine learning algorithms in particle physics for simulation, reconstruction, and analysis are naturally deployed on such platforms. The acceleration of machine learning inference as a web service represents a heterogeneous computing solution for particle physics experiments that requires minimal modification to the current computing model. Coprocessors deployed as an edge or cloud service for the particle physics computing model can have a higher duty cycle and are potentially much more cost-effective. Initial results with Microsoft Brainwave FPGAs and Nvidia GPUs show more than an order of magnitude reduction in inference latency and high throughput. The demonstrated performance is suitable to address the computing challenges faced by both energy frontier and intensity frontier experiments, including the HL-LHC detectors and DUNE.