RESULTS:

Microbiome profiling tools rely on reference catalogues which significantly affect their performance. Comparing them is however challenging mainly due to differences in their native catalogues. In this study we present a novel standardized benchmarking framework that makes such comparisons more accurate. We decided not to customize databases but to translate results to a common reference to use the tools with their native environment. Specifically we conducted two realistic simulations of gut microbiome samples each based on a specific taxonomic profiler and used two different taxonomic references to project their results namely the Genome Taxonomy Database and the Unified Human Gastrointestinal Genome. To demonstrate the importance of using such a framework we evaluated four established profilers as well as the impact of the simulations and that of the common taxonomic references on the perceived performance of these profilers. Finally we provide guidelines to enhance future profiler comparisons for human microbiome ecosystems: (i) use or create realistic simulations tailored to your biological context (BC) (ii) identify a common feature space suited to your BC and independent of the catalogues used by the profilers and (iii) apply a comprehensive set of metrics covering accuracy (sensitivity/precision) overall representativity (richness/Shannon) and quantification (UniFrac and/or Aitchison distance).