Parameter tuning is a key part of dimensionality reduction via deep variational autoencoders for single cell RNA transcriptomics

Qiwen Hu; Casey S. Greene

doi:10.1101/385534

CARVIEW

MOTORHOMES

Select Language

HTTP/2 301 date: Sun, 18 Jan 2026 10:00:42 GMT content-length: 0 location: https://doi.org/10.1101/385534 server: cloudflare vary: Origin expires: Mon, 19 Jan 2026 10:00:42 GMT permissions-policy: interest-cohort=(),browsing-topics=() cf-cache-status: DYNAMIC nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} strict-transport-security: max-age=31536000; includeSubDomains; preload report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=jOAiYyw8CKnTeZBIhNNwnvIJfveWkv2SJJoUvSYd8i1Voqr%2BX4U%2BBr2XIivv3Th%2Fd3x6lZ0ZjOu6NnXvDlrabKO%2FuIoETw%3D%3D"}]} cf-ray: 9bfd3f3119853585-BLR alt-svc: h3=":443"; ma=86400 HTTP/2 302 date: Sun, 18 Jan 2026 10:00:42 GMT content-type: text/html;charset=utf-8 location: https://biorxiv.org/lookup/doi/10.1101/385534 server: cloudflare vary: Origin vary: Accept expires: Sun, 18 Jan 2026 10:07:48 GMT permissions-policy: interest-cohort=(),browsing-topics=() cf-cache-status: DYNAMIC nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} strict-transport-security: max-age=31536000; includeSubDomains; preload report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=EXErkyZsv4HZTvgGieeTQNmG%2BnNFVVvF4gIPLqy4C9zPjJQqUo%2FR1k9TelwAUnaXruWZhBvUs%2FTUkxLb059tjV%2FKxpXgFg%3D%3D"}]} cf-ray: 9bfd3f32cbab3585-BLR alt-svc: h3=":443"; ma=86400 HTTP/1.1 302 Found Date: Sun, 18 Jan 2026 10:00:43 GMT Content-Type: text/html; charset=iso-8859-1 Transfer-Encoding: chunked Connection: keep-alive server: cloudflare location: https://www.biorxiv.org/lookup/doi/10.1101/385534 cf-cache-status: DYNAMIC Nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} Report-To: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=zNubkas7XudUBbBvCFxUa6TVZItyfcuol3XWhtwPc6DlJmDqd2VCX1Hv7Q4e20APZ%2B2gwWhQiiC1K25m0w8lCNnaCmxa%2FhPARGry"}]} CF-RAY: 9bfd3f334c93ff6a-BOM alt-svc: h3=":443"; ma=86400 HTTP/2 301 date: Sun, 18 Jan 2026 10:00:44 GMT content-type: text/html; charset=UTF-8 location: https://www.biorxiv.org/content/10.1101/385534v4 cf-ray: 9bfd3f369d0f58e1-BLR x-content-type-options: nosniff x-content-type-options: nosniff x-drupal-cache: MISS expires: Sun, 18 Jan 2026 10:30:44 GMT cache-control: public, max-age=1800 pragma: no-cache vary: Accept-Encoding x-highwire-sitecode: biorxiv x-highwire-smart-code: biorxiv_production x-varnish: 1892489322 via: 1.1 varnish x-varnish-ttl: x-varnish-cache: cf-cache-status: MISS set-cookie: __cf_bm=FfCEnu0A4e.sIJwTF87xFqGdDM3YfhwLh_PYjwAuXWU-1768730444-1.0.1.1-DxVtwLbUSBtT2sSR0Unvn16Tm2MkSB8ZjT.hSsWhM5YzgmSzjvvq5yJGDjdsrHMw7U7ZM_g_gNnXHoSX._S4JRRZLnFhZZY.h2.OLUu.5XY; path=/; expires=Sun, 18-Jan-26 10:30:44 GMT; domain=.www.biorxiv.org; HttpOnly; Secure; SameSite=None server: cloudflare HTTP/2 200 date: Sun, 18 Jan 2026 10:00:46 GMT content-type: text/html; charset=utf-8 content-encoding: gzip x-content-type-options: nosniff x-content-type-options: nosniff x-drupal-cache: MISS expires: Sun, 19 Nov 1978 05:00:00 GMT cache-control: no-cache, must-revalidate set-cookie: SSESS1dd6867f1a1b90340f573dcdef3076bc=PyUpel3jizvNxurw9u9RHXu_6MbSZx5aMn9tdwQSYP4; expires=Tue, 10-Feb-2026 13:34:04 GMT; path=/; domain=.biorxiv.org; secure; HttpOnly content-language: en x-frame-options: SAMEORIGIN x-generator: Drupal 7 (https://drupal.org) link: ; rel="canonical",; rel="shortlink" vary: Accept-Encoding x-highwire-sitecode: biorxiv x-highwire-smart-code: biorxiv_production x-varnish: 694617580 age: 0 via: 1.1 varnish x-varnish-ttl: x-varnish-cache: cf-cache-status: DYNAMIC server: cloudflare cf-ray: 9bfd3f3ede1a58e1-BLR Parameter tuning is a key part of dimensionality reduction via deep variational autoencoders for single cell RNA transcriptomics | bioRxiv

Abstract

Single-cell RNA sequencing (scRNA-seq) is a powerful tool to profile the transcriptomes of a large number of individual cells at a high resolution. These data usually contain measurements of gene expression for many genes in thousands or tens of thousands of cells, though some datasets now reach the million-cell mark. Projecting high-dimensional scRNA-seq data into a low dimensional space aids downstream analysis and data visualization. Many recent preprints accomplish this using variational autoencoders (VAE), generative models that learn underlying structure of data by compress it into a constrained, low dimensional space. The low dimensional spaces generated by VAEs have revealed complex patterns and novel biological signals from large-scale gene expression data and drug response predictions. Here, we evaluate a simple VAE approach for gene expression data, Tybalt, by training and measuring its performance on sets of simulated scRNA-seq data. We find a number of counter-intuitive performance features: i.e., deeper neural networks can struggle when datasets contain more observations under some parameter configurations. We show that these methods are highly sensitive to parameter tuning: when tuned, the performance of the Tybalt model, which was not optimized for scRNA-seq data, outperforms other popular dimension reduction approaches – PCA, ZIFA, UMAP and t-SNE. On the other hand, without tuning performance can also be remarkably poor on the same data. Our results should discourage authors and reviewers from relying on self-reported performance comparisons to evaluate the relative value of contributions in this area at this time. Instead, we recommend that attempts to compare or benchmark autoencoder methods for scRNA-seq data be performed by disinterested third parties or by methods developers only on unseen benchmark data that are provided to all participants simultaneously because the potential for performance differences due to unequal parameter tuning is so high.

Footnotes

Address reviewer concerns. The primary change is that there is now a discussion of the No Free Lunch theorem. Also there are clarity improvements throughout.

The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY 4.0 International license.