Topologically associating domains (TADs) are important building blocks of three-dimensional genome architectures. The formation of TADs has been shown to depend on cohesin in a loop-extrusion mechanism. Recently, advances in an image-based spatial genomics technique known as chromatin tracing lead to the discovery of cohesin-independent TAD-like structures, also known as single-cell domains, which are highly variant self-interacting chromatin domains with boundaries that occasionally overlap with TAD boundaries but tend to differ among single cells and among single chromosome copies. Recent computational modeling studies suggest that epigenetic interactions may underlie the formation of the single-cell domains. Here we use chromatin tracing to visualize in female human cells the fine-scale chromatin folding of inactive and active X chromosomes, which are known to have distinct global epigenetic landscapes and distinct population-averaged TAD profiles, with inactive X chromosomes largely devoid of TADs and cohesin. We show that both inactive and active X chromosomes possess highly variant single-cell domains across the same genomic region despite the fact that only active X chromosomes show clear TAD structures at the population level. These X chromosome single-cell domains exist in distinct cell lines. Perturbations of major epigenetic components and transcription mostly do not affect the frequency or strength of the single-cell domains. Increased chromatin compaction of inactive X chromosomes occurs at a length scale above that of the single-cell domains. In sum, this study suggests that single-cell domains are genome architecture building blocks independent of the tested major epigenetic components.