Here, we evaluated biospecimens of combined infrapatellar fat pad (IPFP) and synovial tissue from human healthy donors (HDs) or patients with osteoarthritis (OA). Using both single-cell RNA sequencing (scRNA-seq) and single-nucleus RNA sequencing (snRNA-seq) approaches, we identified mesenchymal and myeloid cell subsets in IPFP and synovial tissues and profiled pathological changes in OA. By incorporating previously reported scRNA-seq datasets of human articular cartilage, we conducted in silico analysis to predict cell-cell interactions within the joint. Transcriptomic data, in silico signaling predictions, and a murine collagenase-induced arthritis model suggest a critical role for apolipoprotein E (APOE) in the IPFP and synovium in OA-induced cartilage degeneration. Our results generated a cell atlas of human IPFP and synovium under healthy and OA conditions and revealed a potential target for future research on OA treatment.