Galaxy sizes and their evolution over cosmic time have been studied for decades and serve as key tests of galaxy formation models. However, at z ensuremathgtrsim 1 these studies have been limited by a lack of deep, high-resolution rest-frame infrared imaging that accurately traces stellar mass distributions. Here, we leverage the new capabilities of the James Webb Space Telescope (JWST) to measure the 4.4 ensuremathμm sizes of åisebox-0.5ex~1000 galaxies with $mathrmlogM_* /M_ødot ≥slant 9$ and 1.0 ensuremathłeq z ensuremathłeq 2.5 from public CEERS imaging in the Extended Groth Strip deep field. We compare the sizes of galaxies measured from NIRCam imaging at 4.4 ensuremathμm (ensuremathłambda $_rest$ i̊sebox-0.5ex~ 1.6 ensuremathμm) with sizes measured at 1.5 ensuremathμm (ensuremathłambda $rest$ rs̊ebox-0.5ex~ 5500 r̊). We find that, on average, galaxy half-light radii are raib̊ox-0.5ex~9% smaller at 4.4 ensuremathμm than 1.5 ensuremathμm in this sample. This size difference is markedly stronger at higher stellar masses and redder rest-frame V - J colors: galaxies with M $*$ raiso̊x-0.5ex~ 10$^11$ M $_ensuremathødot$ have 4.4 ensuremathμm sizes that are raisex̊-0.5ex~30% smaller than their 1.5 ensuremathμm sizes. Our results indicate that galaxy mass profiles are significantly more compact than their rest- frame optical light profiles at cosmic noon, and demonstrate that spatial variations in age and attenuation are important, particularly for massive galaxies. The trend we find here impacts our understanding of the size growth and evolution of galaxies, and suggests that previous studies based on rest-frame optical light may not have captured the mass-weighted structural evolution of galaxies. This paper represents a first step toward a new understanding of the morphologies of early massive galaxies enabled by JWST’s infrared window into the distant universe.