Thermal subsidence is the primary driver for sediment accumulation in passive continental margins. Accurately characterizing this process is essential for predicting basin evolution, sediment accumulation patterns, and hydrocarbon generation. However, traditional methods for constraining thermal subsidence often lack independent constraints on lithospheric stretching, limiting their predictive capabilities. We present a novel methodological approach for analyzing continental margin evolution. Integrating sediment and crustal thickness observations we calibrate a comprehensive analytical model. Our Non-Uniform, Magma-Assisted Rifting with Varying Thermal Diffusivity (NUMAR-VTD) model simultaneously parameterizes three key processes: non-uniform lithospheric stretching, crustal magmatic addition, and spatially variable thermal diffusivity. Applied to the Eastern North American Continental Margin, we use seismically-constrained crustal thicknesses as stretching indicators and Early-Middle Jurassic sediment distribution as a subsidence proxy. Through systematic parameter optimization, we identify parameter distributions that best reconcile model predictions with observations, uniquely quantifying the non-uniformity of stretching, magmatic contribution, and spatial variations in the effective thermal diffusivity. The results reveal anomalously fast cooling of the margin that cannot be explained by conductive heat transfer alone. Based on these findings, we suggest that hydrothermal fluid circulation accelerated cooling during the early post-rift phase, providing new insights for basin evolution prediction and hydrocarbon maturation modeling. The crustal-constrained subsidence modeling approach represents a significant advancement in quantitative basin analysis. It can readily be applied to passive margins worldwide, offering a powerful tool for unraveling the thermal histories of continental margins.