This study investigates the stability of a mixed convective flow of a nanofluid through a horizontal porous layer using an unsteady Jeffrey-Darcy model. Linear stability theory is employed to assess the stability of a system, where the base fluid is modeled as a Jeffrey fluid with dispersed nanoparticles in a state of thermal equilibrium. The stability equations are derived as an eigenvalue problem using Fourier decomposition, solved using the higher order Weighted Residual Galerkin Method (WRGM), and validated analytically. The results are presented in terms of critical values of the Darcy-Rayleigh number, wave number, and wave speed over nondimensional parameters. Further, the impact of nondimensional numbers like horizontal pressure gradient, thermal diffusivity ratio, and nanoparticle volume fraction has stabilizing effects, whereas the Jeffrey parameter and the Vadasz number have the opposite effect. Moreover, it has been observed that the increase in the Jeffrey parameter reduces the stability region. The variation Jeffrey parameter causes a change in the flow and thereby discards the analytical proof of stability even under the limit of an infinite Vadasz number. The inquiry into the stability or instability of the fundamental flow is addressed by solving the eigenvalue problem numerically over a finite range of the Jeffrey parameter and horizontal pressure gradient. These results indicate that the oscillatory convection mode is advantageous for estimating the required volume fraction of nanoparticles in the base fluid to improve the thermal efficiency of Jeffrey nanofluids. Numerical and graphical analyses explore the impacts of dimensionless parameters on physical systems, providing insights into the system’s stability properties under varying conditions.