We present a new mechanism of small-scale transition via core dynamics instability (CDI) in an incompressible plane mixing layer, a transition which is not reliant on the presence of longitudinal vortices (‘ ribs’) and which can originate much earlier than rib-induced transition. Both linear stability analysis and direct numerical simulation are used to describe CDI growth and subsequent transition in terms of vortex dynamics and vortex line topology. CDI is characterized by amplifying oscillations of core size non-uniformity and meridional flow within spanwise vortices (‘rolls’), produced by a coupling of roll swirl and meridional flow that is manifested by helical twisting and untwisting of roll vortex lines. We find that energetic CDI is excited by subharmonic oblique modes of shear layer instability after roll pairing, when adjacent rolls with out-of-phase undulations merge. Starting from moderate initial disturbance amplitudes, twisting of roll vortex lines generates within the paired roll opposing spanwise flows which even exceed the free-stream velocity. These flows collide to form a nearly irrotational bubble surrounded by a thin vorticity sheath of a large diameter, accompanied by folding and reconnection of roll vortex lines and local transition. We find that accelerated energy transfer to high wavenumbers precedes the development of roll internal intermittency this transfer, inferred from increased energy at high wavenumbers and an intensification of roll vorticity, occurs prior to the development of strong opposite-signed (to the mean) spanwise vorticity and granularity of the roll vorticity distribution. We demonstrate that these core dynamics are not reliant upon special symmetries and also occur in the presence of moderate-strength ribs, despite entrapment of ribs within pairing rolls. In fact, the roll vorticity dynamics are dominated by CDI if ribs are not sufficiently strong to first initiate transition thus CDI may govern small-scale transition for moderate initial 3D disturbances, typical of practical situations. Results suggest that CDI constitutes a new generic mechanism for transition to turbulence in shear flows.