Performing machine learning with analog signals offers advantages in speed and energy efficiency, but is extremely sensitive to component and measurement imperfections, which grow in importance when the desired update signal (gradient of the error) is successfully reduced, often requiring extensive digital modeling to overcome. Here we take a different perspective, accepting these inherent imperfections and ultimately overcoming them without digital models. We train an analog network of self-adjusting resistors—a contrastive local learning network—for regression and classification, and observe limit cycles and scaling behaviors driven by these imperfections that limit precision, erase memory of previous tasks, and are absent in ‘perfect’ systems. We develop an analytical model capturing these phenomena as a consequence of an uncontrolled learning bias continuously modifying the underlying representation of learned tasks, reminiscent of representational drift in the brain. Finally, we introduce and demonstrate a system-agnostic training method, which we call “overclamping”, that greatly suppresses these effects by shrinking the update duration (learning rate) rather than the magnitude of the gradient signal, preventing the dominance of unwanted dynamics and leaving desired updates largely unchanged. Our work points to a new, scalable analog approach that eschews precise modeling and instead thrives in the mess of real systems.