The OED confirms that the phrase construction “x by x” first appears in English in Langland’s Piers Plowman (the year: 1393). The phrase “little by little” itself first appears in English in 1483. In this: Catholicon Anglicum: an English-Latin wordbook.
Little by little, the phrase “little by little” seemed to have entered the English vernacular c. 1865.