A few years back, I read an article that theorized that plants evolved their distinctive edges so that amphibians that were trying to eat them would hurt their soft mouths and go on to easier-to-eat plants. I, of course, cannot find the article today.
While I think the idea is fascinating, my fundamental problem with it is that I haven’t seen much, outside of certain oaks and holly, that show transition from a jagged-edge leaf to something more like a thorn. In fact, thorns seems to either be adapted from the whole leaf or from a branch, not from leaf “feature creep”, so to speak.
Granted, I’m not a botanist, and there could be whole families of plants I’m not thinking about. If you know of any family that shows evolution from a relatively flat and smooth leaf to something that clearly seems designed for defense, I’d be very interested in hearing about it.