That's a very good point. If the seeds people get from a seed library tend to be unreliable the idea will not thrive in the long term. A regular education offer about seed saving will help people and plants. Or people could note on their contributed seeds if their seed saving efforts are advanced or just 'I collected a seed'.

I could imagine a sort of 'plant patronage' in a seed library where groups of people specialize in and care for certain plants or groups of plants.