40,000!!!! Yikes! You win! Dont think have seen anyone with that many books before!
Trying to "scroll" through that many documents would be painful (even if it didnt lock up.)
1) Have you tried the "Search" feature? Finding what you want that way?
2) What about the "Open Explorer" app? (Yes, I am assuming that you are using some kind of sub-file structure to manage that many documents/books.)
3) Since you're sideloading most of them, you could try the Calibre Companion app. Have no idea how it will respond to that many documents though. If the app works, would give you ability to manage/find docs through use of metadata.
SD card classes are write speed, not read speed, so the class has nothing to do with how fast your device indexes the files.
Class 10 cards are rated on the basis of streaming a single large file (video) to a freshly formatted card.
Class 6/4/2, OTOH, are rated for accessing a fragmented, multiple file condition (photos and music, where one may have deleted some items then copied more stuff to the card).
While it would be nice to assume that a Class 10 will be faster than a Class 6 when using it for general small file I/O, the specification does not require that.
Probably you should just forget about using the Library thing. Install ES File Explorer and use that to search for and open books. Alternatively, some reader apps have their own library management (but they can be slow too - FBReader is a decent app in general, for example, but it can't handle lots of books without slowing to a crawl).