Since I've not used the NookPress formatter yet, and probably won't, I couldnt tell you how to make your TOC work like you want it to. However, it sounds to me like you probably need to hire a formatter. Someone who knows what they're doing and can do as you want.
Out of curiosity, how are you labeling all your hyperlinks, all as H1? or are you using the sub headings 2,3,4 etc? It makes a difference.
I can't be of any help with Open Office, as I don't use anything but Word. . . Sorry.
I may be telling you what you already know but I use OpenOffice too, so here's what I do...
I click beside each place in the doc that I will be linking from the TOC to and I use the "insert" and dropdown option - "bookmark". I then name them things such as "P1" (Part One) and "P1C1" (Part One, Chapter One), to place in the bookmark-naming bar and then press "OK". I always save the bookmarks before I use the hyperlinking tool.
When I use the "hyperlink" tool, also in the "Insert" dropdown menu (after first highlighting what I want to turn into a hyperlink), it always brings up one little window first and inside of that one, I click the little target icon (looks like a bullseye target). This then brings up another little window that allows me to click the bookmarks dropdown menu. I then click the correct bookmark and then the "apply" prompt, afterward, also clicking the "apply" prompt found in the other little window (first click "apply" in the small window & secondly in the larger window). This then makes the hyperlink and they become blue-underlined. Sometimes the blue-underlining doesn't appear until you close the doc and reopen it.
You then just repeat the process for each hyperlink. I's funny how these type things are difficult to describe in writing. Must be why business writers who write product descriptions and directions make good money.
Sorry if you didn't need this but it might help someone else reading-in.
Thanks, Y'all. I thought about adding h1, h2 and h3 tags to each section, chapter and exercise to create a hierarchy, but then got caught up in other experirements. That's probably the way to do it. Indents in the TOC could be causing problems, too. I'm working on four projects, but I'll get to this eventually and report back.
Indents are really a problem at Amazon KDP because they display the free viewing portion, from the perspective of regular Kindles, rather than Kindle Fires. They incorporated a default indentation into the publishing platform which I don't like. It indents the blurb portion of ebooks I submit from OpenOffice, in wacky ways if I don't go in and use the "Manual Break" prompt for line breaks and page breaks.
You can look at a doc and it looks like these things are separated correctly but the mock kindle viewer in the publishing platform doesn't properly recognize breaks, unless done via the "Manual Break" tool. I find the opposite to be true with the former Pubit platform -- it sometimes ran paragraphs and lines together if they weren't separated using the "Manual Break" tool. The point being that word programs vary a lot in their ability to help you format ebooks and can be frustrating.