I'm a night-owl so I almost never get up for family breakfast. Schedules work really well for my kids, but I chafe under them. I wanted to totally "unschool" 12 years ago when I first read about it, but my little ones craved a routine. It's there to help them, but I don't obsess over it.
Some mornings I'm still not awake even when the snacks are over and it's nearly 10 a.m. and the kids want to get going with school. The oldest ones start doing their next assignments. The littler ones play. They all know we'll start opening exercises when Mommy is good and ready.
After that, anything goes. I keep an order in my head of what to do next, just so I can refer to it when people are hanging around doing nothing and need some guidance, and to keep me from leaving out anything vital. This morning at 11 I felt terribly hungry and wondered why and then realized I had completely forgotten to ever have any breakfast.
Sometimes we switch around the order of subjects and I just have to keep track of what's been done and what hasn't. Sometimes I spend all morning giving a little (little?) lecture of reminiscences, imparting my lifetime of wisdom to my darlings.
Then I might skip that day's lesson for everyone on whatever subject I decide to count my lecture as. Sometimes they beg for more chapters at lunch and we spend half the afternoon on reading aloud just for the joy of it!
My laxness of schedule has occasionally worried DH. He has questioned me (grilled me) at times, harangued me at other times. But I've always been able to make him see reason.
If I lie in bed for a half hour tossing and turning at night, I follow my mom's great advice and get up again and do one more useful thing. Sometimes that one thing takes 3 minutes, other times it takes 3 hours. Whatever!
I never wake anyone up unless they have somewhere to be. If they sleep through chores, either those who are awake take up the slack or else the chores get done later. The kids do seem to thrive on schedules for the most part though.
I don't think you're alone, and I really doubt that you're ruining your DD.