Figured it out. When they did the extension they added two home runs. One to the hot tub, and one to the basement breaker. Since I hadn't looked at the main box in a while I knew they added 100A, but I forgot they were split at the main, not at the "new" box. I picked up a 30A breaker for the charger and will run a 50A home run in a few months when my wife has the work she wants done.
I think I see you first problem. In the sub panel the wires to you 14-50 outlet seem to be wired one to the 50 amp 2 pole breaker and one to a 15 amp single pole above the br2 pole breaker. But The sub panel is only a 40 amp feed from your main panel. Then you have all the single pole loads to add. Also the wires from the main to the sub are smaller than your charging outlet.