You may have very well made it with the same carry on bag but were the number of passengers, and their carry ons the same too? The size restrictions are enforced by the Department of Transportation. We dont make the rules. In addition, you can't expect the agents to break the rules because another agent somehwere else broke the rules.
You are allowed a carry on that is up to 45 linear inches. It is stated on the contract of carraige when you purchased the ticket.
http://www.aa.com/aa/i18nForward.do?...nAllowance.jsp
After a few years of "eye balling" bags some of us are pretty good at being eble to tell if the bag is too big or not. If you have doubt, then ask that the bag be measured. However, if the bag still does not fit becuase the flight is full, you will still have to check it.
Extortion would be if you have absolutely no choice and had to pay. This is America, you don't have to fly if you don't like the rules. The other carriers will have the same rules too.