I'm a non-smoker and don't like to be around smoke. If a business is even the least bit smoky, I won't do business with them. That's my choice. However, I also consider the property rights of the business owners. If they clearly indicate that theirs is a smoking establishment, then anyone who enters know what they're getting. That also goes for anyone who chooses to work there. By the same token, if a business decides to be smoke-free, then smokers know they aren't welcome to light up inside.

Just as you described, people don't have the right to go into a privately owned public establishment and hurt others. By the same token, why should they have to right to dictate how the owners operate their business? As long as the customers have the free will to take their business elsewhere (which is a good reason why government buildings should be smoke free), then the rights of the owners should be taken into consideration.