This article points to a connection between chlorinated pools and asthma The authors state that "The respiratory effects of chlorine on swimmers are mainly caused by irritant compounds in the water or air just at the surface, which swimmers actively inhale as water, aerosols or gases. In backyard outdoor pools, the main irritant is probably free chlorine itself. In public pools, other irritants are released on reaction of chlorine with organic matter." I think what the authors mean to say is that we use the word "chlorine" to mean the whole parade of horrible chlorine-containing chemicals at a pool... including chlorine as well as chlorinated organic materials like chloramines.
Dr. Bernard explained that "[p]hysicians should recommend to their patients with asthma, to avoid poorly managed pools with excessive chlorine levels. Clues to identifying these pools are a strong chlorine smell in pool air and irritating effects on the skin, eyes or upper airways." As far as I can tell, the smell comes from the same body of chlorine-containing compounds that irritate lungs, skin, eyes, hair, etc.
Interestingly, the effect of chlorine on the airways of young children is not immediate, said Dr. Bernard, which may be why some studies showed no association between chlorinated pool attendance and asthma in schoolchildren. All the more reason to be careful before symptoms occur. Also, given the health problems linked to chlorine, it would probably make sense to get the chlorine off after swimming. If you smell like chlorine, it's because you are still covered in chlorine.