Infinities that converge towards a finite number are not an issue, nor are infinite limits in calculus (indeed, they are immensely useful, such as for the integration of the Gaussian Distribution e^(-x^2) which is extremely common in wavefunction calculus yet has no known solution if not integrated with infinite limits). I feel that infinities in mathematics are more "How big/small can we make this until making it any bigger/smaller has no more bearing on our final answer?", so they aren't technically infinite in that sense, to me. This is pretty much the crux of calculus wherein we toy about with values that, small as they may be, still aren't exactly infinity's alter ego 0. I feel that infinity (and 0?) are simply useful mathematical tools, they don't really have a place in nature.

As for the size of the universe... Mmm, don't actually know that one. I'm not nearly confident enough in the physics of cosmology to give an answer to call my own! Maybe that could be considered a different sort of infinity, though. In dimensions of infinite expanse, physical laws acting within are not comprimised (i.e. You don't need to put limits on your Cartesian set of axis when plotting the curve of a projectile), but if infinities occur in certain observable properties then we definitely have a problem (i.e. The projectile must have a boundary on its velocity or mass, else infinities arrise in Kinetic Energy)