For the '86-'03 models, the swingarm has only one bearing. It is on the right side pivot. The left side pivot uses a bushing, not a bearing. For the 2004-later models, the swingarm uses a bearing in each side.

Notes gathered from the XLFORUM:

• There is a pivot shaft that supports the swing arm bearings/bolts on the outside, and goes through the rear mounting lug on the tranny case. If the pivot shaft is loose in the rear mount lug, it will need to be replaced. ¹⁾
• Symptoms of swingarm problems: ²⁾
  • Excessive play in the rear wheel, difficulty keeping in a straight line in slow moving traffic.
  • Spherical Bearings having play in them.
  • Rear rubber mounts well out of shape.
  • Torx bolt head shearing off.
• All of the rubber mounts are open to this problem… Because of the fact that the swing arm is mounted to the mount… An it's rubber-mounted with a spherical bearing… It's got some “flex” stock most don't notice it… Till its really bad! ³⁾

• Replace the swing arm bearing when you have it apart… and above all else… follow the TSB for reinstalling the rear brake don't cross thread the pivot bolt. ⁴⁾

• When you have the swingarm out, use a few sockets and some all thread (a bolt could work as well) in a vise to press the bearings out. ⁵⁾ There might be some surface rust in there, just soak it with some rust solvent for a couple of hours before you press it. ⁶⁾

Broken Pivot Bolts:

• I think the design issue is that the pivot shaft is too small for the hole it goes through in the engine. It should be a tighter fit like an axle through a bearing. The purpose of the lock plate is to keep the pivot shaft from spinning so you can tighten the swingarm bolts. I think it does that job just fine. But, because the pivot shaft is too small, when you install the lock plate, the lock plate centers the pivot shaft in its hole. So the engine transmits its forces not directly into the pivot shaft, but rather into the lock plate bolts, then into the lock plate, then finally into the pivot shaft. Those little lock plate bolts cant handle that kind of stress so one or two of them break. I think a better fix would be a larger pivot shaft, or a shim or bushing of some sort to take up the space between the pivot shaft and its bore. ⁷⁾
• I agree that a tighter pivot shift fit would help but the plate also holds against lateral shift of the shaft to the left under hard cornering loads. The combination of that force plus wiggling of the shaft in the motor opening definitely contributes to the problem ⁸⁾

If you are replacing the bearings, it's worth reading this extensive thread about replacing the engine rubber mounts: http://xlforum.net/forums/showthread.php?t=1526368