An inguinal hernia may extend to the scrotum, but not always. Many are exactly where you describe, and having pain or discomfort doesn't occur in all hernias.
I can't think of a single other explanation for swelling in that area. Best to see your doctor soon. :D
Shannon Morgan, CMA
Certified Medical Assistant, Internal Medicine.