It's more that all the interesting things that come from prime numbers don't really work if you include 1 as a prime. So every time you talked about primes you would have to say "something something prime number, other than 1, yadda yadda".
For example: The Fundamental Theorem of Arithmetic states that every natural number can be expressed as a finite product of primes, and that prime factorization is unique.
So
12 = 2^2 x 3
is the only way to express 12 as a product of primes. But if we include 1 as a prime, then
12 = 1 x 2^2 x 3
= 1^2 x 2^2 x 3
= 1^3 x 2^2 x 3
etc. There are infinite prime factorization of every natural number, and the most interesting part of the Fundamental Theorem of Arithmetic no longer holds.
So you're saying I also need to eat half a jar of pickles