Throw out bearing or Input shaft bearing?

So for some time now my 03 Jeep Tj has made this awful grinding noise. It is
hard to determine where exactly it is coming from but it is definitely between
the transfer case and the engine... which could be just about anything.
When it is in neutral it makes a slight grinding or knocking noise and when I
push in the clutch the noise sometimes becomes less noticeable but this is
not always the case, sometimes there will be no noise at all. the noise is the
worst when put in reverse and I let off the clutch. It also happens when put
into first gear and I let off the clutch but it is not as loud. I am thinking it is
the throw out bearing considering the clutch has 160,000 miles on it and
has not been replaced. the clutch slips on occasion which is why I was
leaning more in that direction. Any Ideas of what it could be?