I'm afraid I drained both the transmission and differential in my wife's '04 xB automatic.I drained diff by also removing the vertical plug near the transmission drain plug. From these posts it appears differential and trans do not share fluid and needs gear-oil in diff:

https://www.scionlife.com/forums/sci...change-109452/
Differential Oil? Auto? Manual? - Scion xB Forum

Where is the fill plug for differential?

Thanks.

 

Looks as if one of the links you posted has the answer you were looking for...

"...the automatics (both U340E and U441E) use gear oil (with extreme pressure additive) in the diff, which has a separate oil supply from the autobox itself. The plug on the middle of the rear edge of the diff housing is both level and filler plug, to drain the old gear oil out, you'd need a syphon pump, refill with either EP90, or Redline's excellent EP140."

Fill (and drain plug according to this post) is on the middle of the rear edge of the diff housing.

Although looking at the Service Manual for the xB it doesn't show a separate differential for an automatic that I can see and according to the following document there is no call-out for differential fluid on a 2006 xB with an automatic.

Toyota and Scion Fluid Requirements

Can you snap a picture of where you think you drained the 'diff fluid' and did it look significantly different than the tranny fluid?