Log into your yahoo account, then go to the site explorer page.
https://siteexplorer.search.yahoo.com/mysites
Submit your site, verify your site, and then submit your site map or the RSS feed.
This is kinda like yahoos version of google webmaster tools. Once you get the site explorer up and running, you will be able to get more information directly from yahoo. Then you will have some real information to work with. This would be a lot better then trying to guess what the problem is.
Site explorer says it takes about 24 hours to verify your site, but once it does you will be able to see all kinds of information - such as backlinks, indexed pages, known pages,,,,,,,,
Under the Statistics of each site you will be able to see:
Indexed Pages:
Crawled Pages:
Known Pages:
Known Hosts on This Site:
Hosts linking to Site:
Domains linking to Site:
Hosts Outlinked from Site:
Domains linked from Site: