Can't Eject External USB HDD

I often open content on my (Toshiba) portable hard drive from applications on the local hard drive. In WinXP, I didn't have any problems ejecting the portable drive once I was done and closed the application. ...To be clear, when I did get an "in use" error message, I would simply go into Task Manager and see what offending application processes were still running; once they were stopped, the drive ejected.

Now on Windows 7, the system will not let me properly eject the drive even though the Task Manager shows that I have no applications open on the computer. Each time I try, whether from the taskbar icon or the Devices window, it gives the message that the device cannot be ejected at this time because it's in use. As DaPiit said, this is really annoying because attaching and ejecting USB devices is something I do many times per day! The Windows 7 is not on my machine - I still run WinXP, and this more than anything else is making me REALLY consider finally switching to a Mac instead. (And I've been a Windows user for over 15 years...)

Is there some Windows process that needs to be stopped that I don't know about? I'm really reluctant to just "pull the plug" every time I need to remove the drive.