Although I don't know the program intimately, one would assume that you're trying to convert a copy protected dvd, which is encrypted to prevent you from doing so, so you would first need to circumvent the protection to do so.
If indeed it is NOT a copywritten file, then the name of the program assumes that you would be converting avi's to another type, not other types TO avi!
Your best bet would be to get tmpgenc, it's demoware, and if your .vob is convertable, it'll do it for sure.
http://www.tmpgenc.net/en/index.html