First, have you formatted your player to make sure it is the true memory? If not, follow the instructions posted here
How To: Removing Memory Hack
Second, does your screen size match what you are using in the converter?
Also, for a reason I have yet to figure out, some movies need to be split into two parts. I've had longer movies convert with no problem, but every once in a while I get an error. Can't remember if it was converter 3.68, but I had trouble with one of them. Try downloading
Virtual Dub to split the movie into two parts.
Lastly, you might try downloading another AMV converter from the downloads section and seeing if that does the trick.
Hope this helps>