I recently migrated a Plone 3.1.7 website to Plone 5, now that was a task. A lot of the migration, especially between Plone 4 and 5 resulted in me writing my own migration steps to handle content. One thing that'll catch users off guard is that between Plone 3, 4 and 5 you have to import the correct profiles, it's amazing how many times I tried to debug an issue when in fact it was only because I hadn't imported something required for that release.
You'll probably also find packages that have become outdated, there were a few instances where I had to actually go in and update / modify the imports in files simply because that package release was outdated or not available.
Make sure to remove interfaces, especially old ones otherwise you'll get a crap load of PicklingError issues and so on, this is usually caused by packages not uninstalling fully. Unfortunately you just have to handle this, either by keeping the legacy package available in the upgraded site (not installing it) or by using something like wildcard.fixpersistentutilities to remove old or conflicting junk.
It's very possible you'll have to create your own migration steps, I usually handle this by creating a view, form and then some migration code to move or modify content which allows me to continue with the default migration path.
Don't forget @@custom_migrator or the @@atct_migrator, the former allowing you to generally migrate custom Content-Types to Plone defaults or to registered new ones.
Finally, don't forget to check the Plone portal_tools for old or unused content, old templates, theming files, overrides, installed products and so on. It's better to start the migration process with as clean of a Plone as you can, then work your way up migrating the content as you go. Anything else can be remade or fixed once the content is in Plone 5.
Though, you should provide a little more info so that we can help. "It never worked well" is a bit non-descriptive, do you have some logs or else..? Start with the migration and when you get an error, come here and post it - I'm sure we'll all be happy to help you through the process!