Simple answer: You should use RecyclerView in a situation where you want to show a lot of items, and the number of them is dynamic. ListView should only be used when the number of items is always the same and is limited to the screen size.
You find it harder because you are thinking just with the Android library in mind.
Today there exists a lot of options that help you build your own adapters, making it easy to build lists and grids of dynamic items that you can pick, reorder, use animation, dividers, add footers, headers, etc, etc.
Don't get scared and give a try to RecyclerView, you can starting to love it making a list of 100 items downloaded from the web (like facebook news) in a ListView and a RecyclerView, you will see the difference in the UX (user experience) when you try to scroll, probably the test app will stop before you can even do it.
I recommend you to check this two libraries for making easy adapters:
FastAdapter by mikepenz
FlexibleAdapter by davideas