Apple forbids using private or undocumented APIs in iOS apps. Any calls you make to methods that have the same name as private or undocumented API methods will be flagged as a private API use, even if the method being called is something you have defined yourself.
App Loader does an initial scan, checking for method names, instance variable access, and even @selector usage with private method names. App Loader doesn't always do a great job, and the more source files you have the more likely it is to give you the warning that the API analysis file it has generated is "too large".
Fortunately, you can still submit your application, despite of the warning. Apple will check it internally, and if something gets kicked back because of overlapping names, you'll have to wade through the review process again.
Erika Sadun tried to make an app called API Kit that would do the scanning for you, but she appears to have abandoned her work and removed any trace of the application from her website.
Chimp Studios created App Scanner to do the same thing, but it hasn't been updated since 2011. Unfortunately, for large projects -- and this includes projects with a lot of extra pods from CocoaPods -- there is no current (2014) good way of solving this problem other than proactively naming things such that they won't conflict with private API method and instance names.
You can proactively learn about Apple's Cocoa Naming Conventions and try to anticipate. That will reduce future headaches. Until Apple introduces something like namespaces, however, we may continue to run into this problem from time to time.
The "invalid binary" error can come from a number of causes, but it is entirely unrelated to the API analysis document created by App Loader.
You should know that even with the scanning, there are still ways to get around the prohibition on using private/undocumented APIs. :)