I would like to know when one should not use Transformable but should
instead create another entity and use To-many relationship.
You should only use transformable attributes when you absolutely have to. They are not a convenience or a short-cut but a resource intensive necessity in some circumstances.
You seldom use Core Data to store data structures like arrays or dictionaries because Core Data is primarily used for not for storage/persistence but rather for modeling/simulating. It's kind of useless for modeling data to turn an data structure into a big, logic-less blog of data.
Transformable attributes are usually used to store some class that itself actively manages the data it holds e.g. transforming a UIImage so that you can take an UIImage straight from the UI store and get it back all in one piece.
In answer to your main question:
I am more curious about how big the data can be until it affects
performance and it's better to normalize them.
It depends mostly on the combination of size and complexity. Whenever you transform a bunch of existing objects into a data blob, you have to read the entire blog in by transforming. So if you store a 1mb array by transformation you get a 1mb array back in memory when run the reverse transform. Each transform, no matter how small, takes more processing time than accessing a normal attribute or even finding another managed object. Therefore, having a large number of small transforms that are often accessed will also cause quite a performance hit.
It is always better to decompose big lumps of data into entities, attributes and relationships. Doing so gives you all the flexibility and optimizations of Core Data for free. I find myself using Core Data in place of arrays and dictionaries because once you really wrap your head around Core Data, its just easier to use.
I would never use Core Data to store a transformed array of strings or the like. If the strings have no logic and there is just few dozen of them, you might as well write the array out to plist file. It will be quicker and easier than messing around with a transformable attribute.