How to use Boost Spirit with variant with more than 20 types?

Question

I'm parsing a quite complex grammar with Boost Spirit and I'm facing a problem with a variant that have more than 20 types (21 here):

namespace eddic { namespace ast {

typedef boost::mpl::vector<
            Integer,
            IntegerSuffix,
            Float,
            Litteral,
            VariableValue,
            DereferenceValue,
            Expression,
            Unary,
            Null,
            True,
            False,
            ArrayValue,
            FunctionCall,
            MemberFunctionCall,
            Cast,
            BuiltinOperator,
            Assignment,
            SuffixOperation,
            PrefixOperation,
            Ternary
        > types_initial;

typedef boost::mpl::push_back<types_initial, New>::type types;
typedef boost::make_variant_over<types>::type Value;

}}

Boost Spirit does not recognize the last type (eddic::ast::New) added with push_back. When I parse something that have this element, it fails with this error:

eddic: /usr/include/boost/variant/detail/visitation_impl.hpp:264: typename Visitor::result_type boost::detail::variant::visitation_impl(int, int, Visitor&, VoidPtrCV, mpl_::false_, NoBackupFlag, Which*, step0*) [with Which = mpl_::int_<0>; step0 = boost::detail::variant::visitation_impl_step, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0l>, boost::mpl::v_iter, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, boost::mpl::v_item, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 0>, 21l> >; Visitor = boost::variant, boost::mpl::vector, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Null, eddic::ast::True, eddic::ast::False, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred >, 0> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>::assigner; VoidPtrCV = const void*; NoBackupFlag = boost::variant, boost::mpl::vector, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Null, eddic::ast::True, eddic::ast::False, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred, eddic::ast::Deferred >, 0> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>::has_fallback_type_; typename Visitor::result_type = void; mpl_::false_ = mpl_::bool_]: Assertion `!"Boost.Variant internal error: 'which' out of range."' failed.

If I swap two elements (Ternary and New for example), New is correctly recognized, but not Ternary. It is only the last element that fails.

I already tried using push_front or vector21, but it doesn't change anything, it's always the last element that have problem. In my opinion it comes from the fact that Spirit uses a variant internally before copying it to my variant_over type.

Is there a workaround to this problem ?

I could probably reduce the number to 20, but the problem is that I will certainly have more than that in the future.

Thanks a lot for any idea

Any reason not to mix with classic OO? Roll all operations in a single AST type (polymorphically), all function calls likewise etc. It almost looks as though you are including details at the tokenizer level in your ast (IntegerSuffix - really?) — sehe
I'm not sure I do understand. You mean an inheritance tree ? Like Value being the base class of Integer ? — Baptiste Wicht
I think you get the idea. The thing is, having a humongous variant is pretty much a code smell in my experience. — sehe
@BaptisteWicht, How did your AST end up? I am designing my AST at the moment and will soon be experiencing the very same issue. — PeteUK
@PeteUK I've been able to simplify it a bit (github.com/wichtounet/eddic/blob/develop/include/ast/Value.hpp). I still think it can be improved, especially regarding the constants and literals that could be merged probabled. However, I don't want to make the processing of the AST more complex. — Baptiste Wicht

Ylisar Ylisar · Accepted Answer · 2012-07-24T12:30:41

Define BOOST_MPL_LIMIT_VECTOR_SIZE to whatever new limit you want, but going this high is usually a hint at design problems so it might be worth to give it some consideration.

How to use Boost Spirit with variant with more than 20 types?

1 Answers