Building up on @Batuu's reply, to be just left with the content as text just update the style to
Updated
<Style x:Key="OnOffToggleStyle" TargetType="ToggleButton">
<Setter Property="Content" Value="Off"/>
<Setter Property="Template">
<Setter.Value>
<ControlTemplate TargetType="{x:Type ToggleButton}">
<ContentPresenter />
</ControlTemplate>
</Setter.Value>
</Setter>
<Style.Triggers>
<Trigger Property="IsChecked" Value="True">
<Setter Property="Content" Value="On">
</Setter>
</Trigger>
</Style.Triggers>
</Style>
Addition here compared to original reply is
<Setter Property="Template">
<Setter.Value>
<ControlTemplate TargetType="{x:Type ToggleButton}">
<ContentPresenter />
</ControlTemplate>
</Setter.Value>
</Setter>
You basically set the template of the button to just be the content that it holds.