Try (just cut and paste this):
Needs["Notation`"]
Notation[ParsedBoxWrapper[
RowBox[{"A_", " ", "\[FilledSmallCircle]", " ",
"B_"}]] \[DoubleLongLeftRightArrow] ParsedBoxWrapper[
RowBox[{"Which", "[",
RowBox[{
RowBox[{
RowBox[{"(",
RowBox[{
RowBox[{"MatrixQ", "[", "A_", "]"}], "||",
RowBox[{"VectorQ", "[", "A_", "]"}]}], ")"}], "&&",
RowBox[{"(",
RowBox[{
RowBox[{"MatrixQ", "[", "B_", "]"}], "||",
RowBox[{"VectorQ", "[", "B_", "]"}]}], ")"}]}], ",",
RowBox[{"A_", " ", ".", "B_"}], ",",
RowBox[{
RowBox[{"!",
RowBox[{"(",
RowBox[{
RowBox[{"MatrixQ", "[", "A_", "]"}], "||",
RowBox[{"VectorQ", "[", "A_", "]"}]}], ")"}]}], "&&",
RowBox[{"(",
RowBox[{
RowBox[{"MatrixQ", "[", "B_", "]"}], "||",
RowBox[{"VectorQ", "[", "B_", "]"}]}], ")"}]}], ",",
RowBox[{"A_", "[", "B_", "]"}], ",",
RowBox[{
RowBox[{"(",
RowBox[{
RowBox[{"MatrixQ", "[", "A_", "]"}], "||",
RowBox[{"VectorQ", "[", "A_", "]"}]}], ")"}], "&&",
RowBox[{"!",
RowBox[{"(",
RowBox[{
RowBox[{"MatrixQ", "[", "B_", "]"}], "||",
RowBox[{"VectorQ", "[", "B_", "]"}]}], ")"}]}]}], ",",
RowBox[{"Transpose", "[",
RowBox[{"B_", "[",
RowBox[{"Transpose", "[", "A_", "]"}], "]"}], "]"}]}], "]"}]]]
Now I've entered this with the Notation palette, so actually it looks like this on screen:
(the palette inserts various boxes where necessary). It just looks horrible when I cut and paste due to the explicit string representation of everything.
EDIT: That is: type "Needs["Notation"]`, causing a palette to appear. Click on the first button, whereupon this

appears. Inside the first yellow box type A_ \[FilledSmallCircle] B_, and in the second,
Which[(MatrixQ[A_]||VectorQ[A_])&&(MatrixQ[B_]||VectorQ[B_]),A_ .B_,!(MatrixQ[A_]||VectorQ[A_])&&(MatrixQ[B_]||VectorQ[B_]),A_[B_],(MatrixQ[A_]||VectorQ[A_])&&!(MatrixQ[B_]||VectorQ[B_]),Transpose[B_[Transpose[A_]]]]
The result looks like this

and, when evaluated, defines what you want. Alternatively, after the Needs bit, just cut and paste what I gave above.
A_and the\[FilledSmallCircle]) asTimes[\[FilledSmallCircle], Pattern[A, Blank[]], Pattern[B, Blank[]]]. You need to create an infix notation. See the various answers below. - Simon