The $ operator is not yet documented, but in the examples you can see it's usage. It seems to be similar to MongoDB's $ operator, if you're familiar with that one. In MongoDB, and apparently here, $ stands for "the value of the current element in the list". E.g., the documentation snippet will create six jobs, three running on Ubuntu 16.04 and three on Ubuntu 18.04.
The documentation isn't quite complete, and that the way $ operator is used in the documentation snippet you quoted will change in the future. The snippet should have been:
runs-on: ${{ matrix.os }}
strategy:
matrix:
node: [6, 8, 10]
os: [ubuntu-16.04, ubuntu-18.04]
steps:
- uses: actions/setup-node@v1
with:
version: ${{ matrix.node }}
Although it would be pretty simple for the GitHub Actions parser to figure out which list of strings is supposed to be assigned to the runs-on parameter (since 6, 8, and 10 are not valid values for runs-on), it cannot know what the valid values are for the version input of setup-node. (Which, BTW, has been renamed to node-version as of 2019-08-14, but the documentation snippet hasn't been updated yet). There is nothing in the action.yml config file that lists valid values of inputs, only names and descriptions.
And so since this documentation snippet wouldn't work as-is, I'm guessing that it will be updated at some point to say that $ is the current set of selected matrix variables, and ${{ variableName }} (e.g., ${{ matrix.node }} or ${{ matrix.os }}) is what you should put in the parts of your job config that use those variables.
As for the second part of your question, "How would a custom action look like that works with the build matrix option?", I think it would look like a normal action, documenting its inputs in action.yml (see the action.yml for setup-node v1 for an example of what this looks like). Then when you call it in your job, you'd do:
strategy:
matrix:
someVariable: [foo, bar]
steps:
- uses: my-actions/[email protected]
with:
someInput: ${{ matrix.someVariable }}
