Identity mask for numpy ndarray

Question

I would have expected True to preserve a ndarray when used as a mask, however, it adds a dimension, just like None.

arr = np.arange(16).reshape(2, 4, 2)
np.all(arr[True] == arr)         # outputs: True

Close enough, however looking closer:

arr[True].shape                  # outputs: (1, 2, 4, 2)
arr[None].shape                  # outputs: (1, 2, 4, 2)

I found two ways to set an identity mask: using slice(None) or Ellipsis.

np.all(arr[slice(None)] == arr)  # outputs: True
arr[slice(None)].shape           # outputs: (2, 4, 2)

np.all(Ellipsis == arr)          # outputs: True
arr[Ellipsis].shape              # outputs: (2, 4, 2)

Nothing really surprising here as this is how slicing works in the first place. slice(None) is a tad ugly and Ellipsis seems a wee bit faster.
However, going through:

NumPy's doc on indexing

I am not sure I fully understand this:

Deprecated since version 1.15.0: In order to remain backward compatible with a common usage in Numeric, basic slicing is also initiated if the selection object is any non-ndarray and non-tuple sequence (such as a list) containing slice objects, the Ellipsis object, or the newaxis object, but not for integer arrays or other embedded sequences.

I understand that the best way to preserve an array is not to mask it, but say I really want to setup a default value for a mask... ;-)

Question: Which is the preferred way to setup an identity mask ? And if I may, is True adding a dimension the intended behavior ?

The 2 that work are the same as arr[:] and arr[...]. The return a view. Boolean/mask indexing produces a copy. — hpaulj
@anon01 they are two distinct objects pointing to the same memory. In other words they share data but not metadata. — Paul Panzer
@anon01 You can try looking at arr.flags vs q.flags If arr was a "clean" array they should have different OWNDATA bits. Also, their .base attributes could be different. But even if their metadata happen to have identical values they are not shared. — Paul Panzer
Ellipsis is the most general way of indexing an array, and returning the same values. It won't be the same array (in the is sense), but will be an identical view. slice(None) is pretty general as well, but fails with 0d arrays ('scalar'). — hpaulj
I'm not saying the behavior is useful, but it is perfectly logical: The invariant is that if data are ND and mask is MD then each bit in the mask applies to an (N-M)D chunk of data. The resulting array has 1+N-M dimensions, because masking flattens, i.e. the masked dimensions are changed into a linear index. In your example True is a scalar aka 0D array. Output should therefore have 1+N-0 dimensions which is what you are seeing. — Paul Panzer

user2357112 supports Monica user2357112 supports Monica · Accepted Answer · 2020-10-09T01:41:56

You keep saying "mask", but it doesn't sound like you really want a masking operation at all, even an "identity" mask. A mask array would typically be a boolean array of the same shape as the original array, and indexing with the mask would produce a 1D array with items selected by the mask. Even an all-true mask would produce a flattened copy of the array it was applied to. It wouldn't be an identity operation. It's possible to do weirder things with masks, but not an identity operation.

If you want an indexer that outputs an equivalent array to the original, the typical, most general way to do that would be ... - a literal ellipsis:

arr[...]

Unlike :, this also works for 0-dimensional arrays. Note that this produces a view, not a copy. There is no indexer that would produce a copy and work properly for all input dimensions.

arr[True] works like it does primarily out of a desire to have 0-dimensional arrays follow the same boolean indexing rules as positive-dimensional arrays. As mentioned above, if you index an n-dimensional array with an n-dimensional mask, the result is a 1-dimensional array. If you index a 0-dimensional array with a 0-dimensional mask, the result is again a 1-dimensional array:

In [1]: import numpy

In [2]: x = numpy.array([[1, 2], [3, 4]])

In [3]: x[x % 2 == 0]
Out[3]: array([2, 4])

In [4]: y = numpy.array([1, 2, 3, 4])

In [5]: y[y % 2 == 0]
Out[5]: array([2, 4])

In [6]: z = numpy.array(5) # 0-dimensional!

In [7]: z[z % 2 == 0]
Out[7]: array([], dtype=int64)

In [8]: z[z % 2 == 1]
Out[8]: array([5])

Indexing a 0-dimensional array with a 0-dimensional mask increases the dimensionality by 1. Generalized to higher dimensions, indexing an n-dimensional array with a 0-dimensional mask produces an n+1-dimensional array. If the mask is True, the extra dimension has length 1; if the mask is False, the extra dimension has length 0, and the output has no elements. This generalized behavior is rarely useful, but it's what fits best with the (rarely useful) rules for applying a positive-dimension mask to an array with mismatching dimensions.

Identity mask for numpy ndarray

2 Answers