Julia - dataframe - Same column in a multiple input function with by()

Question

I am trying to explore the capabilities of the DataFrames.jl module. I ran into the following issue when trying to pass the same column into a multiple input variable in the by() function.

My base example is :

df = DataFrame(grp = rand(["a","b"], 100), x= rand(100), y = rand(100), z=rand(100))
by(df, :grp,result= (:x, :z) => ((x, y),) -> cov(x, y))

Giving the following dataframe

2×2 DataFrame
│ Row │ grp    │ result     │
│     │ String │ Float64    │
├─────┼────────┼────────────┤
│ 1   │ a      │ -0.0111914 │
│ 2   │ b      │ -0.0184773 │

Now lets assume that I am not necessarily working with cov() and that I would like to pass the same column x as both the x and y inputs of that function. In the case of cov() , it is a trivial one, but trying to be as generic as possible.

I have tried the following two possibilities

by(df, :grp,result= (:x) => ((x, y),) -> cov(x, y))

Which gives the following error message :

ERROR: MethodError: no method matching cov(::Float64, ::Float64)

The error in this case is expected and I imagine that method refers to the data provided to the function object

I have also tried the following

by(df, :grp,result= (:x,:x) => ((x, y),) -> cov(x, y))

Which gives the following error message

ERROR: ArgumentError: Elements of Symbol[:x, :x] must be unique

This time I understand the error message, but I don't understand why Symbol must be unique. I have checked ?Symbol but couldn't find more details on why or how to bypass this issue (limitation?). Effectively this prevents me from passing the same column programatically for both inputs.

So what would be the way to address this issue and be able to pass the same column twice for a function with f(x,y) ?

PS: Ahead of questions or comments on in this case (x)->cov(x,x) will work. I am aware that it will. But lets say I have a function that will compute the cov() or other functions for the (selected) columns of a dataframe I would prefer not to have to handle special cases for the diagonal items.

Bogumił Kamiński Bogumił Kamiński · Accepted Answer · 2020-02-15T11:08:11

The error with

by(df, :grp,result= (:x) => ((x, y),) -> cov(x, y))

is due to the following way Julia processess arguments:

julia> f((x,y),) = @show (x,y)
f (generic function with 1 method)

julia> f([1,2,3,4,5])
(x, y) = (1, 2)
(1, 2)

so x gets a first element of the passed vector stored in column :x and :y gets the second value.

Note that if your df were shorter than 2 rows you would even get an error:

julia> by(first(df,2), :grp,result= (:x) => ((x, y),) -> println((x,y)))
ERROR: BoundsError: attempt to access 1-element view(::Array{Float64,1}, [1]) with eltype Float64 at index [2]

Now moving to the duplicate column issue. The problem is that by passes to your function a NamedTuple, which you can check by running e.g. the following code:

julia> df = DataFrame(x=[:a,:b,:a,:b], y=1:4)
4×2 DataFrame
│ Row │ x      │ y     │
│     │ Symbol │ Int64 │
├─────┼────────┼───────┤
│ 1   │ a      │ 1     │
│ 2   │ b      │ 2     │
│ 3   │ a      │ 3     │
│ 4   │ b      │ 4     │

julia> by(df, :x, (:x, :y) => x -> @show typeof(x));
typeof(x) = NamedTuple{(:x, :y),Tuple{SubArray{Symbol,1,Array{Symbol,1},Tuple{Array{Int64,1}},false},SubArray{Int64,1,Array{Int64,1},Tuple{Array{Int64,1}},false}}}
typeof(x) = NamedTuple{(:x, :y),Tuple{SubArray{Symbol,1,Array{Symbol,1},Tuple{Array{Int64,1}},false},SubArray{Int64,1,Array{Int64,1},Tuple{Array{Int64,1}},false}}}

and NamedTuple does not allow duplicate columns:

julia> (a=1,a=2)
ERROR: syntax: field name "a" repeated in named tuple

Actually we are discussing in DataFrames.jl currently to switch from passing a NamedTuple to auto-splatting, see the following issue.

Julia - dataframe - Same column in a multiple input function with by()

1 Answers