PyTorch: Sigmoid of weights?

Question

I'm new to neural networks/PyTorch. I'm trying to make a net that takes in a vector x, first layer is h_j = w_j^T * x + b_j, output is max_j{h_j}. The only thing is that I want the w_j to be restricted between 0 and 1, by having w_j = S(k*a_j), where S is the sigmoid function, k is some constant, and a_j are the actual weight variables (w_j is just a function of a_j). How do I do this in PyTorch? I can't just use a torch.nn.Linear layer, there has to be something else/additional to add in the sigmoid function on the weights?

Side question, for that last output layer, can I just use torch.max to get the max of the previous layer's outputs? Does that behave nicely, or is there some torch.nn.Max or some pooling stuff that I don't understand that needs to happen?

Thanks!

I am pretty confused - the layers output can be put through a sigmoid or any monotonic function that varies between 0 and 1. This is called an activation function, and there are a various set of functions that you can call (though I am not familiar with pytorch but I am almost positive that this will be built in - see here) - the last layer output can be put through a softmax function which is essentially a weighted exponential with the sum. — Chinny84
Softmax is what I was looking for (for the side question)! For the first part, the thing is that I don't want to put the layers output through the sigmoid, only the weight variables, which will then be used in the linear layer. — AWhite
Why would you want the weights between 0 and 1? Thats usually not how it works as far as I know — David Montgomery
@DavidMontgomery I was just about to comment something similar. Unless we are looking at some sort of regularisation process as per reg regression etc. — Chinny84
@Chinny84 It would also alter backpropagation so you'd have to think about that too — David Montgomery

Umang Gupta Umang Gupta · Accepted Answer · 2019-05-29T19:21:07

I am really not sure why would you do that but you can declare a custom layer as below to apply sigmoid to weights.

class NewLayer(nn.Module): 
    def __init__ (self, input_size, output_size): 
        super().__init__() 
        self.W = nn.Parameter(torch.zeros(input_size, output_size)) 
        # kaiming initialization (use any other if you like)
        self.W = nn.init.kaiming_normal_(self.W) 
        self.b = nn.Parameter(torch.ones(output_size)) 
    def forward(self, x): 
        # applying sigmoid to weights and getting results 
        ret = torch.addmm(self.b, x, torch.sigmoid(self.W)) 
        return ret

Once you do this, you can use this as you would use linear layer in your code.

PyTorch: Sigmoid of weights?

1 Answers