How to predict a layer's weights using another model in an end to end fashion?

Question

I am working on visual question answering task and i want to reproduce the work in this paper https://arxiv.org/pdf/1511.05756v1.pdf where they predict a layer's weights in a network using the question embedding. i wonder can i do that in keras?
i tried that and i know that for each tensor there is an argument weight ex: Dense(128 , weight= ) but it expect a real number, and i want to train the network in an end to end.