1
$\begingroup$

I have a script in R/RStudio which creates random datasets of binomial variables, feeds them through a neural network, and calculates their likelihood ratio statistic and deviance. I'd like the script to loop with the seed incremented by 1, and collect the value from each run into a data frame, from which the values get averaged. For example, if the script returns 5, 10, and 15 on the first, second, and third iterations of the loop, I'd like the script to create a data frame with those values and then calculate their mean.

Here is my script, annotated:

library(neuralnet)

x=1021

#change nrow to number of times code is looped
result <- data.frame(matrix(nrow = 1000, ncol = 2)) 
colnames(result) <- c("LogL", "D")

repeat{

set.seed(x)

#2 inputs, max number of terms 3
x1=rbinom(2000,1,0.5)
x2=rbinom(2000,1,0.5)
y=rbinom(2000,1,0.5)

data=data.frame(x1,x2,y)

head(data)

#hidden units = 2
nnet=neuralnet(y~x1+x2,data,hidden=2,err.fct = "ce",linear.output = F,likelihood = T)

predictions=compute(nnet,data[,1:2])

predictions$net.result

#likelihood ratio stat
argument=y*log(predictions$net.result)+(1-y)*log(1-predictions$net.result)

head(argument)

#sum of likelihood ratio stat
LogL=sum(argument)

LogL

D=-2*LogL

D

result[x-1022, 1] <- LogL
result[x-1022, 2] <- D

result

#results <- data.frame(LogL, D)

x=x+1
if (x==2021) {
  break
}

print(x)


}

result

My repeat loop works, but a thousand iterations, for example, takes multiple minutes. Is there a more efficient way to code such a loop?

$\endgroup$
3
  • 2
    $\begingroup$ Your repeat termination conditions will never be satisfied. Every round, x will be set to 1021, incremented by 1 and compared to 1023, which will never be true. How will the repeat ever stop? $\endgroup$
    – Ram RS
    Jun 22 '20 at 23:01
  • $\begingroup$ There are other approaches to parameterisation in ANN. @RamRS question on terminating recursion is definitely a good question. $\endgroup$
    – M__
    Jun 22 '20 at 23:26
  • $\begingroup$ Thanks, edited that and it works now, but it's super slow. I'll update the question! $\endgroup$
    – Equinox
    Jun 22 '20 at 23:44
1
$\begingroup$

The concern with your approach is called 'leakage' because you are parameterising the same data set that you are training. This can easily lead to overtraining is in ANN (artificial neural networks) and is not cool. Overtraining is where the ANN very tightly wraps around the training data set that it performs poorly on any other data set. In other words it is overtightened.

The way they get around it is to split the data set usually 60:20:20 into a training split, parameterisation (validation) split and test (testing) split. If the parameterisation split gets mixed with the training set - leakage happens and overtraining results. Normally the data is trained, and the trained model parameters are assessed/optimised in the parameter split. The trained data and optimised parameters are then tested in the test split.

As long as you are aware of the approach you are using, that I guess that is ok. If you separate your training and parameterisation (validation) data that would be cool.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.