evaluation-function

I'm using R's caret package to do some grid search and model evaluation. I have a custom evaluation metric that is a weighted average of absolute error. Weights are assigned at the observation level. X <- c(1,1,2,0,1) #feature 1 w <- c(1,2,2,1,1) #weights Y <- 1:5 #target, continuous #assume I run a model using X as features and Y as target and get a vector of predictions mymetric <- function(predictions, target, weights){ v <- sum(abs(target-predictions)*weights)/sum(weights) return(v) } Here an example is given on how to use summaryFunction to define a custom evaluation metric for caret's

I write programs to play board game variants sometimes. The basic strategy is standard alpha-beta pruning or similar searches, sometimes augmented by the usual approaches to endgames or openings. I've mostly played around with chess variants, so when it comes time to pick my evaluation function, I use a basic chess evaluation function. However, now I am writing a program to play a completely new board game. How do I choose a good or even decent evaluation function? The main challenges are that the same pieces are always on the board, so a usual material function won't change based on position,