Help me in H2o Timeout error

Achu
Achu Member Posts: 1

Hello h2o experts!

I am running a deep learning model (binary classification) with following grid search code

#

hyper_params <- list(
activation = c("Rectifier", "Maxout", "Tanh", "RectifierWithDropout", "MaxoutWithDropout", "TanhWithDropout"),
hidden = list(c(5, 5, 5, 5, 5), c(10, 10, 10, 10), c(50, 50, 50), c(100, 100, 100)),
epochs = c(50, 100, 200),
l1 = c(0, 0.00001, 0.0001),
l2 = c(0, 0.00001, 0.0001),
rate = c(0, 01, 0.005, 0.001),
rate_annealing = c(1e-8, 1e-7, 1e-6),
rho = c(0.9, 0.95, 0.99, 0.999),
epsilon = c(1e-10, 1e-8, 1e-6, 1e-4),
momentum_start = c(0, 0.5),
momentum_stable = c(0.99, 0.5, 0),
input_dropout_ratio = c(0, 0.1, 0.2),
max_w2 = c(10, 100, 1000, 3.4028235e+38)
)

search_criteria <- list(strategy = "RandomDiscrete",
max_models = 100,
max_runtime_secs = 12000,
stopping_tolerance = 0.001,
stopping_rounds = 15,
seed = 42)

dl_grid <- h2o.grid(algorithm = "deeplearning",
x = x,
y = y,
grid_id = "dl_grid",
training_frame = df,
validation_frame = tst,
nfolds = 25,
fold_assignment = "Stratified",
hyper_params = hyper_params,
search_criteria = search_criteria,
seed = 42
)

However I am getting the error "na .h2o.doSafeREST(h2oRestApiVersion = h2oRestApiVersion, urlSuffix = urlSuffix, : Unexpected CURL error: Timeout was reached: [localhost:54321] Resolving timed out after 10180 milliseconds [1] "Job request failed Unexpected CURL error: Timeout was reached: [localhost:54321] Resolving timed out after 10180 milliseconds, will retry after 3s."

R version details are
platform x86_64-w64-mingw32
arch x86_64
os mingw32
crt ucrt
system x86_64, mingw32
status
major 4
minor 2.0
year 2022
month 04
day 22
svn rev 82229
language R
version.string R version 4.2.0 (2022-04-22 ucrt)
nickname Vigorous Calisthenics

Kindly help me
Thank you

Tagged:

Answers

  • tomas_fryda
    tomas_fryda Member Posts: 1

    I believe your error is caused by running out of memory.

    Is it possible that the input has a high dimensionality after one-hot encoding?

    The amount of memory necessary for representing weight can get quite high especially if your data contain categorical columns with high cardinality, those columns are encoded so from one categorical column with cardinality C you get C-1 columns and then you can multiply the total number of columns by the number of neurons in the first hidden layer to get some idea of minimal amount of memory you'll need. Since you use at most 3 hidden layers with 100 neurons and at most 100 models, it doesn't seem to me to be enough to run out of memory unless you have a high input dimensionality.

    If you think that is not the case, it would help if you could post exceptions/errors from the log file from H2O process. The log can be obtained as described in https://docs.h2o.ai/h2o/latest-stable/h2o-docs/logs.html. Note that the log can contain sensitive information such as column names, file names etc.