python - liblinear memory cost too much -

i have run liblinear modeling model file.

the python code here:

y, x = svm_read_problem(vector_file) prob = problem(y, x) param = parameter('-s 2 -c 1') m = train(prob, param) save_model(model_file, m) 

the problem when vector_file 247mb, total cost of memory when running liblinear 3.08gb. why cost much?

and in project, vector_file large 2gb, how can use liblinear train problem, can model file?

okey, know why problem is.

when read problem, python interface of liblinear use:

prob_y = [] prob_x = []  line in open(data_file_name):     line = line.split(none, 1)     # in case instance 0 features     if len(line) == 1: line += ['']     label, features = line     xi = {}     e in features.split():         ind, val = e.split(":")         xi[int(ind)] = float(val)     prob_y += [float(label)]     prob_x += [xi]  return (prob_y, prob_x) 

in python, int costs 28 bytes , float costs 24 bytes, out of imagination.

i post such cases author.