Darknet : No weights created after training custom objects

4
votes

The darknet training command does not produce any output and exits too early (compared to other CNN training projects)

I have followed the instructions for "how to train (to detect your custom objects)".
The yolo-obj.cfg is configured accordingly.
The darknet.exe has been compiled and build succesfully using MSVS 2017.

I have 3 new custom classes in:

obj.data file: 

classes= 3  
train = data/train.txt  
valid = data/train.txt  
names = data/obj.names  
backup = backup/

obj.names file: 

ring  
watch  
necklace

I ran the yolo_mark for approx 500 images for each class, resulting in the corresponding *.txt files.
I put all the jpg and txt files in the obj directory.
The train.txt file contains the path to the *.jpg files, e.g.: "data/obj/necklace 013311.jpg"

Downloaded and put the darknet53.conv.74 file in "x64" directory

Running the command (from virtual machine, hence no GPU) as admin: 

C:\Users\claw\Downloads\darknet-master\darknet-master\build\darknet\x64>darknet_no_gpu.exe detector train data/obj.data yolo-obj.cfg darknet53.conv.74
yolo-obj

command line output:

layer filters size input output  
0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BF  
1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 1.595 BF  
2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 0.177 BF  
3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BF  
4 Shortcut Layer: 1  
5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 1.595 BF  
6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF  
7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF  
8 Shortcut Layer: 5  
9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF  
10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF  
11 Shortcut Layer: 8  
12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 1.595 BF  
13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF  
14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF  
15 Shortcut Layer: 12  
16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF  
17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF  
18 Shortcut Layer: 15  
19 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF  
20 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF  
21 Shortcut Layer: 18  
22 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF  
23 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF  
24 Shortcut Layer: 21  
25 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF  
26 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF  
27 Shortcut Layer: 24  
28 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF  
29 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF  
30 Shortcut Layer: 27  
31 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF  
32 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF  
33 Shortcut Layer: 30  
34 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF  
35 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF  
36 Shortcut Layer: 33  
37 conv 512 3 x 3 / 2 52 x 52 x 256 -> 26 x 26 x 512 1.595 BF  
38 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF  
39 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF  
40 Shortcut Layer: 37  
41 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF  
42 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF  
43 Shortcut Layer: 40  
44 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF  
45 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF  
46 Shortcut Layer: 43  
47 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF  
48 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF  
49 Shortcut Layer: 46  
50 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF  
51 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF  
52 Shortcut Layer: 49  
53 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF  
54 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF  
55 Shortcut Layer: 52  
56 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF  
57 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF  
58 Shortcut Layer: 55  
59 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF  
60 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF  
61 Shortcut Layer: 58  
62 conv 1024 3 x 3 / 2 26 x 26 x 512 -> 13 x 13 x1024 1.595 BF  
63 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF  
64 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF  
65 Shortcut Layer: 62  
66 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF  
67 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF  
68 Shortcut Layer: 65  
69 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF  
70 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF  
71 Shortcut Layer: 68  
72 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF  
73 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF  
74 Shortcut Layer: 71  
75 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF  
76 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF  
77 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF  
78 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF  
79 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF  
80 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF  
81 conv 24 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 24 0.008 BF  
82 yolo  
83 route 79  
84 conv 256 1 x 1 / 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BF  
85 upsample 2x 13 x 13 x 256 -> 26 x 26 x 256  
86 route 85 61  
87 conv 256 1 x 1 / 1 26 x 26 x 768 -> 26 x 26 x 256 0.266 BF  
88 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF  
89 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF  
90 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF  
91 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF  
92 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF  
93 conv 24 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 24 0.017 BF  
94 yolo  
95 route 91  
96 conv 128 1 x 1 / 1 26 x 26 x 256 -> 26 x 26 x 128 0.044 BF  
97 upsample 2x 26 x 26 x 128 -> 52 x 52 x 128  
98 route 97 36  
99 conv 128 1 x 1 / 1 52 x 52 x 384 -> 52 x 52 x 128 0.266 BF  
100 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF  
101 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF  
102 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF  
103 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF  
104 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF  
105 conv 24 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 24 0.033 BF  
106 yolo  

Total BFLOPS 65.304  
Loading weights from darknet53.conv.74...  
seen 64  
Done!  

Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005  
If error occurs - run training with flag: -dont_show  
Resizing  
416 x 416  

Cannot load image "data/img/ring chic-criss-cross-adjustable-ad-ring.jpg"  
Loaded: 1.143984 seconds  
Used AVX  
Region 82 Avg IOU: 0.333570, Class: 0.602019, Obj: 0.402860, No Obj: 0.528741, .5R: 0.000000, .75R: 0.000000, count: 4  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.521660, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.514523, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.329878, Class: 0.570290, Obj: 0.611294, No Obj: 0.528309, .5R: 0.250000, .75R: 0.000000, count: 4  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.521499, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.514392, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.575794, Class: 0.539979, Obj: 0.316475, No Obj: 0.528604, .5R: 0.500000, .75R: 0.500000, count: 2  
Region 94 Avg IOU: 0.312451, Class: 0.125449, Obj: 0.238739, No Obj: 0.521500, .5R: 0.000000, .75R: 0.000000, count: 1  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.514025, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.257590, Class: 0.547629, Obj: 0.447064, No Obj: 0.527685, .5R: 0.000000, .75R: 0.000000, count: 3  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.521665, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.515411, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.297573, Class: 0.436722, Obj: 0.389306, No Obj: 0.528302, .5R: 0.500000, .75R: 0.000000, count: 4  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.521452, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.513978, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.191856, Class: 0.645887, Obj: 0.364560, No Obj: 0.528137, .5R: 0.000000, .75R: 0.000000, count: 5  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.521575, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.514143, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.475039, Class: 0.419801, Obj: 0.578539, No Obj: 0.527876, .5R: 0.500000, .75R: 0.500000, count: 2  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.521085, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.514371, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.264798, Class: 0.416162, Obj: 0.462117, No Obj: 0.527412, .5R: 0.000000, .75R: 0.000000, count: 5  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.521446, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.514205, .5R: -nan(ind), .75R: -nan(ind), count: 0  


1: 1003.093994, 1003.093994 avg loss, 0.000000 rate, 1056.320056 seconds, 64 images  
Loaded: 0.000000 seconds  
Cannot load image "data/img/necklace 570239071_2906.jpg"  
Cannot load image "data/img/necklace 570239072_2906.jpg"  
Cannot load image "data/img/necklace 10019367_no_place_like_roam_necklace_green_main.jpg"  
Region 82 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.527527, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.521694, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.514430, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.213376, Class: 0.587271, Obj: 0.565966, No Obj: 0.528763, .5R: 0.000000, .75R: 0.000000, count: 5  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.522077, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.515318, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.314485, Class: 0.501796, Obj: 0.458959, No Obj: 0.528414, .5R: 0.000000, .75R: 0.000000, count: 2  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.521397, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.514781, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.278535, Class: 0.518696, Obj: 0.510300, No Obj: 0.528529, .5R: 0.000000, .75R: 0.000000, count: 5  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.521170, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.514448, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.270750, Class: 0.498121, Obj: 0.530221, No Obj: 0.528569, .5R: 0.000000, .75R: 0.000000, count: 2  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.521003, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.513312, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.235287, Class: 0.480098, Obj: 0.517194, No Obj: 0.527906, .5R: 0.000000, .75R: 0.000000, count: 4  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.521571, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.513103, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.368155, Class: 0.552764, Obj: 0.482865, No Obj: 0.528044, .5R: 0.200000, .75R: 0.000000, count: 5  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.521782, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.514365, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 82 Avg IOU: 0.393099, Class: 0.568679, Obj: 0.534074, No Obj: 0.528130, .5R: 0.000000, .75R: 0.000000, count: 2  
Region 94 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.522459, .5R: -nan(ind), .75R: -nan(ind), count: 0  
Region 106 Avg IOU: -nan(ind), Class: -nan(ind), Obj: -nan(ind), No Obj: 0.515186, .5R: -nan(ind), .75R: -nan(ind), count: 0  


2: 1002.576904, 1003.042297 avg loss, 0.000000 rate, 1043.121191 seconds, 128 images  
Loaded: 0.000000 seconds

After this I checked the backup directory, where only a *.tmp file was created (0 kb)
No weigth file was created...

What am I doing wrong ?

3
pythondarknet

3 Answers

1
votes

By default, the weights are recorded every 100 iterations. You have to wait to train YOLO for long hours (especially without GPU) before making inferences with your weights.

1
votes

I do not think your training set is correctly configured. Most of your results are -nan(ind)

There may be a problem with your train.txt.

1: 1003.093994, 1003.093994 avg loss, 0.000000 rate, 1056.320056 seconds, 64 images ^ This is the iteration number. By default darknet writes the weight to backup folder after 100 iterations. If you want weights before that open your detector.c file in src and modify 

    if (i % 1000 == 0 || (i < 1000 && i % 3 == 0)) {
    //if (i % 100 == 0) {
    //if(i >= (iter_save + 100)) {

line 204 like i have done for mine and make the number 1(instead of 3) if you want a weight on your first iteration then build again.

Try training on a smaller net like yolo-voc like in this tutorial: https://timebutt.github.io/static/how-to-train-yolov2-to-detect-custom-objects/ Even i am facing a problem when using the yolov3.cfg based net (getting the same output)

0
votes

The issue was that the object txt files (created by yolomarkup) were almost all empty. I had added 3 new objects; necklace, ring, watch and for each object approx 500 jpg images, which I used in yolomarkup.exe. For a lot of those marked up images the corresponding txt file was empty ! I dropped the training alltogether for those objects