ok [[my backprop SGD from scratch 2022-Aug]] looking over results from last time, indeed so strange how microbatch loss was going back and forth and eventually trending that the plot of my change in loss, is increasing. deltas = [x["loss_after"] - x["loss_before"] for x in metrics["micro_batch_updates"]] although initially the values were some negatives, as well. But I wonder does it indeed something is terribly wrong if this number ever goes up at all?...
I have no idea how I managed to get on my bus to DC this morning. My Citibike station I planned on, was full, then my phone is Unavailable until like 10:42 . Bus boards at 11:00am . Still got to find a citibike dock to park the bike, because the stolen citibike fee is $1,250. At 10:42 of course I open phone and have absolutely no cell service. The citibike station does not have a printed map on it....
my backprop SGD from scratch 2022-Aug 14:13 ok reviewing from last time , Yea so I had switched from relu to sigmoid on commit b88ef76daf , but yea log loss is still going up during training, so for sure got rid of the bug of how it did not make sense to map that relu output to a sigmoid since a relu only produces positive numbers and so the sigmoid therefore was only able to produce values greater than 0....
[[my back prop SGD from scratch 2022-Aug]] 13:35 yea so last time I had noticed , hey on a random initialization why was the y_prob 0.5 ? I had literally just initialized a new network and got this first example, ipdb> p x, y (array([10.31816265, -8.80044688]), 1) while running the ipdb debug mode, and inside of train_network() , ran --> 186 y_prob = feed_forward(x, model.layers, verbose=False) and got ipdb> p y_prob 0....
[[my back prop SGD from scratch 2022-Aug]] 16:38 why no learning going on hmm look at this network import network as n import dataset import plot X, Y = dataset.build_dataset_inside_outside_circle(0.5) model = n.initialize_model({"learning_rate": 0.01}) ( loss_vec, model, artifacts, X_validation, Y_validation, Y_prob ) = n.train_network(X, Y, model) 17:02 wondering if I can inspect the gradient, to see if it is pointing where it should 23:02 so for a random weight initialized network, curious at least here the response should be nonlinear right?...