Posts

my backprop SGD from scratch 2022-Aug 14:13 ok reviewing from last time , Yea so I had switched from relu to sigmoid on commit b88ef76daf , but yea log loss is still going up during training, so for sure got rid of the bug of how it did not make sense to map that relu output to a sigmoid since a relu only produces positive numbers and so the sigmoid therefore was only able to produce values greater than 0.5 anyway. So at this point one thought I have for sure is whether this network is just one layer more complicated than would be needed for a problem set this simple. The thought arose after seeing that weight output from last training, ...

[[my back prop SGD from scratch 2022-Aug]] 13:35 yea so last time I had noticed , hey on a random initialization why was the y_prob 0.5 ? I had literally just initialized a new network and got this first example, ipdb> p x, y (array([10.31816265, -8.80044688]), 1) while running the ipdb debug mode, and inside of train_network() , ran --> 186 y_prob = feed_forward(x, model.layers, verbose=False) and got ...

[[my back prop SGD from scratch 2022-Aug]] 16:38 why no learning going on hmm look at this network import network as n import dataset import plot X, Y = dataset.build_dataset_inside_outside_circle(0.5) model = n.initialize_model({"learning_rate": 0.01}) ( loss_vec, model, artifacts, X_validation, Y_validation, Y_prob ) = n.train_network(X, Y, model) 17:02 wondering if I can inspect the gradient, to see if it is pointing where it should 23:02 so for a random weight initialized network, curious at least here the response should be nonlinear right? hmm model = n.initialize_model({"learning_rate": 0.01}) # X_validation # from earlier Y_prob, total_loss = loss(model.layers, X_validation, Y_validation) plot.scatter_plot_by_z(X_validation, Y_prob) # 2022-09-04T031837-scatter.png ...

[[my back prop SGD from scratch 2022-Aug]] 12:38 so what happened last time? well let me look at the 2022-08-21.html notes I created. 13:20 darn so ok spent bunch of time figuring out why I couldnt view all the images in that html but basically combination of the html references images in log seq dir and also I have to copy them to my git repo area for this repo. Anyway, 13:35 finally looking, so the weird issue was my log loss was getting worse with training when only affecting the final layer , so then I plotted the raw input data with a 3d surface plot but it was really weird looking and shapeless . I plotted this on a 2d plot instead and yea looked reasonable. But yea the 95% label=0 to to 5% label=1 maybe was contributing to why the 3d surface plot looked formless and uninteresting. And for fun I tweaked my 2d plotting code to use a spectrum of colors so I can perhaps look at my output data. But then oh wow oops I realized all the predictions were basically just 0.5 . So my main thought then was that haha probably training just the last layer of a network is basically not useful. 13:44 ok so let me continue with a strategy to train the full network and not just the final layer , look back at my network, 13:48 quick capture: Just going to write up the partial derivative parts of the gradient one at a time, 16:53 ok I have added this to the code now . let me try that train loop again then import network as n import plot X, Y = n.build_dataset_inside_outside_circle() layers = n.initialize_network_layers() loss_vec, layers = n.train_network(X, Y, layers) plot.plot_loss_vec(loss_vec) ...

hmm blah blah test something #publish-this What So I had a particular need to load a #[[scikit learn sklearn]] model trained with 0.20.4 on Databricks 10.4 and ran into some fun details I was not initially able to use the Databricks 10.4 pinned version of scikit-learn , 0.24.x because of course there were #backwards-compatibility issues where trying to load the model gave me an error saying that ModuleNotFoundError: No module named 'sklearn.preprocessing.imputation' was not found basically because I learned that all the Imputers were moved over to sklearn.impute ...

will this work? when blog vimeo 2022-04-10 foo yea – – – hope so

Today I thought it would take me maybe half an hour to replace my laptop battery. I got a replacement in the mail. Actually funny enough they are called the “function keys” battery because there are three cells . Anyway, turns out they are glued to the inside and you need some kind of epoxy loosener. I randomly stumbled on this video someone put together using some kind of wax line to actually remove batteries. Well I didn’t have wax thread, but I had floss so umm hey why not try that? ...

Wow, I am setting up on a fresh laptop today and I just ran the typical homebrew https://brew.sh setup and when I saw this, my eyes got slightly watery. ==> This script will install: /usr/local/bin/brew ... ... ==> The Xcode Command Line Tools will be installed. Press RETURN to continue or any other key to abort: ... ==> Searching online for the Command Line Tools ==> /usr/bin/sudo /usr/bin/touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress ==> Installing Command Line Tools for Xcode-13.2 ==> /usr/bin/sudo /usr/sbin/softwareupdate -i Command\ Line\ Tools\ for\ Xcode-13.2 Software Update Tool Finding available software Downloading Command Line Tools for Xcode Downloaded Command Line Tools for Xcode Installing Command Line Tools for Xcode Done with Command Line Tools for Xcode Done. ... ==> /usr/bin/sudo /bin/rm -f /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress ==> /usr/bin/sudo /usr/bin/xcode-select --switch /Library/Developer/CommandLineTools ==> Downloading and installing Homebrew... ... etc. ...

Initial Intent The initial intent here is to take three datasets of article titles of technical articles from different sources and try to classify them using an RNN. And also, another goal is to do this in hour long bites. And going to use this resource, https://madewithml.com/courses/foundations/recurrent-neural-networks/ for inspiration and direction on how to do this. The datasets (1) Per the above lesson, adapting the article dataset, “news.csv” , https://raw.githubusercontent.com/GokuMohandas/MadeWithML/main/datasets/news.csv , ( which was originally transformed from http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html ) (2) This set of ML/data science articles from Kaggle ( https://www.kaggle.com/hsankesara/medium-articles ) (3) And the set of stackoverflow questions ( https://www.kaggle.com/imoore/60k-stack-overflow-questions-with-quality-rate) First task: make my own dataset I am going to combine these three datasets. The label will be the source I only want the titles. The first two are technical, but the news articles source has different categories, so I’ll filter by the technical category. Do this import os import pandas as pd sources = { "news": { "loc": f"{os.getenv('SOURCE_NEWS_DIR')}/news.csv", "title_col": "title", "filter": { "col": "category", "val": "Sci/Tech", } }, "medium": { "loc": f"{os.getenv('SOURCE_MEDIUM_DIR')}/articles.csv", "title_col": "title", }, "stackoverflow": { "loc": f"{os.getenv('SOURCE_STACKOVERFLOW_DIR')}/train.csv", "title_col": "Title"} } def build_my_dataset(sources, out_loc, head=False): dfs = [] for source, detail in sources.items(): title_col = detail["title_col"] df = pd.read_csv(detail["loc"]) df = df.rename(columns={title_col: "title"}) if head: df = df.head() df["source"] = source also_filter = detail.get("filter") if also_filter: df = df[df[also_filter["col"]] == also_filter["val"]].copy() dfs.append(df[["title", "source"]]) pd.concat(dfs).to_csv(out_loc, index=False) # workdir = os.getenv("WORKDIR") out_loc = f"{workdir}/data/newdata.small.csv" build_my_dataset(sources, out_loc=out_loc, head=True) Just the sample.. !cat data/newdata.small.csv title,source Hacker Cracks Apple's Streaming Technology (AP),news European Download Services Go Mobile (Reuters),news Open Source Apps Developer SugarCRM Releases Sugar.Sales 1.1 (TechWeb),news Oracle Sales Data Seen Being Released (Reuters),news Sun's Looking Glass Provides 3D View (PC World),news Chatbots were the next big thing: what happened? – The Startup – Medium,medium Python for Data Science: 8 Concepts You May Have Forgotten,medium Automated Feature Engineering in Python – Towards Data Science,medium Machine Learning: how to go from Zero to Hero – freeCodeCamp,medium Reinforcement Learning from scratch – Insight Data,medium Java: Repeat Task Every Random Seconds,stackoverflow Why are Java Optionals immutable?,stackoverflow Text Overlay Image with Darkened Opacity React Native,stackoverflow Why ternary operator in swift is so picky?,stackoverflow hide/show fab with scale animation,stackoverflow out_loc = f"{workdir}/data/newdata.csv" build_my_dataset(sources, out_loc=out_loc, head=False) Next time Following along here, https://madewithml.com/courses/foundations/recurrent-neural-networks/ I think next I should do some preprocessing, so my data is tokenized. ...

This is a notebook of the day to day activity of a look into three years of fasting data.