Spark

https://databricks.com/try-databricks 2021-03-21 Running A quick start notebook Based on the notes here, it is pretty easy to create an auto-scaling cluster. Not sure yet what events prompt the cluster to get more workers. But I would be curious to try a job that uses fewer workers and more workers, to see how the outcomes compare. I also like ethat this notebook supports SQL and also python , using what looks like first line as %python to indicate the language....

Trying out Spark this weekend These are just my casual notes from doing that, updating them as I go along. Following this post to get kubernetes running in Docker for mac Per this post , I just ticked the “Enable Kubernetes” option in the docker settings. Kubernetes is taking quite a while to start up though . several minutes. kind of weird? Download spark image From here 2021-01-24 ok backup my docker images Per notes , I backed up local docker images, Like this… docker save citibike-learn:0....