Home

Handies

This may apply to Athena and or prestodb in general array empty? Per stackoverflow learned the name for this in the docs is cardinality select cardinality(array[]) = 0; This cannot be applied to the output of a json_extract(json_parse(data), '$.blah.flah.clah') since cardinality() takes ARRAY and not JSON. However, that JSON can be cast. For example, if '$.blah.flah.clah' is like [{"hi": "there"}, {"so": "then"}], then this cardinality(cast(json_extract(json_parse(what), '$.blah.flah.clah') as array(map(varchar, varchar)))) will produce the length of those arrays. ...

(updated February 26, 2023) · 1 min · 76 words · Michal Piekarczyk

Make a zip file for a lambda layer From a well written reference here : adjust the python version as needed #!/bin/bash echo -e "blah-lib==2.0\n\ umm-lib==0.45" > requirements.txt export LIB_DIR="python" rm -rf ${LIB_DIR} && mkdir -p ${LIB_DIR} docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.8 \ pip install -r requirements.txt -t ${LIB_DIR} zip -r layer.zip python And I like to use vim layer.zip to look at the contents Get lambda configuration details by boto Super handy client = boto3.client('lambda') out = client.get_function_configuration(FunctionName='myAwesomeLambda', # Qualifier='99', # optional version. ) In [30]: list(out.keys()) Out[30]: ['ResponseMetadata', 'FunctionName', 'FunctionArn', 'Runtime', 'Role', 'Handler', 'CodeSize', 'Description', 'Timeout', 'MemorySize', 'LastModified', 'CodeSha256', 'Version', 'VpcConfig', 'Environment', 'TracingConfig', 'RevisionId', 'Layers', 'State', 'LastUpdateStatus', 'PackageType']

(updated February 26, 2023) · 1 min · 113 words · Michal Piekarczyk

State Machine life cycle Adapting this from my stackoverflow answer to a question on updating state machines in step functions step_function_stack.yaml AWSTemplateFormatVersion: '2010-09-09' Transform: 'AWS::Serverless-2016-10-31' Description: >- A description of the State Machine goes here. Resources: MyStateMachineName: Type: AWS::StepFunctions::StateMachine Properties: RoleArn: "arn:aws:iam::{{aws_account_id}}:role/service-role/StepFunctions-MyStepFunctionRole" StateMachineName: "MyStateMachineName" StateMachineType: "EXPRESS" DefinitionString: Fn::Sub: | {{full_json_definition}} manage_step_functions.py import boto3 import os import time from jinja2 import Environment def do_render(full_json_definition): with open('step_function_stack.yaml') as fd: template = fd.read() yaml = Environment().from_string(template).render( full_json_definition=full_json_definition, aws_account_id=os.getenv('AWS_ACCOUNT_ID')) return yaml def update_step_function(stack_name, full_json_definition,): yaml = do_render(full_json_definition) client = boto3.client('cloudformation') response = client.update_stack( StackName=stack_name, TemplateBody=yaml, Capabilities=[ 'CAPABILITY_AUTO_EXPAND', ]) return response def create_step_function(stack_name, full_json_definition,): yaml = do_render(full_json_definition) client = boto3.client('cloudformation') response = client.update_stack( StackName=stack_name, TemplateBody=yaml, Capabilities=[ 'CAPABILITY_AUTO_EXPAND', ]) return response def get_lambdas_stack_latest_events(stack_name): # Get the first 100 most recent events. client = boto3.client('cloudformation') return client.describe_stack_events( StackName=stack_name) def wait_on_update(stack_name): events = None while events is None or events['StackEvents'][0]['ResourceStatus'] not in ['UPDATE_COMPLETE', 'UPDATE_ROLLBACK_COMPLETE', 'DELETE_COMPLETE', 'CREATE_COMPLETE']: print(events['StackEvents'][0]['ResourceStatus'] if events else ...) events = get_lambdas_stack_latest_events(stack_name) time.sleep(1) return events step_function_definition.json { "Comment": "This is a Hello World State Machine from https://docs.aws.amazon.com/step-functions/latest/dg/getting-started.html#create-state-machine", "StartAt": "Hello", "States": { "Hello": { "Type": "Pass", "Result": "Hello", "Next": "World" }, "World": { "Type": "Pass", "Result": "World", "End": true } } } Create that step function # From a python shell for example # First just set any privileged variables through environmental variables so they are not checked into code # export AWS_ACCOUNT_ID=999999999 # edit step_function_definition.json then read it with open('step_function_definition.json') as fd: step_function_definition = fd.read() import manage_step_functions as msf stack_name = 'MyGloriousStepFuncStack' msf.create_step_function(stack_name, step_function_definition) # If you are ready to update your State Machine, # you can edit step_function_definition.json or you might create a new file for reference, # step_function_definition-2021-01-29.json # (Because at time of this writing Step Functions dont have versions like Lambda for instance) with open('step_function_definition-2021-01-29.json') as fd: step_function_definition = fd.read() msf.update_step_function(stack_name, step_function_definition) terraform step func API Gateway v1 only works w/ StartExecution and not StartSyncExecution , but here is the basic terraform way to do that. This approach assumes a variables.tf file that defines sfn_orchestrater_arn as your step function arn. resource "aws_api_gateway_resource" "potato" { rest_api_id = aws_api_gateway_rest_api.foo-api.id parent_id = aws_api_gateway_rest_api.foo-api.root_resource_id path_part = "potato" } resource "aws_api_gateway_method" "potato-method" { rest_api_id = aws_api_gateway_rest_api.foo-api.id resource_id = aws_api_gateway_resource.potato.id http_method = "POST" authorization = "AWS_IAM" } resource "aws_api_gateway_integration" "potato-foo-integration" { rest_api_id = aws_api_gateway_rest_api.foo-api.id resource_id = aws_api_gateway_resource.potato.id http_method = aws_api_gateway_method.potato-method.http_method type = "AWS" integration_http_method = "POST" credentials = "arn:aws:iam::${var.aws_account_id}:role/MyRoleFoo" uri = "arn:aws:apigateway:${var.aws_region}:states:action/StartExecution" passthrough_behavior = "NEVER" request_templates = { "application/x-amz-json-1.0" = <<EOF { "input": "$util.escapeJavaScript($input.json('$'))", "stateMachineArn": "${var.sfn_orchestrater_arn}" } EOF } } resource "aws_api_gateway_method_response" "potato_response_200" { rest_api_id = aws_api_gateway_rest_api.foo-api.id resource_id = aws_api_gateway_resource.potato.id http_method = aws_api_gateway_method.potato-method.http_method status_code = "200" } resource "aws_api_gateway_integration_response" "potato_integration_response_200" { rest_api_id = aws_api_gateway_rest_api.foo-api.id resource_id = aws_api_gateway_resource.potato.id http_method = aws_api_gateway_method.potato-method.http_method status_code = aws_api_gateway_method_response.potato_response_200.status_code } resource "aws_api_gateway_method_response" "potato_response_400" { rest_api_id = aws_api_gateway_rest_api.foo-api.id resource_id = aws_api_gateway_resource.potato.id http_method = aws_api_gateway_method.potato-method.http_method status_code = "400" }

(updated February 26, 2023) · 3 min · 462 words · Michal Piekarczyk

I Stumbled on this gem of a stack overflow answer ( in here ) , which goes into some great detail but one of the hacks I learned is you can modify IFS to change the following behavior. (For context, I have some files with spaces in the names and they end in RR.jpg . $ for file in $(ls *RR.jpg) ; do echo $file ; done 2021-05-19 09.01.51RR.jpg 2021-05-19 09.02.00RR.jpg 2021-05-19 09.02.07RR.jpg $ echo $IFS $ OIFS="$IFS" $ IFS=$'\n' $ for file in $(ls *RR.jpg) ; do echo $file ; done 2021-05-19 09.01.51RR.jpg 2021-05-19 09.02.00RR.jpg 2021-05-19 09.02.07RR.jpg $ IFS="$OIFS" $ for file in $(ls *RR.jpg) ; do echo $file ; done 2021-05-19 09.01.51RR.jpg 2021-05-19 09.02.00RR.jpg 2021-05-19 09.02.07RR.jpg

(updated February 26, 2023) · 1 min · 118 words · Michal Piekarczyk

atom vec [] and while loop… (let [results-atom (atom [])] ; ; (println "elements: " (count @results-atom)) ; (swap! results-atom conj "hi") ;results-atom (while (< (count @results-atom) 3) (do (println "doing") ; insert (swap! results-atom conj "hi") (println "elements: " (count @results-atom)) )) ; done (println "Done. Now have elements: " (count @results-atom)) ) => doing elements: 1 doing elements: 2 doing elements: 3 Done. Now have elements: 3 nil

(updated February 26, 2023) · 1 min · 70 words · Michal Piekarczyk

basic callbacks… (comment "basic callback" (let [callback-fn #(+ 5 %) ] (callback-fn 10)) ) … user=> (let [callback-fn #(+ 5 %) #_=> ] #_=> (callback-fn 10)) 15 callback w future (comment "future w callback func" (defn use-callback-when-done [callback-fn] (future (callback-fn (+ 4 5)))) (def output (use-callback-when-done #(println "printings.. " % " .end"))) ) => user=> (def output (use-callback-when-done #(println "printings.. " % " .end"))) #'user/output printings.. 9 .end user=> callback and core async… (comment "use a go put onto a channel callback..." (defn use-callback-when-done [callback-fn] (future (callback-fn (+ 4 5)))) (require ['clojure.core.async :refer ['>! '<! '>!! '<!! 'go 'chan 'alts! 'alts!! 'timeout]]) (let [ my-results-chan (chan) callback-fn #(go (>! my-results-chan %)) output (use-callback-when-done callback-fn) ] ; (fn [x] (<!! my-results-chan)) (println "looking at the channel: " (<!! my-results-chan)) (println "done..") ) ) ==> user=> (let [ #_=> my-results-chan (chan) #_=> callback-fn #(go (>! my-results-chan %)) #_=> output (use-callback-when-done callback-fn) #_=> ] #_=> ; (fn [x] (<!! my-results-chan)) #_=> (println "looking at the channel: " (<!! my-results-chan)) #_=> (println "done..") #_=> ) looking at the channel: 9 done.. nil user=>

(updated February 26, 2023) · 1 min · 179 words · Michal Piekarczyk

(Porting over my notes from gists .. here ),.. quick note on imports i originally inferred this (ns blah-namespace (:use clojure.core.blah ) ; this would take all the terms in blah and put them into the current namespace (:refer-clojure :exclude [blarg flarg]) ; but this is supposed to be a way to avoid term clash ; so blarg and flarg will not be used. (not that they are in clojure.core however) ) Then I read https://www.braveclojure.com/organization/#Anchor-3 , and read use is a shortcut for require followed by a refer. hash-maps Many ways to make a hash-map, ...

(updated February 26, 2023) · 9 min · 1908 words · Michal Piekarczyk

keys keywords from map (let [ {:keys [status headers body error]} {:status 0 :headers 1 :body 3 :error 5 :extra 88} ] (println status body) ) your.app=> (let [ #_=> {:keys [status headers body error]} {:status 0 :headers 1 :body 3 :error 5 :extra 88} #_=> ] #_=> (println status body) #_=> ) 0 3 nil your.app=> also keyword args arity (comment "" (defn foof [a b & {:keys [op-fn] :or {op-fn +}}] (op-fn a b)) (foof 4 5 :op-fn *) (foof 4 5 :op-fn #(str %1 ".." %2)) )

(updated February 26, 2023) · 1 min · 90 words · Michal Piekarczyk

print data structures so as to preserve quotation (quotes) wow .. took too long to come across this nugget https://stackoverflow.com/questions/21136766/clojure-printing-functions-pr-vs-print pr/prn is to print/println for human readability. user=> (def d1 {:foo {:nil true :and "yay"}}) #'user/d1 user=> (prn "ok... " d1) "ok... " {:foo {:nil true, :and "yay"}} nil

(updated February 26, 2023) · 1 min · 49 words · Michal Piekarczyk

Dependencies and the repl It appears adding new dependencies into the build.boot file, and then running boot local repl again, downloads required dependencies and makes them useable for in the project. directory structure for a project my-project-root/ VERSION build.boot src/ blah/ foo.clj blarth.clj

(updated February 26, 2023) · 1 min · 43 words · Michal Piekarczyk