Handies

Fetch all events for an issue like this issue = "12345678" events = get_all_issue_events(issue) First set your SENTRY_AUTH_TOKEN as env var With definitions.. import requests import time def get_all_issue_events(organization_slug, project_slug, issue_id): url = 'https://app.getsentry.com/api/0/issues/%s/events/' % issue_id all_data = get_all_data(url) event_ids = [x["id"] for x in all_data] detail_vec = [get_event_data(organization_slug, project_slug, event_id) for event_id in event_ids] return all_data, detail_vec def get_all_data(url): token = os.environ.get('SENTRY_AUTH_TOKEN') headers = {"Authorization": "Bearer %s" % token, 'Content-Type': 'application/json'} next_results = 'true' next_url = url all_data = [] while next_results == 'true': # Do fetch pass response = requests.get(next_url, headers=headers) if response.status_code == 200: data = response.json() if isinstance(data, list): all_data += data elif isinstance(data, dict): all_data.append(data) next_results = response.links.get('next', {}).get('results') next_url = response.links.get('next', {}).get('url') time.sleep(0.4) else: next_results = 'false' return all_data def get_event_data(organization_slug, project_slug, event_id): url = f"https://sentry.io/api/0/projects/{organization_slug}/{project_slug}/events/{event_id}/" token = os.environ.get('SENTRY_AUTH_TOKEN') headers = {"Authorization": "Bearer %s" % token, 'Content-Type': 'application/json'} response = requests.get(url, headers=headers) return response.json()

enter and exit According to stack overflow you can.. In [97]: class Blah(object): ...: def __enter__(self): ...: print("hello") ...: return self ...: def __exit__(self, exc_type, exc_val, exc_tb): ...: print("bye!") ...: my = "stuff" ...: In [98]: with Blah(): ...: print("doing stuff") ...: hello doing stuff bye! The specific example that follows this is the object which is returned by import psycopg2 conn = psycopg2.connect() Help on connection object: class connection(builtins.object) | connection(dsn, ...) -> new connection object | | :Groups: | * `DBAPI-2.0 errors`: Error, Warning, InterfaceError, | DatabaseError, InternalError, OperationalError, | ProgrammingError, IntegrityError, DataError, NotSupportedError | | Methods defined here: | | __enter__(...) | __enter__ -> self | | __exit__(...) | __exit__ -- commit if no exception, else roll back |

Glom does not let you write or at least I couldnt figure out how. Tinkering with a spec based writer… from glom import glom, PathAccessError def nested_assign(target, spec, value): parts = spec.split(".") last = parts[-1] while parts[:-1]: top = parts.pop(0) target = target[top] target[last] = value def _plant(target, spec): """This is the equivalent of mkdir -p blah/flarg/blarg/klarf """ parts = spec.split(".") try: for i, part in enumerate(parts): glom(target, ".".join(parts[:i + 1])) except PathAccessError as e: print(repr(e)) print("stopped at ", i, part) print("going to add remaining", parts[i:]) print("..", list(range(i, len(parts)))) for j in range(i + 1, len(parts)): this_spec = ".".join(parts[:j]) print("this_spec", this_spec) nested_assign(target, this_spec, {}) def transplant(source_dict, mappings): skeleton = {} for m in mappings: _plant(skeleton, m["original"]) value = glom(source_dict, m["new"]) nested_assign(skeleton, m["original"], value) return skeleton Note I saw someone on stack overflow had a similar question but the approach looks a bit complex. May look at this later.

for tar.gz Assuming we have a func read_from_s3 that reads from s3… import tarfile targz = read_from_s3(bucket, s3fn) tar = tarfile.open(fileobj=io.BytesIO(targz), mode="r:gz") blahstream = tar.extractfile('blah-filename') for zip files Nice doc here from zipfile import ZipFile with ZipFile('foo.zip') as zip_archive: foo = zip_archive.read('some/file.txt')

Macvim brew install macvim Then cp -R /usr/local/Cellar/macvim/8.2-171_1/MacVim.app ~/Applications And add alias mvim=/Users/${username}/Applications/MacVim.app/Contents/bin/mvim to ~/.bash_profile If above not possible, then download MacVIM from macvim github ( which was forked from here originally I think ) other vim stuff ctrlp, from https://github.com/ctrlpvim/ctrlp.vim mkdir -p ~/.vim/pack/plugins/start git clone --depth=1 https://github.com/ctrlpvim/ctrlp.vim.git ~/.vim/pack/plugins/start/ctrlp theme, solarized8 sometimes is good (also slate too ) mkdir -p ~/.vim/pack/themes/opt/ cd ~/.vim/pack/themes/opt/ git clone git@github.com:lifepillar/vim-solarized8.git Vim notes Vim doesnt know about “terraform” files like .tf and the hashivim github helps with that. vim-terraform actually has a magical :TerraformFmt command that inspects the syntax of your .tf file too.

Add reverse lookup to zsh bindkey '^R' history-incremental-search-backward ( read on Stack Exchange > Unix&Linux > “How to enable reverse search in zsh?”

Edit files in place This -i '' was necessary on MacOs to avoid creating a greatings.txt.bak file as a backup $ sed -i '' 's/hello/bonjour/' greetings.txt xargs into vim Per this helpful answer , you can xargs into vim on macos x with xargs -o … find . -name 'blahfile*py' |head -1 |xargs -o vim xargs to md5 This is nice too, quickly md5 files. $ fd spark-wee en/post/2021-01-23-spark-weekend.md posts/2021-01-23-spark-weekend.md $ $ fd spark-wee|xargs -o md5 MD5 (./en/post/2021-01-23-spark-weekend.md) = 5534f81599860e239340b41ffa5aee09 MD5 (./posts/2021-01-23-spark-weekend.md) = 5534f81599860e239340b41ffa5aee09

import datetime def date_from_date_str(date_str): return datetime.datetime.strptime(date_str, '%Y-%m-%d').date() def make_start_end_clauses(start, end): '''Make sql to take advantage of Athena date partitioning. Example WHERE ((year = 2017 AND month = 10 AND day >=30) OR (year = 2017 AND month = 11 AND day = 1))''' assert start <= end month_tuples = make_start_end_month_tuples(start, end) clauses = [] if len(month_tuples) == 1: clause_raw = ('(year = {} AND month = {} AND day BETWEEN {} AND {})') clause = clause_raw.format(start.year, start.month, start.day, end.day) clauses.append(clause) else: # First month. start_clause = '(year = {} AND month = {} AND day >= {})'.format( start.year, start.month, start.day, ) clauses.append(start_clause) # Middle months, if any. months_in_between = month_tuples[1:-1] for year, month in months_in_between: clauses.append( '(year = {} AND month = {})'.format(year, month)) # Last month. end_clause = '(year = {} AND month = {} AND day <= {})'.format( end.year, end.month, end.day, ) clauses.append(end_clause) all_clause = ' OR '.join(clauses) final_clause = ' ( {} ) '.format(all_clause) return final_clause def make_date_clause(start_str=None, end_str=None, start_date=None, end_date=None): '''If given a start, return sql from start until today. Otherwise, return sql which only goes back 7 days. ''' if start_str is None and end_str is None and start_date is None and end_date is None: # do 7 days... today = utc_today() date_clause = date_clause_from_days_ago(today, days_ago=7) return date_clause if start_str is not None and end_str is not None: start_date = date_from_date_str(start_str) end_date = date_from_date_str(end_str) date_clause = make_start_end_clauses(start=start_date, end=end_date) return date_clause if start_date is not None and end_date is not None: date_clause = make_start_end_clauses(start=start_date, end=end_date) return date_clause def make_start_end_month_tuples(start, end): assert start <= end tuples = set([(start.year, start.month)]) next_date = start while next_date <= end: tuples |= set([(next_date.year, next_date.month)]) next_date += datetime.timedelta(days=1) return sorted(list(tuples)) Examples import datetime make_start_end_clauses(datetime.date(2021,4,29), datetime.date(2021, 5, 5)) # ' ( (year = 2021 AND month = 4 AND day >= 29) OR (year = 2021 AND month = 5 AND day <= 5) ) ' make_start_end_clauses(datetime.date(2021,4,29), datetime.date(2022, 5, 5)) # ' ( (year = 2021 AND month = 4 AND day >= 29) OR (year = 2021 AND month = 5) OR (year = 2021 AND month = 6) OR (year = 2021 AND month = 7) OR (year = 2021 AND month = 8) OR (year = 2021 AND month = 9) OR (year = 2021 AND month = 10) OR (year = 2021 AND month = 11) OR (year = 2021 AND month = 12) OR (year = 2022 AND month = 1) OR (year = 2022 AND month = 2) OR (year = 2022 AND month = 3) OR (year = 2022 AND month = 4) OR (year = 2022 AND month = 5 AND day <= 5) ) '

[[TOC]] List indexes From here SELECT tablename, indexname, indexdef FROM pg_indexes WHERE schemaname = 'public' ORDER BY tablename, indexname; Disk Usage per table from the postgresql wiki except one minor change … for ('user_blah', 'user_blar', 'schema1', 'schema2') schemas only … SELECT *, pg_size_pretty(total_bytes) AS total , pg_size_pretty(index_bytes) AS INDEX , pg_size_pretty(toast_bytes) AS toast , pg_size_pretty(table_bytes) AS TABLE FROM ( SELECT *, total_bytes-index_bytes-COALESCE(toast_bytes,0) AS table_bytes FROM ( SELECT c.oid,nspname AS table_schema, relname AS TABLE_NAME , c.reltuples AS row_estimate , pg_total_relation_size(c.oid) AS total_bytes , pg_indexes_size(c.oid) AS index_bytes , pg_total_relation_size(reltoastrelid) AS toast_bytes FROM pg_class c LEFT JOIN pg_namespace n ON n.oid = c.relnamespace WHERE relkind = 'r' and nspname in ('user_blah', 'user_blar', 'schema1', 'schema2') ) a ) a detect blocked queries? This didnt exactly work for me as expected, but colleague had mentioned this … SELECT COALESCE(blockingl.relation::regclass::text,blockingl.locktype) as locked_item, now() - blockeda.query_start AS waiting_duration, blockeda.pid AS blocked_pid, blockeda.query as blocked_query, blockedl.mode as blocked_mode, blockinga.pid AS blocking_pid, blockinga.query as blocking_query, blockingl.mode as blocking_mode FROM pg_catalog.pg_locks blockedl JOIN pg_stat_activity blockeda ON blockedl.pid = blockeda.pid JOIN pg_catalog.pg_locks blockingl ON( ( (blockingl.transactionid=blockedl.transactionid) OR (blockingl.relation=blockedl.relation AND blockingl.locktype=blockedl.locktype) ) AND blockedl.pid != blockingl.pid) JOIN pg_stat_activity blockinga ON blockingl.pid = blockinga.pid AND blockinga.datid = blockeda.datid WHERE NOT blockedl.granted AND blockinga.datname = current_database() hmmmm how about this select blockingl.relation, blockingl.pid, blockingl.mode, blockingl.granted, pgclass.relname, stat.usename, stat.application_name, stat.wait_event_type, stat.wait_event, stat.state, stat.query from pg_catalog.pg_locks blockingl join pg_class pgclass on blockingl.relation = pgclass.oid join pg_stat_activity stat on stat.pid = blockingl.pid check role membership select rr.*, pam.* from pg_catalog.pg_roles rr join pg_catalog.pg_auth_members pam on rr."oid" = pam.roleid in line table using a CTE … with datar(col1,col2) as ( values (1,2), (2,3), (4,5) ) select col1, col2 from datar COALESCE uses early stopping I had a pretty expensive COALESCE(col1, col2, ..., col50) with 50 arguments recently. And in testing whether my non-null value was first or last made a big difference! round does not work on double precision Wow interesting. I was comparing features generated in two tables to do some QA. Table 1 was the original or “target gold” table and table 2 was a table generated by production code. Using a simple table1.col1 = table2.col1 comparison, a particular float column was coming back as 0.092 , for about 100k rows. Hand inspecting, this looked like a difference of precision, but when I applied round(table1.col1, 3) = round(table2.col1, 3) instead, I got UndefinedFunction: function round(double precision, integer) does not exist And surely enough after reading the docs , oddly enough round with precision is only defined for numeric and not float8 (aka double precision). After casting to numeric my result of round(table1.col1::numeric, 3) = round(table2.col1::numeric, 3) was 0.990. Can dig deeper about the implementation later! ...

Interesting The operator ~~ is equivalent to LIKE, and ~~* corresponds to ILIKE. There are also !~~ and !~~* operators that represent NOT LIKE and NOT ILIKE. All of these operators are PostgreSQL-specific. per 7.3 doc. not sure if outdated