Interesting links

This page is a compilation of links I found interesting while learning Python and while solving

everyday problems in project management and maintenance, and of course scientific computing. As I keep on learning this list evolves continuously :-)

“the most dangerous thought you can have as a creative person, is that you know what you are doing.” Bret Victor - The Future of Programming.

“Every great developer you know got there by solving problems they were unqualified to solve until they actually did it.” - Patrick McKenzie

It´s better to wait for a productive programmer to become available than it is to wait for the first available programmer to become productive. - Steve McConnell

Debugging is twice as hard as writing the code in the first place.Therfore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. - Rajanand

Always code as if the guy who ends up maintaning your code will be a violent psychopath who knows where you live. - Rick Osborne

The trouble with programmers is that you can never tell what a programmer is doing until it’s too late. - Seymour Cray

First, solve the problem. Then write the code. - John Johnson

Code is like humor. When you have to explain it, it’s bad. – Cory House

Simplicity is the soul of efficiency. – Austin Freeman

Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris.

C makes it easy to shoot yourself in the foot; C++ makes it harder, but when you do, it blows your whole leg off.

A system administrator has two problems: 1. Dumb users. 2. Smart users.

“Never trust a programmer in a suit.” - Anonymous

“Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it.” - Alan J. Perlis

“Theory and practice sometimes clash. And when that happens, theory loses. Every single time.” - Linus Torvalds

For Beginners

• Learn Python: 7 of my favorite resources

• De Programmeursleerling - Pieter Spronck (in Dutch)

• Slither into Python

• Learn Python by building 5 games

• How do I start learning Python?

• Learn Python Programming

• Python cheat sheet

• Getting started with Python

• The full Python tutorial

• Python & PyGame Tutorial – Code a Duck Hunt Game

Python as a language

• What makes Python a great language?

• Python is known for being a language that’s easy to read, quick to develop in, and applicable to a wide range of scenarios

• Writing your first Python program

• How long did it take you to learn Python Wait, don’t answer that. It doesn’t matter. Ned Batchelder

Software engineering

• Cognitive Biases In Software Development

• What scientists must know about hardware to write fast code A simplified view - but not over-simplified - on how hardware affects performance. Written with Julia in mind rather than Python, but the principles remain valid.

• Clean architecture

• The Grand Unified Theory of Software Architecture

• best practices for writing code comments

• Software Engineering’s Greatest Hits Very interesting.

• SOLID principles for software design

• Software engineering practices

Python internals

• Boolean short-circuiting

• How to run a python script

• Cpython source code guide

• Know thy self - Methods and method binding - PyCon 2017

• Namespaces and Scope in Python

• The many ways to pass code to Python from the terminal

• Unpacking in Python: Beyond Parallel Assignment

• Pragmatic Unicode - Ned Batchelder - PyCon 2012

• Tracing the Python GIL

• Constant folding in Python (constant expressions)

• Python behind the scenes

• Python behind the scenes #7: how Python attributes work

• Unravelling the import statement

• Episode 40: How Python Manages Memory and Creating Arrays With np.linspace

• RealPython podcasts

• Syntactic sugar

• Python Pitfalls - Expecting The Unexpected

• Unravelling Python classes

• Python range

• Python Lambda functions

• Modify Iterables While Iterating in Python

• Understanding Attributes, Dicts and Slots in Python

• Strict Python function parameters

• sockets for dummies

• Python bytecode explained

• A tour of Python’s itertools library

• This Is Why Python Data Classes Are Awesome

• understanding class and instance variables

• Type annotations

• Python behind the scenes #13: the GIL and its effects on Python multithreading

• Blog on Python internals with many interesting topics

• Python List sort(): An In-Depth Guide to Sorting Lists

Python for HPC

Here’s a list of approaches that rely on low-lever programming languages, as C, C++ and Fortran, for speeding up Python (sequential) code. Some of these approaches, e.g. Numba rely on automatic code transformation from Python, so there is no need to write low-level code yourself.

• Performance Python: Seven Strategies for Optimizing Your Numerical Code

• High performance Python 1

• High performance Python 2

• High performance Python 3

• Python Bindings: Calling C or C++ From Python

• Implementing C++ Virtual Functions in Cython

• Wrapping C++ with Cython: intro

• How vectorization speeds up your Python code Quite a few interesting ideas: * self-instrumenting a python program for performance measurements * using pypy

• Speeding up your code

• Cython, Rust, and more: choosing a language for Python extensions

• PyCon 22 Talk - Henry Fredrick Schreiner III: Building a binary extension

• Deep CMake for library authors CppCon 2019, Interesting CMake stuff by craig scott, the author of Professional CMake - a practical guide

Approaches mimicking or wrapping OpenMP and MPI:

• Pymp – OpenMP-like Python Programming A really interesting concept, not as efficient as OpenMP itself (which incurs quite a bit of overhead itself), and, of course, limited to a single node. As the number of cores per node keeps increasing, pymp may be a good solution for problems that can do with a single node.

• High performance Python 4 Mpi4py, doing mpi from Python.

Other parallel processing approaches:

• Sequential Execution, Multiprocessing, and Multithreading IO-Bound Tasks in Python

• Common Issues Using Celery (And Other Task Queues)

• The Parallelism Blues: when faster code is slower

• Dask

• Visualize multi-threaded Python programs with an open source tool

GPU

• Accelerating Python on GPUs with nvc++ and Cython

Concepts and ideas:

• Does it ever make sense to use more concurrent processes than processor cores? You can have as many threads as you want as long as they’re doing nothing.

Code modernization

• Improving performance with SIMD intrinsics in three use cases

Profiling

• Profiling python

• Python profiling with blackfire

• Python 3.9 StatsProfile

• Profiling Python Code

• Disassemble Your Python Code

• Counting FLOPS and other CPU counters in Python

• A Comprehensive Guide to Profiling Python Programs

• Yet Another Python Profiler, but this time thread&coroutine&greenlet aware

• scalene

• Memory profiler for Python

• Python timer functions

• How to Benchmark (Python) Code

Memory profiling

• Optimizing Memory Usage in Python Applications

Resource monitoring

• Remora

• REMORA: REsource MOnitoring for Remote Applications

• My favorite Linux top command options

Python idioms and readability

• The Elements of Python Stylez

• Practical decorators Reuven Lerner

• Useful Python decorators for Data Scientists

• Elegant Solutions For Everyday Python Problems - PyCon 2018

• Yes, It’s Time to Learn Regular Expressions - PyCon 2017

• Decorators, unwrapped How do they work - PyCon 2017

• Decorators and descriptors decoded - PyCon 2017

• The Dictionary Even Mightier - PyCon 2017

• Looping Like a Pro in Python - PyCon 2017

• Readable Regular Expressions - PyCon 2017

• Passing Exceptions 101 Paradigms in Error Handling - PyCon 2017

• Readability Counts - PyCon 2017

• Modern Python Dictionaries: A confluence of a dozen great ideas - PyCon 2017

• Gang of 4 inspired decorators

• Python module of the week

• Type hints for busy programmers

• Exceptions

• Python Tips and Tricks, You Haven’t Already Seen - part 1

• Python Tips and Tricks, You Haven’t Already Seen - part 2

• 30 Python Best Practices, Tips, And Tricks

• pythonic things

• 71 Python Code Snippets for Everyday Problems

• Clean Code Concepts Adapted for Python

• The place of the ‘is’ syntax in Python

• 5 Things You’re Doing Wrong When Programming in Python

• 10 Python Tips and Tricks For Writing Better Code

• Tour of Python Itertools

• Getting the most out of Python collections

• Unpacking in Python: Beyond Parallel Assignment

• When Python Practices Go Wrong About the use of exec() and eval(). A presentation, so, the logic isn`t always obvious, but definitely an interesting topic. Here’s the corresponding video When Python Practices Go Wrong - Brandon Rhodes - code::dive 2019

• The Curious Case of Python’s Context Manager

• Demystifying Python’s Descriptor Protocol

• Why You Should Use More Enums In Python

• Regular Expressions: Regexes in Python (Part 1)

• Regular Expressions: Regexes in Python (Part 2)

• Novice to Advanced RegEx in Less-than 30 Minutes + Python

• 10 Awesome Pythonic One-Liners Explained

• Stop writing classes

• Generators, Iterables, Iterators in Python: When and Where

• New Features in Python 3.9 You Should Know About

• Python 101 – Working with Strings

• A Guide to Python Lambda Functions

• Pythonic code review

• Python args and kwargs: Demystified

• Python Dictionary Iteration: Advanced Tips & Tricks

• Python Code style and pythonic idioms

• Learn something new about Python every day in less than 1 minute

• The pass Statement: How to Do Nothing in Python

• 73 Examples to Help You Master Python’s f-strings

• Python f-strings Are More Powerful Than You Might Think

• The Correct Way to Overload Functions in Python

• Singleton is a bad idea

• The unreasonable effectiveness of f-strings and re.VERBOSE how to construct readable and documented regular expressions.

• Organize Python code like a PRO

• Coding 102: Writing code other people can read

Useful packages

• safer: a safer file writer

• sproc: subprocesses for subhumanses

• The 22 Most-Used Python Packages in the World

• Five Amazing Python Libraries you should be using!

• The most underrated python packages

• No Really, Python’s Pathlib is Great

• Python 101 – Creating Multiple Processes

• Python Packages: Five Real Python Favorites

• Python and PDF: A Review of Existing Tools

• A cross-platform Python module for copy and paste clipboard functions

• The Python pickle Module: How to Persist Objects in Python

• Pickle’s nine flaws

• Taichi:a programming language designed for high-performance computer graphics

• rich: rich text and beautiful formatting in the terminal

• Awesome pattern matching (apm) for Python

• Scheduling All Kinds of Recurring Jobs with Python

• bidict

Exceptions

• Better Python tracebacks with Rich

• Write Unbreakable Python

• pretty-errors: Prettifies Python exception output to make it legible

• Python KeyError Exceptions and How to Handle Them

Type checking in Python

• Type-checked Python in the real world - PyCon 2018 mypy

• Applying mypy to real world projects

• Types at the Edges in Python

• Exhaustiveness (enum) Checking with Mypy

Design patterns

• Design Patterns in Python for the Untrained Eye - PyCon 2019

• Python patters

• Refactoring and Design patterns

• Pyton anti-patterns

• Coding problems

Testing

• Getting Started Testing: pytest edition

• tox nox and invoke Break the Cycle: Three excellent Python tools to automate repetitive tasks

• Hypothesis

• Escape from auto-manual testing with Hypothesis!

• Beyond Unit Tests: Taking Your Testing to the Next Level - PyCon 2018

• How to mock in Python? – (almost) definitive guide

• Why your mock doesn’t work

• Visual Testing with PyCharm and pytest - PyCon 2018

• “WHAT IS THIS MESS?” - Writing tests for pre-existing code bases - PyCon 2018

• Python Testing 201 with pytest

• 8 great pytest plugins

• Pytest Features, That You Need in Your (Testing) Life

• An Introduction To Test Driven Development

• How To Write Tests For Python

• How I’m testing in 2020

• Building Good Tests

• Property-based tests for the Python standard library (and builtins)

• a pytest plugin designed for analyzing resource usage

• ward - A modern Python test framework

• The Clean Architecture in Python - How to write testable and flexible code

• Effective Python Testing With Pytest

• Document your tests

• 15 amazing pytest plugins and more (an episode on an interesting blog).

• ARRANGE-ACT-ASSERT: A PATTERN FOR WRITING GOOD TESTS

• There’s no one right way to test your code

• Why you should document your tests

• Property-Based Testing with hypothesis, and associated use cases

• Testing Python Applications with Pytest [Guide]

• Learning Python Test Automation These days, there’s a wealth of great content on Python testing. Here’s a brief reference to help you get started.

• How to write doctests in Python

• A Gentle Introduction to Testing with PyTest

• unittest’s new context methods in Python 3.11

Debugging

• pdb - The Python debugger

• Python debugging with pdb

• Python 101 – Debugging Your Code with pdb

• tutorial on sys.settrace

• Liran Haimovitch - Understanding Python’s Debugging Internals - PyCon 2019

• bdb - debugger framework

• pudb for Visual Debugging

• Cyberbrain: Python debugging, redefined

• Python Traceback (Error Message) Printing Variables

• Introspection in Python

• Learn to debug code with the GNU Debugger

• GDBGUI - A browser-based frontend to gdb

• GDB Tutorial - a walkthrough with examples

• Debugging Python and C(++) extensions with gdb and pdb

• When C extensions crash: easier debugging for your Python application

• Debugging Python C extensions with GDB

• Add custom windows to GDB: Programming the TUI in Python

• All about gdb, TUI and python inside gdb:

  • CppCon 2015: Greg Law “Give me 15 minutes & I’ll change your view of GDB

  • CppCon 2016: Greg Law “GDB - A Lot More Than You Knew

  • Getting the Most Out of GDB - Mark Williamson & Greg Law - C++ on Sea 2022

  • CppCon 2018: Greg Law “Debugging Linux C++”

  • Cool New Stuff in Gdb 9 and Gdb 10 - Greg Law - CppCon 2021

  • Cool New Stuff in GDB 9, 10 and 11 - Greg Law - ACCU 2022

  • ` <>`_

Logging

• Python logging tutorial

• Writing custom profilers for Python

• Do not log

• Understanding Python’s logging library

• Logging like a pro

Scientific Python

• Array Oriented Programming with Python NumPy

• Numeric and Scientific Python Packages built on Numpy

• Symbolic Maths in Python

• How to use HDF5 files in Python

• A Gentle Introduction to Serialization for Python (pickle, hdf5)

• A free course on Numpy

• Generating Stl Models with Python (CAD)

• Matplotlib Tutorial (2022): For Physicists, Engineers, and Mathematicians

• SciPy Tutorial (2022): For Physicists, Engineers, and Mathematicians

• NumPy Tutorial (2022): For Physicists, Engineers, and Mathematicians

• SymPy Tutorial (2022): For Physicists, Engineers, and Mathematicians

• Scientific Visualization using Python and Matplotlib

Artificial intelligence, machine learning and data science

• Scikit-learn, wrapping your head around machine learning - PyCon 2019

• Applied Deep Learning for NLP Using PyTorch

• Data Science Best Practices with pandas - PyCon 2019

• Thinking like a Panda: Everything you need to know to use pandas the right way

• Plotnine: Grammar of Graphics for Python

• Top 10 Python libraries of 2019

• Top 10 Python Packages for Machine Learning

• streamz: Build Pipelines to Manage Continuous Streams of Data

• nfstream - A flexible network data analysis framework

• A series how to turn machine learning models into production-ready software solutions

• A free course on Python Pandas

• Neural Networks Explained from Scratch using Python

• Machine learning made easy withe Python

• Speeding up your pandas code

• NN template

• Object localization using PyTorch : part 1

• Object localization using PyTorch : part 2

• µTransfer: A technique for hyperparameter tuning of enormous neural networks

• Practical Quantization in PyTorch

• Natural language processing demistified

• Neural Networks: Zero to Hero

• This guide is designated to anybody with basic programming knowledge or a computer science background interested in becoming a Research Scientist with 🎯 on Deep Learning and NLP

• pyGWalker

Pandas

• The great Python dataframe showdown, part 1: Demystifying Apache Arrow

• Tutorial: Indexing DataFrames in Pandas

• Efficient pandas Dataframes

• Speed up your slow pandas python code by 2500x…

• How to iterate over DataFrame rows (and should you?)

• Python for Data Analysis, 3d Edition

• Effective Pandas (Matt Harrison)

CLIs and scripting

• Building a CLI for Firmware Projects using Invoke

• Click

• QUICK: A real quick GUI generator for click

• When laziness is efficient: Make the most of your command line

• Messing with the python shell

• Converting shell scripts to python scripts

• a Python shell environment that combines the expressiveness of shell pipelines with the power of python iterators

• build a command line text editor with Python and curses

• Show progress in your Python apps with tqdm

• Questionary is a Python library for effortlessly building pretty command line interfaces

• Command Line Interface Guidelines

• iterm2 plugins written in python

• typer: Python library for building CLI applications Built on top of click

GUI

• Use PyQt’s QThread to Prevent Freezing GUIs

• CustomTkinter UI-Library

• Learn Python GUI Development for Desktop – PySide6 and Qt Tutorial

• Create a modern user interface with the Tkinter Python library

Packaging

• packaging with setuptools (nov 2021)

• Inside the Cheeseshop: How Python Packaging Works - PyCon 2018 historical overview with thorough explanation

• Share Your Code! Python Packaging Without Complication - PyCon 2017

• A Python alternative to Docker

• The Python Packaging Ecosystem

• Python Packaging Is Good Now

• Conda: Myths and Misconceptions

• The private PyPI server powered by flexible backends

• Packaging without setup.py

• PDM - Python Development Master

• Python Packaging Made Better: An Intro to Python Wheels

• Options for packaging your Python code: Wheels, Conda, Docker, and more

• What the heck is pyproject.toml?

• 4 Things Tutorials Don’t Tell You About PyPI

• How to improve Python packaging, or why fourteen tools are at least twelve too many

Graphics

• matplotlib

• “Cyberpunk style” for matplotlib plots

• Effectively using matplotlib

• ModernGL : a python wrapper over OpenGL 3.3+

• Magnum: Lightweight and modular C++11/C++14 graphics middleware for games and data visualization

• Grammar of graphics for Pyhon (using plotnine and pandas)

• plotly Express

• widgets in matplotlib

• How to build beautiful plots with Python and Seaborn

• HiPlot is a lightweight interactive visualization tool to help discover correlations and patterns in high-dimensional data

Installing packages

• A quick-and-dirty guide on how to install packages for Python

Tools

• Software Development Checklist for Python Applications

• IPython and Jupyter in Depth: High productivity, interactive Python Matthias Bussonier

• Faster Python Programs - Measure, don’t Guess - PyCon 2019

• Python Tooling Makes a Project Tick

• Life Is Better Painted Black, or: How to Stop Worrying and Embrace Auto-Formatting

• Using GitHub, Travis CI, and Python to Introduce Collaborative Software Development - PyCon 2018

• What’s in your pip toolbox - PyCon 2017

• How can I get tox and poetry to work together to support testing multiple versions of a Python dependency?

• Understanding Best Practice Python Tooling by Comparing Popular Project Templates

• My unpopular meaning about Black code formatter

• Python static analysis tools

• Leverage Sublime project folders to eashttps://martinheinz.dev/blog/34e your work

• Deep dive into how pyenv actually works by leveraging the shim design pattern

• Explore binaries using this full-featured Linux tool

• How to write a configuration file in python

• How to automatically set up a development machine with Ansible

• `direnv -- unclutter your .profile <https://github.com/direnv/direnv?utm_source=tldrnewsletter.`_ direnv is an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory.

• A list of new(ish) command line tools

git and other VCS

• lazygit: A simple terminal UI for git commands

• Introduction to Git In 16 Minutes

• 9 useful tricks of git branch

• gitutor

• Git Commands to Live By - The cheat sheet that goes beyond Git basics

• Things You Want to Do in Git and How to Do Them

• Helpful git commands for beginners

• understanding git: commits are snapshots not diffs

• Pijul, a sound and fast distributed version control system based on a mathematical theory of asynchronous work.

• Getting The Most Out Of Git

• Git is my buddy: Effective Git as a solo developer

• A practical guide to using the git stash command

• gitlab

• git flight rules

• HUBFS · File System for GitHub

• Git for professionals: Tools & Concepts for Mastering Version Control with Git

• Create changelog from git history

• 10 Git tips we can’t live without

• idiot proof git

• how to use git squash, git rebase, and git cherry-pick

Development environment, developement workflow

• pyenv+poetry+pipx <https://jacobian.org/2019/nov/11/python-environment-2020/>

• Improving Python Dependency Management With pipx and Poetry

• https://sourcery.ai/blog/python-best-practices/

• https://pypi.org/project/create-python-package/ a micc ‘light’

• Managing Python Environments

• Using Sublime Text for python

• How to Set Up a Python Project For Automation and Collaboration

• Hypermodern Python

• Thoughts on where tools fit into a workflow

• poetry

• Rewriting your git history, removing files permanently - cheatsheet & guide

• pipupgrade

• How to Set Environment Variables in Linux and Mac: The Missing Manual

• GitHub Actions: the full course

CI/CD

• Blazing fast CI with GitHub Actions, Poetry, Black and Pytest

• A beginner’s guide to CI/CD and automation on GitHub

Problem solving

• The mental game of Python - Raymond Hettinger - pybay 2019

• You should be reading academic computer science papers

Documentation

• Writing Documentation with Sphinx and reStructuredText

• Practical Sphinx - PyCon 2018

• Write the Docs is a global community of people who care about documentation

• How documentation works, and how to make it work for your project - PyCon 2017

• How to document Python code with Sphinx

  interesting section about tox

• Scott Meyers’ advise on writing

• A “How to” Guide for Sphinx + ReadTheDocs

• Write documentation as code

• sphinx-autodoc-typehints

Django

• Understanding django

Fortran/C/C++ Syntax

• https://www.fortran90.org

• http://www.cplusplus.com

• http://cppreference.com

C++

• A friendly guide to the syntax of C++ method pointers

• How Many Strings Does C++ Have?

• Johnny’s software lab very thorough site on C++ performance issues.

Compilers

• CppCon 2017: Matt Godbolt “What Has My Compiler Done for Me Lately? Unbolting the Compiler’s Lid”

• A Complete Guide to LLVM for Programming Language Creators

Notebooks

• Jupyter Notebooks in the IDE

• Jupyter everywhere

• 8 surprising ways how to use Jupyter Notebook

Containers

• Building Python Data Science Container using Docker

Windows

• Using WSL to Build a Python Development Environment on Windows This is promising: maybe we finally have a an environment on Windows with a minimal difference from Linux an MacOSX.

Linux

• 2020: The Year of the Linux Desktop - Moving from Macbook to Linux

Programming blogs

• julien danjou

• Patrick’s software blog

• Ruslan Spivak

• https://rhodesmill.org/brandon/

• testandcode

QUOTES

• “The code you write makes you a programmer. The code you delete makes you a good one. The code you don’t have to write makes you a great one.” - Mario Fusco

• “It’s hard enough to find an error in your code when you’re looking for it; it’s even harder when you’ve assumed your code is error-free.” - Steve McConnell