Python and Astronomy

Style Guide

Python style guide are in two PEPs (Python Enhancement Proposals):

• PEP 8 for the main text;

• PEP 257 for docstring conventions.

Tkinter.Toplevel(master, class_='ClassName')

print("Why, hello there wonderful "
      "stackoverflow people!")

import codecs
f0 = open(file)
# DO NOT WORK
# lines = f0.read().decode('utf-8-sig').encode('utf-8')
lines = f0.readlines()
f0.close()
if lines[0].startswith(codecs.BOM_UTF8):
    lines[0] = lines[0].replace(codecs.BOM_UTF8, '', 1)

Useful Python libraries

numpy compared to R

# R:
fx <- function(x) {x**2}

# Python:
def fx(x):
    return x**2

Python super() considered super!

https://rhettinger.wordpress.com/2011/05/26/super-considered-super/

flat a double (!) nested list:

result = []
for x in oidata.vis2:
    for item in x._vis2data:
        result.append(item)

# is equivalent to:
tmp2 = [item for x in oidata.vis2 for item in x._vis2data]

There is a general answer for arbitrary nested case in Python Cookbood (3rd ed).

os.path

This is the right way of dealing with paths, filenames, extensions...

observ = '/media/moser/SAMSUNG2TB/interf/interf_PIO/2015/2015-11-29_SCI_alp_Eri_oidataCalibrated.fits'
os.path.split(observ) == (os.path.dirname(observ), os.path.basename(observ))

input vs raw_input

raw_input does not exists in Python 3.x. It was renamed to input.

pip

pip install --install-option="--prefix=$PREFIX_PATH" package_name

python -m ensurepip
# OR
python3.6 -m pip install --user pipenv

python3 -m pip install --user --upgrade pip

sudo pip install pip -U
sudo pip install setuptools -U

pip freeze > requirements.txt
pip install -r requirements.txt

head -n1 $(which pip)

curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | /usr/local/bin/python

python get-pip.py --user

ipython

• ipython is not calling the python version I want. What should I do?

  # You can discover the `ipython` you are calling typing
  which ipython
  # ~/.local/bin/ipython
  
  # Then type
  cat ~/.local/bin/ipython
  
  # The first line tells you the python ipython is calling
  #!/usr/local/bin/python
  # You may want to change to
  #!/usr/bin/env python

  Remember: ipython is equivalent to python -m IPython.

• ipython v1.0 is the most updated one for Python version equal or smaller than 2.6 ou 3.2.

• Magic functions: starting with %. time is particurlarly interesting. %time measures execution time in a single line. %%time measures in a line block.

Encoding

#!/usr/bin/env python
# -*- coding:utf-8 -*-

from io import open

What IDE to use for Python?

This is a religious question.

http://stackoverflow.com/questions/81584/what-ide-to-use-for-python

Matplotlib

can't invoke "event" command: application has been destroyed
while executing "event generate $w <<ThemeChanged>>"
(procedure "ttk::ThemeChanged" line 6)
invoked from within
"ttk::ThemeChanged"

Python GUI

"Always" the first option is to use Tkinter because it is part of the standard Python module and runs in most of the systems. However it is not so beautiful as the Qt library.

If someone needs pretty graphics, interactively, one may think using pyqtgraph (http://www.pyqtgraph.org/).

Errors

# DO NOT USE THIS!
# import sys
# from __future__ import print_function
#
# def eprint(*args, **kwargs):
#     print(*args, file=sys.stderr, **kwargs)

# USE THIS:
import warnings

warnings.warn('Be aware of what can happen when you read this...')

raise TypeError('A `TypeError` happened here! Program stops')

• More about warnings: https://pymotw.com/2/warnings/

• Following the updated recipe, the warnings (and the errors) will be printed automatically on sys.stderr

• The nuilt-in error classes are listed here: https://docs.python.org/2/library/exceptions.html

• raise by default stops the program (so does raise Warning('Message') )

VO tables

https://github.com/astropy/astropy/blob/master/docs/io/votable/index.rst

import astropy.io.votable as votable
votable = votable.parse("/data/Downloads/simbad")  # xml file
table = votable.get_first_table()
# table  # prints the table
data = table.array
# data[0] will NOT work! (It is a np structured array)
datacols = list(data.dtype.names)
arr = np.array(data[datacols[0]])

Astropy affiliated packages

Matt Craig, has created this list of Astropy affiliated packages to help improve your experience exploring astronomy using Python.

https://www.numfocus.org/blog/leveling-up-with-open-astronomy-astropy-affiliated-packages/

Status line (printing over the same line)

def fnPrintLine(tag, msg, cols=None, sameLine=False, align='left', flush='', full=False):
    """
    prints a formated line with a tag, message and time to the screen:
    [   TAG    ] This is a message....................................... [ 22:36:39 ]

    :author: J. Humberto
    """
    if align == 'center':
        halign = '^'
    elif align == 'right':
        halign = '>'
    else:
        halign = '<'

    if cols == None:
        try:
            cols = get_terminal_width()
            if cols < 80:
                raise
        except:
            cols = 100

    if len(msg) > cols - 34:
        msg = textwrap.wrap(msg, width=cols - 34)
        if tag == None:
            string = '{0:^16} {1:{flush}{halign}{w}}'.format('', msg[0], w=cols - 34, halign=halign, flush=flush)
            for line in msg[1:]:
                string += '\n{0:^18} {1:{flush}{halign}{w}}'.format('', line, w=cols - 34, halign=halign, flush=flush)
        else:
            string = '[{0:^16}] {1:{flush}{halign}{w}} [{2:^12}]'.format(tag, msg[0],                               datetime.now().strftime('%H:%M:%S'), w=cols - 34, halign=halign, flush=flush)
            for line in msg[1:]:
                string += '\n{0:^18} {1:{flush}{halign}{w}} {2:^14}'.format('', line, '', w=cols - 34, halign=halign,
                                                                            flush=flush)

    else:
        if tag == None:
            string = '{0:^18} {1:{flush}{halign}{w}}'.format('', msg, w=cols - 34, halign=halign, flush=flush)
        else:
            string = '[{0:^16}] {1:{flush}{halign}{w}} [{2:^12}]'.format(tag, msg, datetime.now().strftime('%H:%M:%S'),
                                                                         w=cols - 34, halign=halign, flush=flush)

    if sameLine == True:
        sys.stdout.write('{} \r'.format(string))
        sys.stdout.flush()
    elif sameLine == False:
        print string
    return

Check if a variable is string

In Python 2.x, one would do for the s variable

isinstance(s, basestring)

to check for str or unicode objects. In Python 3.x, it would be

isinstance(s, str)

If you're writing 2.x-and-3.x-compatible code, you'll probably want to use six:

from six import string_types
isinstance(s, string_types)

emcee and other nice stuff

http://eso-python.github.io/ESOPythonTutorials/ESOPythonDemoDay8_MCMC_with_emcee.html

http://eso-python.github.io/ESOPythonTutorials/

https://github.com/ESO-python/ESOPythonTutorials/tree/master/notebooks

http://www.sc.eso.org/~bdias/pycoffee/refs.html

Kernel Density Estimation in Python

Kernel density estimation is a way to estimate the probability density function (PDF) of a random variable in a non-parametric way.

• https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/

• https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/

Adding nice help to your program

Use the module ``optparse` ` (depricated).

Use argparse: https://docs.python.org/2/library/argparse.html

Date & Time

import time

## Regular and 12 hour format ##
print (time.strftime("%H:%M:%S"),time.strftime("%I:%M:%S"))

## Date with full and short year ##
print (time.strftime("%Y/%m/%d"), time.strftime("%y-%m-%d"))

Profile

python -m cProfile script.py

Broadcasting

import numpy as np
from itertools import product as itprod

a = np.arange(120.).reshape(3, 2, 5, 2, 2)
b = np.arange(120.).reshape(3, 2, 5, 2, 2)
fact = np.linspace(1, 1.4, 15).reshape((3, 5))

for i, j in itprod(range(3), range(5)):
    a[i, :, j] *= fact[i, j]

b *= fact[:, np.newaxis, :, np.newaxis, np.newaxis]

Pandas

import pandas

df = pandas.read_csv(csvfilename, sep=',') #,header=None)
df.values[:10,2]

idx = df['col3'].str.contains(regex)
subdf = df[idx]

# Create a DataFrame and save a CSV file
full_data = {'first_name': ['Jason', 'Molly', 'Tina', 'Jake', 'Amy'],
        'last_name': ['Miller', 'Jacobson', 'Ali', 'Milner', 'Cooze'],
        'age': [42, 52, 36, 24, 73],
        'preTestScore': [4, 24, 31, 2, 3],
        'postTestScore': [25, 94, 57, 62, 70]}

data = [['Jason', 'Molly', 'Tina', 'Jake', 'Amy'],
        ['Miller', 'Jacobson', 'Ali', 'Milner', 'Cooze'],
        [42, 52, 36, 24, 73],
        [4, 24, 31, 2, 3],
        [25, 94, 57, 62, 70]]

df1 = pandas.DataFrame(data, columns = ['first_name', 'last_name', 'age', 'preTestScore', 'postTestScore'])

df2 = pandas.DataFrame(full_data)

df3 = pandas.DataFrame(data)

dfn.to_csv('filename.csv')#, sep=',', encoding='utf-8')

Capturing the shell output in Python

This is way easier, but only works on Unix (including Cygwin).

import commands
print commands.getstatusoutput('wc -l file')

it returns a tuple with the (return_value, output).

https://stackoverflow.com/questions/4760215/running-shell-command-from-python-and-capturing-the-output

regex Python tip

The '.'' (dot) doesn't have the original regex meaning with the default re in Python.

So, we need to enable it using the flag re.DOTALL. Example:

outgroups = re.findall(rule, string, flags=re.DOTALL)

The re.DOTALL flag tells python to make the '.'' (dot) special character match all characters, including newline characters. This is very important when working with multi-line strings.

http://www.thegeekstuff.com/2014/07/advanced-python-regex/

List of reserved words in Python

General notes

class A(object):
    def foo(self, x):
        print(f"executing foo({self}, {x})")

    @classmethod
    def class_foo(cls, x):
        print(f"executing class_foo({cls}, {x})")

    @staticmethod
    def static_foo(x):
        print(f"executing static_foo({x})")

a = A()
A.foo(1)
# error!
a.foo(1)
# executing foo(<__main__.A object at 0xb7dbef0c>, 1
A.class_foo(1)
# executing class_foo(<class '__main__.A'>, 1)
a.class_foo(1)
# executing class_foo(<class '__main__.A'>, 1)
A.static_foo(1)
# executing static_foo(1)
a.static_foo(1)
# executing static_foo(1)

Definitions

• Thread: independent process, managed by the operational system.

• Daemon thread: by default, the main program waits the end of all threads before closing itself. However, this condition can be relaxed, and define the so-called "daemon threads".

• Event: an object to communicate event between the threads.

• Semaphore: an object to flux control (generally, controls the available resources, as CPUs).

• Queue: structure that allows safe sharing of data between threads.

• Locking: process that makes that threads be launched or interrupted under specific circumstances.

• Block: Is a kind of locking. An inactive threading, or a thread without available resources, is put to sleep in the system, until an event reactivates it or a required resource becomes available. In python, this is the standard described as (block=True, timeout=None). If timeout > 0, timeout defines the maximum allowed time that a thread can sleep before raising an exception (or error). If block=False a thread can not be put to sleep.

• Sleep: state of an inactive thread.

David Beazley - Python Concurrency From the Ground Up (PyCon 2015).

curve_fit

Consider errors for fitting. The solution returns the covariation matrix. Its diagnonal is the variance (the squared root, σ)!!

from scipy.optimize import curve_fit

def gauss(x, *p):
    A, mu, sigma = p
    return A*_np.exp(-(x-mu)**2/(2.*sigma**2))+1

p0 = [1., vels[i0], 40.]
coeff0, cov = curve_fit(gauss, x, y, p0=p0, sigma=yerr)

print('# Best coefficients are:')
print(coeff0)

leastsq

Consider errors for fitting. The solution, however, has no errors in the parameters.

http://wiki.scipy.org/Cookbook/FittingData

PyHdust + leastsq

import pyhdust.phc as phc

def polfunc(p, phi=np.linspace(0,1,21)):
    """
    P(phi) = P0+A cos[4 pi(phi-delt)] """
    P0, A, delt = p
    return P0+A*np.cos(4*np.pi*(phi-delt))

mag = sst.BlobDiskMod()
mag = sst.BlobDiskMod(Qis=-.348, Uis=0.040)

p0 = [.0471,.021,-.17]
p, c2r = phc.optim(p0,mag.phiobs,mag.P2,mag.sigP,polfunc)

PyHdust + curve_fit

import pyhdust.phc as phc

def polfunc2(phi=np.linspace(0,1,21), *p):
    """ P(phi) = P0+A cos[4 pi(phi-delt)] """
    P0, A, delt = np.array(p).flatten()
    return P0+A*np.cos(4*np.pi*(phi-delt))

mag = sst.BlobDiskMod()
mag = sst.BlobDiskMod(Qis=-.348, Uis=0.040)

p0 = [.0471,.021,-.17]
p, perr, c2r = phc.optim2(p0,mag.phiobs,mag.P2,mag.sigP,polfunc2)

Start

# In ~/.bashrc
export PYTHONPATH=$PYTHONPATH:~/pyhdust

PATH=~/.local/bin/:$PATH

LD_LIBRARY_PATH="$HOME/.local/lib"
export LD_LIBRARY_PATH PATH

export LDFLAGS="-L$HOME/.local/lib"
export CPPFLAGS="-I$HOME/.local/include"
export CXXFLAGS=$CPPFLAGS
export CFLAGS=$CPPFLAGS
export LD_RUN_PATH=$LD_LIBRARY_PATH

$ python setup.py install --user
$ ./configure --prefix="~/.local"

MS-Windows

When dialing with binary files in Windows (e.g., struct, xdrlib) open/write the files with the appendix 'b' (i.e., rb, wb, r+b...).

Starting at version 2.7.9, Python comes with pip!!!

Unofficial Windows Binaries for Python Extension Packages

http://www.lfd.uci.edu/~gohlke/pythonlibs/

Newer Python versions for Ubuntu

# double check if you have this package:
sudo apt install software-properties-common

# add repo:
sudo add-apt-repository ppa:deadsnakes/ppa

# enforce update
sudo apt update

# install you version
sudo apt install python3.12 python3.12-dev

# set default python version:
python3 --version
# 3.10.12
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.10 1
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.12 2
sudo update-alternatives --config python3

Other interesting packages:

• python3.12-venv: provides the standard library venv module.

• python3.12-distutils: provides the standard library distutils module.

• python3.12-lib2to3: provides the 2to3-3.12 utility and the standard library lib2to3 module.

• python3.12-gdbm: provides the standard library dbm.gnu module.

• python3.12-tk: provides the standard library tkinter module.

Compiling Python

Compiling Python on Ubuntu:

• Download the source from Python website

• edit the setup.py and add '/usr/lib/x86_64-linux-gnu' to the lib_dirs list:

  lib_dirs = self.compiler.library_dirs + [
          '/lib64', '/usr/lib64',
          '/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  
  # http://stackoverflow.com/questions/10654707/no-module-named-zlib-found

• edit Modules/setup.py and uncomment the lines for the module CSV, socket, SSL (set SSL=/usr/), curses, zlib...

• install a list of dev libraries

  apt-get install libreadline-dev
  apt-get install libssl-dev
  apt-get install libbz2-dev
  apt-get install build-essential
  apt-get install sqlite3
  apt-get install tk-dev
  apt-get install libsqlite3-dev
  apt-get install libc6-dev
  apt-get install libgdbm-dev
  apt-get install libncurses-dev
  
  # http://stackoverflow.com/questions/19148564/getting-failed-to-build-these-modules-curses-curses-panel-ssl-while-instal

• If you get the following message, there is a bug with SSL. Comment all the lines with "ethod_v2" in the files ssl.py and _ssl.c:

  "ImportError: cannot import name HTTPSHandler"

• In the end, you should get something like this:

  Failed to find the necessary bits to build these modules:
  _bsddb             _sqlite3           _tkinter
  bsddb185           dl                 imageop
  linuxaudiodev      ossaudiodev        sunaudiodev
  To find the necessary bits, look in setup.py in detect_modules() for the module's name.
  
  
  Failed to build these modules:
  readline

• Remember: zlib and ssl modules are required for pip.

# sudo apt-get install cmake

mkdir mybuiltdir
cd mybuiltdir

cmake path/to/llvm/source/root

cmake --build .

cmake -DCMAKE_INSTALL_PREFIX=$HOME/.local/ -P cmake_install.cmake
# cmake --build . --target install

How to clean up conda environments

• https://til.hashrocket.com/posts/9ohfsktorj-squeaky-clean-anaconda-environments

• https://conda.io/docs/user-guide/tasks/manage-environments.html#removing-an-environment

conda info -e
conda remove --name myenv —all
conda clean -a -y

General use Python packages in Astronomy

• PyHdust (Python tools for hdust code)

  http://astroweb.iag.usp.br/~moser/doc/

• AstroPy (community Python library for Astronomy)

  http://www.astropy.org/

  • AstroQuery: http://astroquery.readthedocs.org/en/latest/

  • PhotUtils: https://photutils.readthedocs.io/en/stable/

• astroplan: An Open Source Observation Planning Package in Python

  https://arxiv.org/abs/1712.09631

• sgp4: Track Earth satellites given TLE data, using up-to-date SGP4 routines

  https://pypi.org/project/sgp4/

• Skyfield: positions for the stars, planets, and satellites in orbit around the Earth

  https://rhodesmill.org/skyfield/

• SpecViz: 1D Spectral Visualization Tool

  http://specviz.readthedocs.io/

• astLib (set of Python modules that provides some tools for research astronomers)

  http://astlib.sourceforge.net/

• PyAstronomy (collection of astronomy related packages)

  http://www.hs.uni-hamburg.de/DE/Ins/Per/Czesla/PyA/PyA/

• Astropysics

  https://pythonhosted.org/Astropysics/

• spectral-cube

  https://github.com/radio-astro-tools/spectral-cube

• Trendvis

  https://github.com/matplotlib/trendvis

• Kapteyn package

  http://www.astro.rug.nl/software/kapteyn/

• Python time series analysis (pytseries)

  http://pytseries.sourceforge.net/

• scikit-learn (Machine Learning in Python)

  http://scikit-learn.org/stable/

• PyQt-Fit (regression toolbox in Python)

  http://pythonhosted.org/PyQt-Fit/

• PyData tools

  http://pydata.org/downloads.html

  • The Blaze Ecosystem: http://blaze.pydata.org/

• statsmodels

  http://statsmodels.sourceforge.net/

• SpectRes: A Fast Spectral Resampling Tool in Python

  https://github.com/ACCarnall/SpectRes

Python learning for astronomers

• Python para zumbis

  https://www.pycursos.com/python-para-zumbis/

• Python para desenvolvedores

  http://ricardoduarte.github.io/python-para-desenvolvedores/

• CodeCAdemy

  http://www.codecademy.com/en/tracks/python

• Coursera

  https://www.coursera.org/course/interactivepython1

• Python4astronomers

  https://python4astronomers.github.io/

• Machine learning in Python

  http://www.scipy-lectures.org/packages/scikit-learn/index.html

• Matplotlib tutorial

  http://www.labri.fr/perso/nrougier/teaching/matplotlib/#other-types-of-plots

• Python Fundamentos para Análise de Dados

  https://www.datascienceacademy.com.br/public-course?courseid=python-fundamentos

• Series of videos on how to build a #DataAnalytics library from scratch in #Python by @DunderData https://t.co/GwgCavd8AM #DataScience https://t.co/7MA1TeKfo0

  (https://twitter.com/ga_braganca/status/1101569058829414401)

• Scaler Topics on Python

  https://www.scaler.com/topics/python/

Python distributions

• Ureka*

  http://ssb.stsci.edu/ureka/

• Anaconda

  http://continuum.io/downloads

• Enthought Canopy

  http://www.enthought.com/products/canopy/

• Python(x,y)

  http://code.google.com/p/pythonxy/

• WinPython

  http://winpython.github.io/

• Pyzo

  http://www.pyzo.org/

Python environments

• IPython

  http://ipython.org/

• The Jupyter Notebook

  http://ipython.org/notebook.html

• Geany

  http://www.geany.org/

• PyCharm

  http://www.jetbrains.com/pycharm/

Utilities