Installation

Micc2 can be installed in different ways on your system. Here is a simple decision scheme:

  • if your OS is MACOS or Linux:

  • If your OS is Windows, go to Windows. Micc2 was initially developed for HPC purposes. As most HPC clusters have a Linux OS, Micc2 was developed for Linux, and, by extension, MACOS. However, many HPC users are running Windows on their desktop, so after Micc2 settled, (some) support for Windows was added.

Installation on your desktop or laptop

The most practical way to install micc2 on your desktop or laptop is

> python -m pip install [--user] et-micc2

Note

If you have different Python versions, you need to do this for each Python version in which you need access to micc2.

Note

If the current python is located in the system directories, and you do not have root access, the --user flag is necessary. Otherwise, not, but it is nevertheless wise to separate your “additions” from the system installation, in case you screw up.

Depending on which micc2 functionality you need, micc2 depends on other tools, which are not automatically installed when installing micc2. Micc2 will warn you if you need to install them:

  • Micc2 sets up version control using git. If you want another VCS, you must do that manually. If git is available, the command micc create will add a local git repository for your project. If gh is also available and you have added a personal access token for your github account, the micc create command can also automatically creeate a remote github repository (public by default) for your project. Git and gh are usually installed system-wide.

  • Building binary extensions from Fortran requires CMake and numpy. CMake can be installed system-wide, but also using pip:

    > python -m pip install [--user] cmake

    If you have different Python versions, it is recommended to do a system-wide installation. Numpy is installed with pip:

    > python -m pip install [--user] numpy

    You also need a Fortran and a C compiler on your PATH.

  • Building binary extensions from C++ requires CMake and pybind11. The latter is installed as:

    > python -m pip install [--user] pybind11

    You also need a C++ compiler on your PATH.

  • Tests and unit-tests can be done with pytest. Install as:

    > python -m pip install [--user] pytest

  • Building documentation requires sphinx and a suitable html theme, typically sphinx_rtd_theme:

    > python -m pip install [--user] sphinx
> python -m pip install [--user] sphinx-rtd-theme

    If you have want to extract documentation from command line applications (CLIs) based on click, you also need sphinx_click:

    > python -m pip install [--user] sphinx-click

  • Publishing your project to PyPI is easily achieved with poetry_(poetry publish --build). We recommend to install it separately for each Python version:

    > python -m pip install [--user] poetry

Installation on a HPC cluster

On typical HPC clusters (such as those of the VSC) application software must me made available by loading cluster modules. Even if the operating system of the login nodes and compute nodes posses versions of the above tools, they are typically not the most recent ones and have not been compiled for optimal performance on HPC systems, and you must load cluster modules for:

  • Python

  • git

  • gh

  • CMake

  • numpy

  • pytest

Some cluster modules expose several tools in one go. E.g., on the VSC cluster Leibniz one can use:

> module load Python
> module load buildtools
> module load gh
> module list
Currently Loaded Modules:
  1) leibniz/supported              6) Tcl/8.6.10-intel-2020a       11) METIS/5.1.0-intel-2020a-i32-fp64
  2) GCCcore/9.3.0                  7) X11/2020a-GCCcore-9.3.0      12) SuiteSparse/5.7.1-intel-2020a-METIS-5.1.0
  3) binutils/2.34-GCCcore-9.3.0    8) Tk/8.6.10-intel-2020a        13) Python/3.8.3-intel-2020a
  4) intel/2020a                    9) SQLite/3.31.1-intel-2020a    14) buildtools/2020a
  5) baselibs/2020a-GCCcore-9.3.0  10) HDF5/1.10.6-intel-2020a-MPI  15) gh/1.8.0

Together with Python 3.8.3, this gives us most of the tools and packages micc2 needs:

> git --version
git version 2.26.0

> gh --version
gh version 1.8.0 (2021-03-30)
https://github.com/cli/cli/releases/tag/v1.8.0

> cmake --version
cmake version 3.17.0
CMake suite maintained and supported by Kitware (kitware.com/cmake).

> pytest --version
This is pytest version 5.4.1

> python
Python 3.8.3 (default, May 27 2020, 11:32:18)
[GCC Intel(R) C++ gcc 9.3 mode] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.18.3'

Some packages are missing, however, notably the sphinx related ones, but for a reason: it is strongly discouraged to use HPC clusters for building documentation. Building documentation is not computationally intensive enough to engage a supercomputer. In is on the contrary I/O intensive and may put the cluster’s file system under stress. So do not install any sphinx related packages on your cluster. Documentation should be built on a desktop/laptop system.

Poetry is also missing. As long as the virtual environments created by poetry have problems with honoring the --system-site-packages flag, we do not recommend poetry for use on the cluster. Publishing can be done in a desktop environment, just like building documentation.

The other tools can just be installed as in Installation on your desktop or laptop, but the --user flag must be added, since you have no privileges for writing to the cluster system directories. So:

> python -m pip install --user et-micc2
> python -m pip install --user pybind11

What about virtual environments?

Virtual environments are extremely usefull to isolate the dependencies of different projects. If the your project has dependencies which are incompatible with those of micc2, you can separate them into two virtual environments, one for running micc2 and managing the project, and one for the project itself and its dependencies. Both environments can even be based on different Python versions.

You create a virtual environment like this:

> python -m venv venv-name

This creates a directory venv-name in the current directory containing a complete virtual Python environment based on your current python executable. It provides just a bare Python installation, no site packages. If you also want the system site packages to be available in the virtual environment, specify the --system-site-packages flag:

> python -m venv venv-name --system-site-packages

Note

The --system-site-packages flag is highly recommended when working on the cluster because it enables Python Packages which are built for optimal performance on HPC systems, e.g. numpy, scipy, hdf5py, … Installing these packages yourself, will consume a lot of your disk space and yield suboptimal performance.

Note

venv uses symbolic links as much as possible, as to use disk space efficiently. If symbolic links are not the default for your system, add the --symlinks flag

A virtual environment is activated as:

> source venv-name/bin/activate
(venv-name) >

Note how the prompt is modified, as to show the active virtual environment.

Note

Do not use the --user flag when installing in a virtual environment.

To deactivate an activated virtual environment, run:

(venv-name) > deactivate
>

The prompt returns back to normal. If you do not need your venv anymore, you can delete it like any other directory:

> rm -rf venv_name

Now you can use the virtual environment, and install packages in it without i mpacting anything else. E.g. a micc2 virtual environment could be created in a terminal as:

> python -m venv venv-micc2 --system-site-packages
> source venv-micc2/bin/activate
(venv-micc2) > python -m pip install et-micc2
(venv-micc2) > python -m pip install pybind11
(venv-micc2) >

Here, you issue the project management commands, add components, modify version, build binary extensions, etc. In another terminal you create a virtual environment for the project you are working on, say project FOO. Typically, it is created inside the FOO project directory itself:

> cd path/to/FOO > python -m venv .venv-FOO –system-site-packages > source .venv-FOO/bin/activate (.venv-FOO) >

Note, that the Python version to create the virtual and need not be the same as that of the micc2 environment.

So far, this virtual environment .venv-FOO does not know about FOO. We must still install FOO. Typically, during development, you want an editable install, so that any changes to``FOO``’s source code are instantly visible in the virtual environment. Micc2 provides a script for that purpose:

(.venv-FOO) > python ~/.et_micc2/scripts/install-e.py
Create editable install of project `FOO` in current Python environment (=/Users/etijskens/.pyenv/versions/3.8.5)?
Proceed (yes/no)? y

Proceeding ...
> /Users/etijskens/.pyenv/versions/3.8.5/bin/python -m pip install --user .
Processing /Users/etijskens/software/dev/workspace/FOO
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: foo
  Building wheel for foo (PEP 517) ... done
  Created wheel for foo: filename=FOO-0.0.0-py3-none-any.whl size=10686 sha256=153875968adc059610dba8cd1184fcd8ba93b658f67039fa6af8d02b07127a13
  Stored in directory: /private/var/folders/rt/7h5lk6c955db20y1rzf1rjz00000gn/T/pip-ephem-wheel-cache-x03e7s9q/wheels/a9/a4/e2/b61066293a36081a4330481401f731ee167914546c9f637bff
Successfully built foo
Installing collected packages: foo
  Attempting uninstall: foo
    Found existing installation: foo 0.0.0
    Uninstalling foo-0.0.0:
      Successfully uninstalled foo-0.0.0
Successfully installed foo-0.0.0
WARNING: You are using pip version 21.0.1; however, version 21.1 is available.
You should consider upgrading via the '/Users/etijskens/.pyenv/versions/3.8.5/bin/python -m pip install --upgrade pip' command.
Package `foo` installed at `/Users/etijskens/.local/lib/python3.8/site-packages`.

Removing package: /Users/etijskens/.local/lib/python3.8/site-packages/foo
Replacing package with symbolic link: /Users/etijskens/software/dev/workspace/FOO/foo
Editable install of FOO is ready.
(.venv-FOO) >

Now, the package FOO is available in .venv-FOO and any code changes to FOO’s source code are immediately visible in .venv-FOO.

Note

Micc2 projects lack a setup.py file, and consequently pip install -e path/to/FOO will fail. The install-e.py script provides a work around for this. For non-editable installs pip install path/to/FOO works flawless.

Note that this works exactly the same way on the cluster, provided you load the appropriate cluster modules to expose the cluster tools that you need, prior to creating and activating the virtual environments.

Windows

If you want to use Micc2 on Windows, you have a few options, most of which are incompletely tested.

Native Windows Python environment

To use Micc2 in a native Windows environment, the following components are needed

For building binary extension modules from C++:

  • The C++ compiler from Microsoft Visual Studio <- https://visualstudio.microsoft.com/downloads/ . Make sure you install the desktop development tools.

    _images/vs_DesktopDevelopment.png

    Every time you open a cmd prompt to execute micc2 build you must run vcvarsall.bat x86_amd64 to set the environment variables for the Visual Studio compiler.

For building binary extension modules from Fortran:

  • The C++ compiler from Microsoft Visual Studio <- https://visualstudio.microsoft.com/downloads/ (as above).

  • The Fortran compiler from Intel. At the time of writing this was part of the Intel oneAPI HPC Toolkit, which also requires the installation of the Intel oneAPI base Toolkit. The section below describes a few issues that need your attention to get things working.

The documentation below is based on Visual Studio 2019, and Intel oneAPI toolkits 2021.2.0.

Installation issues of with the Intel oneAPI toolkits

This is only needed for building binary extensions from Fortran.

  1. Do NOT install the Intel oneAPI toolkits in the default location, because that path contains spaces (C:Program Files (x86)\Intel\oneAPI), which causes f2py to fail with an uninformative error. I installed it at C:\Intel\oneAPI.

  2. The executables of the Intel oneAPI toolkits are added to the path by calling (using the above installation location):

    > C:\Intel\oneAPI\setvars.bat

  3. The Intel oneAPI toolkits v2021.2.0 come with their own Python distribution, v3.7.9, and include a number of useful packages, e.g. Numpy. The versions are typically not the latest, but they are well optimized. We recommend it. It is recommended to install other packages in user space, to avoid modifying the oneAPI installation:

    > pip install <some_package> --user

    If you install in user space, and the installed packages also installs an executable file or CLI, like Micc2, you must add the directory %UserProfile%\AppData\Roaming\Python\Python37\Scripts to the PATH. Pip will warn you if it installs CLIs in a location that is not on the path. Note that the path depends on the Python version of the Intel Python distribution, here 3.7.x.

  4. Because f2py calls vcvarsall.bat of Visual Studio every time, the PATH variable can get too long and micc2 build will fail, again with an uninformative message. This can be circumvented by renaming vcvarsall.bat to, e.g. vcvarsall-orig.bat and putting a vcvarsall.bat in place that returns immediately. These commands do the trick:

    > cd C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build
> rename vcvarsall.bat vcvarsall_orig.bat
> echo @exit /B 0 > vcvarsall.bat

  5. To make everything run smoothly, here is a batch script that sets the neccessary environment variables for Visual Studio, the Intel oneAPI toolkits, and adds the location of scripts in user space to the PATH:

    call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall_orig.bat" x86_amd64
call \Intel\oneAPI\setvars.bat
set "PATH=%UserProfile%\AppData\Roaming\Python\Python37\Scripts;%PATH%"

    Run this script every time you open a cmd prompt to work with these

In principle it should be possible to build Fortran binary extension modules for other Python distributions than the Intel Python distribution as well. Unfortunately you are in for a headache figuring out how to exclude the Intel Python distribution from the path set by \Intel\oneAPI\setvars.bat.

WSL-2

Modern Windows versions are equiped with Windows Subsystem for Linux, a compatibility layer for running Linux binary executables (in ELF format) natively on Windows 10. It provides you with a (virtual) Linux terminal which should be able to run all the tools just as in a MACOS or Linux terminal. It should be possible to build binary extension from C++ and Fortran code using the Gnu Compiler Collection (GCC).

Cygwin

Cygwin is a large collection of GNU and Open Source tools which provide functionality similar to a Linux distribution on Windows. It provides all the necessary tools for building binary extension from C++ and Fortran code using the Gnu Compiler Collection (GCC).

First time Micc2 setup

When you run Micc2 for the first time, it will complain that it has not been set up yet, and that you need to run micc2 setup. In order to facilitate your project management as much as possible, Micc2 needs to know some personal data such as your name, e-mail address, github account, … If you do not have a github account yet, create a github account at https://github.com>/join/.

Then, run:

> micc2 setup

The dialogue that follows asks for your full name, your e-mail address, your github username, a default html theme for Sphinx documentation, and a default license. For the last two questions you can select a default answer by just hitting enter. Finally, Micc2 shows you the chosen preferences and asks for permission to save the results. The dialogue goes like this:

Enter your full name
Enter `^^` to exit.
>: John Doe

Enter your e-mail address
Enter `^^` to exit.
>: john.doe@example.com

Enter your github username (leave empty if you do not have one,
  or create one first at https://github.com/join). Default = []
Enter `^^` to exit.
>: jdoe

Enter Html theme for sphinx documentation. Default = [sphinx_rtd_theme]
Enter `^^` to exit.
>:

Enter the default software license [GNU General Public License v3]
Choose between:
  0: GNU General Public License v3 (default)
  1: MIT license
  2: BSD license
  3: ISC license
  4: Apache Software License 2.0
  5: Not open source
Enter `^^` to exit.
>:
>: GNU General Public License v3
These preferences are saved to /Users/etijskens/.micc2/micc2.cfg:
{
  "full_name": "John Doe",
  "email": "john.doe@example.com",
  "github_username": "jdoe",
  "sphinx_html_theme": "sphinx_rtd_theme",
  "software_license": "GNU General Public License v3",
  "version": "0.0.0",
  "github_repo": "{{cookiecutter.project_name}}",
  "git_default_branch": "main",
  "minimal_python_version": "3.7",
  "py": "py"
}
Continue? yes/no >:
Preferences saved to /Users/etijskens/.micc2/micc2.cfg.

Micc2 uses your name and e-mail address to configure git.

Finally, if you want to be able to automatically create remote github repositories, you need also a personal access token for your github account. Follow these instructions At point 7 check at least these boxes:

  • repo

  • read:org

At point 9 copy the token to the clipboard and paste it in file ~/.micc2/<your-github-username>.pat, like this:

> echo <paste> > ~/.micc2/<your-github-username>.pat

Here <paste> stands for your OS’s keyboard combination for pasting in the terminal. ( Typically: Linux: shift+ctrl+v, MACOS: command+v, Windows: ctrl+v).