10. Software Management#
In addition to writing a program for each computing task, knowldge and skills are needed for designing and managing the entire data analysis or simulation procedure, testing and revising the codes, and sharing the data and tools among collaborators.
Unlike commercial software development, research computing often starts from a simple exploratory code created by a single researcher. However, even for a single-person project, it is beneficial to follow the standard practices in software development because.
If your project is successful, it will be succeeded by other members of your lab or the research community world wide.
You, after a few months, do not remember where you put which file, what was this file, or why you wrote this code.
Reference:#
Greg Wilson, et al. (2017). Good enough practices in scientific computing. PLOS Computational Biology, 13(6): e1005510 (https://doi.org/10.1371/journal.pcbi.1005510)
Coding Style#
In writing programs, keep in mind:
Make them modular and aviod duplicate codes.
Give explanation at the beginning of each file/function.
Use file/function/variable names that you can comprehend a year later.
Never write a numeric parameter in an equation; define it as a variable/argument.
Give comments for significant parts/lines of codes.
Turn comment/uncomment into
if-else
for different modes of operation.Verify your code with a simple input for which the correct output is known.
Prepare documentation even before somebody asks you, as you yourself will need that after a few months.
In some projects, all you need is to download pre-existing tools and apply them to the data. Even in that case, it is better to record the procedure as a script for
avioding/detecting manual errors
reproducibility of the result
re-analysis with new data, tool, or parameters
Scritping#
On Unix-like systems, the common way is a shell script, which is a file containing a series of commands you would type into a terminal.
For a more elaborate processing, a Python script is often preferred.
Version Control System#
Software development is repetitions of coding, testing, and improving. A version control system (VCS) allows
parallel development of parts and re-integration
trace back to previous versions when a problem is detected
Git#
The most popular VCS today is Git, created by Linus Torvalds for developing Linux.
After creating/editing your files, you stage them for management and commit for a certain version.
If git
has not been installed, follow one of these to install.
Mac:
Install XCode from the App Store
or install XCode Command Line Tools by
xcode-select --install
or install HomeBrew and run
brew install git
Windows:
Install Git for Windows
Detailed documentations can be found at https://git-scm.com/docs
odesim#
As an example of version control, let us take a simple ODE simulator, odesim
%ls odesim
__pycache__/ first.py odesim.py odesim.py~ second.py
%cd odesim
/Users/doya/OIST Dropbox/kenji doya/Python/iSciComp/odesim
%cat odesim.py
# odesim.py
# An Ordinary Differential Equation Simulator
# 2022 by Kenji Doya
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt
import importlib # import module from string
class odesim():
"""An ODE Simulator"""
time = 0
trun = 10
dt = 0.1
def __init__(self, odename):
"""create a new ODE"""
print('Importing ODE:', odename)
self.ode = importlib.import_module( odename)
importlib.reload(self.ode) # for updated module
self.state = self.ode.initial_state
#self.reset()
plt.ion()
def reset(self):
"""reset the state"""
self.time = 0
self.state = self.ode.initial_state
plt.clf()
def simulate(self):
"""simulate the ODE"""
# +dt/2 to include time+trun
self.t = np.arange(self.time, self.time+self.trun+self.dt/2, self.dt)
self.y = odeint(self.ode.dynamics, self.state, self.t, args=(self.ode.parameters,))
# update the time and state
self.time = self.t[-1]
self.state = self.y[-1,:]
print("t=", self.time, "; state=", self.state)
def plot(self):
"""plot in time"""
plt.plot(self.t, self.y)
plt.xlabel('t'); plt.ylabel('y')
def run(self):
"""simulate the ODE"""
self.simulate()
self.plot()
Here is its example usage
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import odesim
import importlib
#importlib.reload(odesim) # when odesim.py is updated
sim = odesim.odesim('first')
Importing ODE: first
sim.run()
t= 10.0 ; state= [-0.36787947]
![_images/284e4cac6d9370899e72b59adaa762547e398dd037e906be863b20207aad3881.png](_images/284e4cac6d9370899e72b59adaa762547e398dd037e906be863b20207aad3881.png)
sim2 = odesim.odesim('second')
Importing ODE: second
sim2.run()
t= 10.0 ; state= [ 0.52920895 -0.32397961]
![_images/1516023ebdc8feeaf93e23b2bce5ee89e6e9fba3b9f9ff23f1ba0f32f8647a34.png](_images/1516023ebdc8feeaf93e23b2bce5ee89e6e9fba3b9f9ff23f1ba0f32f8647a34.png)
Starting a repository#
Let us go to your working folder and start a new repository by git init
%pwd
'/Users/doya/OIST Dropbox/kenji doya/Python/iSciComp/odesim'
!git init
Reinitialized existing Git repository in /Users/doya/OIST Dropbox/kenji doya/Python/iSciComp/odesim/.git/
This creates an invisible folder .git
for book keeping.
%ls -a
./ .git/ __pycache__/ odesim.py second.py
../ .gitignore first.py odesim.py~
# The contents of .git folder
%ls .git
COMMIT_EDITMSG config index objects/
HEAD description info/ refs/
ORIG_HEAD hooks/ logs/
You can check the status by git status
!git status
On branch main
nothing to commit, working tree clean
Staging and Commiting files#
Use git add
to add files for tracking.
And then git commit
to save a version.
!git add *.py
!git status
On branch main
nothing to commit, working tree clean
Register the current version by git commit
with a message by -m
.
!git commit -m "first version"
!git status
On branch main
nothing to commit, working tree clean
On branch main
nothing to commit, working tree clean
You can list files that need not to be tracked in .gitigonore
file
!echo '.*\n__*\n*~\n' > .gitignore
!cat .gitignore
!git status
.*
__*
*~
On branch main
nothing to commit, working tree clean
Registering changes#
After editing a file, you can register a new version by git commit
.
Try changing a parameter or initial state, e.g., in dynamics/first.py
!git status
On branch main
nothing to commit, working tree clean
Use git add
to stage updated files.
!git add first.py
!git status
On branch main
nothing to commit, working tree clean
And the git commit
the changes
!git commit first.py -m "first.py updated"
!git status
On branch main
nothing to commit, working tree clean
On branch main
nothing to commit, working tree clean
You can see what was changed by git show
.
!git show
commit a121e1fa5fe692dec10c125390d633b23d42a6d9 (HEAD -> main)
Author: Kenji Doya <doya@oist.jp>
Date: Wed Aug 16 22:36:50 2023 +0200
first version
diff --git a/odesim.py b/odesim.py
index 02e7d0e..4d5391b 100644
--- a/odesim.py
+++ b/odesim.py
@@ -20,13 +20,16 @@ class odesim():
print('Importing ODE:', odename)
self.ode = importlib.import_module( odename)
importlib.reload(self.ode) # for updated module
- self.reset()
-
+ self.state = self.ode.initial_state
+ #self.reset()
+ plt.ion()
+
def reset(self):
"""reset the state"""
self.time = 0
- self.state = self.ode.initial_state
-
+ self.state = self.ode.initial_state
+ plt.clf()
+
def simulate(self):
"""simulate the ODE"""
# +dt/2 to include time+trun
@@ -41,7 +44,6 @@ class odesim():
"""plot in time"""
plt.plot(self.t, self.y)
plt.xlabel('t'); plt.ylabel('y')
- plt.show()
def run(self):
"""simulate the ODE"""
You can check the revision history by git log
!git log
commit a121e1fa5fe692dec10c125390d633b23d42a6d9 (HEAD -> main)
Author: Kenji Doya <doya@oist.jp>
Date: Wed Aug 16 22:36:50 2023 +0200
first version
commit 9b97d69bb5a0872bf342a2cba9e53a6ec6c5f590 (myBranch)
Author: Kenji Doya <doya@oist.jp>
Date: Tue Dec 6 12:24:44 2022 +0900
second.py updated
commit 8b01220bd8dfe78f71297e62e962758e4977caa4
Author: Kenji Doya <doya@oist.jp>
Date: Tue Dec 6 12:24:17 2022 +0900
first.py updated
commit 9a1b45a6daceb403255b30febd3b0dc8fb3f58ee
Author: Kenji Doya <doya@oist.jp>
Date: Tue Dec 6 12:22:25 2022 +0900
first version
Branch#
You can create a new branch and checkout a particular branch.
!git branch myBranch
!git checkout myBranch
fatal: A branch named 'myBranch' already exists.
Switched to branch 'myBranch'
Make a change, e.g., editing second.py
.
And then git add
and git commit
.
!git add second.py
!git commit -m "second.py updated"
!git status
On branch myBranch
nothing to commit, working tree clean
On branch myBranch
nothing to commit, working tree clean
!git show
commit 9b97d69bb5a0872bf342a2cba9e53a6ec6c5f590 (HEAD -> myBranch)
Author: Kenji Doya <doya@oist.jp>
Date: Tue Dec 6 12:24:44 2022 +0900
second.py updated
diff --git a/second.py b/second.py
index 785645a..2c8638b 100644
--- a/second.py
+++ b/second.py
@@ -15,4 +15,4 @@ def dynamics(y, t, a):
parameters = [-0.1, -1, 0]
# Default initial state
-initial_state = [1, 0]
+initial_state = [-1, 0]
!git log --all --graph
* commit a121e1fa5fe692dec10c125390d633b23d42a6d9 (main)
| Author: Kenji Doya <doya@oist.jp>
| Date: Wed Aug 16 22:36:50 2023 +0200
|
| first version
|
* commit 9b97d69bb5a0872bf342a2cba9e53a6ec6c5f590 (HEAD -> myBranch)
| Author: Kenji Doya <doya@oist.jp>
| Date: Tue Dec 6 12:24:44 2022 +0900
|
| second.py updated
|
* commit 8b01220bd8dfe78f71297e62e962758e4977caa4
| Author: Kenji Doya <doya@oist.jp>
| Date: Tue Dec 6 12:24:17 2022 +0900
|
| first.py updated
|
* commit 9a1b45a6daceb403255b30febd3b0dc8fb3f58ee
Author: Kenji Doya <doya@oist.jp>
Date: Tue Dec 6 12:22:25 2022 +0900
first version
You can go back to a previous branch by checkout.
!git checkout main
!git log --all --graph
Switched to branch 'main'
* commit a121e1fa5fe692dec10c125390d633b23d42a6d9 (HEAD -> main)
| Author: Kenji Doya <doya@oist.jp>
| Date: Wed Aug 16 22:36:50 2023 +0200
|
| first version
|
* commit 9b97d69bb5a0872bf342a2cba9e53a6ec6c5f590 (myBranch)
| Author: Kenji Doya <doya@oist.jp>
| Date: Tue Dec 6 12:24:44 2022 +0900
|
| second.py updated
|
* commit 8b01220bd8dfe78f71297e62e962758e4977caa4
| Author: Kenji Doya <doya@oist.jp>
| Date: Tue Dec 6 12:24:17 2022 +0900
|
| first.py updated
|
* commit 9a1b45a6daceb403255b30febd3b0dc8fb3f58ee
Author: Kenji Doya <doya@oist.jp>
Date: Tue Dec 6 12:22:25 2022 +0900
first version
!git branch
* main
myBranch
You can merge another branche to the current branch by git merge
!git merge myBranch
!git log --all --graph
Already up to date.
* commit a121e1fa5fe692dec10c125390d633b23d42a6d9 (HEAD -> main)
| Author: Kenji Doya <doya@oist.jp>
| Date: Wed Aug 16 22:36:50 2023 +0200
|
| first version
|
* commit 9b97d69bb5a0872bf342a2cba9e53a6ec6c5f590 (myBranch)
| Author: Kenji Doya <doya@oist.jp>
| Date: Tue Dec 6 12:24:44 2022 +0900
|
| second.py updated
|
* commit 8b01220bd8dfe78f71297e62e962758e4977caa4
| Author: Kenji Doya <doya@oist.jp>
| Date: Tue Dec 6 12:24:17 2022 +0900
|
| first.py updated
|
* commit 9a1b45a6daceb403255b30febd3b0dc8fb3f58ee
Author: Kenji Doya <doya@oist.jp>
Date: Tue Dec 6 12:22:25 2022 +0900
first version
GitHub#
GitHub is currently the most popular cloud service for sharing software. It is free for open software.
This is a good platform for sharing programs, or in some cases text data and manuscripts, among collaborators. It is also helpful for a single-person project, for succession by a future member of your lab, for open access after publication, or for yourself after some time.
These are typical steps in contributing to a project in GitHub.
Join as a member of a repository.
Copy the existing files and see how they work.
Make a new branch and add or modify the codes.
After tesing locally, commit the new version.
Open a pull request for other members to test your revision.
Your pull request is merged into the master branch.
See “Hello World” in GitHub Guide for details (https://guides.github.com).
Cloning a repository#
If you just use a copy of a stable software, and not going to contribute your changes, just downloading a zip file is fine.
But if you would congribute to joint development, or catch up with updates, git clone
is the better way.
Cloning ComputationalMethods repository#
To download a copy of the repository, run
git clone git@github.com:oist/ComputationalMethods2022.git
You are asked to input the passphrase you set in creating your SSH Key.
This should create a folder ComputationalMethods2022
.
%pwd
'/Users/doya/OIST Dropbox/kenji doya/Python/iSciComp/odesim'
!git clone git@github.com:oist/ComputationalMethods2022.git
Cloning into 'ComputationalMethods2022'...
remote: Enumerating objects: 578, done.
remote: Counting objects: 0% (1/232)
remote: Counting objects: 1% (3/232)
remote: Counting objects: 2% (5/232)
remote: Counting objects: 3% (7/232)
remote: Counting objects: 4% (10/232)
remote: Counting objects: 5% (12/232)
remote: Counting objects: 6% (14/232)
remote: Counting objects: 7% (17/232)
remote: Counting objects: 8% (19/232)
remote: Counting objects: 9% (21/232)
remote: Counting objects: 10% (24/232)
remote: Counting objects: 11% (26/232)
remote: Counting objects: 12% (28/232)
remote: Counting objects: 13% (31/232)
remote: Counting objects: 14% (33/232)
remote: Counting objects: 15% (35/232)
remote: Counting objects: 16% (38/232)
remote: Counting objects: 17% (40/232)
remote: Counting objects: 18% (42/232)
remote: Counting objects: 19% (45/232)
remote: Counting objects: 20% (47/232)
remote: Counting objects: 21% (49/232)
remote: Counting objects: 22% (52/232)
remote: Counting objects: 23% (54/232)
remote: Counting objects: 24% (56/232)
remote: Counting objects: 25% (58/232)
remote: Counting objects: 26% (61/232)
remote: Counting objects: 27% (63/232)
remote: Counting objects: 28% (65/232)
remote: Counting objects: 29% (68/232)
remote: Counting objects: 30% (70/232)
remote: Counting objects: 31% (72/232)
remote: Counting objects: 32% (75/232)
remote: Counting objects: 33% (77/232)
remote: Counting objects: 34% (79/232)
remote: Counting objects: 35% (82/232)
remote: Counting objects: 36% (84/232)
remote: Counting objects: 37% (86/232)
remote: Counting objects: 38% (89/232)
remote: Counting objects: 39% (91/232)
remote: Counting objects: 40% (93/232)
remote: Counting objects: 41% (96/232)
remote: Counting objects: 42% (98/232)
remote: Counting objects: 43% (100/232)
remote: Counting objects: 44% (103/232)
remote: Counting objects: 45% (105/232)
remote: Counting objects: 46% (107/232)
remote: Counting objects: 47% (110/232)
remote: Counting objects: 48% (112/232)
remote: Counting objects: 49% (114/232)
remote: Counting objects: 50% (116/232)
remote: Counting objects: 51% (119/232)
remote: Counting objects: 52% (121/232)
remote: Counting objects: 53% (123/232)
remote: Counting objects: 54% (126/232)
remote: Counting objects: 55% (128/232)
remote: Counting objects: 56% (130/232)
remote: Counting objects: 57% (133/232)
remote: Counting objects: 58% (135/232)
remote: Counting objects: 59% (137/232)
remote: Counting objects: 60% (140/232)
remote: Counting objects: 61% (142/232)
remote: Counting objects: 62% (144/232)
remote: Counting objects: 63% (147/232)
remote: Counting objects: 64% (149/232)
remote: Counting objects: 65% (151/232)
remote: Counting objects: 66% (154/232)
remote: Counting objects: 67% (156/232)
remote: Counting objects: 68% (158/232)
remote: Counting objects: 69% (161/232)
remote: Counting objects: 70% (163/232)
remote: Counting objects: 71% (165/232)
remote: Counting objects: 72% (168/232)
remote: Counting objects: 73% (170/232)
remote: Counting objects: 74% (172/232)
remote: Counting objects: 75% (174/232)
remote: Counting objects: 76% (177/232)
remote: Counting objects: 77% (179/232)
remote: Counting objects: 78% (181/232)
remote: Counting objects: 79% (184/232)
remote: Counting objects: 80% (186/232)
remote: Counting objects: 81% (188/232)
remote: Counting objects: 82% (191/232)
remote: Counting objects: 83% (193/232)
remote: Counting objects: 84% (195/232)
remote: Counting objects: 85% (198/232)
remote: Counting objects: 86% (200/232)
remote: Counting objects: 87% (202/232)
remote: Counting objects: 88% (205/232)
remote: Counting objects: 89% (207/232)
remote: Counting objects: 90% (209/232)
remote: Counting objects: 91% (212/232)
remote: Counting objects: 92% (214/232)
remote: Counting objects: 93% (216/232)
remote: Counting objects: 94% (219/232)
remote: Counting objects: 95% (221/232)
remote: Counting objects: 96% (223/232)
remote: Counting objects: 97% (226/232)
remote: Counting objects: 98% (228/232)
remote: Counting objects: 99% (230/232)
remote: Counting objects: 100% (232/232)
remote: Counting objects: 100% (232/232), done.
remote: Compressing objects: 0% (1/152)
remote: Compressing objects: 1% (2/152)
remote: Compressing objects: 2% (4/152)
remote: Compressing objects: 3% (5/152)
remote: Compressing objects: 4% (7/152)
remote: Compressing objects: 5% (8/152)
remote: Compressing objects: 6% (10/152)
remote: Compressing objects: 7% (11/152)
remote: Compressing objects: 8% (13/152)
remote: Compressing objects: 9% (14/152)
remote: Compressing objects: 10% (16/152)
remote: Compressing objects: 11% (17/152)
remote: Compressing objects: 12% (19/152)
remote: Compressing objects: 13% (20/152)
remote: Compressing objects: 14% (22/152)
remote: Compressing objects: 15% (23/152)
remote: Compressing objects: 16% (25/152)
remote: Compressing objects: 17% (26/152)
remote: Compressing objects: 18% (28/152)
remote: Compressing objects: 19% (29/152)
remote: Compressing objects: 20% (31/152)
remote: Compressing objects: 21% (32/152)
remote: Compressing objects: 22% (34/152)
remote: Compressing objects: 23% (35/152)
remote: Compressing objects: 24% (37/152)
remote: Compressing objects: 25% (38/152)
remote: Compressing objects: 26% (40/152)
remote: Compressing objects: 27% (42/152)
remote: Compressing objects: 28% (43/152)
remote: Compressing objects: 29% (45/152)
remote: Compressing objects: 30% (46/152)
remote: Compressing objects: 31% (48/152)
remote: Compressing objects: 32% (49/152)
remote: Compressing objects: 33% (51/152)
remote: Compressing objects: 34% (52/152)
remote: Compressing objects: 35% (54/152)
remote: Compressing objects: 36% (55/152)
remote: Compressing objects: 37% (57/152)
remote: Compressing objects: 38% (58/152)
remote: Compressing objects: 39% (60/152)
remote: Compressing objects: 40% (61/152)
remote: Compressing objects: 41% (63/152)
remote: Compressing objects: 42% (64/152)
remote: Compressing objects: 43% (66/152)
remote: Compressing objects: 44% (67/152)
remote: Compressing objects: 45% (69/152)
remote: Compressing objects: 46% (70/152)
remote: Compressing objects: 47% (72/152)
remote: Compressing objects: 48% (73/152)
remote: Compressing objects: 49% (75/152)
remote: Compressing objects: 50% (76/152)
remote: Compressing objects: 51% (78/152)
remote: Compressing objects: 52% (80/152)
remote: Compressing objects: 53% (81/152)
remote: Compressing objects: 54% (83/152)
remote: Compressing objects: 55% (84/152)
remote: Compressing objects: 56% (86/152)
remote: Compressing objects: 57% (87/152)
remote: Compressing objects: 58% (89/152)
remote: Compressing objects: 59% (90/152)
remote: Compressing objects: 60% (92/152)
remote: Compressing objects: 61% (93/152)
remote: Compressing objects: 62% (95/152)
remote: Compressing objects: 63% (96/152)
remote: Compressing objects: 64% (98/152)
remote: Compressing objects: 65% (99/152)
remote: Compressing objects: 66% (101/152)
remote: Compressing objects: 67% (102/152)
remote: Compressing objects: 68% (104/152)
remote: Compressing objects: 69% (105/152)
remote: Compressing objects: 70% (107/152)
remote: Compressing objects: 71% (108/152)
remote: Compressing objects: 72% (110/152)
remote: Compressing objects: 73% (111/152)
remote: Compressing objects: 74% (113/152)
remote: Compressing objects: 75% (114/152)
remote: Compressing objects: 76% (116/152)
remote: Compressing objects: 77% (118/152)
remote: Compressing objects: 78% (119/152)
remote: Compressing objects: 79% (121/152)
remote: Compressing objects: 80% (122/152)
remote: Compressing objects: 81% (124/152)
remote: Compressing objects: 82% (125/152)
remote: Compressing objects: 83% (127/152)
remote: Compressing objects: 84% (128/152)
remote: Compressing objects: 85% (130/152)
remote: Compressing objects: 86% (131/152)
remote: Compressing objects: 87% (133/152)
remote: Compressing objects: 88% (134/152)
remote: Compressing objects: 89% (136/152)
remote: Compressing objects: 90% (137/152)
remote: Compressing objects: 91% (139/152)
remote: Compressing objects: 92% (140/152)
remote: Compressing objects: 93% (142/152)
remote: Compressing objects: 94% (143/152)
remote: Compressing objects: 95% (145/152)
remote: Compressing objects: 96% (146/152)
remote: Compressing objects: 97% (148/152)
remote: Compressing objects: 98% (149/152)
remote: Compressing objects: 99% (151/152)
remote: Compressing objects: 100% (152/152)
remote: Compressing objects: 100% (152/152), done.
Receiving objects: 0% (1/578)
Receiving objects: 1% (6/578)
Receiving objects: 2% (12/578)
Receiving objects: 3% (18/578)
Receiving objects: 4% (24/578)
Receiving objects: 5% (29/578)
Receiving objects: 6% (35/578)
Receiving objects: 7% (41/578)
Receiving objects: 8% (47/578)
Receiving objects: 9% (53/578)
Receiving objects: 10% (58/578)
Receiving objects: 11% (64/578)
Receiving objects: 12% (70/578)
Receiving objects: 13% (76/578)
Receiving objects: 14% (81/578)
Receiving objects: 15% (87/578)
Receiving objects: 16% (93/578)
Receiving objects: 17% (99/578)
Receiving objects: 18% (105/578)
Receiving objects: 19% (110/578)
Receiving objects: 20% (116/578)
Receiving objects: 21% (122/578)
Receiving objects: 22% (128/578)
Receiving objects: 23% (133/578)
Receiving objects: 24% (139/578)
Receiving objects: 25% (145/578)
Receiving objects: 26% (151/578)
Receiving objects: 27% (157/578)
Receiving objects: 28% (162/578)
Receiving objects: 29% (168/578)
Receiving objects: 30% (174/578)
Receiving objects: 31% (180/578)
Receiving objects: 32% (185/578)
Receiving objects: 33% (191/578)
Receiving objects: 34% (197/578)
Receiving objects: 35% (203/578)
Receiving objects: 36% (209/578)
Receiving objects: 37% (214/578)
Receiving objects: 38% (220/578)
Receiving objects: 39% (226/578)
Receiving objects: 40% (232/578)
Receiving objects: 41% (237/578)
Receiving objects: 42% (243/578)
Receiving objects: 43% (249/578)
Receiving objects: 44% (255/578)
Receiving objects: 45% (261/578)
Receiving objects: 46% (266/578)
Receiving objects: 47% (272/578)
Receiving objects: 48% (278/578)
Receiving objects: 49% (284/578)
Receiving objects: 50% (289/578)
Receiving objects: 51% (295/578)
Receiving objects: 52% (301/578)
Receiving objects: 53% (307/578)
Receiving objects: 54% (313/578)
Receiving objects: 55% (318/578)
Receiving objects: 56% (324/578)
Receiving objects: 57% (330/578)
Receiving objects: 58% (336/578)
Receiving objects: 59% (342/578)
Receiving objects: 60% (347/578)
Receiving objects: 61% (353/578)
Receiving objects: 62% (359/578)
Receiving objects: 63% (365/578)
Receiving objects: 64% (370/578)
Receiving objects: 65% (376/578)
Receiving objects: 66% (382/578)
Receiving objects: 67% (388/578)
Receiving objects: 68% (394/578)
Receiving objects: 69% (399/578)
Receiving objects: 70% (405/578)
Receiving objects: 71% (411/578)
Receiving objects: 72% (417/578)
Receiving objects: 73% (422/578)
Receiving objects: 74% (428/578)
Receiving objects: 75% (434/578)
Receiving objects: 76% (440/578)
Receiving objects: 77% (446/578)
Receiving objects: 78% (451/578)
Receiving objects: 79% (457/578)
Receiving objects: 80% (463/578)
Receiving objects: 81% (469/578)
Receiving objects: 82% (474/578)
Receiving objects: 83% (480/578)
Receiving objects: 84% (486/578)
Receiving objects: 85% (492/578)
Receiving objects: 86% (498/578)
Receiving objects: 87% (503/578)
Receiving objects: 88% (509/578)
Receiving objects: 89% (515/578)
Receiving objects: 90% (521/578)
Receiving objects: 91% (526/578)
Receiving objects: 92% (532/578)
Receiving objects: 93% (538/578)
Receiving objects: 94% (544/578)
Receiving objects: 95% (550/578)
Receiving objects: 96% (555/578)
remote: Total 578 (delta 126), reused 150 (delta 76), pack-reused 346 (from 1)
Receiving objects: 97% (561/578)
Receiving objects: 98% (567/578)
Receiving objects: 99% (573/578)
Receiving objects: 100% (578/578)
Receiving objects: 100% (578/578), 360.27 KiB | 897.00 KiB/s, done.
Resolving deltas: 0% (0/280)
Resolving deltas: 1% (3/280)
Resolving deltas: 2% (6/280)
Resolving deltas: 3% (9/280)
Resolving deltas: 4% (12/280)
Resolving deltas: 5% (14/280)
Resolving deltas: 6% (17/280)
Resolving deltas: 7% (20/280)
Resolving deltas: 8% (23/280)
Resolving deltas: 9% (26/280)
Resolving deltas: 10% (28/280)
Resolving deltas: 11% (31/280)
Resolving deltas: 12% (34/280)
Resolving deltas: 13% (37/280)
Resolving deltas: 14% (40/280)
Resolving deltas: 15% (42/280)
Resolving deltas: 16% (45/280)
Resolving deltas: 17% (48/280)
Resolving deltas: 18% (51/280)
Resolving deltas: 19% (54/280)
Resolving deltas: 20% (56/280)
Resolving deltas: 21% (59/280)
Resolving deltas: 22% (62/280)
Resolving deltas: 23% (65/280)
Resolving deltas: 24% (68/280)
Resolving deltas: 25% (70/280)
Resolving deltas: 26% (73/280)
Resolving deltas: 27% (76/280)
Resolving deltas: 28% (79/280)
Resolving deltas: 29% (82/280)
Resolving deltas: 30% (84/280)
Resolving deltas: 31% (87/280)
Resolving deltas: 32% (90/280)
Resolving deltas: 33% (93/280)
Resolving deltas: 34% (96/280)
Resolving deltas: 35% (98/280)
Resolving deltas: 36% (101/280)
Resolving deltas: 37% (104/280)
Resolving deltas: 38% (107/280)
Resolving deltas: 39% (110/280)
Resolving deltas: 40% (112/280)
Resolving deltas: 41% (115/280)
Resolving deltas: 42% (118/280)
Resolving deltas: 43% (121/280)
Resolving deltas: 44% (124/280)
Resolving deltas: 45% (126/280)
Resolving deltas: 46% (129/280)
Resolving deltas: 47% (132/280)
Resolving deltas: 48% (135/280)
Resolving deltas: 49% (138/280)
Resolving deltas: 50% (140/280)
Resolving deltas: 51% (143/280)
Resolving deltas: 52% (146/280)
Resolving deltas: 53% (149/280)
Resolving deltas: 54% (152/280)
Resolving deltas: 55% (154/280)
Resolving deltas: 56% (157/280)
Resolving deltas: 57% (160/280)
Resolving deltas: 58% (164/280)
Resolving deltas: 59% (166/280)
Resolving deltas: 60% (168/280)
Resolving deltas: 61% (171/280)
Resolving deltas: 62% (174/280)
Resolving deltas: 63% (177/280)
Resolving deltas: 64% (180/280)
Resolving deltas: 65% (182/280)
Resolving deltas: 66% (185/280)
Resolving deltas: 67% (188/280)
Resolving deltas: 68% (192/280)
Resolving deltas: 69% (194/280)
Resolving deltas: 70% (196/280)
Resolving deltas: 71% (199/280)
Resolving deltas: 72% (202/280)
Resolving deltas: 73% (205/280)
Resolving deltas: 74% (208/280)
Resolving deltas: 75% (210/280)
Resolving deltas: 76% (213/280)
Resolving deltas: 77% (216/280)
Resolving deltas: 78% (219/280)
Resolving deltas: 79% (222/280)
Resolving deltas: 80% (224/280)
Resolving deltas: 81% (227/280)
Resolving deltas: 82% (230/280)
Resolving deltas: 83% (233/280)
Resolving deltas: 84% (236/280)
Resolving deltas: 85% (238/280)
Resolving deltas: 86% (241/280)
Resolving deltas: 87% (244/280)
Resolving deltas: 88% (247/280)
Resolving deltas: 89% (250/280)
Resolving deltas: 90% (252/280)
Resolving deltas: 91% (255/280)
Resolving deltas: 92% (258/280)
Resolving deltas: 93% (261/280)
Resolving deltas: 94% (264/280)
Resolving deltas: 95% (266/280)
Resolving deltas: 96% (269/280)
Resolving deltas: 97% (272/280)
Resolving deltas: 98% (275/280)
Resolving deltas: 99% (278/280)
Resolving deltas: 100% (280/280)
Resolving deltas: 100% (280/280), done.
%ls
ComputationalMethods2022/ first.py odesim.py~
__pycache__/ odesim.py second.py
Move into the folder and test odesim.py
program.
%cd ComputationalMethods2022
/Users/doya/OIST Dropbox/kenji doya/Python/iSciComp/odesim/ComputationalMethods2022
%ls
README.md odesim.py second.py test.py
first.py outputs/ simple_growth.py
lorenz.py pendulum.py tatsuo.py
From the console you can run interactively after reading the module as:
python -i odesim.py
sim = odesim('first')
sim.run()
from odesim import *
sim = odesim('first')
Importing ODE: first
sim.run()
t= 10.0 ; state= [0.36787947]
![_images/2e71ffd92071be535558426342887020fdd90a5db6588dbfcec01e2cde3d577d.png](_images/2e71ffd92071be535558426342887020fdd90a5db6588dbfcec01e2cde3d577d.png)
Your branch#
Now make your own branch, check it out, and add your own ODE module.
!git branch myname
!git checkout myname
Switched to branch 'myname'
Make a copy of a dynamics file first.py
or second.py
, implement your own ODE, and save with a new name, e.g. vdp.py
.
Run odesim and confirm that your ODE runs appropriately.
Then you can add and commit your change.
!git status
On branch myname
nothing to commit, working tree clean
!git add vdp.py
fatal: pathspec 'vdp.py' did not match any files
!git commit -m "adding my model vdp.py"
On branch myname
nothing to commit, working tree clean
!git log --graph --oneline --all
* c7c6fe1 (HEAD -> myname, origin/main, origin/HEAD, main) Merge pull request #5 from oist/Sougato
|\
| * 3784ba9 (origin/Sougato) Adding pendulum.py after fixing errors
| * 5d6b385 Adding a pendulum ODE module
* | cdce9fb Merge pull request #4 from oist/Amy
|\ \
| * | 79900ea (origin/Amy) changed into a cubic nonlinearity
| * | 4c16aca edited test.py
| * | 831f82f adding my model test.py
| * | 0ef7d6a adding my model test.py
* | | 86c34e3 Merge pull request #6 from oist/lily
|\ \ \
| * | | dc9432c (origin/lily) Create simple_growth.py
|/ / /
* | | 84f49b7 Merge pull request #2 from oist/rui_fukushima
|\ \ \
| * | | e6b32f9 (origin/rui_fukushima) fix parameter part according prof. comments
| * | | f73f9c2 remove vdp module
| * | | 56110fe remove project notebook
| * | | 7b64069 add lorenz module
| * | | 0e84f70 adding personal project notebook
| * | | 28d072a adding my model vdp.py
| | |/
| |/|
* | | d546ed1 Merge pull request #3 from oist/tatsuo
|\ \ \
| |_|/
|/| |
| * | 919412b (origin/tatsuo) adding my model tatsuo.py
| * | 7109914 adding my model ti.py
| * | ad683ef adding my model tatsuo.py
| |/
| | * c6538a0 (origin/fhn) added pplot method to odesim.py
| | * 0fb56f7 fhn.py module and run_fhn.py for testing it in scripts folder
| | * 37e5425 (origin/kazuma-class2022AY1) Feat: add henonmap and cleanup fhn model
| | * 409d7b5 Feat: add FHN model in odesim.py
| | * d510ba8 Fix: move all images to outputs dir
| | * ed3afff Merge branch 'main' into kazuma-class2022AY1
| | |\
| |_|/
|/| |
* | | 576a0aa added outputs folder
|/ /
| * 16b8943 Feat: add phase plot and bifuraction diagram
| * 4ef681d Feat: add simple fhn plot
| * 5d2b871 Feat: add FHN model script
| * 64e1fbc Feat: add cell executer comment
| * a93c0e4 Feat: create a new script
|/
| * 2ffd690 (origin/kenji) adding my model vdp.py
|/
* 19a1a40 (origin/initial) added .gitignore
* eab5bf5 initial form
* 96d27bf (origin/clean) Delete old folders
* 680c44b Delete old folders
* d9e432e Delete old folders
* a630596 Created for 2022
* 1dc906b first three dynamics functions
* ef02eb6 original dynamics viewer
* 9251e5d Cleanup
* 4ed4c6e Create Readme.md
* d505312 Update README.md
* 205f2b2 Cleanup
* bfb02e6 Cleanup
* f9de384 Cleanup
* c00ceb6 Cleanup
* ccf6534 Cleanup
* 7aa7977 Delete Masato-first.py
* f20fa86 Merge pull request #50 from oist/Masato
|\
| * 89fedbd Update Masato-first.py
| * e5d01a5 Add files via upload
|/
* e86fea4 Merge pull request #38 from oist/deleting-the-files
|\
| * 8c71105 Delete jan-second.py
| * 00bb569 Delete jan-first.py
|/
* 84971c4 Merge pull request #35 from oist/jan-grasic
|\
| * f1d48db Add files via upload
|/
* 12c9d16 Dynamics viewer
* 8470b8e Delete DynaView.py
* b32f02a sample dynamics files
* eae595e Uploaded the same code as 2018
* 523a383 Create Readme.md
* 5b429b5 Update Readme.md
* f887d7f Create Readme.md
* fbf5185 Merge pull request #32 from oist/hoching
|\
| * 5f628fb changed the dynamic function
|/
* 56e6fc2 Merge pull request #28 from oist/Darshini
|\
| * 51bc33d Add files via upload
* | 602fddf Merge pull request #27 from oist/aisulu_maipas
|\ \
| * | 65845a7 Add files via upload
| * | c3c4e9d adding my script
* | | 096fe7b Merge pull request #31 from oist/AnzhelikaKoldaeva
|\ \ \
| * | | 3750870 added new functions to ODE solver
|/ / /
* | | f2455b3 Merge pull request #25 from oist/Angela-Kirykowicz
|\ \ \
| |/ /
|/| |
| * | fd56c93 Add files via upload
| |/
* | d6fe3f4 Merge pull request #26 from oist/lazzat
|\ \
| |/
|/|
| * 08c9647 adding my script
* | 7f5ef37 Delete Lewis_wavy.py due to incorrect location
|/
* 478e839 Example
* eaf500d Update lewis_wavy.py
* c91b30a Add files via upload
* c0188d2 Update lewis_wavy.py
* a609a28 Merge pull request #24 from oist/module-scan
|\
| * a1ac524 Merge branch 'master' into module-scan
| |\
| |/
|/|
* | 00b93e4 Update DynaView.py
* | 628b8c5 Add files via upload
| * 6aa4601 parameters format changed
| * c18d6e0 Parameters and state entry boxes added
| * 9fd54f3 Update Readme.md
| * 4650244 moved to "dynamics"
| * d5f6914 moved to "dynamics"
| * 18871c5 Scan modules in "dynamics" folder
| * e9080a9 first three modules
| * eab81cb Create Readme.md
|/
* 9e9ed1b 2017 files
* 5319297 Delete third_lewis.py
* 9a6d07d Delete wataru.py
* fad7706 Delete second.py
* beee04f Delete logistic.py
* e26d14a Delete first_nonlinear.py
* 309db7b Delete first.py
* 751c472 Delete Sarah.py
* fed919f Delete Pendulum.py
* b1e1da9 Delete FHN.py
* 617279c Delete DynaView.py
* fb5acc5 2017 dynamics files merged manually
* aa5fb68 Merge pull request #11 from oist/darya
|\
| * 1e05d8c Merge branch 'master' into darya
| |\
| |/
|/|
* | 598ce08 Merge pull request #14 from oist/Rachapun
|\ \
| * | b760678 Update first_Rachapun.py
* | | 38f4e08 Merge pull request #15 from oist/linlingru
|\ \ \
| * \ \ ae13d99 Merge branch 'master' into linlingru
| |\ \ \
| |/ / /
|/| | |
* | | | 8c6b6db Merge pull request #16 from oist/Charles_W_Edits
|\ \ \ \
| * | | | 0135d42 Update DynaView.py
| * | | | 2860ee0 Update first.py
| * | | | 4e09dca Update second.py
| * | | | f723342 Update first.py
* | | | | 8f4bdb9 Merge pull request #17 from oist/Osamu
|\ \ \ \ \
| * \ \ \ \ a30c13b Merge branch 'master' into Osamu
| |\ \ \ \ \
| |/ / / / /
|/| | | | |
* | | | | | 668a73e Merge pull request #19 from oist/Wataru
|\ \ \ \ \ \
| * \ \ \ \ \ bce065e Merge branch 'master' into Wataru
| |\ \ \ \ \ \
| |/ / / / / /
|/| | | | | |
* | | | | | | e81837b Merge pull request #23 from oist/Mizuki_Kato
|\ \ \ \ \ \ \
| * \ \ \ \ \ \ 8f3bb8f Merge branch 'master' into Mizuki_Kato
| |\ \ \ \ \ \ \
| |/ / / / / / /
|/| | | | | | |
* | | | | | | | 7497aa6 Merge branch 'lewis'
|\ \ \ \ \ \ \ \
| * | | | | | | | b771b88 Add files via upload
| * | | | | | | | 459f682 Add files via upload
* | | | | | | | | e3357a6 Update Readme.md
* | | | | | | | | ab0307c Delete touseef_second.py
* | | | | | | | | fc11022 Delete touseef_first.py
* | | | | | | | | 1b0b636 Delete hsiaochiao
* | | | | | | | | 8241eb7 Delete DynaView.py
* | | | | | | | | 05bbfb0 Delete Readme.md
* | | | | | | | | 94bf611 Delete Readme.md
* | | | | | | | | 3b346d4 2018 course material
* | | | | | | | | 8b2443b Initial files by KD
* | | | | | | | | 9431195 Update Readme.md
* | | | | | | | | a58e3e4 Create Readme.md
* | | | | | | | | 2b2726b Create Readme.md
* | | | | | | | | e2a4ffa Create Readme.md
| | * | | | | | | cf01dbd Add files via upload
| |/ / / / / / /
|/| | | | | | |
* | | | | | | | a886358 Merge pull request #22 from 123miles/123miles-patch-9
|\ \ \ \ \ \ \ \
| * | | | | | | | b5262f5 Add files via upload
| | |_|_|/ / / /
| |/| | | | | |
* | | | | | | | 36feb34 Merge pull request #21 from 123miles/readme-edits
|\ \ \ \ \ \ \ \
| |/ / / / / / /
|/| | | | | | |
| * | | | | | | 4bc04b8 Merge pull request #1 from 123miles/123miles1
|/| | | | | | |
| * | | | | | | 3e58714 Update README.md
|/ / / / / / /
| | * / / / / 7fc4940 Add files via upload
| |/ / / / /
| | * | | | f965764 Create logistic.py
| | * | | | d21975f Update DynaView.py
| |/ / / /
|/| | | |
* | | | | 31214f0 Merge pull request #13 from oist/Rachapun
|\ \ \ \ \
| | |_|/ /
| |/| | |
| * | | | d544025 Update DynaView.py
| * | | | 11fa29b Add files via upload
|/ / / /
| | * | c1ee017 Update DynaView.py
| | * | 77b19a1 Add files via upload
| | * | a880143 Delete linlingru_first.py
| | * | cf7ef67 update first.py
| | * | c0c018a Add files via upload
| |/ /
|/| |
| | * de06314 Add files via upload
| | * aa0c972 Update DynaView.py
| |/
|/|
* | 558a07e Update Readme.md
* | 5bf9e0f Update Readme.md
* | dc803f7 Some notes on the exercise
* | 8e2b144 Merge pull request #9 from otistwo/master
|\ \
| * | c140ae9 changed first and second to otis
| |/
* | 73cdbca Merge pull request #8 from oist/Touseef
|\ \
| * | 66bf463 Add files via upload
| |/
* / c456b82 i updated first_Tabbal
|/
* 69e0154 Create hsiaochiao
* fd4d5c5 DynaView.py and two dynamics modules
* 3d5b491 Create Readme.md
* baf2bb6 First version for "Software Management"
* 50cabde Initialized Readme.md
* b8dc4bb Initial commit
Now push your branch to GitHub repository by, e.g.
git push origin myname
!git push origin myname
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: Create a pull request for 'myname' on GitHub by visiting:
remote: https://github.com/oist/ComputationalMethods2022/pull/new/myname
remote:
To github.com:oist/ComputationalMethods2022.git
* [new branch] myname -> myname
Check the status on GitHub: oist/ComputationalMethods2022
and make a pull request for the repository administrator to check your updates.
The administrator may reply back with a comment for revision or merge your change to the main branch.
Pulling updates#
While you are working on your local code, the codes on the origial repository may be updated. You may also want to check the branches other people have created.
You can use git pull
to reflect the changes in the GitHub to your local repository.
You can use git branch
to see what branches are there and git checkout
to try with the codes in other branches.
!git pull
There is no tracking information for the current branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.
git pull <remote> <branch>
If you wish to set tracking information for this branch you can do so with:
git branch --set-upstream-to=origin/<branch> myname
!git branch
main
* myname
Optional) In addition to adding a new module, you are welcome to improve the main program odesim.py
itself. For example,
add other visualization like a phese plot.
fix any bugs or improve error handling.
add documentation.
…
Software/Data Licenses#
Today, increasingly more journals and agencies request that you make the data and programs publicly accessible for
reproducibility of research results
enable meta-analysis
facilitate reuse of data and programs
You should set an appropriate condition in making your data or program public, to facillitate their use and to keep your (and your organization’s) intellectural property. Points of consideration in making your data/programs public include:
copyright
acknowledgement
revision
re-distribution
commercial use
It is also important to know the licenses of the software you use for your development, as that can limit the way you can use/distribute your programs.
Creative Commons#
Creative Commons (https://creativecommons.org) is an emerging standard using combination of three aspects:
Attribution (BY): request aknowldgement, e.g., citing a paper
NonCommercial (NC): no commercial use
ShareAlike (SA) or NoDerivs (ND): allow modification and re-distribution or not
See https://creativecommons.org/licenses/?lang=en for typical combinations.
GPL, BSD, MIT, Apache, etc.#
In open software community, several types of licensing have been commonly used:
Gnu General Public Licence (GPL): redistribution requires access to source codes in the same license. Called copy left.
BSD and MIT license: do not require source code access or succession of the same license.
Apache License: does not even require the license terms.
Public Domain (CC0): no copyright insisted. Free to use/modify/distribute.
See https://en.wikipedia.org/wiki/Comparison_of_free_and_open-source_software_licenses for further details.
Data Management#
Most research start with obtaining raw data, continues on with a series of pre-processing, visualization and analyses, and complete with paper writing. Handling all different files without confusion and corruption takes some good thoughts and habits.
Keep the raw data and metadata and take back up.
Store data as you wish to see when receiving.
Record all the steps of processing, better with a script.
References:#
Hart EM, et al. (2016). Ten simple rules for digital data storage. PLoS Comput Biol, 12, e1005097. https://doi.org/10.1371/journal.pcbi.1005097
Ouellette F, et al. (2018). A FAIR guide for data providers to maximise sharing of human genomic data. PLoS Comput Biol, 14. https://doi.org/10.1371/journal.pcbi.1005873
Eke DO, Bernard A, Bjaalie JG, Chavarriaga R, Hanakawa T, Hannan AJ, Hill SL, Martone ME, McMahon A, Ruebel O, Crook S, Thiels E, Pestilli F (2021). International data governance for neuroscience. Neuron, 10.1016/j.neuron.2021.11.017. https://doi.org/10.1016/j.neuron.2021.11.017
Always Backup#
As soon as you obtain data, don’t forget to take a backup with appropriate documentation.
For a small scale data, DropBox is an easy solution for data backup and sharing.
At OIST, for storing large scale data, you can use the bucket drive. See: https://groups.oist.jp/it/research-storage
In a Unix like system, rsync
is the basic command to take a backup of a folder.
There are options for incremental backup, by searching for new files in the folder and copy them.
Data sharing#
Data sharing is an important emerging issue in the scientific community, as science today is becoming more and more data intensive. In good old days, each researcher did an experiment, gathered data, wrote a paper, and that was the end of the story. Nowadays, each experiment can produce Giga to Tera bytes of data, which are much more than just one researcher to analyze by him/herself. We nee efficient and reliable methods to share data within each lab, across collaboration labs, and the entire research community.
Data Governance#
When making data public, especially human subject data, a good care has to be taken for the privacy. In general
data should be anonymized so that the identity of subject cannot be obtained or inferred.
prior consent must be obtained from the subject regarding the way their data are made public.
Metadata#
Metadata is data about data. It usually includes:
Time and date of creation
Creator or author of the data
Method for creating the data
File format
File size
…
Different research communities have their own standards of metadata, such as
ISO-TC211 for geographic data: https://www.isotc211.org
ISA for biomedical data: https://www.isacommons.org
Following such a standard can help you using common data processing tools, and your data to be found and utilized by more people.
Data File Formats#
It is always better to save your data in a common file format so that they can be read by many data processing tools.
CSV, TSV#
Values separated by comma or tab, in multiple lines like a table. These are still commonly used for simplicity.
XML#
https://www.xml.org Keys and values stored in a form similar to html. Often used to store metadata.
JSON#
https://www.json.org/json-en.html Common in exchanging large data with multiple compnents.
HDF5#
https://www.hdfgroup.org Hierarchical datar format that can also store binary data.
Some domain-specific data formats are based HDF5, such as Neurodata Without Borders (NWB) https://www.nwb.org
Pipeline Tools#
Once your procedures for data processing is determined, such as filtering, visualization, and statistical tests, the sequence should be defined as a script with folders, filenames, and parameters.
A classic way in Unix-like system is shell script, but you can use Python for data processing scripts. There are dedicated packages for data processing pipelines, such as:
scikit-learn.pipeline: https://scikit-learn.org/stable/modules/compose.html
Luigi: spotify/luigi
Prefect: https://www.prefect.io
Snakemake: https://snakemake.github.io