backup.py

Fri, 26 Jan 2018 10:23:38 +0000

author
Tuomo Valkonen <tuomov@iki.fi>
date
Fri, 26 Jan 2018 10:23:38 +0000
changeset 70
3f794760d52e
parent 64
6cfe6a89e810
child 71
a8a5ebb64e02
permissions
-rw-r--r--

Do not set error status from borg output if loglevel = WARNING, only >= ERROR

1
4cdc9c1f6b28 basic scheduler structure draft, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents:
diff changeset
1 #
4cdc9c1f6b28 basic scheduler structure draft, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents:
diff changeset
2 # Borgend Backup instance
4cdc9c1f6b28 basic scheduler structure draft, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents:
diff changeset
3 #
4cdc9c1f6b28 basic scheduler structure draft, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents:
diff changeset
4
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
5 import config
4
d72c4844e791 Better borg output processing and some logging
Tuomo Valkonen <tuomov@iki.fi>
parents: 3
diff changeset
6 import logging
5
4c5514b2fa76 basic schedule calculation
Tuomo Valkonen <tuomov@iki.fi>
parents: 4
diff changeset
7 import time
20
fdfbe5d7b677 Keychain support and random fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 17
diff changeset
8 import keyring
34
9fce700d42de Log window and other logging improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 32
diff changeset
9 import borgend
54
cfcaa5f6ba33 Basic repository queue
Tuomo Valkonen <tuomov@iki.fi>
parents: 50
diff changeset
10 import repository
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
11 from enum import IntEnum
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
12 from instance import BorgInstance
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
13 from threading import Thread, Lock, Condition
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
14 from scheduler import TerminableThread
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
15
34
9fce700d42de Log window and other logging improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 32
diff changeset
16 logger=borgend.logger.getChild(__name__)
31
b4b4bb7a2ec5 More logging detail
Tuomo Valkonen <tuomov@iki.fi>
parents: 30
diff changeset
17
64
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
18 JOIN_TIMEOUT=60
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
19
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
20 #
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
21 # State and operation related helper classes
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
22 #
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
23
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
24 class State(IntEnum):
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
25 # State
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
26 INACTIVE=0
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
27 SCHEDULED=1
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
28 QUEUED=2
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
29 ACTIVE=3
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
30
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
31
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
32 class Errors(IntEnum):
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
33 OK=0
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
34 BUSY=1
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
35 OFFLINE=2
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
36 ERRORS=3
45
aa2a95dc6093 Further improvements to state reporting; indicate busyness if repository lock cannot be acquired
Tuomo Valkonen <tuomov@iki.fi>
parents: 38
diff changeset
37
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
38 def combine(self, other):
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
39 return max(self, other)
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
40
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
41 def ok(self):
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
42 return self==self.OK
45
aa2a95dc6093 Further improvements to state reporting; indicate busyness if repository lock cannot be acquired
Tuomo Valkonen <tuomov@iki.fi>
parents: 38
diff changeset
43
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
44 def __str__(self):
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
45 return _errorstring[self]
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
46
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
47 _errorstring={
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
48 Errors.OK: 'ok',
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
49 Errors.BUSY: 'busy',
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
50 Errors.OFFLINE: 'offline',
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
51 Errors.ERRORS: 'errors'
6
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
52 }
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
53
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
54 class Operation:
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
55 CREATE='create'
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
56 PRUNE='prune'
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
57 def __init__(self, operation, when_monotonic, **kwargs):
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
58 self.operation=operation
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
59 self.when_monotonic=when_monotonic
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
60 self.detail=kwargs
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
61
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
62 def when(self):
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
63 return self.when_monotonic-time.monotonic()+time.time()
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
64
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
65
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
66 class Status(Operation):
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
67 def __init__(self, backup, op=None):
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
68 if op:
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
69 super().__init__(op.operation, op.when_monotonic,
63
Tuomo Valkonen <tuomov@iki.fi>
parents: 62
diff changeset
70 **op.detail)
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
71 else:
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
72 super().__init__(None, None)
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
73
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
74 self.name=backup.name
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
75 self.state=backup.state
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
76 self.errors=backup.errors
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
77
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
78 #
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
79 # Miscellaneous helper routines
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
80 #
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
81
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
82 loglevel_translation={
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
83 'CRITICAL': logging.CRITICAL,
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
84 'ERROR': logging.ERROR,
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
85 'WARNING': logging.WARNING,
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
86 'DEBUG': logging.DEBUG,
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
87 'INFO': logging.INFO
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
88 }
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
89
6
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
90 def translate_loglevel(x):
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
91 if x in loglevel_translation:
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
92 return loglevel_translation[x]
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
93 else:
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
94 return logging.ERROR
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
95
15
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
96 def safe_get_int(t, x):
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
97 if x in t:
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
98 tmp=t[x]
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
99 if isinstance(tmp, int):
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
100 return tmp
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
101 return None
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
102
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
103 #
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
104 # The Backup class
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
105 #
15
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
106
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
107 class Backup(TerminableThread):
1
4cdc9c1f6b28 basic scheduler structure draft, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents:
diff changeset
108
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
109 def __decode_config(self, cfg):
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
110 loc0='backup target %d' % self.identifier
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
111
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
112 self._name=config.check_string(cfg, 'name', 'Name', loc0)
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
113
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
114 self.logger=logger.getChild(self._name)
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
115
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
116 self.loc='backup target "%s"' % self._name
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
117
54
cfcaa5f6ba33 Basic repository queue
Tuomo Valkonen <tuomov@iki.fi>
parents: 50
diff changeset
118 reponame=config.check_string(cfg, 'repository',
cfcaa5f6ba33 Basic repository queue
Tuomo Valkonen <tuomov@iki.fi>
parents: 50
diff changeset
119 'Target repository', self.loc)
cfcaa5f6ba33 Basic repository queue
Tuomo Valkonen <tuomov@iki.fi>
parents: 50
diff changeset
120
cfcaa5f6ba33 Basic repository queue
Tuomo Valkonen <tuomov@iki.fi>
parents: 50
diff changeset
121 self.repository=repository.get_controller(reponame)
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
122
3
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
123 self.archive_prefix=config.check_string(cfg, 'archive_prefix',
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
124 'Archive prefix', self.loc)
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
125
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
126 self.archive_template=config.check_string(cfg, 'archive_template',
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
127 'Archive template', self.loc)
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
128
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
129 self.backup_interval=config.check_nonneg_int(cfg, 'backup_interval',
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
130 'Backup interval', self.loc,
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
131 config.defaults['backup_interval'])
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
132
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
133 self.retry_interval=config.check_nonneg_int(cfg, 'retry_interval',
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
134 'Retry interval', self.loc,
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
135 config.defaults['retry_interval'])
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
136
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
137 self.paths=config.check_nonempty_list_of_strings(cfg, 'paths', 'Paths', self.loc)
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
138
6
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
139 self.common_parameters=config.check_list_of_dicts(cfg, 'common_parameters',
3
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
140 'Borg parameters', self.loc,
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
141 default=[])
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
142
6
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
143 self.create_parameters=config.check_list_of_dicts(cfg, 'create_parameters',
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
144 'Create parameters', self.loc,
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
145 default=[])
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
146
3
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
147 self.prune_parameters=config.check_list_of_dicts(cfg, 'prune_parameters',
6
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
148 'Prune parameters', self.loc,
3
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
149 default=[])
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
150
21
c36e549a7f12 Errors as rumps notifications
Tuomo Valkonen <tuomov@iki.fi>
parents: 20
diff changeset
151 self.__keychain_account=config.check_string(cfg, 'keychain_account',
c36e549a7f12 Errors as rumps notifications
Tuomo Valkonen <tuomov@iki.fi>
parents: 20
diff changeset
152 'Keychain account', self.loc,
c36e549a7f12 Errors as rumps notifications
Tuomo Valkonen <tuomov@iki.fi>
parents: 20
diff changeset
153 default='')
c36e549a7f12 Errors as rumps notifications
Tuomo Valkonen <tuomov@iki.fi>
parents: 20
diff changeset
154
32
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
155 self.__passphrase=None
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
156
30
3dd525652dc8 Added no_quit_menu_entry option
Tuomo Valkonen <tuomov@iki.fi>
parents: 21
diff changeset
157 if config.settings['extract_passphrases_at_startup']:
32
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
158 try:
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
159 self.extract_passphrase()
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
160 except Exception:
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
161 pass
20
fdfbe5d7b677 Keychain support and random fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 17
diff changeset
162
21
c36e549a7f12 Errors as rumps notifications
Tuomo Valkonen <tuomov@iki.fi>
parents: 20
diff changeset
163 def extract_passphrase(self):
c36e549a7f12 Errors as rumps notifications
Tuomo Valkonen <tuomov@iki.fi>
parents: 20
diff changeset
164 acc=self.__keychain_account
32
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
165 if not self.__passphrase:
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
166 if acc and acc!='':
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
167 self.logger.debug('Requesting passphrase')
32
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
168 try:
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
169 pw=keyring.get_password("borg-backup", acc)
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
170 except Exception as err:
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
171 self.logger.error('Failed to retrieve passphrase')
32
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
172 raise err
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
173 else:
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
174 self.logger.debug('Received passphrase')
32
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
175 self.__passphrase=pw
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
176 else:
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
177 self.__passphrase=None
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
178 return self.__passphrase
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
179
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
180 def __init__(self, identifier, cfg, scheduler):
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
181 self.identifier=identifier
1
4cdc9c1f6b28 basic scheduler structure draft, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents:
diff changeset
182 self.config=config
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
183 self.__status_update_callback=None
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
184 self.scheduler=scheduler
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
185 self.logger=None # setup up __decode_config once backup name is known
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
186
3
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
187 self.borg_instance=None
7
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
188 self.thread_log=None
8
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
189 self.thread_res=None
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
190
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
191 self.current_operation=None
10
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
192 self.scheduled_operation=None
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
193 self.lastrun_when=None
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
194 self.state=State.INACTIVE
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
195 self.errors=Errors.OK
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
196
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
197 self.__decode_config(cfg)
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
198
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
199 super().__init__(target = self.__main_thread, name = self._name)
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
200 self.daemon=True
3
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
201
7
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
202 def is_running(self):
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
203 with self._cond:
8
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
204 running=self.__is_running_unlocked()
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
205 return running
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
206
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
207 def __is_running_unlocked(self):
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
208 running=self.current_operation
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
209 if not running:
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
210 # Consistency check
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
211 assert((not self.borg_instance and not self.thread_log and
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
212 not self.thread_res))
7
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
213 return running
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
214
3
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
215 def __block_when_running(self):
7
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
216 running=self.is_running()
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
217 assert(not running)
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
218
7
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
219 def __log_listener(self):
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
220 self.logger.debug('Log listener thread waiting for entries')
7
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
221 success=True
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
222 for msg in iter(self.borg_instance.read_log, None):
64
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
223 self.logger.info(str(msg))
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
224 t=msg['type']
15
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
225
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
226 errormsg=None
15
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
227 callback=None
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
228
4
d72c4844e791 Better borg output processing and some logging
Tuomo Valkonen <tuomov@iki.fi>
parents: 3
diff changeset
229 if t=='progress_percent':
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
230 current=safe_get_int(msg, 'current')
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
231 total=safe_get_int(msg, 'total')
15
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
232 if current is not None and total is not None:
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
233 with self._cond:
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
234 self.current_operation.detail['progress_current']=current
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
235 self.current_operation.detail['progress_total']=total
14
5a988a2c2624 UI: progress percentange support (Borg doesn't seem to be reporting) + error indicator in tray: B?
Tuomo Valkonen <tuomov@iki.fi>
parents: 12
diff changeset
236 status, callback=self.__status_unlocked()
15
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
237
4
d72c4844e791 Better borg output processing and some logging
Tuomo Valkonen <tuomov@iki.fi>
parents: 3
diff changeset
238 elif t=='archive_progress':
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
239 original_size=safe_get_int(msg, 'original_size')
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
240 compressed_size=safe_get_int(msg, 'compressed_size')
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
241 deduplicated_size=safe_get_int(msg, 'deduplicated_size')
15
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
242 if original_size is not None and original_size is not None and deduplicated_size is not None:
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
243 with self._cond:
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
244 self.current_operation.detail['original_size']=original_size
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
245 self.current_operation.detail['compressed_size']=compressed_size
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
246 self.current_operation.detail['deduplicated_size']=deduplicated_size
15
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
247 status, callback=self.__status_unlocked()
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
248
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
249 elif t=='progress_message':
4
d72c4844e791 Better borg output processing and some logging
Tuomo Valkonen <tuomov@iki.fi>
parents: 3
diff changeset
250 pass
15
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
251
4
d72c4844e791 Better borg output processing and some logging
Tuomo Valkonen <tuomov@iki.fi>
parents: 3
diff changeset
252 elif t=='file_status':
d72c4844e791 Better borg output processing and some logging
Tuomo Valkonen <tuomov@iki.fi>
parents: 3
diff changeset
253 pass
15
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
254
4
d72c4844e791 Better borg output processing and some logging
Tuomo Valkonen <tuomov@iki.fi>
parents: 3
diff changeset
255 elif t=='log_message':
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
256 if 'levelname' not in msg:
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
257 msg['levelname']='ERROR'
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
258 if 'message' not in msg:
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
259 msg['message']='UNKNOWN'
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
260 if 'name' not in msg:
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
261 msg['name']='borg'
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
262 lvl=translate_loglevel(msg['levelname'])
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
263 self.logger.log(lvl, msg['name'] + ': ' + msg['message'])
70
3f794760d52e Do not set error status from borg output if loglevel = WARNING, only >= ERROR
Tuomo Valkonen <tuomov@iki.fi>
parents: 64
diff changeset
264 if lvl>=logging.ERROR
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
265 errormsg=msg
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
266 errors=Errors.ERRORS
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
267 if ('msgid' in msg and
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
268 (msg['msgid']=='LockTimeout' or # observed in reality
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
269 msg['msgid']=='LockErrorT' or # in docs
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
270 msg['msgid']=='LockErrorT')): # in docs
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
271 errors=Errors.BUSY
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
272 with self._cond:
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
273 self.errors=self.errors.combine(errors)
21
c36e549a7f12 Errors as rumps notifications
Tuomo Valkonen <tuomov@iki.fi>
parents: 20
diff changeset
274 status, callback=self.__status_unlocked()
c36e549a7f12 Errors as rumps notifications
Tuomo Valkonen <tuomov@iki.fi>
parents: 20
diff changeset
275 else:
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
276 self.logger.debug('Unrecognised log entry %s' % str(status))
15
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
277
aaf1a281a3fe More error reporting
Tuomo Valkonen <tuomov@iki.fi>
parents: 14
diff changeset
278 if callback:
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
279 callback(status, errors=errormsg)
6
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
280
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
281 self.logger.debug('Waiting for borg subprocess to terminate in log thread')
4
d72c4844e791 Better borg output processing and some logging
Tuomo Valkonen <tuomov@iki.fi>
parents: 3
diff changeset
282
6
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
283 self.borg_instance.wait()
46c89e5a219f subprocess improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 5
diff changeset
284
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
285 self.logger.debug('Borg subprocess terminated; terminating log listener thread')
7
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
286
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
287 def __result_listener(self):
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
288 self.logger.debug('Result listener thread waiting for result')
7
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
289
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
290 res=self.borg_instance.read_result()
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
291
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
292 self.logger.debug('Borg result: %s' % str(res))
7
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
293
64
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
294 with self._cond:
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
295 if res is None:
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
296 self.logger.error('No result from borg despite no error in log')
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
297 if errors.ok():
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
298 self.errors=self.errors.combine(Errors.ERRORS)
7
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
299
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
300
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
301 def __do_launch(self, op, archive_or_repository, *args):
32
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
302 passphrase=self.extract_passphrase()
21
c36e549a7f12 Errors as rumps notifications
Tuomo Valkonen <tuomov@iki.fi>
parents: 20
diff changeset
303
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
304 inst=BorgInstance(op.operation, archive_or_repository, *args)
32
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
305 inst.launch(passphrase=passphrase)
3
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
306
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
307 self.logger.debug('Creating listener threads')
31
b4b4bb7a2ec5 More logging detail
Tuomo Valkonen <tuomov@iki.fi>
parents: 30
diff changeset
308
7
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
309 t_log=Thread(target=self.__log_listener)
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
310 t_log.daemon=True
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
311
7
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
312 t_res=Thread(target=self.__result_listener)
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
313 t_res.daemon=True
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
314
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
315 self.thread_log=t_log
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
316 self.thread_res=t_res
3
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
317 self.borg_instance=inst
64
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
318 self.current_operation=op
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
319 # Update scheduled time to real starting time to schedule
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
320 # next run relative to this
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
321 self.current_operation.when_monotonic=time.monotonic()
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
322 self.state=State.ACTIVE
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
323 # Reset error status when starting a new operation
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
324 self.errors=Errors.OK
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
325 self.__update_status()
3
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
326
7
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
327 t_log.start()
e189d4a6cb8c Also listen to stdout
Tuomo Valkonen <tuomov@iki.fi>
parents: 6
diff changeset
328 t_res.start()
3
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
329
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
330 def __launch(self, op):
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
331 self.logger.debug("Launching '%s'" % str(op.operation))
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
332
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
333 if op.operation==Operation.CREATE:
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
334 archive="%s::%s%s" % (self.repository.repository_name,
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
335 self.archive_prefix,
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
336 self.archive_template)
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
337
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
338 self.__do_launch(op, archive,
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
339 self.common_parameters+self.create_parameters,
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
340 self.paths)
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
341 elif op.operation==Operation.PRUNE:
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
342 self.__do_launch(op, self.repository.repository_name,
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
343 ([{'prefix': self.archive_prefix}] +
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
344 self.common_parameters +
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
345 self.prune_parameters))
8
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
346 else:
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
347 raise NotImplementedError("Invalid operation '%s'" % str(op.operation))
2
e343594c0014 basic config processing
Tuomo Valkonen <tuomov@iki.fi>
parents: 1
diff changeset
348
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
349 # This must be called with self._cond held.
64
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
350 def __launch_and_wait(self):
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
351 op=self.scheduled_operation
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
352 if not op:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
353 self.logger.debug("Queued operation aborted")
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
354 else:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
355 self.scheduled_operation=None
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
356
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
357 self.__launch(op)
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
358
64
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
359 self.__wait_finish()
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
360
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
361 def __wait_finish(self):
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
362 # Wait for main logger thread to terminate, or for us to be terminated
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
363 while not self.terminate and self.thread_res.is_alive():
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
364 self._cond.release()
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
365 self.thread_res.join(JOIN_TIMEOUT)
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
366 self._cond.acquire()
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
367
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
368 # If terminate has been signalled, let outer termination handler
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
369 # take care of things (Within this Backup class, it would be cleanest
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
370 # to raise an exception instead, but in most other places it's better
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
371 # to just check self._terminate, so we don't complicate things with
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
372 # an extra exception.)
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
373 if self._terminate:
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
374 return
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
375
64
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
376 self.logger.debug('Waiting for borg and log subprocesses to terminate')
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
377
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
378 self._cond.release()
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
379 self.thread_log.join()
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
380 self._cond.acquire()
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
381
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
382 if not self.borg_instance.wait():
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
383 self.logger.error('Borg subprocess did not terminate')
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
384 self.errors=self.errors.combine(Errors.ERRORS)
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
385
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
386 if self.current_operation.operation=='create':
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
387 self.lastrun_when=self.current_operation.when_monotonic
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
388 self.thread_res=None
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
389 self.thread_log=None
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
390 self.borg_instance=None
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
391 self.current_operation=None
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
392 self.state=State.INACTIVE
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
393 self.__update_status()
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
394
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
395 def __main_thread(self):
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
396 with self._cond:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
397 try:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
398 while not self._terminate:
64
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
399 assert(not self.current_operation)
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
400 self.__main_thread_wait_schedule()
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
401 if not self._terminate:
64
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
402 self.__main_thread_queue_and_launch()
32
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
403 except Exception as err:
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
404 self.logger.exception("Error with backup '%s'" % self._name)
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
405 self.errors=Errors.ERRORS
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
406
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
407 self.state=State.INACTIVE
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
408 self.scheduled_operation=None
3
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
409
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
410 # Clean up to terminate: kill borg instance and communication threads
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
411 if self.borg_instance:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
412 self.logger.debug("Terminating a borg instance")
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
413 self.borg_instance.terminate()
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
414
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
415 # Store threads to use outside lock
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
416 thread_log=self.thread_log
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
417 thread_res=self.thread_res
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
418 self.thread_log=None
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
419 self.thread_res=None
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
420
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
421 self.logger.debug("Waiting for log and result threads to terminate")
8
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
422
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
423 if thread_log:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
424 thread_log.join()
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
425
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
426 if thread_res:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
427 thread_res.join()
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
428
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
429 # Main thread/2. Schedule next operation if there is no manually
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
430 # requested one
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
431 def __main_thread_wait_schedule(self):
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
432 op=None
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
433 if not self.scheduled_operation:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
434 op=self.__next_operation_unlocked()
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
435 if op:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
436 now=time.monotonic()
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
437 delay=max(0, op.when_monotonic-now)
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
438 self.logger.info("Scheduling '%s' (detail: %s) in %d seconds" %
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
439 (str(op.operation), op.detail or 'none', delay))
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
440
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
441 self.scheduled_operation=op
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
442 self.state=State.SCHEDULED
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
443 self.__update_status()
3
4cad934aa9ce Can launch borg now; output not yet processed
Tuomo Valkonen <tuomov@iki.fi>
parents: 2
diff changeset
444
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
445 # Wait under scheduled wait
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
446 self.scheduler.wait_until(now+delay, self._cond, self._name)
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
447 else:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
448 # Nothing scheduled - just wait
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
449 self.logger.info("Waiting for manual scheduling")
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
450
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
451 self.state=State.INACTIVE
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
452 self.__update_status()
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
453
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
454 self._cond.wait()
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
455
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
456 # Main thread/3. If there is a scheduled operation (it might have been
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
457 # changed manually from 'op' created in __main_thread_wait_schedule above),
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
458 # queue it on the repository, and launch the operation once repository
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
459 # available
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
460 def __main_thread_queue_and_launch(self):
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
461 if self.scheduled_operation:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
462 self.logger.debug("Queuing")
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
463 self.state=State.QUEUED
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
464 self.__update_status()
64
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
465 res=self.repository.queue_action(self._cond,
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
466 action=self.__launch_and_wait,
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
467 name=self._name)
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
468 if not res and not self._terminate:
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
469 self.logger.debug("Queueing aborted")
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
470 self.scheduled_operation=None
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
471 self.state=State.INACTIVE
6cfe6a89e810 Repository queuing fixes
Tuomo Valkonen <tuomov@iki.fi>
parents: 63
diff changeset
472 self.__update_status()
8
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
473
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
474 def __next_operation_unlocked(self):
7b2d2eac6a48 basic scheduling
Tuomo Valkonen <tuomov@iki.fi>
parents: 7
diff changeset
475 # TODO: pruning as well
5
4c5514b2fa76 basic schedule calculation
Tuomo Valkonen <tuomov@iki.fi>
parents: 4
diff changeset
476 now=time.monotonic()
32
06fc14211ba9 Error handling improvements
Tuomo Valkonen <tuomov@iki.fi>
parents: 31
diff changeset
477 if not self.lastrun_when:
10
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
478 initial_interval=self.retry_interval
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
479 if initial_interval==0:
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
480 initial_interval=self.backup_interval
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
481 if initial_interval==0:
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
482 return None
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
483 else:
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
484 return Operation(Operation.CREATE, now+initial_interval,
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
485 reason='initial')
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
486 elif not self.errors.ok():
10
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
487 if self.retry_interval==0:
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
488 return None
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
489 else:
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
490 return Operation(Operation.CREATE,
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
491 self.lastrun_when+self.retry_interval,
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
492 reason='retry')
5
4c5514b2fa76 basic schedule calculation
Tuomo Valkonen <tuomov@iki.fi>
parents: 4
diff changeset
493 else:
4c5514b2fa76 basic schedule calculation
Tuomo Valkonen <tuomov@iki.fi>
parents: 4
diff changeset
494 if self.backup_interval==0:
10
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
495 return None
5
4c5514b2fa76 basic schedule calculation
Tuomo Valkonen <tuomov@iki.fi>
parents: 4
diff changeset
496 else:
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
497 return Operation(Operation.CREATE,
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
498 self.lastrun_when+self.backup_interval)
10
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
499
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
500 def __status_unlocked(self):
21
c36e549a7f12 Errors as rumps notifications
Tuomo Valkonen <tuomov@iki.fi>
parents: 20
diff changeset
501 callback=self.__status_update_callback
38
085a635f23f5 Improved error indicators
Tuomo Valkonen <tuomov@iki.fi>
parents: 34
diff changeset
502
10
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
503 if self.current_operation:
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
504 status=Status(self, self.current_operation)
61
bc6c3d74e6ea Made combination error-state into an error-state matrix, as well as Enum
Tuomo Valkonen <tuomov@iki.fi>
parents: 58
diff changeset
505 elif self.scheduled_operation:
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
506 status=Status(self, self.scheduled_operation)
10
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
507 else:
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
508 status=Status(self)
10
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
509
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
510 return status, callback
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
511
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
512 def __update_status(self):
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
513 status, callback = self.__status_unlocked()
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
514 if callback:
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
515 #self._cond.release()
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
516 try:
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
517 callback(status)
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
518 except Exception:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
519 self.logger.exception("Status update error")
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
520 #finally:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
521 # self._cond.acquire()
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
522
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
523 #
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
524 # Interface functions
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
525 #
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
526
21
c36e549a7f12 Errors as rumps notifications
Tuomo Valkonen <tuomov@iki.fi>
parents: 20
diff changeset
527 def set_status_update_callback(self, callback):
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
528 with self._cond:
21
c36e549a7f12 Errors as rumps notifications
Tuomo Valkonen <tuomov@iki.fi>
parents: 20
diff changeset
529 self.__status_update_callback=callback
10
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
530
49
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
531 def status(self):
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
532 with self._cond:
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
533 res=self.__status_unlocked()
db33dfa64ad6 Improved scheduler
Tuomo Valkonen <tuomov@iki.fi>
parents: 48
diff changeset
534 return res[0]
10
76dbfb06eba0 Semi-working menu items.
Tuomo Valkonen <tuomov@iki.fi>
parents: 8
diff changeset
535
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
536 def create(self):
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
537 op=Operation(Operation.CREATE, time.monotonic(), reason='manual')
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
538 with self._cond:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
539 self.scheduled_operation=op
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
540 self._cond.notify()
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
541
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
542 def prune(self):
62
b7d13b2ad67e Turned Status and Operation into classes instead of dictionaries
Tuomo Valkonen <tuomov@iki.fi>
parents: 61
diff changeset
543 op=Operation(Operation.PRUNE, time.monotonic(), reason='manual')
55
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
544 with self._cond:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
545 self.scheduled_operation=op
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
546 self._cond.notify()
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
547
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
548 # TODO: Decide exact (manual) abort mechanism. Perhaps two stages
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
549 def abort(self):
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
550 with self._cond:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
551 if self.borg_instance:
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
552 self.borg_instance.terminate()
407af23d16bb Improved backup main thread loop, etc.
Tuomo Valkonen <tuomov@iki.fi>
parents: 54
diff changeset
553

mercurial