Add tus upload backend via bfabricpy

README.md

@@ -6,7 +6,9 @@ BioBeamer is a Python-based data collection system for scientific instruments. I
 
 ## Features
 
-- **Multiple transfer protocols**: robocopy, scp, sftp (with paramiko support)
+- **Multiple transfer protocols**: robocopy, scp, sftp (with paramiko support), and
+  tus (resumable HTTP upload straight into B-Fabric -- nothing needs to be mounted,
+  and the workunit is created at upload time)
 - **XML-based configuration**: Flexible host and instrument configuration
 - **File filtering**: Pattern-based file selection and filtering
 - **Monitoring**: Comprehensive logging and syslog integration
@@ -15,9 +17,19 @@ BioBeamer is a Python-based data collection system for scientific instruments. I
 
 ## Requirements
 
-- Python 3.7+
+- Python 3.9+
 - For SFTP support: `paramiko` library
 - For scp/robocopy: respective system tools installed
+- For tus support: **Python 3.11+** and the `tus` extra (`pip install -e ".[tus]"`),
+  which pulls in `bfabric[transfer]>=1.21.0`. Kept optional on purpose: bfabric is a
+  large dependency tree and requires 3.11+, while BioBeamer must stay installable on
+  the older instrument PCs. On Python < 3.11 the extra resolves to nothing and
+  `tool="tus"` fails at startup with a message saying so -- those hosts keep using
+  robocopy/scp/sftp.
+
+  1.21.0 is a hard floor, not a preference: 1.20.0 had an incompatible
+  `upload_files` signature (files passed positionally plus a `force` boolean) and no
+  `on_duplicate="link"`.
 
 ## Installation
 
@@ -90,14 +102,85 @@ BioBeamer uses XML configuration files to define hosts, instruments, and transfe
 
 #### Configuration Parameters
 
-- **`tool`**: Transfer protocol (`robocopy`, `scp`, `sftp`)
+- **`tool`**: Transfer protocol (`robocopy`, `scp`, `sftp`, `tus`)
 - **`pattern`**: Regex pattern for file filtering
 - **`min_size`**: Minimum file size in bytes
 - **`min_time_diff`**: Minimum file age in seconds before transfer
 - **`max_time_diff`**: Maximum file age for transfer
 - **`simulate_copy`**: Test mode without actual transfers
 - **`source_path`**: Source directory path
-- **`target_path`**: Destination path (can include user@host: for remote)
+- **`target_path`**: Destination path (can include user@host: for remote). Not used by
+  `tool="tus"`, which uploads to `tus_endpoint` instead.
+
+##### TUS parameters (`tool="tus"` only)
+
+- **`tus_endpoint`**: tus server URL, e.g. `http://localhost:1337/files` (required)
+- **`tus_container_pattern`**: regex whose first group yields the B-Fabric container id.
+  Defaults to matching a `p<digits>` or `C<digits>` path segment.
+- **`tus_track_job`**: `true`/`false`. Creates an `UPLOAD` job under the workunit whose
+  status the tus server's hooks maintain.
+- **`tus_on_duplicate`**: what to do when the container already stores identical bytes --
+  `upload` (default, send anyway), `skip` (leave out of the workunit), or `link`
+  (register a resource pointing at the existing bytes without transferring).
+- **`<b-fabric><applicationID>`**: required. The B-Fabric application is the uploading
+  instrument.
+
+The B-Fabric container is **not** configured per host: it is read from each file's path,
+because one instrument writes data for many projects. The OAuth credentials come from the
+launcher (`bfabric_base_url`, `bfabric_client_id`, `bfabric_scope` in `launcher.ini`) with
+the client secret supplied through the `BFABRIC_CLIENT_SECRET` environment variable -- never
+as a command-line argument, so it cannot leak into logs or the process list. The scope must
+include `tus`; B-Fabric's default scope does not grant it.
+
+##### Legal source path formats for `tool="tus"`
+
+**This is an operational contract, not a convention.** There is deliberately no fallback
+container: filing data under the wrong project is worse than a failed run, so a path with no
+recognisable container is a hard error and nothing is uploaded. Instrument folders must
+therefore always carry the container in the path:
+
+| Format | Example | Container |
+|---|---|---|
+| `p<digits>` (project) | `D:/Data2San/p1234/Proteomics/EXPLORIS_1/run_A/f.raw` | 1234 |
+| `C<digits>` (order) | `D:/Data2San/C4321/Proteomics/QDA_1/f.raw` | 4321 |
+
+Anything else fails with:
+
+```
+could not determine B-Fabric container for '<path>': the path must contain a container
+segment: either 'p<digits>' (project, e.g. '/Data2San/p1234/...') or 'C<digits>' (order,
+e.g. '/Data2San/C1234/...')
+```
+
+If an instrument writes a different layout, set `tus_container_pattern` for that host rather
+than renaming folders by hand.
+
+##### Storage is confirmed before a source file is deleted
+
+A completed tus transfer is **not** confirmed storage. The storage service runs its virus scan,
+checksum verification and disk checks in a *post-finish* hook, after the transfer is already
+complete, and that hook reports to B-Fabric rather than to BioBeamer -- it cannot fail the transfer
+that produced it. So a file BioBeamer recorded as copied can still end up with its resource marked
+`failed`, holding no usable bytes.
+
+Because `max_time_delete` eventually deletes source files, trusting the copied-files ledger alone
+would risk destroying the only copy of data B-Fabric rejected. For `tool="tus"` BioBeamer therefore:
+
+- records which resource each uploaded file became, in `tus_resources.json` beside the ledger
+- re-reads those statuses at **deletion** time and deletes only what B-Fabric reports `available`;
+  anything still `pending`, unreadable, or unaccounted for is kept
+- drops rejected files from the ledger on the next run, so they are uploaded again instead of being
+  skipped for ever
+
+Deletion is gated at the moment the decision is made, not just after upload, because verification
+runs on the server's schedule: a resource can still be `pending` when the upload returns.
+
+##### One workunit per acquisition
+
+Files are grouped by acquisition folder, and each group becomes one B-Fabric workunit named
+after the folder. Instruments that write a *directory* per acquisition (Bruker `.d`, Waters
+`.PRO`) are uploaded as a single directory entry, so the bundle stays one workunit with its
+internal structure preserved as resource names.
 
 #### XML Validation
 

pyproject.toml

@@ -8,14 +8,31 @@ authors = [
     { name  = "Claudio Cannizzaro", email= "claudio.cannizzaro@fgcz.uzh.ch"}    
 ]
 readme = "README.md"
-requires-python = ">=3.7"
+# 3.9 is the honest floor for the core: cli.py already uses importlib.resources.files()
+# (3.9+) and shlex.join() (3.8+). The `tus` extra needs more still - see below.
+requires-python = ">=3.9"
 dependencies = [
     "lxml",
     "platformdirs>=4.0.0",
     "paramiko",
 ]
 
 [project.optional-dependencies]
+# tool="tus" only. Kept out of the core dependencies on purpose: bfabric pulls a large tree
+# (pydantic, httpx, polars, suds, zeep) and requires a much newer Python than the robocopy
+# instrument PCs run, so requiring it would block installs on hosts that will never upload
+# over tus. Install with: pip install -e ".[tus]"
+# The marker is load-bearing: bfabric itself requires Python >=3.11, while BioBeamer must stay
+# installable on the 3.9/3.10 instrument PCs. Without it, a resolver solving for the whole
+# declared range (uv sync) fails outright, because it must satisfy every extra on every
+# supported Python. With it, `pip install -e ".[tus]"` on 3.9 is a silent no-op -- so tus hosts
+# must run 3.11+, which is enforced at runtime by the ImportError guard in tusupload.py.
+tus = [
+    # 1.21.0 is the floor: it is the first release with UploadFilesParams.files /
+    # UploadFileParam(on_duplicate=...) and on_duplicate="link", which this code uses.
+    # 1.20.0 had an incompatible signature (files as a positional arg plus a `force` bool).
+    "bfabric[transfer]>=1.21.0; python_version >= '3.11'",
+]
 development = [
     "pytest>=7.4.4",
     "pytest-mock",