Separating file transfer from file reconciliation in PXP

Portal has had file conflict resolution since v0.10.0.

It worked, but while working on v0.13.0, I noticed that the way conflicts were being handled had an unintended side effect: conflict resolution was happening in the middle of a file transfer.

That shouldn't have anything to do with transfer progress.

So in v0.14.0, I changed the architecture.

How receiving worked before

When Portal received a file, the data was streamed from the sender into a temporary location before being moved into the final directory:

sender → stream → temp → final directory

If the destination already contained a file with the same name, Portal would detect the conflict and ask how to handle it.

The available actions were:

• overwrite
• overwrite all for this transfer
• rename
• rename all for this transfer
• skip
• skip all for this transfer

This worked correctly from a filesystem perspective.

The problem was when the conflict was detected.

Because the file was being streamed as it arrived, the receiver could discover the conflict while the sender was still transferring the file.

At that point, the receiver had to stop and wait for the conflict to be resolved.

The sender, however, had no idea that this was happening.

That created a particularly ugly interaction with the progress display.

v0.13.0

A conflict could leave the receiver looking something like this:

Portal: Receiving item 1 of 1 [━━━━━━━━━━━━━━] 1/1
Receiving introducing-pxp.mdx ░░░░░░░░░░░░ 0% | 0 B/s | 0s
WARN Conflict detected for path: "./introducing-pxp.mdx"
Portal: Receiving item 1 of 1 [━━━━━━━━━━━━━━] 1/1
Portal: Receiving item 1 of 1 [━━━━━━━━━━━━━━] 1/1
Portal: Receiving item 1 of 1 [━━━━━━━━━━━━━━] 1/1
Portal: Receiving item 1 of 1 [━━━━━━━━━━━━━━] 1/1
Portal: Receiving item 1 of 1 [━━━━━━━━━━━━━━] 1/1  Receiving introducing-pxp.mdx ░░░░░░░░░░░░ 0% |
Portal: File 'introducing-pxp.mdx' received successfully!
Portal: All item(s) have been received successfully! Saved to '.'

The transfer itself and the filesystem decision had become coupled.

They needed to be independent.

The constraint

The obvious solution would be to receive the entire file somewhere first, check for conflicts, and then copy it into the destination:

receive → save → check conflict → copy → destination

But that introduces another problem.

Portal streams files specifically to avoid unnecessary disk usage and copying. If the destination is an external drive, receiving into one location and then copying the entire file to another can introduce significant additional I/O.

I didn't want to solve the progress problem by creating a different disk-I/O problem.

So the staging location needed to live alongside the destination.

Staging

In v0.14.0, received files are staged inside the final directory:

final/
└── .portal/
    └── staging/
        └── <sessionId>/

The transfer and the final placement of the file are now two separate operations.

During the transfer:

sender
   ↓
stream
   ↓
.portal/staging/<sessionId>/

Once the transfer session has finished, Portal reconciles what was received:

.portal/staging/<sessionId>/
   ↓
reconcile()
   ↓
final destination

This is the main change in PXP.

Transfer first, reconcile afterward

The transfer phase no longer needs to care about whether a file can immediately be placed in its final location.

Its job is simply to receive the data.

Once the transfer is complete, reconcile() takes over.

That's where Portal can determine:

• which files were received
• where they should go
• whether a destination already exists
• what to do when a conflict occurs

Conflict resolution still works the same way from the user's perspective.

The difference is that it no longer interrupts the transfer itself.

v0.14.0

The same situation now looks more like:

Portal: Receiving item 1 of 1 [━━━━━━━━━━━━━━] 1/1
Receiving introducing-pxp.mdx ░░░░░░░░░░░░ 0%
Portal: File 'introducing-pxp.mdx' received successfully!
> Portal: 'introducing-pxp.mdx' exists. Action? Overwrite
Portal: All item(s) have been received successfully! Saved to '.'

The transfer can finish before Portal asks what should happen to the file.

The progress display no longer has to compete with conflict resolution.

Reconciliation also helps with interrupted transfers

Separating these two phases had another useful consequence.

A transfer doesn't always end normally.

The sender might disconnect, the connection might be interrupted, or the session might otherwise end before everything has been transferred.

Because the received data is staged by session, the receiver has something it can inspect after the connection is gone.

reconcile() can check the staging area and determine what was received before the session ended, then handle those files accordingly.

So reconciliation isn't only a mechanism for resolving filename conflicts.

It also gives Portal a defined place to deal with the state left behind by an interrupted transfer — see interrupted transfers.

See staging and conflicts for more details.

A cleaner separation of responsibilities

The change in v0.14.0 is ultimately less about adding another directory and more about separating two responsibilities that had previously been mixed together.

Transfer is responsible for getting the data from the sender to the receiver.

Reconciliation is responsible for deciding what happens to that received data on the filesystem.

That separation means conflict resolution no longer has to interfere with streaming, and the transfer itself doesn't have to wait for filesystem decisions.

PXP can receive first.

Then it can reconcile.

Release: v0.14.0 · Diff: v0.13.0...v0.14.0 · Docs: Staging and conflicts