Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ public class TakeBackupCommand extends Command {
private List<String> volumePaths;
@LogLevel(LogLevel.Log4jLevel.Off)
private String mountOptions;
private boolean executeInSequence = false;

public TakeBackupCommand(String vmName, String backupPath) {
super();
Expand Down Expand Up @@ -89,6 +90,13 @@ public void setVolumePaths(List<String> volumePaths) {

@Override
public boolean executeInSequence() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if a host has 20 VMs on the same backup schedule, do all 20 get sent at once now? is there anything that caps how many run together on one host?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not any more. f0096a6 adds a per-host gate in the provider: at most backup.nas.parallel.max.per.host (default 2) take-backup commands are in flight on a host, and the rest wait on the management server, in the async job thread, bounded by backup.nas.parallel.queue.timeout (default 2 h). A wait that runs out fails cleanly without leaving a BackingUp row behind. So 20 VMs on one schedule on one host run two at a time.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok that covers it

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the agent has a small pool of worker threads and everything the mgmt server sends goes through it. if long backups fill it up, does anything else sent to that host wait behind them?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was the real gap, thank you. The cap is deliberately below the agent's default of 5 workers, so at least 3 workers stay free for start, stop, reboot and migrate at all times; an operator who raises the agent's workers can raise the cap to match. The waiting never happens on the agent side.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is that cap per management server? with two of them a host could still get 4 backups at once and leave only one worker free

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is per management server. The count is kept in memory on each one, so with two management servers a host can see up to 2 x the cap, which is 4 at the default and leaves one of the agent's 5 workers free. Making it cluster-wide would mean either a host_id on the backups table (a schema change) or holding a DB lock for the whole backup, which ties up a dedicated MySQL connection for hours per in-flight backup. I did not want either in this PR.

So in 3340496 I have documented it instead. The setting's description now says the cap applies per management server and gives the sizing rule: cap x number of management servers must stay below the agent's workers (agent.properties, default 5). With two management servers, a cap of 1 keeps 3 workers free; with one, the default of 2 does. If you or the other maintainers would rather have a cluster-wide count, I am happy to do it as a follow-up.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, the sizing rule in the setting covers it for now. fine to leave a cluster wide count for a follow up

return true;
// Parallel by default: each backup uses its own per-VM on-NAS path, so concurrent
// runs do not contend. Operators can force sequential execution per zone via the
// backup.nas.parallel.execution.enabled setting.
return executeInSequence;
}

public void setExecuteInSequence(boolean executeInSequence) {
this.executeInSequence = executeInSequence;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
package org.apache.cloudstack.backup;

import org.junit.Assert;
import org.junit.Test;

public class TakeBackupCommandTest {

@Test
public void testExecuteInSequenceDefaultsToParallel() {
TakeBackupCommand command = new TakeBackupCommand("vm-1", "/backups/vm-1");
// Default: run in parallel with other backup/delete commands.
Assert.assertFalse(command.executeInSequence());
}

@Test
public void testExecuteInSequenceIsSettable() {
Comment thread
DaanHoogland marked this conversation as resolved.
TakeBackupCommand command = new TakeBackupCommand("vm-1", "/backups/vm-1");

command.setExecuteInSequence(true);
Assert.assertTrue(command.executeInSequence());

command.setExecuteInSequence(false);
Assert.assertFalse(command.executeInSequence());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -35,5 +35,6 @@ public interface BackupDao extends GenericDao<BackupVO, Long> {
List<Backup> syncBackups(Long zoneId, Long vmId, List<Backup> externalBackups);
BackupVO getBackupVO(Backup backup);
List<Backup> listByOfferingId(Long backupOfferingId);
List<BackupVO> listByZoneAndStatus(long zoneId, Backup.Status status);
BackupResponse newBackupResponse(Backup backup);
}
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ public class BackupDaoImpl extends GenericDaoBase<BackupVO, Long> implements Bac
BackupOfferingDao backupOfferingDao;

private SearchBuilder<BackupVO> backupSearch;
private SearchBuilder<BackupVO> zoneStatusSearch;

public BackupDaoImpl() {
}
Expand All @@ -72,6 +73,19 @@ protected void init() {
backupSearch.and("backup_offering_id", backupSearch.entity().getBackupOfferingId(), SearchCriteria.Op.EQ);
backupSearch.and("zone_id", backupSearch.entity().getZoneId(), SearchCriteria.Op.EQ);
backupSearch.done();

zoneStatusSearch = createSearchBuilder();
zoneStatusSearch.and("zone_id", zoneStatusSearch.entity().getZoneId(), SearchCriteria.Op.EQ);
zoneStatusSearch.and("status", zoneStatusSearch.entity().getStatus(), SearchCriteria.Op.EQ);
zoneStatusSearch.done();
}

@Override
public List<BackupVO> listByZoneAndStatus(long zoneId, Backup.Status status) {
SearchCriteria<BackupVO> sc = zoneStatusSearch.create();
sc.setParameters("zone_id", zoneId);
sc.setParameters("status", status);
return listBy(sc);
}

@Override
Expand Down
Loading
Loading